Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization
Authors: Juntao Dai
Taiye Chen
Yaodong Yang
Qian Zheng
Gang Pan
Venue: ICLR 2025 Poster (2025)
Keywords: Reward Over-Optimization
Reinforcement Learning from Human Feedback
Large Language Model
Primary area: reinforcement learning

Abstract: Reinforcement learning from human feedback (RLHF) is an effective method for aligning large language models (LLMs) with human values. However, reward over-optimization remains an open challenge leading to discrepancies between the performance of LLMs under the reward model and the true human objectives. A primary contributor to reward over-optimization is the extrapolation error that arises when the reward model evaluates out-of-distribution (OOD) responses. However, current methods still fail to prevent the increasing frequency of OOD response generation during the reinforcement learning (RL) process and are not effective at handling extrapolation errors from OOD responses. In this work, we propose the *Behavior-Supported Policy Optimization* (BSPO) method to mitigate the reward over-optimization issue. Specifically, we define *behavior policy* as the next token distribution of the reward training dataset to model the in-distribution (ID) region of the reward model. Building on this, we introduce the behavior-supported Bellman operator to regularize the value function, penalizing all OOD values without impacting the ID ones. Consequently, BSPO reduces the generation of OOD responses during the RL process, thereby avoiding overestimation caused by the reward model’s extrapolation errors. Theoretically, we prove that BSPO guarantees a monotonic improvement of the supported policy until convergence to the optimal behavior-supported policy. Empirical results from extensive experiments show that BSPO outperforms baselines in preventing reward over-optimization due to OOD evaluation and finding the optimal ID policy.

Published as a conference paper at ICLR 2025
MITIGATING REWARD OVER-OPTIMIZATTION IN RLHF VIA SEHAVIOR-SUPPORTED REGULARIZATION
Juntao Dai Taiye Chen', Yaodong Yang"*, Qian Zheng!2, Gang Pan
College of Computer Science and Technology. Zhejiang University
The State Key Lab of Brain-Machine Intelligence, Zhejiang University
LLMM Safety Centre, Beijing Academy of Artificial Intelligence *Center for AI Safety and Governance, Peking University (juntaodai, qianzheng, gpan @0 jrzzjuu.uuu.. edu. com
(yeyuta haan hane @stu. .ku. edu. en, (yaodong yyaodong.yaagg edu.
ABSTRACT
Reinforcement learning from human feedback (RLHF) is an effective method for aligning large language models (LLMs) with human values. However, reward over-optimization remains an open challenge leading to discrepancies between the performance of LLMs under the reward model and the true human objec- tives. A primary contributor to reward over-optimization is the extrapolation error that arises when the reward model evaluates out-of-distribution (OOD) responses. However, current methods still fail to prevent the increasing frequency of OOD response generation during the reinforcement learning (RL) process and are not effective handling extrapolation errors from OOD responses. In this work, we propose the Behavior-Supported Policy Optimization (BSPO) method to mitigate the reward over-optimization issue. Specifically, we define behavior policy as the next token distribution of the reward training dataset to model the in-distribution (ID) region of the reward model. Building on this, we introduce the behavior- supported Bellman operator to regularize the value function, penalizing all OOD values without impacting the ID ones. Consequently, BSPO reduces the gener- ation of OOD responses during the RL process, thereby avoiding overestimation caused by the reward model's extrapolation errors. Theoretically, we prove that BSPO guarantees monotonic improvement of the supported policy until conver- gence to the optimal Sahavior-supporter policy. Empirical results from extensive experiments show that BSPO outperforms baselines in preventing reward over- optimization due to OOD evaluation and finding the optimal ID policy.
INTRODUCTION
effective method for aligning large language models (LLMs) with human values (Ouyang et al., 2022b; Bai al., 2022; Achiam et al., 2023; Yang al., 2023; Ji al., 2023b). A key phase of RLHF is reward modeling, where the reward model trained on preference datasets to approximate human preferences (Stiennon al., 2020; Ganguli et al., 2022). The reward model subsequently used to evaluate the responses of LLMs during the reinforcement learning (RL) phase. Despite its empirical success, RLHF criticized for its vulnerability and instability (Casper al., 2023). One of the open challenges in RLHF is the reward overooptinization issue (Gao et al., 2023; Coste et al., 2023). As shown in Figure ((a), although the performance of LLMs may seem improve under the reward model (proxy reward), it can deviate from the actual human objectives (gold reward).
Reinforcement Learning from Human Feedback (RLHF) has been demonstrated
One of the primary causes of reward over-optimization is the extrapolation error that arises when the reward model evaluates cittof-diitributiin (OOD) responses (Eisenstein et al., 2023; Laidlaw et al., 2024; Yang et al., 2024b). Due to the exploratory nature of RL, LLMs may generate responses that fall outside the training data distribution of the reward model. Lacking the ability to accurately assess these unseen responses, the reward model may overestimate the reward signal, leading to
"Corresponding author.

---

Published as a conference paper at ICLR 2025
Supported Respansee
1.0
Proxy Reward
In In-Distribution Region
1.01
Gold Reward
Connect
0.8
BSPO
Ratio
0.5
Incerroct
MWW
0.6
Unsapported Reeponses
1.0
0.2
Ratio
0.5
Standard PPO
0.0
MD
Out-of-Disrrbution Region
0.0
50
100
150
200
100
1.50
200
Step
Step
(a) Phenomenon of Reward Over-optimization
(b) Policy Optimization within
(c) Reward Model Fails
In- D--Ditrribution Region
to Evaluate OOD Responses
(a) Reward over-optimization. Although the performance of LLMSs may seem to improve under the reward model (proxy reward), deviates from the actual human objectives (gold reward). (b) Search in ID region. Our algorithm guides policy iteration within the ID region of the reward model, whereas others may enter the OOD region, suffering from extrapolation errors. (c) Hard to evaluate unsupported responses. Responses are categorized as supported or unsupported, depend- ing on whether they include actions unsupported by the behavior policy (B(als) = 0). As policy iterates, the occurrence of unsupported responses increases. "Correct/Incorrect indicates whether the proxy model's 's evaluation of a generated response aligns with the gold model. The proxy model predicts preference pairs well for supported responses but struggles with unsupported ones.
Figure 1:
severe overestimation in the value function an incorrect policy evaluation. As the policy iteration, the frequency of OOD responses tends to increase, further amplifying the over-optimization issue.
Previous work (Gao et al., 2023) finds that enlarging the reward model and dataset may mitigate over-optimization, but it is often impractical in real-world scenarios. Thus, many others focuses on enhancing the RL process using reward regularization. A common approach is to use the Kullback- Leibler (KL) divergence as a penalty(Ouyan) et al., 2022b; Touvron et al., 2023). This method limits the policy updates to remain close the initial policy (Gao et al., 2023), such in the dotted inner circle in Figure 1(b). However, due to the insensitivity to the in-distribution (ID) region of the reward model. it is difficult to determine an appropriate penalty strength, and it prevents full exploration of the ID region to identify the optimal solution. Similar limitations exist in methods that introduce a maximum reward constraint (Moskovitz et al., 2023) Other approaches include using uncertainty quantifiers (Zhang et al., 2024b) or ensembles (Coste et al., 2023). Yet, uncertainty quantifiers may hard to generalize to OOD regions (Nalisnick et al., 2018) while ensemble method may suffer from consistent overestimation across ensembles and higher computational costs (Eisenstein et al., 2023). Furthermore, a problem across above methods is that while they handle OOD responses pessimisti- cally, they also impact the evaluation of ID ones, potentially leading to suboptimal solutions.
In this work, we propose the Rehavior-Supporte Policy Optimization (BSPO). a novel approach to address reward over-optimization. The core idea is to use value regularization to guide policy iteration only within the in-distribution (ID) region, as shown in Figure (b). Specifically, we use the distribution of the next token to characterize the action distribution in the reward training dataset. Inspired by offline RL (Levine et al., 2020), we refer to this distribution as the behavior policy. Due to the auto-regressive nature of LLMs, any OOD action results in the accumulation of extrapolation errors within unsupported responses, which invalidate the reward model, as shown in Figure I(c). Then, we introduce a behavior supported Bellman operator to regularize the value function. The regularized value function derived from this operator penalizes all OOD values without affecting the ID ones. Our method leverages this value regularization to reduce OOD generation during RI., thereby preventing overestimation caused by the extrapolation errors of reward prediction. Theoret- ically, BSPO guarantces monotonic improvement of the supported policy until convergence to the optimal behavior-supported policy, while other methods lack this convergence guarantee.
We summarize our primary contributions as three folds:
We propose a novel method that leverages the next-token distribution to characterize the reward training dataset, enabling the detection of whether response is OOD for the reward model.
2

---

Published as a conference paper at ICLR 2025
We introduce the BSPO algorithm, the first method that uses value regularization to address ward over-optimization. It penalizes OOD values without affecting ID ones, thereby theoretically achieving the same solution as standard policy evaluation.
We provide a lightweight implementation of BSPO using the ScoreL. model to predict rewards and behavior distributions. Through extensive experiments, we empirically show that BSPO out- performs baselines in avoiding reward over-optimization and finding the optimal ID policy.
2 BACKGROUND
Token-level MDP When LLM is modeled as an RL agent, its actions and states can be repre- sented by tokens or token sequences. Thus, we formalize language generation tasks as a token-level MDP M = (S, AS,A, T, X, 7)) Given the vocabulary A of an LLM, it also represents the action set in the MDP, where an action a corresponds to generating a token of the vocabulary. X' is the set of prompts. μ is the distribution of the prompt given to the LLM, and S is the set of states, where each state s is composed of a prompt x E X' and sequence of generated tokens a0:- = U az- Specifically, s = x U 00:t-1. Thus, the state transition is defined as T(z J d0:--1, at) = x U doa- The r denotes the reward function and the y is the discount factor. In this framework, a station- ary policy, is probability distribution indicating the likelihood of the next token as given state 8t = x U Go:t-1 at step t. Then, II denotes the set of all stationary policies.
The goal of reinforcement learning is to maximize performance measure, [(T], which is typ- ically defined as an infinite horizon discounted total return. [(T) Eysa 1Er=o r'r.]. Here, 100 ~ x denotes the distribution over trajectories generated by æ, where 8g = xx= ~~~a μ.,
T a,, rt)
(s,,a,,
as T(. 81), 81+1 = T(s,, (t), and r, = r(st, at). We express the state value function of x as (B) E ET EEr= 7 Tr. so = s] and the state-action value function as Q a) =
E Et=D 7'r 80 = 8, 0g = a). The advantage function is A A7 a) = (s, - (s)..
Preference Modelling The RLHF method improves the quality of LLM responses by leveraging human preference data through a reward model (Ouyang et al., 2022a; Bai et al., 2022). The re- ward model denoted as R(x,y) y) is designed to align with human preferences, where represents the input prompt and y is the corresponding response generated by the LLM. Human preferences are captured as pairs of responses, symbolized as yu > 3/ Jr, where the (win) denotes response more preferred by humans than z1 (lose). Using the Bradley-Terry model (Bradley & Terry, 1952). the likelihood of a preference pair can be estimated as p(y yr [x) of[(, Vve - R(x, y)) where (Ir) = 1
I4e-= is the logistic sigmoid function. Consequently, given the preference dataset D= (x", . v/ 1 the reward model is trained by minimizing the negative log- big-liilldood loss, D) = - -E(z.g.. >-D (log a (R, yw) - [I g:)l. In the context of LLMs, the reward model typically includes linear layer after the final transformer layer. During the RL stage, the reward model assigns a reward to the final token of the sequence, which is typically the EOS token.
Related Work of Reward Optimmptimization This phenomenon, where the policy language model exploits imperfects in the reward model, is commonly known as reward over-optimization (Gao et al., 2023), and is also called reward hacking (Amodei et al., 2016; Skalse et al., 2022) or reward gaming (Pang et al., 2023). Given the high cost of evaluations for studying reward over- optimization, most studies adopt the synthetic setup that employs a powerful gold model to substitute for human labeling and evaluation (Gao al., 2023; Moskovitz et al., 2023; Coste et al., 2023).
The work most closely related to ours includes a series of reward regularization methods, such as adding a KL penalty the reward (Kullback & Leibler, 1951) or utilizing a reward ensemble (Coste et al., 2023). KL penalty method uses a per-token KL penalty from the SFT model to mitigate over-optimizationn of the reward model, which is first proposed by Stiennon et al. (2022) and widely employed in the context of RLHF (Ouyang et al., 2022b; Bai et al., 2022). Similarly, building on the concept of early stopping, Moskovitz et al. (2023) introduces an approach called Constrained PPO to prevent the policy from surpassing each reward model's threshold of usefulness while minimizing KL divergence in the token distribution. However, due to insensitivity to the reward model's 's in- distribution (ID) region, these methods conservatively iterate near the initial model. In contrast, our algorithm leverages the behavior policy B. allowing relaxed distance constraints and broader exploration of regions where valid reward model predictions exist.
3

---

Published as a conference paper at ICLR 2025
On the other hand, previous work in machine learning demonstrates that combining multiple estima- tors can enhance robustness (Du & Swamy, 2019). Coste et al. (2023) train several proxy models and develop ensemble-basee conservative optimization methods, such as using variance for uncertainty- weighted optimization (UWO) and applying the worst reward for conservative worst-case optimiza- tion (WCO). Ramé et al. (2024) propose more efficient ensemble-based approach by leveraging weight-averaged reward models. Zhang et al. (2024b) introduce a lightweight uncertainty-weighter optimization method that quantifies uncertainties of rewards by utilizing only the last layer embed- dings of the reward model. However, without additional information about OOD responses, reward model ensembles mitigate but do not eliminate reward hacking (Nalisnick al., 2018; Eisenstein et al., 2023). All ensemble proxy models may exhibit consistent error patterns in the OOD region.
Another route to mitigating reward over-optimization is by training more generalized and robust re- ward models (Wang et al., 2024a;b; Shen et al., 2024). Gao et al. (2023) demonstrated in synthetic setup that increasing the reward model size and the training data volume can mitigate this over- optimization issue. Notably, methods aimed at enhancing the generalization at the reward modeling stage run parallel to our approach and can be combined. In our implementation, the training of the reward model incorporates the hidden state regularization technique (Yang et al., 2024b).
ANALYSIS: TEHAVIOR-SUPPORTEE METHOD
3.1 BEHAVIOR POLICY FOR OOD DETECTION OF REWARD PREDICTION
Since reward learning is inherently data-driven, it faces significant challenges in evaluating sponses that lie outside the distribution of the training dataset. However, current OOD detection techniques (Yang et al., 2024a) require additional information for the OOD part (Nalisnick et al., 2018), and are not applicable to reward prediction in LLM. Therefore, we introduce an OOD detec- tion approach based on the power of well ceel-pretr LLMs for next-token prediction.
Based on the token-level MDP modeling in RL for LLMs, we use the distribution of the next token, 3 : S A + [0, 1]. characterize the action distribution in the preference dataset. Inspired by offline RL (Levine et al., 2020; Wu et al., 2022), we refer to this distribution as the behavior policy. The behavior policy divides actions at a given state s into two categories, defined as follows:
Definition 1 (Behavior-Suupported Action). In a given state s, an action a is considered supported by the behavior distribution 3 if and only if 3(als) > O.
Due to the auto-regressive nature of L.Mss, we believe that any OOD action (i.e., one not a behavior- supported action) leads to the accumulation of extrapolation errors in the resulting unsupported responses. To validate this hypothesis, we collect all responses generated during the RLHF process under the same experimental settings of the experiment section. These responses are categorized into supported responses and unsupported responses, depending on whether they contain behavior- unsupported actions (i.e., B(als) 0). These responses, along with those generated by the initial model given the same prompt, construct comparison pairs. We use these comparison pairs and the gold model as ground truth to test whether the proxy model can correctly predict preferences.
Figure I(c) shows that as the policy iterates, the proportion of unsupported responses generated by the L.M increases. For comparison pairs consisting of supported responses, the proxy model ac- curately predicts preferences, achieving an average accuracy of 75.91% which is comparable to its performance on the test set (Figure 2(b)). However, for comparison pairs composed of unsupported responses, the proxy model's predictive accuracy drops significantly to 58.10% 10%. These results in- dicate whether the response contains only behavior sehavior-supported actions is an effective indicator for measuring whether it is OOD for reward prediction. More results are provided in Appendix D.
3.2 SHHAVIOR-SUPPOREEE REGULARIZATION
A natural idea to mitigate reward over-optimization during the RL is to avoid taking actions that are not supported by the behavior policy. In this paper, we consider applying regularization to the value function in order to reduce the ranking of actions that are not supported by the behav- ior distribution during policy evaluation. Specifically, we define Tmin = min (r(s,a) and
4

---

Published as a conference paper at ICLR 2025
Qmin = Et= =1=0 T'Tmin
Suin
Then, we propose the behavior-supporte Bellman operator:
TzQ(s,a)= { (T if B(als) > 0,
(1)
Qmin:
otherwise.
where T (s) = = + (((--) (Q(s', a')] represents the standard Bellman operator, and s' = T(s,a) a) is the next state. This new operator has the following properties:
Theorem 1 (Contraction of TT), For any policy π E II, the operator T3 is y-contraction with respect to the Lo norm over the space S x A.
The proofs are provided the Appendix A. The 7-contraction property of the operator 7 guaran- tees that, for any initial V-values, iterative application of T3 converges to unique fixed point at rate of 7- Thus, we find this fixed point by iteratively solving the problem, ming Er (Q(, TTQ(s as) and use it for policy evaluation optimize the policy
Intuitively, an observation from Equation that, at each iteration k, T3 yields the same Q++1 (s, a) the standard operator T for any behavior-supporter actions (i.e., ID actions). However, for actions that are not supported (i.e., OOD actions), Ty yields smaller Q (s, Through this iterative process, the convergence to fixed point ensures the validity of the following theorem:
Theorem (Fixed Points). For any policy π E II, the fixed point Q7 Tf satisfies
Qmin S (3(0,)) S (s, if B(als) > 0, (s) = Qmin
(2)
otherwise.
where (=,ssa) a) fixed point of standard Bellman operator T
We refer to the fixed point Q3 as the behavior-supporter Q-value function. Theorem 2 indicates that for any policy т,, using the behavior-supported Q-value function for policy evaluation underestimates the future returns of OOD actions, thereby disadvantaging these actions across the action space. This leads the policy iteration process to reinforce ones. Now, we consider a common policy itcration using behavior suppooted value, as follows:
actions and weakens unsupported
T(aels:)
T arg max E =EIl
A (st, (s)
(3)
T (ayls))
Ea 177 (s, (s))). and A7" (s, = ( As the policy iterates, the action selection will eventually consist only of behavior-supported actions. We define such a policy as a behavior suppoore- policy. Specifically, for any state 8, the supported action space of state is denoted as supp (B(-ls)) = {a E A| B(als) > 0}. The set of all behavior-supported policies is defined as
where (s)
11 - (Tt E II (als)) = O, Vs S, a & supp (B(-))))).
Furthermore, the following corollary guarantees that the policy optimization method in Equation IIg through regularized policy evaluation Q7-
yields behavior-supporter supported policies
Corollary (Supported Policy Optimization), The policy optimization method mentioned in Equa- sion 3 yields behavior-supporte policy x E 1T
Theoretically, we demonstrate that policy iteration with the regularized value function guarantees that each iteration results in behavior-supporee policy. This restricts the search for the optimal policy the ID region of the reward model. As a result, the reward model does not need to evaluate OOD responses generated with rehavior-unsuppon optimization issue caused by extrapolation errors.
actions, thereby preventing the reward over-
Furthermore, the following corollary holds for behavior-suppprted policy iterations:
Corollary 2. For any behavior- policy π E II the fixed point Q3 of To satisfies
(s)(s,)), if B(als) 0, Qmin: otherwise.
(3(s,a)=
(5)
5

---

Published as a conference paper at ICLR 2025
Behavoir Distribution [ dpu-1)
85.0
ScoreLM
LM Part
82.5
Standard
LM head
80.0
Accuracy
Next prediction
dist hidden
Score
77.5
Transformer
Reward R(y(2)
75.0
72.5
RM Part
70.0
774M
1.1B
2:7B
Query x
Answer QO.T
Proxy Model Size
(a) Structure of ScoreLM
(b) Prediction Accuarcy
Figure 2: (a) Structure of our ScoreLM model. We retain the original language model head to predict the next-token distribution and initialize score head to predict the reward. (b) Compare with Standard RM. The performance of ScoreLM comparable standard reward models under three scales on the test set. The short vertical lines indicate the standard deviation of four repetitions.
Corollary indicates that the behavior-supported policy evaluation provides the same unbiased Q- values for all ID actions the standard Bellman operator T Together with Corollary 1, for any state s, the optimal behavior-supporte action is the same under the Q and Q value functions during behavior supported policy iteration. This ensures that the method Equation converges to the optimal behavior-supporte policy Tÿ. Specifically, the following theorem holds:
Theorem 3 (Monotonicity to Optimality). The behavior behavior-supported policy optimization method men- tioned Equation 3 results in a strictly monotonic improvement of policy performance, continuing until the optimal behavior-suppon supported policy Fig is attained.
Notably, all proofs of this section are provided in the Appendix A.
IMPLEMENTATION: BEHAVIOR SERHAVIOR-SUPPRRE SUPPORTED POLICY OPTIMIZATION
We introduce the behavior-supported method that uses regularized Q-values guide the policy iteration within the ID region of the reward model. In this section, we present an implementation of the Behavior- Policy Optimization (BSPO) algorithm for the RLHF training of LLMs.
Behavior Distribution Prediction Since pre-trained LLMs have already been exposed to an ex- tensive amount of data, we can directly fit the distribution of the next token the reward training dataset without introducing additional OOD response information. In the standard RLHF paradigm, reward model is derived by converting the LLMs, replacing the language model head with scalar head for scoring. In our work, we retain the original language model head, as shown in Figure 2, to predict the behavior distribution. Specifically, we train this ScoreLM model by minimizing the following loss function, which consists of two components:
EsCOreLM (Ø: EE log a (Rlzyywi ywi 6) - R(x, uu: +))] -a ED log B(a, lr (u:--1: Ø)
Preference Loss
Supervised Loss
(6)
where a is hyperparameter to balance the two losses. We integrate reward prediction and behavior distribution prediction into the same model for two main reasons. First, experimental results indicate that this integration has almost no impact on prediction accuracy, as shown in Figure 2(b). Addition- ally, the supervised learning loss helps preserve the language capabilities of the transformer, thereby improving the generalization of the reward model (Yang et al., 2024b). Second, compared to stan- dard PPO, using ScoreLM introduces negligible additional memory and computational overhead.
Suppirrr- Value Function In our implementation of value regularization, we pre- dict behavior-supporte V-values instead of Q-values to achieve greater stability while maintaining

---

Published as a conference paper at ICLR 2025
equivalent policy evaluation. The deterministic state transition of the token-level MDP (i.e., given state 8 = Qu:t-1 and a next token a, the next state s' du.z determined) ensures this equivalence. It is important to clarify that this determinism inherently holds due to the auto-regressive nature of next-token prediction in LLMs. Specifically, the behavior-supported Bellman V-operator is
V U Jaot-1);;
if 8(a--1/x an 0,
T.y (x ao:
10 r(x a0u-2: ar a--1), otherwise,
(xUtttt11) = |a [r(x: U d0:-11, a1) + y (x du::) ] is the standard Bell- man operator for the V-value function. Then, the following theorem and corollary hold:
where T3F V VFVVx
Theorem (Contraction of TTyy). ). For any policy π E II, the operator Tž.v is "-contraction with respect to the Loo, norm over the state space
Corollary (Equivalent Policy Evaluation). Vπ E II. denote Vī as the fixed point of the Bell- man operator Tøyv in Equation Then, its corresponding state-action value function Q7 a) = r(s,a) a) + vV3 (T(s, a)) is equal to the fixed point of the Bellman operator Tÿ in Equation I.
The proofs of Theorem 4 and Corollary are provided in the Appendix B. These results guarantee that the VV.VV. converges unique fixed point, VT, which provides policy evaluation equivalent that of the behavior-supported Q-values, Q7. For a parameterized critic model Vip(2 ao:-1), train it by minimizing the loss funetion with Tg. Vi calculated from ScoreLM R.g. step k:
[(Vo -1 - Tg. V (Uu ag:- aoa-1) 27
Lv (yp: π) = ET
(8)
Behavior-Supported Policy Optimization We combine the behavior-supporte method with the widely used Proximal Policy Optimization (PPO) (Schulman et al., 2017) algorithm. Specifically, for the parameterized LLM TT. we optimize it by minimizing the following loss function step k:
(0) - ET [min A0 (st, clip(p: 1 +) (81,4:))
me, [s) and T represents the advantage estimates, which are calculated based on predictions from behavior-supported value model Vo..-
where pe(0) me (e(o)(s))
Importantly, PPO ensures the accuracy of gradient prediction when reusing data by clipping the gradients where the policy ratio difference exceeds a certain threshold. thereby enhancing sample efficiency. If we focus soiely on the portion used to update the policy gradient, Equation 9 provides an unbiased estimate of the optimization objective in Equation 3. Together with the equivalent policy evaluation demonstrated in Corollary 3, our implementation of the SehaaviorSSupporee Policy Optimization algorithm (BSPO) is theoretically consistent with the analysis Section 3.
Bringing everything together, the pseudo-code of our algorithm is shown in Algorithm 1.
EXPERIMENTS
In this section, present experiments to demonstrate the effectiveness of the BSPO algorithm. Specifically, we focus on the following three aspects:
BSPO outperforms baseline algorithms, proving its capacity to better mitigate reward over- optimization and find the optimal in-distribution policies (Section 5.2).
BSPO reduces the generation OOD responses during the RI., thereby avoiding overestimation caused by the extrapolation errors of the reward prediction (Section 5.3).
BSPO effectively avoids over-optimization at larger KI. divergence distances (Section 5.4).
5.1 EXPERIMENTAL SETTINGS
To rigorously evaluate our algorithm, we compare it with five baseline methods across three proxy model scales in the synthetic setup. The specific experimental settings follows:
7

---

Published as a conference paper at ICLR 2025
ENS-UWO
ENS-WCO
CPPO
KL-Penalty
Standard PPO
Ours
Proxy Reward
Gold Reward
2
0
50
100
150
200
50
100
150
200
0
50
100
150
200
Step
Step
Step
(a)
774M Proxy RM
(b) 1.1B Proxy RM
(c)
2.7B Proxy RM
Figure 3: Main results. The training curves of various algorithms across three experimental settings show upward trends in proxy rewards. Most baselines suffer from reward over-optimization. In contrast, BSPO algorithm effectively mitigates this issue and achieves the highest gold reward.
Synthetic Setup Given the high cost of human evaluations for studying reward over-optimization, employ widely used synthetic setup in over-optimization research (Gao et al., 2023; Moskovitz et al., 2023; Coste et al., 2023). In this setup, labels are generated by a "gold-standard" reward model (gold model) rather than by humans. Mcanwhile, the proxy model is proxy of the ground truth, trained to fit the labels from the gold model.
In experiments, train the gold model based Llama3-8B (Dubey al., 2024). For the proxy model, we utilize three smaller models: 774M (GPT-22llrre (Radford et al., 2019)), 1.1B (TinyLlama (Zhang al., 2024a)), and 2.7B (ShearedLlama (Xia et al., 2023)). employing the McoreIM architecture as described in Section 4. The gold model is trained using 57k preference pairs from the binarized UltraFeedback dataset (Cui al., 2023). The gold model re-annotates 30k data points, which are then used for training the proxy models, whose training curves provided in Appendix D.2. During RL, Alpaca-75 7B (Taori et al., 2023) is employed as the initial actor model. All algorithms are trained using the same proxy model and evaluated against the same gold model.
Baselines
We implement five baseline algorithms for comparison (details in Appendix C.1):
Standard PPO: The standard implementation of Proximal Policy Optimization (PPO) (Schul- man et al., 2017) without the KL penalty the LLM RLHF (Ouyang et al., 2022a).
KL- KL-Penalty: Add per-token KL penalty to the standard PPO algorithm (Gao al., 2023).
CPPO: Constrained PPO identifies potentially over-optimized proxy points through experimenta- tion and constrains the reward value be below those points during RL (Moskovitz al., 2023).
ENS-UWO & ENS-WCO: The ensemble baseline (ENS) integrates four reward models, utilizing variance for uncertainty-weiglee optimization (UWO) or employing the worst reward for con- servative worst-case optimization (WCO) (Eisenstein et al., 2023; Coste et al., 2023).
5.2 MAIN RESULTS
Figure presents the training curves different algorithms across three proxy model scales. Our BSPO algorithm effectively mitigates reward over-optimization acToss all parameter sizes, as evi- denced by the consistent upward trends in both the proxy reward and the gold reward. One possible explanation is that BSPO reduces the OOD generation, thereby minimizing extrapolation emrors from the proxy model, which we empirically validate Section 5.3. We also observe that BSPO more reliably finds the optimal policies achieving the highest gold reward.
In contrast, most baseline algorithms continue to suffer from the reward over-optimization problem. First, the inability of ensemble methods to handle OOD) responses, noted in Eisenstein et al.
8

---

Published as a conference paper at ICLR 2025
Win
Lose
Gold Reward
M
KL-000
0.005
Our
0.24
0.26
CPPO
R008
Gold Reward
KL.0,05
Unsappnna o.
WC0
Curs
Namber
KL
UWO
PPO
0.506
Outs
0,866
100 Step
200
1.0
K.L Divergence
(a) Win Rate
(b) Average Number of Behavior- Unsupported Actions
(c) BSPO vs. Different KL Coefficients
vs. Initial Model
Figure 4: (a) Win Rates. The win rates of different algorithms against the initial SFT model using the 2.7B proxy model. (b) Count behavior-unsup unnspppooeed actions during RL. We track the average number of actions (tokens) that are not supported by the behavior policy for each response during RL. Over-optimization in traditional methods leads to a sharp rise in unsupported actions, as shown by the dashed line, while our BSPO algorithm keeps these actions consistently low during training. (c) Compare with KL penalty. Compared with the KL penalty method using different penalty coefficients, BSPO effectively avoids reward over-optimization at larger KL divergence distances.
(2023), remains a concern These methods mitigate overestimation within the ID setting, but for OOD responses, multiple models may still consistently overestimate. As the policy iteration, the frequency OOD responses tends to increase, further amplifying this issue. Second, methods that apply KL penalties maximum constraints to the reward can be viewed as limiting the policy search to specific region. However, unlike BSPO, these methods do not explicitly model the ID region of the reward model, and thus may only mitigate over-optimization in the overlapping region (typically near the initial policy). Third, while the constraints of CPPO successfully keep the proxy reward the threshold, the gold reward continues to change. Therefore, relying solely the proxy reward may be insufficient to establish a threshold that indicates when over-optimization occurs. Furthermore, CPPO introduces instability inherent the Lagrangian approach (Platt & Barr, 1987).
Finally, to further validate the results, we evaluate the final model on the test set and calculate its win rate against the initial Alpaca-7B, as shown in Figure 4(a). Unlike other baseline methods that adjust the requence-wise reward, BSPO penalizes action-wise OOD values without affecting ID ones, allowing to find the same optimal ID policies as standard policy evaluation. This may explain why the model trained with BSPO outperforms the other baselines shown in the figure.
5.3 ABLATION ON THE BEHAVIOR SUPPORTED PREDICTION
We employ the behavior policy 3. which denotes the next token distribution in the reward model's training dataset, method for OOD detection of the reward model. To experimentally illustrate how BSPO mitigates the issue of over-optimization, we track the average number of actions (tokens) that are not supported by the behavior policy for each response during the RL process.
The results are shown in Figure 4(b). We observe that the occurrence of the over-optimization phe- nomenon in traditional methods is accompanied by significant increase the number of behavior- unsupported actions, as indicated by the dashed line. In contrast, our BSPO algorithm maintains a consistently low number of such actions throughout the training process. This supports one of our core insights: regularizing the value function reduces the generation of OOD responses, thereby avoiding the overestimation problem caused by the extrapolation error from OOD evaluation.
5.4 FURTHER COMPARISON WITH DISTANCE CONTROL
We further highlight the necessity of modeling the ID region of the reward model. Figure 4(c) presents a comparison between the KL penalty method, using different penalty coefficients, and BSPO. The KI. penalty method can be seen approach that constrains the policy from deviating

---

Published as a conference paper at ICLR 2025
too far from the initial one under distance measure (Gao al., 2023). The results show that BSPO avoids over-optimization even at larger KL divergence distances, whereas the KL penalty method fails at closer distances, only ensuring consistent increase of gold reward within proximal region.
This phenomenon suggests that the region defined by the distance constraint does not align with the ID region of the proxy model. As result, valid reward prediction without extrapolation error can only be guaranteed within the proximal region that is fully covered by the ID region (as the inner dashed circle in Figure I(b)). In contrast, our algorithm models the ID region directly through the behavior policy 3, enabling us to relax the distance constraint and fully explore the entire region where the reward model can generate valid predictions.
CONCLUSION AND DISCUSSION
A primary cause of reward over-optimization is the extrapolation enror that occurs when the reward model evaluates OOD responses. However, current methods still fail to prevent the increasing gen- eration of OOD responses during the RL process and are not effective at handling extrapolation rors from their reward prediction. Furthermore, they apply reward regularization to pessimistically handle OOD responses which, meanwhile, introduces unintended changes to ID ones, potentially leading to suboptimal solutions, In this work, we propose the Behavior-Supported Policy Optimiza- tion (BSPO), a method grounded in two core insights. First, we define behavior policy as the next token distribution of the reward training dataset to model the ID region of the reward model. Second, Bellman operator to regularize the value function, penalizing
we introduce the Sehaaior-supporrte all OOD values without impacting the ID ones. It illustrates that BSPO iterates the policy in the ID region of the reward model, thereby avoiding the reward model to evaluate the OOD response. Theoretically, we prove that BSPO ensures monotonic improvement of the supported policy until convergence to the optimal ID policy, a guarantee that other methods lack. Extensive experiments demonstrate that BSPO outperforms baselines and exhibits two key strengths. First, BSPO reduces the generation of OOD responses during the RL, thereby avoiding overestimation caused by the extrapolation errors. Second, BSPO effectively avoids reward over-optimization larger distances, facilitating the search for the optimal ID policy.
6.1 LIMITATION AND FUTURE WORK
This study has several notable limitations. First, due the unaffordable costs of human analyzing for studying the reward over-optimization, we employ the widely accepted synthetic setup (Gao et al., 2023: Coste al., 2023; Moskovitz al., 2023). However, there are some differences between the synthetic setup and real-world scenarios. For example, model-based evaluations tend to exhibit higher consistency, while human preferences are subject to greater variability. Therefore, further validation experiments in real-world settings are concerned. In Appendix D.5, We emulate human behavior by a LLM to conduct experiments in a realistic setting and engage in further discussions.
Second, our algorithm primarily addresses the issue of reward over-optimization during the RL phase caused by OOD responses in evaluations (Eisenstein et al., 2023; Laidlaw et al., 2024; Yang et al., 2024b), assuming the of well-trained reward model by default. However, if the reward model fails to accurately capture preferences during the reward learning phase, this could also lead to deviations from human values (Miao et al., 2024). It is important recognize that this issue parallels our concerns and is at a different stage of training. Another aspect to be considered additionally is the presence of an OOD prompt in the RL phase, as there may be no response for the reward model ID on these OOD prompt. This requires further study of the data distribution shift between the reward model phase and the RL. phase.
Third, a current research direction in LLM alignment involves multiple reward models (Moskovitz al., 2023; Ji al., 2023a; Dai et al., 2023a). Extending our algorithm to objti-obeecctvv scenarios to mitigate the over-optimization of each reward model presents a promising direction.
ACKNOWLEDGEMENTS
This research is supported by the STI 2030 Major Projects ence Foundation of China (No. 61925603)
and the Natural Sci-
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303, 087744 2023.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp. 22-31, PMLR, 06-11 Aug 2017. URL ht tps: //ppo/eeeonnnggeeeeeeeeeeeeeeeee mlr. www.ppeess.......m.............e wwwss/v7000. achiamiiim...mmmmm 7 htmi.
Eitan Altman.
Constrained Markov Decision Processes, volume 7. CRC Press, 1999.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety, 2016. URL ht http/axxvvvvv....a aas///b///// 1606. 06565.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.0086) 2021.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0586,, 2022.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39:324, 1952. URL https://api. semanticschold org
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.152177 2023.
Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf, 2024. URL https: /ttpaaaiivv/////////////a//aaaaa org/ abs /2022440222220 07319.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint urXiv:2310.02744 2023.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback, 2023a. URL https:
//arxiv.crg/ www.rr///b.//eeeeeee..eeeeeeeeee attp//b///// 22300 12773.
Juntao Dai, Jiaming Ji, Long Yang, Qian Zheng, and Gang Pan. Augmented proximal policy optimization for safe reinforcement learning. Proceedings of the AAAI Conference on Artifi- cial Intelligence, 3766):288-72955 Jun. 2023b. doi: (0.1669/aaai.vv7i66 v37i6.25888. URL https:
//ojs. aaai org index. php/ AAAI
Juntao Dai, Yaodong Yang. Qian Zheng, and Gang Pan. Safe reinforcement learning using finite- horizon gradient-based estimation, 2024. URL https: /ttpppp:///axxiv./// wrw. abs/ 2412 11138.
Ke-Lin Du and M. S. Swamy. Combining Multiple Learners: Data Fusion and Ensemble Learn- ing, pp. 737-767. Springer London, London, 2019. ISBN 978-1-4471-7455-33 doi: 10. 1007/ 778-1-4471-7452-3222 URL http:::/laa..ogg/0.000////111113 25.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Leuman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
11

---

Published as a conference paper at ICLR 2025
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlle alpacaeval: A simple way to debias automatic evaluators, 2024a. URL bttps://arxiv. 04475.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani. Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2024b. URL https: /ataas//arxivii htsgg/abs/2200.5 14387.
Jacob Eisenstein, Chirag Nagpal, Alckh Agarwal, Ahmad Beirami, Alex D' Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herd- ing? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09244 2023.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209007888 2022.
Leo Gao. John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835.1066.. PMLR, 2023.
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety align- ment of Ilm via human-preferenne dataset. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Process- ing Systems, volume 36, pp- 24678-24704. Curran Associates, Inc., 2023a. URL pttps::/prooeeednng..ccm wew.rsn.nuuipeee 4cbbl..b66611eccctaa311177088883 pdf.
faapperrriileesppe//2/2222 file/ Paper- Datasetsaaataaaaae and Benchmarks.
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310. 201.21200. 2023b.
Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: A unified safe reinforcement learning benchmark. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (ods.)..dvancce, in Neural Information Processing Systems, volume 36, pp. 18964-18993. Curran Associates, Inc., 2023c. URL https: http::s:s/ppoceeddnng.ooo wwwp:eeeeeednne...eeeeeeeeeeeeee wttpppssscccceeeeeeooeeeee wwppaper files/ paper fi and_Benchmarke .df.
Jiaming Ji, Jiayi Zhou, Hantao Lou, Boyuan Chen, Donghai Hong. Xuyao Wang, Wenqi Chen, Kaile Wang, Rui Pan. Jiahao Li, Mohan Wang, Josef Dai, Tianyi Qiu, Hua Xu, Dong Li. Weipeng Chen, Jun Song, Bo Zheng, and Yaodong Yang. Align anything: Training all-modality models to follow instructions with language feedback, 2024a. URL https:/ /ttp://axxiv... wwg/abg........ooooo... 2412.158330
Jiaming Ji, Jiayi Zhou, Borong Zhang, Juntao Dai, Xuehai Pan, Ruiyang Sun, Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. Journal of Machine Learning Research, 25(285):1 2024b.
URL http::/t///.. http:://mmmmm org /ttpppaper//eeeaaaeeeeeeeeeeeeee v25/23 0681 html.
Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Tianyi Alex Qiu, Juntao Dai, and Yaodong Yang. Aligner: Efficient alignment by learning correct. Advances in Neural Information Processing Systems, 37:90853-90889 2025.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pp. 267-274, 2002.
12

---

Published as a conference paper at ICLR 2025
S. Kullback and R. A. Leibler. On information and sufficieney. The Annals of Mathematical Statistics, 22(1):79-88) 86, 1951. ISSN 00034851. URL /ttp::/www.// orq/ stable/ 2236703.
Cassidy Laidlaw. Shivam Singhal, and Anca Dragan. Preventing reward hacking with occupancy measure regularization. arXiv preprint 1rxXx::403.03/888 2024.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprini TrXiv:2005.01643 2020.
Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adver- sarial regularizer. arXiv preprint arXiv:2405. 16436, 2024.
Wenjia Meng, Qian Zheng, Yue Shi, and Gang Pan. An off-policy trust region policy optimization method with monotonic improvement guarantee for deep reinforcement learning. IEEE Transac- tions on Neural Networks and Learning Systems, 33(5):2223 -223,, 2021.
Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. Mitigating reward hacking via cntemmttion-hhoreetee reward modeling. arXiv preprint arXiv:2402.09335, 2024.
Ted Moskovitz, Aaditya K Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca D Dragan, and Stephen McAleer. Confronting reward model overoptimization with constrained rlhf. arXiv preprint wrXiv:2310.04373. 2023.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminaraaannan Do deep generative models know what they don't know? arXiv preprint 1rXiv:1810.09136, 2018.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback, 2022a. URL https: / http//awivvvv/////e abs/2205 02155.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022b.
Richard Yuanzhe Pang. Vishakh Padmakumar, Thibault Sellam, Ankur P. Parikh, and He He. Re- ward gaming conditional text generation, 2023. URL https:t//aa/tttti org/ abs/221. 08714.
John Platt and Alan Barr. Constrained differential optimization. In Neural Information Processing Systems, 1987.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models, 2024.
URL https: htt///axxa /ta/avivvvv..///// org/ /b// 12187.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https: / arxiv org/ob//// 170710634007 00647.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation, 2018. URL https: wwwaaiiwwwrggaba//llaaaaaaeeeeee 02438.
13

---

Published as a conference paper at ICLR 2025
Jiaming Shen, Ran Xu, Yennie Jun, Zhen Qin, Tianqi Liu, Carl Yang, Yi Liang, Simon Baumgart- ner, and Michael Bendersky. Boosting reward model with preference-conditiona multi-aspect synthetic data generation, 2024. URL
http//orrg/aabs/2...5000./ 16008.
Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback, 2023. URL https: /aapps///rxxv....// org/ abs /2100. 05199.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf, 2024. https: / /ttpwaawivv//////////.../.aat. org/orgg////......o.. absaaassll000 /tt.//331.01111111111
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and character- izing reward gaming. Advances in Neural Information Processing Systems, 45:9460-9471, 2022.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-300 2020.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. URL
https: / //axivvvvvv. org/ wwwp/ass////ooooo../ooo///s..arr absaaaas/050 2009. 01325.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunri B. Hashimoto. Stanford alpaca: An instruction-following llama model.
https://githtie wwt.eoo/////a//.e ttt.sttatsaaaaaaaa stanford_alpace. 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint MxXiv:2307.09288. 2023.
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Secrets of rlhf in large language models part ii: Reward modeling, 2024a. URL https:/ arxiv.orgei......... abs 2201. 06080.
Haoxiang Wang. Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts, 2024b. URL https::/aarat org/abs/222 2406 12845.
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy opti- mization for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:31278-31291, 2022.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre- pre-training via structured pruning. arXiv preprint 1tth::2310006999 2023.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for I1m alignment? a comprehensive study, 2024. URL https: /twpvv.avvv///.///////.../...... org/ org/abs//////eeoo/eoo////.ae awws////.00 241./11444...1.....1111111111111
Aiyuan Yang, Bin Xiao, Bingning Wang. Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie, Jun Dai, Kun Fang, Lei Su, Liang Song, Lifeng Liu, Liyun Ru, Luyao Ma, Mang Wang. Mickel Liu, MingAn Lin, Nuolan Nie, Peidong Guo, Ruiyang Sun. Tao Zhang, Tianpeng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xiangrong Zeng, Xiaochuan Wang. Xiaoxi Chen, Xin Men, Xin Yu, Xuehai Pan, Yanjun Shen, Yiding Wang, Yiyu Li, Youxin Jiang, Yuchen Gao, Yupeng Zhang, Zenan Zhou, and Zhiying Wu. Baichuan 2: Open large-scale language models, 2023. URI. https: /tt////rxiii org/ alss//////////// 2309. 10305.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distrrbutie detection: A survey. International Journal of Computer Vision, pp. 1-28, 2024a.
14

---

Published as a conference paper at ICLR 2025
Long Yang. Jiaming Ji, Juntao Dai, Linrui Zhang, Binbin Zhou, Pengfei Li, Yaodong Yang, and Gang Pan. Constrained update projection approach to safe policy optimization. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 9111-9124. Curran Associates, Inc., 2022. URL https: /ttoceee////ceeeeeeeeoeo ngs. neurips
pappaper_fllesppppppr_ p222p/202/ Paper Conference pdf.
Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for llms. arXiv preprint arXiv:2406. 102/6, 2024b.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024a. URL https: // //trxvavvvv//////////e////aaaav. wwg.or////oo......oo
erggabs////eeoooooooe/.//e//e.ee abs/ /2401. 02385.
Xiaoying Zhang, Jean-Francois Ton, Wei Shen, Hongning Wang, and Yang Liu. Overcoming ward overoptimization via adversarial policy optimization with lightweight uncertainty estima- tion. arXiv preprini arXiv:2403.05171, 2024b.
15

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
This paper presents work that aims to advance the field of machine learning, specifically focusing on mitigating reward over-optimization in RLHF. There many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
A SUPPLEMENTARY DETAILS OF ANALYSIS SECTION
A.1 PROOFS
Notations. For any state 8, the supported action space of state is denoted supp (3(+ls) {a E A B(als) > 0}. The set of all behavior wwpproor-suppretedddeeee policies is denoted as
=
B = t El (a)s)) = Va & supp (3(-1s))}
(10)
We define Tmin min,es,ac A (r(s, a)l and Qmin Ex= T' = 1-.. following behavior-supported Bellman operator:
Then, we propose the
(T, ( 1minn
B(als) > O, otherwise.
T ()s(s
(11)
where (*Q(s, = r(s, + --ee [Q(s', a')] is the traditional Bellman operator and T(s, a) is the next state. This new operator has the following properties:
Theorem 1 (Contraction of Tg). For any policy E II, the operator T3 is y-contraction with respect to the Co norm over the space S x A.
Proof. For any functions f1, f2 : A- - R. any policy æ, and Vs S,a if = 0 have
T1 a) - T3. f2(8, a) = =mQ Qmin 0 0 S2lloc
(12)
else if B(a)s) > we have
T (s - T3. f2(s, a)l
= -|>Bar (fi(s','') - fa(s', a')l
I BB our (-1))) (i((',d') - f2(8',
(13)
S 7 max (fi(s, fz(
sES, A
= 7 11/1 - S2lloo
where s' = =T)
Thus, we have 1 Ta S r1lf - f2lloo. that is, Ty is 7-contraction.
The 7-contraction property the operator TØ. guarantees that, for any initial Q-values, iterative application of TT"" converges to a unique fixed point at a rate 7. Thus, we find this fixed point by iteratively solving the problem, minQ Ex (Q(s, T3 Q(s, a))², and use it for policy evaluation to optimize the policy π.
The fixed point Q3 has the following properties:
Theorem (Fixed Points). For any policy π E II, the fixed point Q7 Tf satisfies
{ Qmin S (7(3, a) S Q" (s, if B(als) 0, (s) Qmin.
otherwise.
where Q (T(ssa) a) is fixed point of standard Bellman operator T
Proof. By Theorem 1, TT is 7-contraction. Assume the Q7 is the fixed point, then we have (7 T (5
(TT))))))). if B(als) 0, Qmin otherwise.
(14)
16

---

Published as a conference paper at ICLR 2025
where s' = as,
Denoting (8, a) is the minimum point of Q7 a) when B(a]s) > 0, that is, we define (8, a) = arg min (esssccs.s))))((((1))) (3(s, a). Then, we have
()) r(s. à) + y ( (07(5',aa))
where s' =(s,
= = +2
Σ
F(a' 8 Q3((1d))
Σ
T(a' (la'ls)Qmmnn
d'Exupp(0((le())
at o'(supo(3((1o"))
=r(s,a)+7
a(a'ls ((a'ss))Q((s''aa
T<a'ls 5')Qmin
at d'Empp((( (s(1))
supp((((1<0))
r((,a)+ +7
Σ 7(d'ss) a' d'Esup((((1e))
Q3(8,))+ +7
Σ
T(alls')
Qmin
al ((supp((()))))
(15)
The last inequality in Equation 15 holds since (5, â) is the minimum point of Q7(s, a). To simplify the expression, we define À = En (=) (a'ls') Then, X E. E(dgsupp(((11s)) (aa)ss)). Bringing into the above inequality and since E [0, 1), E [0, 1], we have
((,,))
â) + + X)Q min
(1 -0- 2) 2 ((( (1))))mmm)))@mmmmm
(16)
r(s.a) + + - 2)Q
Q3(3,0) V
7
Since a(,,a)= 2 Tmin = (1 - 2m follows that
(1 (1-7) (1-7)Qmin + + - A)Qmin ()(3,))
Qmin
(17)
1-7À
Since ($, â) is the minimum point of Qī(s, a), the following equation holds
vs E S,a ((-ls). (7(s, 2 (3 Quin
(18)
Then, since Vs S,a & supp(3(-1s)). Q7 a) = Qmin. we have Vo E,a (, 2 Qmin-
Next, we will prove the relationship between Q7 a) and Q* (s)
Vss E S, a E A, we have
T=Q = r(s, + = (118)) (77(s'
where 5' =
2 r(s,a) a) + - [Qmin]
(19)
11-7) Qmin + Q
Qmin
Thus, for any & E S,S, a E A, we have
THd Q)))s))))
B(ass) > 0,
Qmi (T)))))
otherwise
(20)
$ (3,))(),
17

---

Published as a conference paper at ICLR 2025
vs S, E supp()(-)s)), it holds that (7(s, = (s,a)
TTQ3(s,0)
= =r(s,a) + +7 - m-(-11")
[Q((s' where - (8,
=r(s,a) + 7B N (-((-1
<r(s,a) + +7 (1(1'')
[T=Q7(6',a)]
= =r(s,a) + (-
[r(s',a')+ + 7
(Q3(d",a"))
where 8" = T(8',
= r(s,a) + + (-))0))))))
[r(s',a') + (
[T ,a"]]
S r((sa) + + / ~ (100))
(r(s',d') + " 1
[T "F((",a"")
< E
rrr(s; 180= 00
= as)
(21)
Combining Equation 18 and Equation 21, we prove that V' E IT, the fixed point Q3 of Ty satisfies
f Qmin I (7(s,a) S (s) if B(als) > 0, (Q((s,)) Quin,
(22)
otherwise.
We refer to the fixed point Q3 as the behavior-supportal Q-value function. It underestimates the future returns of OOD actions, thereby disadvantaging these actions across the action space. Now, consider common policy iteration using behavior-supported value, follows:
TTaa (s)) A (st, ai)
Tk+1 = arg max BT #EIl
(23)
T (7, st))
(s). Furthermore, the following corollary guarantees that the policy optimization method in Equation yields behavior- supported policies π E Iī through regularized policy evaluation.
where Vir (s) = (a Q77 (s, a)) and 47 = Q (s, a) - V VT*(s)
Corollary (Supported Policy Optimization). The policy optimization method mentioned in Equa- tion 3 yields celuvvoorsupporrt policy æ E 1lg.
Proof. For any E S.
t (s) = dtt (x (xxt11== Σ m(x)
II
nh aaaas L Uao--)).
(24)
#EX
==
Then, we have
<(ay(st)) (s)
(at1(als) = arg max EyonE
AT* (st, ai)
Th (atls))
REIl
((als) (als)
arg max
(s)
TA (als) Σ
A (s,
sEll
#ES
aEA
arg max
TH (8)
Σ
T Talss)
[45 (8,
REII
SES
aEA
(25)
arg max ATt (s. as)aa)
0.
e'EA
(Q (s, a') - -V (8) (o))
0 arg max e'EA
0 arg max Q (s, a')
e'EA
18

---

Published as a conference paper at ICLR 2025
Since 23¹h (s. a) is equal to the fixed point of the Tg it holds that, Vs S, Va' supp(3(-1s)),
Va" supp(3((1s)),
(s, 2 Qmin Q3 (s,
(26)
where the inequality holds because of Theorem 2. Therefore, the maximum solution will only choose hehavior-supported actions. It follows that
(als( arg max Q33 (8, a')
arg max (supp((((11s)
Q (0, II
(27)
a'EAA
In conclusion, for arbitrary policy The the policy optimization method mentioned in Equation yields supported next policy Tk-1 llg.
Theoretically, we demonstrate that policy iteration with the regularized value function guarantees that each iteration results in behavior-supported policy. This restricts the search for the optimal policy to the ID region of the reward model. As a result, the reward model does not need to evaluate trajectories generated by OOD actions, thereby preventing the over-optimization issue caused by extrapolation errors. For behavior-supported policy iterations, the following corollary holds: Corollary 2. For any selhavior-supporte policy π E IIg. the fixed point Q3 of Tğ satisfies
(7) = (0 a), if B(als) > 0,
(5)
Qmin:
otherwise.
Proof. By Theorem 1, T3 is --contraction. Assume the Q3 is the fixed point, then we have (T)))() if B(als) 0, otherwise.
()( Tg
(28)
Qmin
where s' = as)
For all supported policy æ E II it holds that Vs E S, a & supp(3(-1s), (als)) = 0. Then, we have Σ (als))mmin
Exxx(-1e) (T()
Σ
Q()))))))(( +
(Esupp(P({s))
a ((sp))((()s))
Σ
T(als)TTQ7
(Es)))(((()))
Σ
(a)s))=(((ss.a)+
Σ
(EsIp)(D((aa)
((Sipp(A((13)
(s,
(29)
Thus, vs E S, a E (upp(3())s)), it holds that
Q7(s, a) (7)))s
-T"Q7(s,a)
r(s,a) + t
[Q](s',a']] where = T(,,
= =r(s,a) + 7 a (-16'')
=r(s,a) + 7B (-16')
[T=Q7(6(,a))
=r(s,a) + 7 w (118)
[r(s',a') (xg"nxx(--
[Q3(6" "o(S",a")]] a")]]
where 8" = T(8',
= r((sa) a) + 7E, (t
[r(s',a') + 7E." (1")
TS" ,a]]
r(s, a) + 7E, ~ (89)
[r(s',a') + yB 1
(36"" ["]]
= E
Σ
* r (:) 1 1800 =
1==
(s)
(30)
19

---

Published as a conference paper at ICLR 2025
Corollary indicates that the behavior-supported policy evaluation provides the same unbiased Q- values for all ID actions as the standard Bellman operator 7 Together with Corollary 1, for any state s, the optimal behavior-supported action is the same under the Q and Q" value functions during behavior-supported policy iteration.
Furthermore, we the following lemma prove the monotonicity of our method:
Lemma 1 (Performance Difference, Lemma 6.1 in (Kakade & Langford, 2002)), For any policies π and xO and any starting state distribution μ.
1 and
J(7')-
an)))))))) (AT(s,
(31)
behavior-supported Theorem 3 (Monotonicity to Optimality). The behavior supported policy optimization method men- tioned Equation 3 results in a strictly monotonic improvement of policy performance, continuing until the optimal behavior-supper supported policy T'g is attained.
Proof. First, we prove the monotonicity. Given sequence of policies ( generated by the policy optimization method in Equation 3. From Lemma 1, for any k 2 0, we have
[(T++1) [TT))
sndft k+s am)T+1((1s) [A** (s**
1
SES
AEA
The (als)
27 277 (s, - V3a (8)
8ES
a
+++
Σ (a= - (s)
SES
LaEA
Σ (1l1
Σ * (als))
LEEA
E
snd
Tkll
- Ea (1)))
7
2 0
(32)
where the last inequality holds since Th+1 is the greedy policy with respect to Q i.c., (als)
Il arg max (supp(3(-ls)) Q7 (s.a')
Second, we prove the convergence. For any & 2 0, if [(TO41) [(TA) = 0, since Vs S,dj dT (8) 0, we have
E
E (11)) (s, = Eanzy (3))
(33)
Since Q77t is equal to the fixed point of the Bellman Operator T.u it holds that
QT (s, B(als) > 0, Qmin
-T Q7 (8,a) =
(34)
otherwise.
20

---

Published as a conference paper at ICLR 2025
As result, Vs E S, A, A(ass))) > O, we have
(8 T (s,0a"
r(s, a) + 7 07 (s'.
where s' (s)
=((s,)) + +, m Q33 (s', (s',aa ad]
from Equation 33
= as( + + Σ
arg max ((
(7' d')
27 (5'(s', from Equation 27
acA
= =r(s,a) + ('Exup)(8(()
max
28 (s',a')
= (8,001,0)
(35)
Thus, Q is the fixed point of the Bellman optimality operator Tg. i.e., Q3² = =Q3- From Equa- tion 27, we have,
(als() =
arg max upppp((())))
03 (s, (s, = a= arg max 23(5, = -Ta(ass) ((-s)p((B(1ss)
(36)
B SUPPLEMENTARY DETAILS OF IMPLEMENTATION SECTION
B. PROOFS
Behavior-Supported Value Function The core of our algorithm involves training critic model to perform behavior-supported policy evaluation. In our implementation, we predict behavior- supported V-values instead of Q-values to achieve greater stability while maintaining equivalent policy evaluation. The deterministic state transition equation of token-level MDP in LLM, T(=Uap:-1: (4) x Ja.r, ensures this equivalence. Specifically, the behavior-supporter Bellman V-operator is defined as:
V2x U ap:-1);
3(0,-122 U 20: > 0,
Tgy V(r 10)
(37)
1/2 Qmin r(x L0u-an-222411 a-1)) otherwise,
where T V (= (a [r(x U aaa--1, ax) VV U aoa)] is the standard Bell- man operator for the V-value function. This operator has the following properties:
Theorem (Contraction of (T..)) For any policy E II, the operator Tÿ. is y-contraction with respect the Lox norm over the state space S.
Proof. Vf1. f2 S + R. VI E 11, and Vs S, denoting x af i
follows that
fi(ss fe (mii(() Vmnn(s)) = 5.005/10 f2lloo
(38)
3(a-1100 U 0u:-2) > 0, we have
- -Tx) = (awat(-lo) [r(8, a) 7f((()) -E (F(8, + +7f2(81))
( [fi(s') f2(5')]
yllaann(-1) [fi(s') f2(s')
(39)
S max (f((s) f2(s)]
SES
y I1[ fell
where s' T(s, a). Thus, V f2llo lol S Yllfi falloc, that is, T5 c-cottrcctoon operator in the Loo norm within the S space.
21

---

Published as a conference paper at ICLR 2025
We refer to the fixed point V the behavior-supported V-value function. Now, we will prove that Q33 and V have the equivalent policy evaluation.
Corollary 3 (Equivalent Policy Evaluation). Vπ E IT, denote V3 as the fixed point of the Bell- man operator Tgy in Equation 7. Then, its corresponding state-action value function (7 a) = r(s, a) + 3V3 (T a)) is equal to the fixed point of the Bellman operator Tÿ in Equation 1.
Proof. By Theorem 4, Vø is y-contraction operator in the Lox norm within the S space, Assume the V is the fixed point, then we have Vs x U 1p:t11 E
T (x U a0 1Qmin r(x: -0a-2; a-1)), otherwise,
if Mar-1/1 aoor > 0,
V (x U 1oo
(40)
ss x U 00:-1 E &,a E A. B(o 10:-1) 0, it holds that Q7(2z U0:--1.0)) = r(x U Qu:-11, a1) + VV (x U Ua:z))
= r(x U Qu:--1, az) ) + [Qmin r(x U a0u-1, a:)l
(41)
=Qmin
B(a, (ax-1) > holds that
(7( U d0 1, az) = rr U 10 + VV (3 U aox)
=r(x U (0:0-1,,;)) (t) + 77, V U ao:t)
=r(x U 0,: t (
[r(x U ouss a141) + +V3 ( (x aw:41)
=r(x U Ua:-U, 0z) + +7
(77( au:2 a(41)l
= =T U0:--1,
(42)
Therefore, we have
07(2 U0.-0,,,), = To (T U0:c-1,,))
[T= Uan-1,
B(a, U Ua > 0,
(43)
1min
otherwise.
which means Q7 is the fixed point of the Bellman operator Tg in Equation 1.
On the other hand, we will prove that if Q3 is the fixed point of the Bellman operator T3 Equa- tion I, then its corresponding state value function VJ is the fixed point of the Bellman operator Vg in Equation 7.
Assume Q7 is the fixed point of the Bellman operator Tj. Vs U a0:-1 E S, Va, A, it holds that
T TT2 Uaor-1, if 8(a, l U Jaoa-1) 0,
07(2 U 10)
(44)
1mmin
otherwise.
Since Q7 U ao.t 11- r(x apn 2, a-1) + VVz apa- 1), Vo x x un E S, if 3(a--1 |r 00.-2) = 0, it follows that
V (a an--1) = [Q] U ag:t (;:0-2 - U an=
(45)
1Qmin r(x U 00-
((;-11/0 U (0.-2)) > 0, follows that
Vī U a0:1-1) = Ba (atuaarr
1T5(1 U a0u-1,
(46)
[r ao.-11, vVV uo::))
Therefore, Vs E S, we have (s) Tg. Vil (s) which means V is the fixed point of the Bellman operator Vy in Equation 7.
22

---

Published as a conference paper at ICLR 2025
B.2 PSEUDO-CODE
We provide the pseudo-code of the implementation of our BSPO algorithm as follows:
Algorithm Behavior- Seapavirr-Suppoteed Policy Optimization
Require: Reward function r(z, y). behavior function {aass), initial policy wo, initial value function V6
2: for step k == ......... K do
3: Generate responses
ÿ (0b,0....;;6777 PTa-1 (yb/ xx))
II
t=
where 86,0 = Tb. and b = 1,... B refers to the index within the batch. 4: Compute reward with KI. penalty
RM r Pb -1 = r(0b, ÿb),
Th (as. 2hh U (6,00=1)
KL
= log
To (ab, 2bb U (0.00=-1)
RM fb, rb,
KL
TT
where v/ represents the KL penalty coefficient. 5: Compute advantage estimates:
A = 6b,t + 7Aba.t11 +... (72) -00--- 61,T-1
where 0b,5 = fb.. + (V - Vo-1(88,11.
6: Compute actor loss:
B T-1
CpPO
min()).c(4)A.... cl)pp.....ipppp(()))1___1 - +c).,h
BT
where (k.) = 1 (b.4))
7: Compute behavior-supported value function and critic losst:
T Valxs U U0b,00-1)= = f + VVa U Ja, if 3(as,-1 2h U (0,0:0-2) > Eg, otherwise,
(mmins
B
1 Cv = BT
-((s,,)-Tav (VV
where Eg and Vmin are constant minimal values representing the unsupported action's 's thresh- old and the minimum of the value function.
8:
Update actor model and critic model with Lrro and Lv
9: end for
10:
Return RK
1: Since value regularization method is compatible with the reward regularization method based on KL divergence control, we preserve this component of the standard RLHF implementation the pseudo-code. Notably, ensure fair comparison against the KL-only method in main experiments and provide a detailed analysis of the differences between the two methods in Section 5.4.
1: In the implementation, parameterized method predicts the behavior policy. To prevent modeling errors, an infinitesimal value Eg is used as a threshold, instead of zero, to determine whether an action is supported.
23

---

Published as a conference paper at ICLR 2025
C SUPPLEMENTARY DETAILS OF EXPERIMENTS
C.I BASELINES
In this section, we elaborate on baseline algorithms and provide details of our implementation.
C.1.1 STANDARD PPO
We adopt the training procedure proposed by Ouyang al. (2022b), with focus on the RL ob- jective. In this setup, the output from the score head of the ScoreLM model serves as the reward function for RL training of baselines to ensure fair comparison. Given prompt x D Prompe: the current actor model (o(y)a) æ)) generates a corresponding response y = 0o:T-1 with length T'. Then the reward for tokens do:r-1 defined as:
0, (2,
0 O<<<T t = 1
'
(47)
In the RLHF fine-tuning phase, we use the Proximal Policy Optimization (PPO) algorithm (Schul- man al., 2017) train the LLM. The surrogate PPO clip loss for the RL training objective is formulated as:
CRL (0: D Proumpp) BeaenDDrrnne Pronp pr (a Ex min (p:(0)A clip -1,0),,--,+,))) A
Ex
(88)
where Pe(O) (a, (euo is the importance sampling weight and Ool is model parameters aoa (=(l)
from the previous gradient update, E (0, 1) is the PPO clip ratio. Åî is the advantage of the reward estimated by the GAE method (Schulman et al., 2018).
C.1.2 KL PENALTY
The primary difference from standard PPO is the addition of per-token KL penalty the reward function used in the RL training. With the prompt x ~ Dprompt and the corresponding response y a1:7, the reward function can be expressed as:
To To(alk U (o:r-1))
KL = - log
S 1),
(49)
Tra a i U (api-1)
F, RM + B KL (0SIST-1),
(50)
where T.e(( ()) denotes the reference model, and 3 2 0 represents the KI, penalty coefficient. For each token, dense reward is applied, penalized by the KL divergence between the current actor model and the reference model. The score head of the ScoreLM model provides a sparse reward only on the final token. The reference model is frozen LLM initialized with the actor model's parameters at the start of the RLHF phase.
C.1.3 ENSEMBLE
We propose to learn an ensemble of reward models {Ri..., R& in the reward model training stage. During policy optimization, we combine the reward estimates from different reward models within the ensemble according to the following two methods which have been proved to be more effective.
Worst-Case Optimization Worst-case optimization (WCO) creates a conservative estimate by choosing the lowest reward from the ensemble at every step. Choosing the lowest reward helps ensure that as long as at least one ensemble member does not overestimate the true reward, policy optimization will not result in over-optimization.
24

---

Published as a conference paper at ICLR 2025
Rwco y) == min R; (2,
(51)
major advantage of WCO is that does not have any hyperparameters that might require tuning. However, it can sometimes result in a performance penalty due to its highly conservative nature.
Uncertainty- -Weeghted Optimization In uncertainty- optimization (UWO), the reward for a sample is calculated by combining the average reward across all models in an ensemble with the intra-ensemble variance, weighted by a coefficient X. Intuitively, UWO works by penalizing the policy for generating responses for which there is high disagreement among reward models within the ensemble. This helps prevent the exploitation of a single faulty reward model which might be erroneously assigning high rewards to incorrect or low-quality responses. Mathematically, this objective is given by:
(e
Σ R,( -x Σ
R2( 1 (((
(52)
k
mean
veriance
where À is hyperparameter which controls the weight of the uncertainty component.
C.1.4 CONSTRAINED PPO
The basic idea of Constrained PPO (Moskovitz et al., 2023) is to model the over-optimization prob- lem constraint in the RLHF process and then solve it using the Safe RL (Ji et al., 2024b; 2023c; Yang et al., 2022; Dai et al., 2023b; Meng et al., 2021; Dai et al., 2024; Achiam et al., 2017) do- main approach. Unlike the original paper which uses several rule-based rewards, LLM-based reward model. To identify proxy point, we train PPO agents (Schulman et al., 2017) maximize the reward model (without KL regularization) and plot the resulting evaluation scores against the gold model. We observe that the evaluation score initially increase before falling, so we identify the point where further optimization will lead to a decrease in the gold score. After per- forming the above steps multiple times, we select the most conservative proxy point to ensure that roo will not occur.
employ an
Once one has identified proxy point for the reward model, the next question is how to train agents to maximize the reward until they hit this critical value. In constrained reinforcement learning, an agent seeks to maximize its value while adhering to constraints on its behavior. Mathematically, this problem is formalized as a constrained MDP (Altman, 1999). For clarity, we will hereafter refer to r0 as the "task reward" rather than just the reward, and the CMDP optimization problem is given by:
max of s.t. U. II
o,, 1 1,..,,
(53)
As mentioned the paper, To is selected as kl divergence, and we have only one reward model, that is, N=1. Given our possible objectives, we can now consider how to optimize them. One popular approach to solving constrained problems is to use Lagrangian relaxation:
max min OO
+
pal0 - -.))
(54)
a20 I
where the weights on the value each RMs μ = [M [NN] E R are the Lagrange multipliers associated with each constraint.
[N]"
We implemented the algorithm referred as a f PPO in the paper, which also the best perforrmnn among the several algorithms proposed. It is designed to stay close to To and ensure reward models hit the target, with an objective the following form:
max UKL S.l. Ui = 0,Vi ti F
(55)
25

---

Published as a conference paper at ICLR 2025
With this objective, then design mixed advantages which convex combination of the task and constraint advantages:
N (= a) - (N Σ
p.) A5 + Σ u (8,
(56)
1=1
This equation has the intuitive interpretation of placing more weight optimizing constraint reward rs > when 14 > is high (indicating constraint violation), and more weight on task reward ro when J1:N low (indicating that constraints satisfied).
The original paper employs tanh function to constrain the Lagrange multipliers between -1 and 1. However, in our experiment, the total number of training steps is relatively low due the large batch size. Using the tanh function slows down the optimization of the Lagrange multipliers, causing them remain elevated for too long when constraints satisfied, and vice versa. Therefore, replace the tanh function with simple clipping of the Lagrange multipliers between -1 and 1.
C.2 HYPER-PARAMETER
In this section, we provide all the hyper-parameters used in experiments.
Table Hlyper-paammetee of three experimental settings of BSPO.
Plyper-parametert
Proxy-774M
Proxy-1.1B
Proxy-2.7B
epochs
3
3
3
mex_length
512
512
512
temperature
1.2
1.2
1.2
top-p
1
1
1
num return return_sequencee sequences
1
1
eretitionnpenaly
1.0
1.0
1.0
prr.device-prompttaacee batchhsztttessieeeeeeeeeeeee
8
8
8
per per_ device train batch size
8
8
8
gradient accumulation steps actor_Ir
1
1
1
1.00E-5
1.00E-5
1.00E-5
actor_weigh decay actor schedulerrerrceeee type
0.01
0.01
0.01
cosine
cosine
cosine
actor cetto..r.warmpprrattt
0.03
0.03
0.03
actor gradiient.checkointinn
TRUE
TRUE
TRUE
critic
5.00E-06
5.00E-06
5.00E-06
critic_weight.do _dccay
0.0
0.0
0.0
critic srfic..r_schee www.:rrsscheduer..yyy...
constant
constant
constant
critic Ir warmup ratio
0.03
0.03
0.03
santicgrradentt.cheee checkpointing
TRUE
TRUE
TRUE
hhiprange_ratte ratio (e)
0.2
0.2
0.2
mnsupported_value
-15
-15
-15
epsilon_beta (cg)
1.00E-4
1.00E-4
1.00E-4
bfl6
TRUE
TRUE
TRUE
tf32
TRUE
TRUE
TRUE
26

---

Published as a conference paper at ICLR 2025
Table 2: Hyper-parameters of Reward Model Training.
Hyper-parameters
Gold-8B
Proxy-774M
Proxy-1.1B
Proxy-2.7B
epochs
2
2
2
2
max length
1024
1024
1024
1024
per_device train_batch.ste
16
16
16
16
per_device_ewe .eaal bat__bath__izt
16
16
16
16
gradient accumulation steps
1
1
1
1
gradient.checkpoinnigg
TRUE
TRUE
TRUE
TRUE
regularization Ir
0.001
0.001
0.001
0.001
2.00E-05
2.00F-05
2.00E-05
2.00F-05
Ir sr.scheduler.type
cosine
cosine
cosine
cosine
Ir dr_warmup_ras ratio
0.03
0.03
0.03
0.03
weight decay Im_coef (ax)
0.1
0.1
0.1
0.1
NULL
0.01
0.01
0.01
bfl6
TRUE
TRUE
TRUE
TRUE
tf32
TRUE
TRUE
TRUE
TRUE
C.3 RUNTIME ENVIRONMENT
All experiments in this paper utilize the following runtime environment. The server's CPU is an Intel(R) Xeon(R) Platinum 8358P CPU @ 2.60GHz with 128 cores, and the graphics cards are NVIDIA A A100-SXM4-80GB x8, with NVLink support and the graphics driver version being 550.54.15.
27

---

Published as a conference paper at ICLR 2025
D MORE EXPERIMENTAL RESULTS
D.1 PROXY MODEL FAILS TO EVALUATE UNSUPPORTED RESPONSES
Since reward learning is inherently data-driven, it faces significant challenges in evaluating sponses that lie outside the distribution the reward training dataset. As described in Section 3, we introduce an OOD detection approach for reward prediction of RLHF based the power of well weet-prer LLMs for next-token prediction.
Reward
Reward
1.0
1.0
1.0
Gold exw. Proxy
Gold
Giold
Reward
Reward
Reward
0.5
0.5
Proxy
0.5
Praxy
M NMAM
Yin Aidpsli Mlllo Ao Mmn
0.0m
0.0
Suppocted Responoos
Supported Responses
Supported Responsos
1.0
Correct Incorrett
Connect
Correct Incorrect
Ratio
Ratio
Incomeet
Ratio
M
Unsupported Responses
Casupporzed Resporses
Unsupportod Respoesee
1.0
1.0
1.0
(N
Ratio
Ratio
Ratio
0.5
0.5
0.5
0.0
0.0
0.01
50
1.50
200
100
150
50
1.50
200
Step
Step
Step
(a)
(b)
(c)
Figure 5: The prediction accuracy of the proxy model pairs of supported and unsupported sponses is evaluated across three repeated standard PPO experiments, each conducted with different random seeds.
We collect all responses generated during RLHF under the same experimental settings and categorize them as supported or unsupported based on whether the behavior policy supports the actions. These responses, along with those from the initial model given the same prompts, form comparison pairs. We then test whether the proxy model can correctly predict preferences using the gold model as ground truth. The results are provided in Figure 5.
Figure 5 shows that as the policy iterates, the proportion of unsupported responses generated by the L.M increases. For comparison pairs consisting of supported responses, the proxy model pre- diets preferences with accuracies of 75.91% %, 71.36%, and 74,81% which are comparable to its performance on the test set (Figure 2(b)). In contrast, for comparison pairs involving unsupported responses, the model's accuracy drops markedly to 58.10%, 62.25%, and 59.01%. %. These findings suggest that whether the response contains only behavior-supported actions is an effective indica- tor for measuring whether it is OOD for reward prediction. Furthermore, the occurrence of reward over-optimization is also directly related to the percentage of unsupported responses and the validity of the evaluation of responses (extrapolation error).
D.2 MORE TRAINING CURVES OF SCORELM
We train three ScoreLM models for our algorithm and baselines using the loss function in Equa- tion 6 with different model scales, shown in Figure 6. The accuracies the evaluation dataset demonstrate a scaling law, where larger pretrained models consistently achieve better performance on the downstream task.
28

---

Published as a conference paper at ICLR 2025
GPT2
TinALLAMA
ShearedLLAMA
1.0
Train Accuracy
Score Loss
Total T Loss
0.6
mn MAmm
0.5
Mmmm
0.4
0.
0.2
0.5
3.3
Lival Accuracy
0.7
Eval Mcan
LMME Loss
2.0
0.4
1.5
1500
1000
1000
1500
Step
Step
Step
Figure 6: Training curves of ScoreLM models
D.3 MORE TRAINING CURVES OF ENSEMBLE BASELINES
The ensemble baselines, ENS-UWO and ENS- WCO, require multiple reward or ScoreLM models. Therefore, we train four reward models for each pretrained model same experimental settings, varying only the random seed. The training curves and evaluation curves are presented in Figure 7.
D.4 RESPONSE LENGTH
Improvements in reward are often driven by longer responses, LLMs exploit human raters' pref- erence for detailed content (Singhal et al., 2024; Shen et al., 2023). Several methods aim to control this length bias (Dubois et al., 2024a; Singhal al., 2024; Shen al., 2023; Chen et al., 2024; Ji et al., 2025; 2024a). Our method provides more comprehensive solution, effectively mitigating various biases, including response length.
We collect response length statistics for different methods during training, as shown in Figure 8. Both the PPO and ENS-UWO algorithms approach the model's maximum generation length, while ENS-WC ENS-WCO generates shorter responses due to conservation optimization. KL-penalty PPO produces shorter responses by strictly enforcing KL-divergence constraints. Constrained PPO is the most unstable, with response lengths varying unpredictably. Although there control for length, proposed approach strikes balance and achieves an appropriately concise and detailed response.
D.5 NON-SYNTHETIC SETTING
Studying reward over-optimization requires continuous evaluation during the training and effective comparison with the proxy reward. However, human evaluation is not only expensive but also unable to provide timely scalar feedback. As result, the best of our knowledge, nearly all prior works (Gao et al., 2023; Coste et al., 2023; Eisenstein et al., 2023) rely on synthetic settings.
The differences between synthetic settings and real-world scenarios section, we emulate human behavior by the GPT-4o (Achiam et al., 2023) to conduct experiments in realistic setting and engage in further discussions.
important concern. In this
29

---

Published as a conference paper at ICLR 2025
GPT2
TinyLLaMA
LacaredLLaMA
1.0
1.0
1.0
Train Accuracy
0.9
0.8
0.8
0.7
0.7
0.7
0.6
0.5
0.5
0.5
0.4
0.4
0.7
0.8
0.6
Train Loss
0.6
0.4
0.4
0.2
0.2
0.3
0.8
0.8
0.8
Accuracy
0.7
0.7
0.6
0.6
0.6
0.5
400
Step
Step
Step
Figure 7: Training curves of four reward models used in ensemble baselines, with variations only in random seeds.
Ours
PPO
KL
CPPO
WCO
UWO
GPT2
TimyLLaMA
ShearedLLaMA
Mcan Length (token)
3.00
300
300
2.50
250
250
200
200
1:0
100
50
100
1.5
200
50
100
150
200
50
100
150
200
Step
Step
Step
Figure 8: Mean generated length in different algorithms
D.5.1 IMPLEMENTATION DETAILS
Assigning a scalar score to responses through human evaluation inherently challenging. There- fore, effective alternative involves comparing the responses generated by different checkpoints on an evaluation dataset. Specifically, we train a 7B reward model from Alpaca-7B (Taori et al., 2023) on the Ultraeeddacck preference dataset (Cui et al., 2023), achieving test set accuracy of 82.94%. This reward model was subsequently used to conduct training for BSPO and the baseline methods (PPO (Ouyang al., 2022a). WCO (Coste al., 2023)).
30

---

Published as a conference paper at ICLR 2025
BSPO
PPO
WCO
0.5
1240
Reward
0.0
1220
ELO
0.5
1200
-1.0
1160
100
200
300
100
300
step
step
Figure 9: The training curves and ELO scores of different algorithms in the non-synthetic setup.
Table 3:
The win rate of different checkpoints
Win Ratio
Initial
PPO:79step
WCO:79step PPO:158step
WCO:158step
BSPO:79step
BSPO:158step
Initial
0.4921
0,5316
0.4683
0148211
0.4246
0.3849
PPO:79step
0,5079
0.5
0.5258
0.4722
0,4942
0.4227
0.3952
PPO:158ster 158step
0.4684
0.4742
0.5
0.4643
0.4603
0.4048
0.369
79stpp
0.5317
0.5278
0.5357
0.5
0,5161
0.4724
0,4365
WCO: 18sstep
0.5179
0,5058
0.5397
0.4839
0.5
0,4782
0,4325
BSPO:79<tep
0.5754
0.5773
0,5952
0.5276
0.5218
0.5
0,4484
ASPO::58step
0.6151
0.6048
0.631
0.5635
0,5675
0,5516
0.5
To quantify evaluation results, we compared the checkpoints on the test set to calculate the win rate between checkpoints and fit an ELO score (Askell et al., 2021) as the scalar evaluation metric. The ELO update formula is as follows:
I2,s = RA KK SAR 1 10(RE- Ra
(57)
where SAB represents the win rate of Model A over Model B the test set, K is the update coefficient, and RA and RB are the ELO scores of Models and B, respectively.
D.5.2 EXPERIMENTAL RESULTS
The training curves and ELO scores can be found in Figure 9 and Table 4. The specific win rates between different checkpoints are presented in Table 3.
Our experiments reveal interesting insights. We observed that while the proxy reward exhibits sistently increases across the three approaches, their performance under human evaluation reveals differences. Naïve PPO shows performance improvement 79 steps but suffers severe decline by 158 steps, highlighting the issue of reward over-optimization. WCO maintains stable human eval- uation performance across both steps, indicating effectiveness in mitigating over-optimization. However, it does not identify the optimal solution effectively as BSPO. BSPO demonstrates con- tinuous improvement both proxy reward and human evaluation, indicating its capability to address over-optimization while identifying optimal policies.
D.6 DATA WITH LABEL NOISE
Many studies (Gao et al., 2023; Coste et al., 2023; Eisenstein et al., 2023) utilize re-annotated data with label noise train their proxy models. Similarly, conducted experiments demonstrate the robustness of our algorithm. By introducing 15% label noise to the training data of the proxy reward models (TinyLlama-L.IB (Zhang al., 2024a)), obtained the results shown in Figure 10.
We observed the following phenomena. The inclusion of additional noisy data results in a decrease in the performance of all algorithms when evaluated the gold reward, compared to the Figure 3 of
31

---

Published as a conference paper at ICLR 2025
Table 4: The ELO scores of different checkpoints
Model
Initial
PPO:79step
PPO:158step
WCO:79sstp
BSPO:79step WCO:158step
58stpp
ELO
1178.30
1181.77
1164.17
1201.77
1195.39
1224.92
1253.68
PPO
ENS WCO
Ours
2
Gold Reward
Reward
-1
50
100
150
200
0
50
100
150
200
Steps
Steps
Figure 10: Training curves of different algorithms with label noise
the paper. Despite this, BSPO consistently mitigates reward over-optimization and achieves better performance than the baseline methods, demonstrating its robustness to noise.
D.7 ABLATION STUDIES OF HYPER-PARAMETERS
D.7.1 ROBUSTNESS VAIN
Vinin is designed suppress the V-values corresponding to OOD actions, necessitating that Vnin be lower than all possible V-values.
In general, Vmin can be determined by calculating the minimum V-value, such Vnin T- Tiins where Mmin denotes the minimum value of the reward function. Specifically, for RLHF of LLMs, where only the final token receives nonzero reward, we have V(s) = (llllr)))) | so min(0 (0, Tmin) = Vmin-
On the other hand, empirical evidence suggests that as long Vmin remains smaller than all pos- sible V-values, the performance of BSPO is not sensitive to the Vmin- To evaluate this, we con- ducted experiments under the condition Fmin 100, testing range of Vnii values: Vnnn 100 15, -20, -25
The resulting experimental outcomes summarized in Figure 11. We observe that, across different values of Vmin- the training curves BSPO exhibit similar behavior, consistently improving both the proxy reward and the gold reward while effectively addressing the issue of over-optimization. These results demonstrate the robustness of the Vmin selection.
D.7.2 ROBUSTNESS OF
Since we use a parameterized model to predict the behavior policy. Thus, to prevent modeling errors, an infinitesimal value eg is used threshold, instead of O, to determine whether response supported. To analyze the sensitivity of the BSPO algorithm to Eg. we conducted experiments
with range of values: 6g le 3, le
le
le 7.
32

---

Published as a conference paper at ICLR 2025
Min
Vmn
Vmin -20
Vmte
Vmn
2
Gold Reward
Reward
0
-2
50
100
150
200
0
100
150
200
Steps
Steps
Figure 11: Training curves of different V Vinin values
EPS 1e
3
EPS
1e
EPS
1e
EPS le 6
EPS 1e
2
Gold Reward
Reward
50
100
150
200
0
50
100
150
200
Steps
Steps
Figure 12: Training curves of different 6g values
The experimental results are presented in the Figure 12. Our observations indicate that for Eg 1e 4, le 5, le 6. BSPO mitigates reward overooptimization and identifies the optimal policy. However, the performance for Eg = le 3 and Eg = le 7 shows slower improvement. This behavior may result from excessively large e values, which can incorrectly classify some actions as behavior-unsupported, or excessively small values, which may fail to account for certain OOD actions. In conclusion, BSPO is robust to eg within a reasonable range, extreme values should be avoided.
D.8 COMPARISON WITH DPO
Direct Preference Optimization (DPO) (Rafailov et al., 2024) is an efficient algorithm for aligning LLMss with human preferences. DPO learns from preference data with the following objective:
Toe To
( (x) 33 log
]
LDPO (TT: T(et) = Eyg yu 3n
D loga 3 log
(58)
Teet y1 (x)
33

---

Published as a conference paper at ICLR 2025
DPO
RPO
0.9
7.6
(twwel Accincyy
Bewod
SPT11cm
0.7
7.0
6.6
750
1000
250
750
1000
1250
500
750
1000
1290
Figure 13: Training curves of DPO and RPO
Table 5: The win rate of DPO and RPO compared to other algorithms
vs. Alpaca-7B (ref)
PPO
vs. BSPO (ours)
DPO
0.6592
0.6226
0.2627
RPO
0.7058
0.6664
0.297
DPO-based algorithms have also been reported to have over-optimization issues (Liu et al., 2024; Xu et al., 2024). However, the over-optimization problems of DPO-based algorithms differ from the reward over-optimization problem that our work addresses:
Over-optimization studies for DPO focus on mitigating overfitting to the training dataset in DPO- based algorithms, which results in reduced generalization to the test set. DPO-based algorithms do not use a reward model to evaluate responses generated from exploration. Instead, the policy learns preferences solely from the "chosen" and "rejected" labels in the training dataset. Consequently, the reward over-optimization problem solved in our work does not arise this context.
contrast, our work addresses a distinct challenge: in RL-bbased algorithms, exploration can produce OOD responses for the reward model, leading to reward overestimation and, ultimately, reward hacking.
To tackle the issue of over-optimization DPO-based algorithm, Liu (2024) proposes Regular- ized Preference Optimization (RPO). This method aims to identify the optimal policy by leveraging an adversarially chosen reward model that minimizes a weighted combination of its expected value and the maximum likelihood estimation (MLE) loss. The corresponding policy objective is
Tee T (--(
CRrO (f) = 17.3 "
[- log( mo (a)[=))] + Lo
log
(59)
The training curves is shown in Figure 13. Although our work does not focus the same problem, we compare the performance of the two approaches with our algorithm and baselines in Table 5.
Both DPO and RPO exhibit improvements compared the original Alpaca-7B model, with the integration of SFT loss further enhancing their performance. The performance of PPO training is relatively poor due severe reward over-optimization issues. In contrast, DPO and RPO do rely on the reward model during training, thereby avoiding reward over-optimization problems and outperforming PPO, Nevertheless, because DPO and RPO lack the ability explore new responses and depend heavily on the quality of the training dataset (Xu et al., 2024), their performance falls short of RL-based BSPO.
For this phenomenon, we believe the following is a plausible discussion for the comparison between BSPO and DPO-based methods:
As demonstrated in Theorem 3, although we apply regularization to the value of OOD actions to avoid overestimation, BSPO can still converge to the same optimal solution for the ID region as the standard value. BSPO retains the exploration capability inherent in RL-based methods. Therefore, algorithms primarily reflects the broader distinc-
the comparison between BSPO and tion between RI-baaed and DPO-based approaches.
34

---

Published as a conference paper at ICLR 2025
As reported in Xu et al. (2024), compared to RL-based methods, DPO-based methods have several limitations: over over-fitting to the training dataset (a different manner of over-opiimization, generat- ing biased policy that favors OOD responses, and sensitivity to the quality of preference data.
D.9 OTHER DATASET
We also applied our pipeline to the AlpacaFarm dataset (Dubois et al., 2024b). Our gold reward model. based on Llama3-8B (Dubey al., 2024), is trained on combined preference dataset com- prising UltraFeedback and a 20k preference dataset annotated by GPT-4 from AlpacaFarm. The gold model with the best performance over three training epochs selected. For the proxy reward model, used TinyLlama (Zhang et al., 2024a), training on the 20k re-annotated AlpacaFarm dataset. This proxy model was then employed the training of PPO, WCO, and BSPO. The results presented in Figure 14.
WCO
Gold Reward
Reward
0
6
-2
2
0
50
100
150
0
50
100
150
Figure 14: Training curves of different algorithms using AlpacaFarm
Due to the relatively smaller size of the AlpacaFarm dataset, the accuracy of the ScoreLM proxy model lower compared to when using the UltraFeedback dataset. This limitation results in an earlier occurrence of the phenomenon in both PPO and WCO. However, only BSPO shows continuous improvement in both proxy rewards and gold rewards, highlighting its ability to mitigate over-optimizationn while identifying optimal policies effectively.
35


Reviews:
[{"review_id": "bohv0rJQvA", "reviewer": "Reviewer_HnWK", "summary": "This paper introduces an approach named Behavior-Supported Policy Optimization (BSPO) to address the challenge of reward over-optimization in Reinforcement Learning from Human Feedback. The core issue addressed is the extrapolation error that arises when the reward model evaluates out-of-distribution responses, leading to discrepancies between the performance of LLMs under the reward model and true human objectives. The authors propose using a behavior policy to model the in-distribution region of the reward model and introduce a behavior-supported Bellman operator to regularize the value function, penalizing out-of-distribution values without impacting in-distribution ones. Theoretical proofs are provided to show that BSPO guarantees monotonic improvement of the supported policy until convergence to the optimal behavior-supported policy. Empirical results demonstrate BSPO's effectiveness in preventing reward over-optimization and finding the optimal ID policy.", "questions": "The paper mentions that using V values instead of Q values provides greater stability, mainly because state transitions are deterministic in the context of LLMs. Specifically, for a given input prompt and a sequence of generated tokens, the generation of the next token is deterministic—given the current state and action, the next state is uniquely determined. In this case, there is a direct relationship between V values and Q values. But what if the next token generation is sampled, introducing uncertainty into the transition function? How does BSPO work in such a situation?\n\nThere is a lack of analysis of the experimental results. What causes the drop observed during the training of CPPO?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-organised and articulated with clarity, the complex concepts are explained in a clear and concise manner. \n2. The paper provides theoretical justifications, and the results are convincing.", "weaknesses": "1. The paper primarily focuses on the synthetic set for evaluating reward over-optimisation, the alignment of the reward with real human annotators’ evaluation remains insufficiently validated.\n2. Although the paper did experiments on the robustness of BSPO with noisy data, a more in-depth analysis of how BSPO performs under various types of noise or distributional shifts could strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ho3TCR5jcM", "reviewer": "Reviewer_PiVm", "summary": "This work addresses reward over-optimization in RLHF by penalizing out-of-distribution (OOD) responses, which are a significant source of extrapolation errors during evaluation. The proposed approach, Behavior-Supported Policy Optimization (BSPO), identifies OOD tokens by checking if the predicted probability from the reward model falls below a defined threshold. It integrates an auxiliary loss based on the supervised fine-tuning (SFT) objective into the reward model’s training. The proposed mechanism helps mitigate OOD overestimation without impacting the model’s performance on in-distribution (ID) data.", "questions": "1. How do you select the value of $V_{min}$ and $\\epsilon_\\beta$?\n\n2. If the SFT loss is not added to the training of reward model, can the reward model still recognize OOD prompts?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper effectively addresses a critical and well-motivated issue in RLHF, reward over-optimization due to OOD responses.\n\n2. The proposed Behavior-Supported Policy Optimization (BSPO) method introduces a unique approach by leveraging a behavior policy for OOD detection and integrating it with value regularization.\n\n3. The empirical validation through synthetic experiments demonstrates that BSPO outperforms baseline methods.\n\n4. The paper’s methodology is rigorously supported by theory, with proofs ensuring the convergence and stability of the behavior-supported value functions.", "weaknesses": "1. A significant concern is that the experiments are conducted solely on synthetic data. This limitation raises questions about the applicability and effectiveness of the proposed method in real-world scenarios.\n\n2. The experimental results lack an ablation study on the sensitivity of the parameter $\\epsilon_\\beta$. \n\n3. The paper does not adequately discuss related works that incorporate SFT loss into the training objective, albeit for different purposes, such as in policy loss. Notable examples include:\n- Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer\n- Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WO6oaujr3r", "reviewer": "Reviewer_DPgD", "summary": "This paper addresses reward over-optimization in Reinforcement Learning from Human Feedback (RLHF), where large language models (LLMs) sometimes align poorly with human objectives due to extrapolation errors when evaluating out-of-distribution (OOD) responses. The authors propose Behavior-Supported Policy Optimization (BSPO), which models the in-distribution (ID) region by defining a behavior policy based on the reward training dataset’s next token distribution. To regularize the value function, BSPO introduces a behavior-supported Bellman operator, penalizing OOD values while leaving ID values unaffected, thus reducing OOD response generation. BSPO is shown theoretically to guarantee monotonic improvement of the policy within the ID region, leading to convergence at the optimal policy. Empirical results confirm BSPO’s superiority over baselines in preventing reward over-optimization, making it effective in aligning LLMs with human intent.", "questions": "- In Eq (6), is the supervised loss calculated with respect to both $y_w$ and $y_l$?\n\n- Regarding the main results in Figure 3, how were the remaining 27k samples, which were not used for training the proxy reward model, utilized? What was the rationale behind splitting the dataset into 30k and 27k?\n\n- As noted in the Weakness section, is there a specific reason for not comparing BSPO with direct preference optimization methods like DPO, KTO, and CTO?\n  - In what ways do you consider BSPO to be superior to DPO-related methods?\n\n- In the first line of Eq (15) in the Appendix, should it be $a' \\sim \\pi(\\cdot|s')$ instead of $a' \\sim \\pi(\\cdot|\\pi)$?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The research focus is clearly presented, providing readers with the essential background needed to understand the main contributions of the paper.\n\n- The core idea of Behavior-Supported Policy Optimization (BSPO) is intuitive and easy to implement, with a strong theoretical ground covering key properties like contractivity, monotonicity, and convergence.\n\n- The synthetic experiments are well-designed to demonstrate the rigor of the proposed method. The results are presented and interpreted clearly, making the comparisons with baseline methods easy to follow and effectively highlighting the strength of BSPO.\n\n- The literature review is thorough, giving readers, even those unfamiliar with reward over-optimization, a clear view of the related research landscape.", "weaknesses": "- The experimental results presented in this paper are demonstrated exclusively on the UltraFeedback dataset, making it essential to evaluate the generalizability of the proposed method by conducting experiments on additional benchmark datasets.\n\n- The paper emphasizes the concept of the \"In-Distribution (ID) region of the reward training dataset\" as a key to avoiding reward over-optimization, with BSPO’s foundational idea built on this concept. Given this emphasis, it would have been helpful to see a performance comparison with DPO-based methods, which directly utilize the ID region of reward training datasets (e.g., win response $y_w$ and lose response $y_l$), to further contextualize BSPO’s effectiveness.\n\n- As noted in the limitations, significant differences may exist between human preferences and model-predicted preferences, underscoring the importance of human evaluation for generated responses. In cases where human judgment is not feasible, using an LLM-as-a-judge (please refer to Reference) could serve as a practical alternative for conducting pseudo-human evaluation.\n\n- Some visual content is challenging to interpret at first glance. For instance, it seems unclear in Figure 1(c) what \"correct/incorrect\" specifically indicates, as this is not explicitly mentioned in the main text. Additionally, Figure 2(a) is somewhat confusing, as it places the \"query+answer (given past tokens)\" at the top and the \"behavior distribution (next tokens)\" at the bottom, unlike the conventional layout where the past context is on the bottom and the future sequence is on the top. A reversed figure layout might have improved clarity.\n\n- Some explanations were not detailed enough. For example, it would be helpful if there were some sentences describing where the (pretrained) proxy reward model $R\\_{\\phi}(\\cdot)$, which is optimized by $\\mathcal{L}\\_{\\text{ScoreLM}}(\\phi; \\mathcal{D})$, is placed at the calculation of $\\mathcal{L}\\_{V}(\\phi; \\pi)$.\n\n**Reference**\n- Zheng, L., Chiang, W. L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., ... & Stoica, I. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 46595-46623.\n\n**Response to Rebuttal**\n- I truly appreciate the authors' efforts in addressing my concerns. I will maintain my original score", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "coCZShPYkS", "reviewer": "Reviewer_JZFB", "summary": "This paper investigates the issue of reward over-optimization in RLHF and introduces Behavior-Supported Policy Optimization (BSPO) to prevent the generation of out-of-distribution responses during RL training. By learning the value function using in-dataset actions, the author refines the policy through PPO updates. Empirical results demonstrate the effectiveness of the proposed method.", "questions": "Please refer to the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The studied problem is important for RLHF and the proposed method is well-motivated.  \n\n* The idea of using reward model training distribution to regularize the value function learning for LLMs is novel, to the best of my knowledge.\n\n* The empirical evidence shows promising results to alleviate the overoptimization issue.", "weaknesses": "Despite the strengths of this paper, I identified several limitations that require further attention:\n\n1. The paper claims that prior constraint optimization methods \"only suppress overestimation in the ID region.\", which I disagree. The worst-case optimization method with ensemble has been shown to mitigate over-optimization for OOD actions [1]. During PPO training, a worst-case reward can also penalize OOD responses generated by the trained LLM. The authors should provide a more accurate comparison of BSPO with previous methods.\n\n2. There seems to be a problem in the proof of Corollary 1. The authors suggest that optimizing the objective in Eq 23 forces the learned policy to choose behavior-support actions. However, there are cases where all actions are OOD for a given prompt $s$, which could inevitably introduce OOD actions. This situation can arise when the SFT model and reward training use different datasets. Thus, the argument may not hold.\n\n3. The current BSPO can only be applied to token-level RLHF, while many studies focus on response-level RLHF. Can the proposed method also be adapted for response-level RLHF?\n\n4. The evaluation is restricted to reward models smaller than 3B. Validating on 7/8B reward models would strengthen the experiments.\n\n5. The authors should explore how the two threshold hyperparameters, $V_{min}, \\epsilon_{\\beta}$, affect the method's performance.\n\n6. Prior work [2][3] considers a noisy label setting, where the over-optimization problem is more severe. Can BSPO be effective in this challenging setting?\n\n7. Typos: In Eq 36, the use of Q function is not correct.\n\n**References**\n\n[1] An G, Moon S, Kim J H, et al. Uncertainty-based offline reinforcement learning with diversified q-ensemble[J]. Advances in neural information processing systems, 2021, 34: 7436-7447.\n\n[2] Ramé A, Vieillard N, Hussenot L, et al. Warm: On the benefits of weight averaged reward models[J]. arXiv preprint arXiv:2401.12187, 2024.\n\n[3] Yang R, Ding R, Lin Y, et al. Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs[J]. arXiv preprint arXiv:2406.10216, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)