Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking
Authors: Cassidy Laidlaw
Shivam Singhal
Anca Dragan
Venue: ICLR 2025 Spotlight (2025)
Keywords: reward hacking
reward gaming
overoptimization
occupancy measures
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We introduce a new definition of reward hacking, which leads to better regularization strategies for preventing it.

Abstract: Because it is difficult to precisely specify complex objectives, reinforcement learning policies are often optimized using proxy reward functions that only approximate the true goal. However, optimizing proxy rewards frequently leads to reward hacking: the optimized reward function ceases to be a good proxy and the resulting policy performs poorly with respect to the unspecified true reward. Principled solutions to reward hacking have been impeded by the lack of a good definition for the problem. To address this gap, we introduce a definition of reward hacking based on the correlation between proxy and true rewards for states and actions seen by a “reference policy” that breaks down under optimization. We show that this definition captures reward hacking behavior across several realistic settings, including in reinforcement learning from human feedback (RLHF). Using our formulation, we show theoretically that regularization to the reference policy can effectively prevent reward hacking. While the current practice in RLHF applies a KL penalty between action distributions for this purpose, our theory suggests regularizing the χ2 divergence between the policies’ occupancy measures can be more effective. We intuitively show the benefits of this type of regularization and demonstrate that it better mitigates reward hacking in practice across four realistic settings, including RLHF. Our code is available at https://github.com/cassidylaidlaw/orpo.

Published as a conference paper at ICLR 2025
CORRELATED PROXIES: A NEW DEFINITION AND IMPROVED MITIGATION FOR REWARD HACKING
Cassidy Laidlaw* Shivam Singhal*
Anca Dragan
Department of Electrical Engineering and Computer Science University of California, Berkeley Berkeley, CA 94709, USA cossidy_laidie shivams shivamsinghal, anca @berkeley. edu
ABSTRACT
Because it is difficult to precisely specify complex objectives, reinforcement learn- ing policies are often optimized using proxy reward functions that only approximate the true goal. However, optimizing proxy rewards frequently leads to reward hack- ing: the optimized reward function ceases to be good proxy and the resulting policy performs poorly with respect to the unspecified true reward. Principled solutions to reward hacking have been impeded by the lack of good definition for the problem. To address this gap, we introduce a definition of reward hacking based the correlation between proxy and true rewards for states and actions seen by a "reference policy" that breaks down under optimization. We show that this defini- tion captures reward hacking behavior across several realistic settings, including in reinforcement learning from human feedback (RLHF). Using our formulation, we show theoretically that regularization to the reference policy can effectively prevent reward hacking. While the current practice in RLHF applies KL penalty between action distributions for this purpose, our theory suggests regularizing the x² diver- gence between the policies' occupancy measures can be more effective. We intu- itively show the benefits of this type of regularization and demonstrate that it better mitigates reward hacking in practice across four realistic settings, including RLHF. Our code is available at https:/ /tt:://gttttttt /github... com/ cassi doiiddlliiiii awwlwoo........o
INTRODUCTION
A major challenge for the designers of goal-oriented Al systems is specifying reward function that robustly captures their goals and values. Manually designing reward functions is difficult due to the ambiguities and complexity underlying real-world scenarios (Ibarz et al., 2018), An alternative is to learn reward functions from human data (Sadigh et al., 2017; Jeon et al., 2020), but these often fail to generalize outside the distribution of behavior seen during training (McKinney et al., 2023: Tien et al., 2023). Thus, a learned or hand-specifien reward function is often just a proxy for the true reward underlying the system designer's intent. Misalignment between the two objectives can lead to reward hacking: learned policy performs well according to the proxy reward function but not according to the true reward function (Russell et al., 2010; Amodei et al., 2016; Pan et al., 2022; Skalse et al., 2022). A reward hacking policy's 's behavior is often undesirable and can be especially catastrophic when deployed in safety-critical scenarios, such as autonomous driving (Krakovna et al., 2019; Turner al., 2019; Knox et al., 2022). Unfortunately, reward hacking common phenomenon (Krakovna, 2018) and has harmful effects in the real world (Lum & Isaac, 2016; Corbett- Daveess et al., 2017; Obermeyer et al., 2019; Milli et al., 2021; Franchi et al., 2023; Kleinberg al., 2023).
The ideal solution to prevent reward hacking would be to perfectly align the specified proxy and unknown true reward; however, in many domains, this is impossible to achieve. For example, imagine trying to design a reward function for a self-driving car. It would need to capture multiple factors: the arrival time, passenger comfort, compliance with traffic laws, and various other considerations-at many of which are difficult to robustly measure and would need to be carefully weighted against each other (Knox al., 2022). In practice, reward hacking can occur even with significant reward engineering
"qqual contribution.

---

Published as a conference paper at ICLR 2025
We present new characterization of reward hacking and method for preventing We define a proxy reward function as one that cor- relates with an unknown true reward function for state-action pairs sampled from some reference policy. However, optimizing the proxy alone can lead to breakdown the correlation and worse true reward than the reference policy. We show
Figure 1:
(8, sampled from Tief
True rewar H(s.a)
(s, a) from optimizing
R with regularization to Tee
(3, from optimizing I? alone
theoretically and empirically that optimizing the proxy with x² occupancy measure regularization the reference policy can allow outperforming the reference policy under the unknown true re- ward.
Proxy reward R(s, a)
efforts. This is evident in the case of recommender systems, where hand-designed reward functions have led to adverse outcomes in terms of user experience (Stray et al., 2022).
Since proxy reward functions for complex tasks are nearly always misspecified in practice, what can be done to avoid reward hacking? There is a lack of principled solutions for preventing reward hacking, stemming from a more fundamental problem: defining reward hacking in formal sense that captures realistic cases. In particular, it is difficult to characterize what makes a proxy reward function "good". We argue that proxies are chosen because they seem to capture the true objective under some reference distribution of behavior. To formalize this, we define a proxy reward as one that correlates with the true reward function under the distribution of states and actions visited by reference policy, Then, we define reward hacking in this setting as when optimizing correlated proxy reward leads to policy obtaining lower true reward than the reference policy (Figure 1). This captures several intuitive cases of reward hacking across realistic environments: traffic control, pandemic mitigation, blood glucose regulation, and reinforcement learning from human feedback (RLHF) (Figure 2).
Furthermore, our definition motivates method for avoiding reward hacking during policy opti- mization by regularizing the optimized policy to be similar to the reference policy. Specifically, we find that optimizing the proxy reward minus a regularization term provides provable lower bound on improvement in true reward. The amount of regularization needed increases as the correlation between the proxy and true rewards becomes weaker. In practice, RLHF already uses this form of regularization through a KL divergence penalty to the reference policy (Stiennon et al., 2020; Bai al., 2022), and Theorem 5.1 provides theoretical justification for why this works. However, our results suggest an important modification: rather than penalizing the KL divergence between the policies' action distributions as currently done in RLHF we should instead penalize the chi-squared (x") divergence between the policies' occupancy measures. We compare both OM to AD regulariza- tion and x" to KL divergence, presenting intuitive reasons why x² OM divergence may be better regularization target in Figures 3 and 4.
Based on our theoretical results, we empirically investigate the benefits of using x2 occupancy measure regularization to prevent reward hacking. Following previous work (Ho & Ermon, 2016), we implement OM regularization using a discriminator network that approximates the OM divergence between policies. We evaluate our approach in multiple reward hacking benchmark environments (Pan et al., 2022). Our results demonstrate that training with occupancy measure regularization leads to better performance under the unseen true reward function in all of the environments, validating our theoretical results. In contrast, we find that it is difficult to tune AD regularization in some environments to both prevent reward hacking and allow meaningful improvement over the reference policy. Furthermore, regularization with x² divergence tends to be more robust to the choice of regularization coefficient and often achieves higher true reward than regularizing with KL divergence.
Our main contributions can be summarized as follows:
1. We provide new formal definition of reward hacking that captures realistic case studies.
2. Using our definition, we establish that optimizing a correlated proxy reward with X2 occu- pancy measure (OM) regularization leads to a provable improvement in the unknown true reward function.
3. We show how to practically implement X2 OM regularization and demonstrate that outperforms the current standard for preventing reward hacking via action distribution (AD) regularization.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
Reward hacking. Some prior works establish theoretical models of reward hacking as a special case of Goodhart's 's Law (Goodhart, 1984; Leike et al., 2018; Manheim & Garrabrant, 2019; Krakovna, 2019; Skalse et al., 2022: Ngo et al., 2023; Fluri et al., 2024; Kwa et al., 2024). Krakovna (2018) 8) catalog many examples of reward hacking. Pan al. (2022) categorize different types of reward misspecification and relate optimization power to reward hacking. See the end of Section 4 for comparison of our definition of reward hacking with previous ones.
Safe reinforcement learning. In the context of reward hacking, regularizing policies to be similar to an offline policy based on their action distribution KL divergence was proposed by Stiennon et al. (2020) and has since been widely employed in the context of optimizing LLMs using RLHF (Ouyang et al., 2022; Bai et al., 2022: Glaese et al., 2022). KL. regularization for RLHF has been further studied by Vieillard et al. (2021), Gao et al. (2022), and Korbak et al. (2022). Nika et al. (2024) propose type of occupancy measure regularization in RLHF but it is limited to deterministic MDPs, while we study general stochastic MDPs. Some alternative approaches avoid reward hacking include quantilizers (Taylor, 2016), "mild" optimization (Taylor et al., 2020), and impact regularization (Turner al., 2020). While constrained RL can prevent the misbehavior of agents that optimize flawed reward functions (Dalal et al., 2018; Chow et al., 2019; Zhang et al., 2020; Roy et al., 2022), it simply shifts the difficulty of designing a reward function to specifying a set of constraints and weights. Robust RL usually considers a misspecified transition model. but some work has explored misspecified reward functions (Derman et al., 2021; Gadot et al., 2024). Other proposals to address the reward specification problem attempt to infer the true reward function based on the given proxy reward function, environment context, and/or feedback from humans (Hadfield -Mnelll et al., 2017: Reddy et al., 2020; Lee et al., 2021). Gleave al. (2021) and Skalse et al. (2024) have previously studied quantifying the similarity of reward functions.
Other divergences for regularization in RLHE. Some work has explored using divergences other than KL divergence for regularization in RLHF. Go et al. (2023) investigate using other f-divergences for aligning language models. Wang et al. (2023a) explore generalizing direct preference optimization (DPO) (Rafailov et al., 2024), a type of RLHF that leverages offline RL, to other f-divergences. The most relevant work to ours in this area is Huang al. (2024), which also proposes using x² divergence as a more theoretically grounded alternative to KL divergence for regularization in RLHF Our work differs from theirs in several key aspects: we address reward hacking in general rather than focussing specifically on learning from preference data; we use online RL instcad of their DPO-like algorithm; and, we validate our approach through experiments in realistic environments while their work is primarily theoretical.
Applications of occupancy measure regularization. Occupancy measure regularization and optimization have been used for a variety of purposes, including imitation learning (Ho & Ermon, 2016) and efficient exploration Kang et al. (2018); Hazan et al. (2019); Lee et al. (2020); Nedergaard & Cook (2023). Various types of distributional regularization used in model-basse RL since learned models may not generalize -ut-o--distribution (Yang et al., 2022).
Offline reinforcement learning. Our work bears superficial resemblance to prior work in offline RL, where algorithms commonly use occupancy measure or action distribution based regularization to constrain the learned policy within the training data distribution (Fujimoto et al., 2019; Lee et al., 2022; Mandal et al., 2023; He, 2023; Cheng et al., 2022; Rashidinejad et al., 2023; Xie et al., 2023). However, the settings are fundamentally different: while offline RL is limited by lack of coverage in the training data, the difficulty in our setting is that the reward function is misspecified. While it might be possible to avoid reward hacking by using offline RL algorithms, we leave this future work and focus on regularization-be based approaches in the online RL setting.
3 PRELIMINARIES
To study reward hacking, we consider the setting of an infinite-horizon Markov decision process (MDP). An agent takes actions a E A to transition between states s E over a series of timesteps t= 0, 1, We assume that S and A are finite for simplicity, but our results can easily generalize to infinite state or action spaces. The first state Sp is sampled from an initial distribution (s))), and when an agent takes action ae in 8, at time the next state 81+1 is reached at timestep + 1 with
3

---

Published as a conference paper at ICLR 2025
Usee: Mhat 18 the ospital of
Stecpptiie Exprood
CGM
abeped
Assistant:
Pump
capital of France La Paris.
Traffic control (r=0.10)
Pandemic mitigation (r=0.75)
Glucose monitoring (r=0.70)
RLHF
AI safety gridworld
(r=0.40)
(r=0.40)
Reference policy
30
0.1
-10-
-5
(4.5, -7.6)
+
-0.1
(000 -0.06)
-100-
6-004 -28)
20
60000. 16.4)
0.05
00.07. 0.06)
-100-
-10-
-1000
12
-0.5
-0.1 -0.05
-0.2
-0.1
0.2
0.4
Reward hacking policy
True reward R(s,
-5-
0.1
-50
-10
(0114)
(1 10.0)
0.05
10:45, 0.02)
19.1.1197.2)
(-0.00, 1066
10
-10
-1000
12
-0.5
-0.1 -0.05
-0.2
-0.1
0.4
Regularized
30
0.1
policy
-10
(4.9. -4.2)
0.1
(-0.05.-000 -0.05)
-100
(.0.11,
20
+
0.05
(0.09, 0.09)
(0.3, 17.0)
-100-
-10
-1000
12
-0.5
-0.05
-0.05
-0.2
-0.1
0.2 0.4
Mean reward
Proxy reward fi(ss a)
Figure 2: Our definition of reward hacking successfully describes reward hacking behavior in four realistic environments and illustrative gridworld. The top row shows the distribution of proxy and true reward values for state-action pairs sampled from a dopaon-appropriate reference policy for each environment; all environments, the proxy and true rewards are correlated. However, as shown the middle row, this correlation breaks down if the proxy is optimized via RL, and the true reward achieved is lower than that of the reference policy, which characterize reward hacking. In line with theoretical results, the bottom shows that RL with occupancy measure regularization the reference policy can prevent reward hacking while enabling an increase true reward.
transition probability p(s++1] St, dt). The agent aims to optimize a reward function It S x A + R. and rewards are accumulated over time with discount factor y [0, 1). A policy ग maps each state 8 to a distribution over actions to take at that state π (a] 8). We define the (normalized) return of policy æ under a reward function R as
JTT, R) = (1 11-10 7) E. EE R(s,, a:)
(1)
where E, refers to the expectation under the distribution of states and actions induced by running the policy π in the environment.
We define the state-action occupancy measure μx a policy и as the expected discounted number of times the agent will be in particular state and take specific action:
Pa (s, a) = (1 - 7) E, > The standard approach to solving an MDP is to find policy æ that maximizes its return J(m, R). However, as we discussed in the introduction, an AI system designer might optimize स using a learned or hand-specified proxy reward function î that imperfectly captures the true reward function R. precise definition of reward hacking would help us to better understand and mitigate the problem.
A a; ====a]]]
Intuitively, reward hacking occurs when when optimizing the proxy reward J (mT, R) of policy ı ultimately leads to low true reward (TT, R). However, formalizing this intuition is surprisingly challenging.
Desiderata for a definition of reward hacking. To understand why defining reward hacking is difficult, consider the case study of RLHF. In RLHF, optimizing a learned reward function over LLM outputs eventually leads to the LLM producing nonsensical responses Stiennon et al. (2020). This clearly satisfies informal definition of "optimizing the proxy makes the true reward decrease."
4

---

Published as a conference paper at ICLR 2025
However, what if were to optimize R(s,a)= -R(s,a)? ? In this case, optimizing R also makes the true reward go down, but arguably this is not reward hacking because R was not good "proxy" in the first place. Thus, a good definition reward hacking should distinguish between "reasonable" proxies and reward functions that are clearly unrelated the true reward function.
In the example of RLHF, when optimization leads to nonsensical outputs, we can clearly identify that the proxy has been "hacked." However, what if optimizing the proxy produced mediocre but not obviously bad outputs? Even if optimizing the proxy does not lead to near-optimal true reward, we wouldn't necessarily classify this behavior as reward hacking. Thus, a good definition of reward hacking should specify a threshold for the true reward: if the true reward falls below the threshold, reward hacking has occurred, and above it, it has not.
To summarize, we argue that a good definition of reward hacking should satisfy two desiderata:
1. Characterizing what makes a proxy reward "reasonable" optimize in the first place.
2. Defining a threshold for true reward below which a policy is reward hacking.
After presenting our definition in the next section, we compare to other definitions using these criteria.
DEFINING REWARD HACKING
Despite the diffieulties in formalizing reward hacking, we argue that both of our desiderata for definition can be met by defining reward hacking with respect to a reference policy. This approach enables both a precise definition of proxy rewards and a natural true reward threshold for determining when reward hacking is occurring.
Characterizing proxy rewards. To find realistic definition of a good proxy reward, consider the process by which system designers create proxies. If they are hand-specified, we argue that designers usually imagine distribution of behavior and design a reward that captures the objective under that distribution. For example, we study a traffic control simulator (Lopez et al., 2018; Vinitsky et al., 2018; Wu et al., 2022; Pan et al., 2022) where a small number of autonomous cars help to regulate traffic among larger population of human drivers. In this case, designer might choose average vehicle speed as a proxy for traffic flow; this is a reasonable proxy because under the distribution of typical human driving behavior, higher speeds are associated with better traffic conditions. We can formalize this intuition by requiring that the proxy and true rewards be correlated over states and actions sampled from reference policy:
Definition 4.1 (Correlated proxy reward). An r-correlated proxy reward R with respect a reference policy Tet is one that has a correlation of r > 0 with the true reward for state-action pairs sampled from the reference policy:
Eppar [( (Blsa)) J( Ri)
R(s,a)- J(H RR
RR
OR
Ep where 0f = Epear (R(s. (TTer R and o = (R(s, [[Trets RI)"
are the variances of proxy and true rewards under the reference policy.
To consider an application of our definition, let's return to the example of our traffic environment, where we define the true reward function as the negative total commute time for all vehicles. Using an autonomous vehicle policy based on a common model of human driving behavior as Tets we plot the distribution of true and proxy rewards in the top-left corner of Figure 2. The clear correlation that we observe between the two reward functions-highe average speed corresponding to lower commute times and vice fersa-confing that average speed is correlated proxy reward according to Definition 4.1.
Definition 4.1 offers key advantages over past formalisms. First, it avoids too strongly constraining proxy rewards by allowing the proxy and true reward functions to diverge arbitrarily at some state- action pairs when those pairs have low it encompasses both learned proxy rewards and hand-specified ones; see Lemma A.8 for a discussion of how learned rewards generally correlated proxies.
occupancy measure under the reference policy. Second,
Choosing a threshold for reward hacking. As discussed in Section 3, defining reward hacking requires specifying a threshold of true reward below which policy behavior is considered poor enough
5

---

Published as a conference paper at ICLR 2025
to be classified as reward hacking. Given that we already characterize proxy rewards with respect to reference policy, this same reference policy provides a natural baseline for evaluating a policy that optimizes the proxy: if optimizing the proxy leads to worse true reward than the reference policy achieves, then the system designer may be better off simply using the reference policy.
Definition 4.2 (Reward hacking), Suppose R is an r-correlated proxy with respect to Tet (Defini- tion 4.1). Then we say reward hacking occurs when a policy π optimized for R has lower true reward than the reference policy Tres i.e., when J (TT, R) < J (Tre, Rt).
If an optimal policy for R. exhibits reward hacking, then we say that R is hackable correlated proxy. for some
J [TT R) < J (Tret, R)
π E arg max ((t, "
For example, in the traffic control environment, optimizing average speed as a proxy leads to autonomous vehicles blocking highway on-ramps, strategy that increases overall average speed by allowing highway traffic to move freely. However, the true reward becomes extremely low because commute times for the cars on the on-ramp are arbitrarily long. This policy is thus reward hacking under Definition 4.2 since the true reward is lower than that of typical human driving.
To test whether Definition 4.2 accurately captures intuitive cases of reward hacking in practice, we consider four realistic environments and an illustrative gridworld. In addition to the aforementioned traffic simulator, we consider pandemic mitigation and glucose monitoring environments, all of which were originally studied by Pan et al. (2022) as examples of reward hacking. PandemicSimulator (Kompella et al., 2020) uses a specialized SEIR (susceptible, exposed, infected, recovered) infection model to simulate the COVID-19 pandemic among a population; the policy controls the level of lockdown restrictions placed on the population by observing the results of testing. The proxy reward omits the political cost of certain decisions. SimGlucose (Man et al., 2014) is based on an FDA-approved simulator of Type Diabetes patients in which a policy monitors glucose levels and administers insulin. The true reward captures patient health while the proxy reward prioritizes reducing the monetary cost of insulin.
Verifying our definition experimentally.
We also study RLHF, in which LLMS are optimized based on a reward function Iearned from human preferences. Following Gao et al. (2022) and Coste et al. (2024), we use a large (more robust) reward model as the true reward function and a smaller (less robust) one as the proxy. Finally, as an interpretable and illustrative example, we include the tomato- watering Al safety gridworld from Leike et al. (2017). Besides the gridworld, all of our environments reflect complex, realistic tasks with large or infinite state state spaces. We construct natural reference policy for each of the five environments; see Appendix C for more details about the environments and reference policies.
The top row of Figure 2 shows the distribution of true and proxy reward values for state-action pairs sampled from these reference policies in each environment. We find that in all cases, the proxy rewards correlate with the true reward, satisfying our definition of correlated proxy. Furthermore, when optimizing the proxies, we see that the true reward drops significantly compared to the reference policy. Thus, these intuitive cases of reward hacking captured by Definition 4.2.
Comparison to other definitions of reward hacking. Skalse et al. (2022) define reward hacking increase in proxy reward accompanied by a drop in true reward. This definition does not satisfy our first desideratum of requiring the proxy to be a reasonable optimization target. Also, the threshold of any decrease in the true reward being considered reward hacking seems overly restrictive; in many cases, optimizing a proxy reward will not lead to a perfect optimization of the true reward. Kwa al. (2024) define the phenomenon of "catastrophic Goodhart" as when optimizing a proxy with low expected етrог under some base policy leads to "arbitrarily high [proxy] reward despite achieving no more [true reward]" than the base policy. This definition is quite similar ours but does not capture that the true reward decreases when reward hacking occurs; furthermore, it is often impossible to obtain arbitrarily high proxy reward in practice. Finally, Fluri et al. (2024) define error-regret mismatch when a proxy reward has low error on training distribution compared to the true reward, but optimizing the proxy produces a suboptimal policy. This definition is also similar to ours but leaves the threshold for reward hacking ambiguous.
Furthermore, none of these works develop a general method for preventing reward hacking. In contrast, will show that definition leads to natural regularization method proxy reward without allowing reward hacking. Also, unlike most of these works, we validate our definition and method through extensive experiments in realistic environments.
for optimizing

---

Published as a conference paper at ICLR 2025
Figure 3: Unlike RLHF, which attempts prevent ward hacking by regularizing action distribution (AD) divergence from the reference policy, our results suggest regularizing using occupancy measure (OM) divergence is more effective. These plots of the glucose monitoring environment show the typical ADs and OMs of two poli- cies. π is close to Fee in AD; it gives slightl less insulin. However, I optimization of the proxy also leads to a vastly different OM with typical glucose levels far outside the healthy range (dotted lines). Thus, regularizing ADs close to The not enough to prevent reward hacking. Instead, divergence between the OMs better captures the reward hacking hehavior.
[T [(ett) = 1.4
x (Wurlluaaa 919.6
sta (0l88 (8)
Mrr (s)
Mem (5)
0.00 a (insulin delivered) (blood glucose level)
0.05
0.10
200
400
5 MITIGATING REWARD HACKING WITH OCCUPANCY MEASURE REGULARIZATION
We now discuss how new definition of reward hacking motivates better methods for preventing it. Ideally, we would like to be able to optimize a proxy reward function and have it translate into an improvement in true reward over the reference policy. To understand how this might be possible, consider again the traffic environment. The reward hacking policy exhibits behavior which is very unlikely under the reference policy: human drivers hardly ever stop indefinitely on on-ramps. That is, optimizing the proxy reward leads to out-of-distributiun states and actions where the correlation that made the proxy good in the first place breaks down. Visually, this can be seen in the second row of Figure 2: the reward hacking policy usually finds state-action pairs with high proxy reward and low true reward that were very unlikely to be reached by the reference policy.
Thus, one solution to prevent reward hacking could be to optimize the proxy reward while avoiding states that are unlikely under the reference policy. The following theorem formalizes this idea.
Theorem 5.1. Suppose that R is an r-correlated-pree -roxx for the true reward function R. and let a it and on be defined as in Definition 4.1. Then for any policy æ such that fir < Peas (i.e., Man (s,a) a)
u (s) = 0), we have
J(n, R)) J (Trof, R)
1 [TT, R)) (TTer R)
V (1-r²)x (M (Mellleme)
(3)
OR
OR
where x2 (HmllHmm)) = 1p
us (s.0) (1,0)
1] the x² divergence between μz and pawe
See Appendix A for the proof. Equation (10) gives lower bound on how much the policy æ improves over the reference policy Ter in terms of the true reward, normalized by the standard deviation of the true reward under the reference policy. Although this exactly what we would like to maximize, we must instead optimize the right-hand side (RHS) of (10) since we cannot access the true reward directly. The RHS consists of two terms: one measuring the normalized improvement of the policy T 's proxy reward over the reference policy and the other penalizing the divergence of xts occupancy measure from that of मet.. Optimizing the first term alone often leads to reward hacking, but by both incentivizing achieve high proxy reward and stay close to the reference policy, и can achieve high true reward.
By scaling the RHS of (10) and removing terms that are constant with respect to æ, arrive the following regularized policy optimization objective to avoid reward hacking:
maximize
J(H, R) Av x2 (M ((
where
A= B=
(4)
The amount of regularization needed to improve on the reference policy depends on the strength of correlation r. The higher the correlation, the lower the regularization strength vI - r Given the prefactor of the RHS of (10), it may appear that a lower correlation leads to larger gain in true reward. However, Lemma A.2 shows that in fact, the lower bound decreases as a function of r.
While Theorem 5.1 does not guarantee that the true reward can be increased by optimizing (4), it does at least allow us to provably avoid reward hacking. In Theorem A.3 in the appendix, we show that is difficult to guarantee an improvement in true reward in general. However, if the lower bound
7

---

Published as a conference paper at ICLR 2025
Prompt: Explain why whales migrate.
Whales migrate for various reasons, including the ability to avoid potential conflicts in their populations, to forage...
Whales migrate for a variety of reasons. They migrate to access food, as well as for security and mating opportunitics...
Whales migrate to and from the ocean because it is the most efficient and pleasant way to travel...
20
divergence
KL divergence
0
-20
80
100
120
log (Ha a)
Figure 4: Our theory also suggests reward hacking can more effectively be prevented by regularizing with x" divergence instead of KL divergence. This plot illustrates how x" regularization more effective at preventing reward hacking in RLHF. Both divergences can be written as the expectation of a function g( log (w=(s, a)/ from zero. The g(-) associated with KI. divergence only increases slowly for large log-ratios, so policies trained with KT. divergence may produce nonsensical text. In contrast, the g(-) for x² divergence increases exponentially, better constraining the L.M to produce text similar to SFT policy.
(s, a)) which increases the penalty on state action pairs based on how far the log- log-ratio
the RHS of (10) can be increased above zero atro-which can be tested empirically- -hen we know that the true reward has also increased. In our experiments in Section 6, we show that in many realistic environments it is possible to increase the true reward by optimizing (4).
Comparison to KL regularization in RLHE. Currently, RLHF calls for the application of KL penalty between the distributions of responses generated by the optimized policy and the SFT policy (Stiennon et al., 2020; Bai et al., 2022). In our setting, we can write this as
(5) That is, in RLHF, the expected KL. divergence between the action distributions of and नie is penalized. Action distribution divergence is easy to calculate and optimize, and training LLMs with (5) seems work well in practice. However, unlike our objective in (4), (5) lacks theoretical guarantees, and it is unclear if it works in other environments.
maximize J(T, R) X(1 - E, [_t 7 DKL 8t) || M |
Our regularized objective in (4) differs from the regularization applied in RLHF in two key ways: our approach uses occupancy measure (OM) divergence instead of action distribution (AD) divergence, and it employs x² divergence instead of KL divergence. In this section, we provide intuition for why choosing OM over AD divergence and x2 over KL. divergence are more effective for preventing reward hacking. Then, in Section 6, we empirically explore applying different types of regularization to prevent reward hacking in the five environments study.
Occupancy measure vs. action distribution regularization. While AD regularization works well for RLHF, this may be because RLHF is essentially contextual bandit problem, meaning that γ = O; in this case, OM and AD divergence are equivalent (see Appendix A.3). However, in other cases, AD divergence may not suffice to prevent reward hacking behavior. This is because, in longer-horizon environments, small change in action distribution at a single state can lead to much higher probability of reaching undesirable states. Figure 3 shows example of this in the glucose monitoring environment: policies that are close in action distribution regularization produce vastly different patient glucose levels. Occupancy measure regularization avoids this issue by directly preventing the distribution of glucose levels from differing too much from the reference policy. In Theorem A.5 in the appendix, we show that in general it is impossible lower bound the improvement in true reward using almost any form of AD regularization, in contrast to our results on OM regularization.
x² vs. KI divergence. Compared to KI. divergence, x² divergence may be more effective for preventing reward hacking because it more strongly penalizes out-of-distiibution state-action pairs. To illustrate this, write both divergences expectations over functions of the log-ratio of the occupancy measures, which we denote d(ss, a):
(px ) Ep,, [d(s, a) + e -e-((s d(s,a)]
x x²(ur prea)) = Eun [ed(sse) + d(s.a))
where d(s,a) a) log (ur (0, a) ()))
(6)
8

---

Published as a conference paper at ICLR 2025
Environment Glucose monitoring (x10²)
Traffic control
Pandemic mitigation
RLHF
Al safety
Method
gridworld
(x10")
Action dist. x²
-1.29 1 0.10
-12.29 1 0.05
-74.8 1 11.8
16.94 0.07
6.24 1 0.09
State occupancy x²
-2.18 + 0.38
-10.88 + 0.15
-54.7
1.0
9.07 + 0.06
State-action occupancy x²
-11.17 0.17
-4.66
9.17 + 0.11
Action dist. KL
0.05
12.20
8.3
16.81
0.27
6.33 1 0.11
State occupancy KL
-1.34
-10.24 0.54
1.4
7.07 0.11
State-action occupancy KL
0.06
-11.73
-8,9 0.5
6.86 1 0.17
Wee
-2.28 0.00
12.26 0.00
-72.6 0.0
16.37 0.00
5.86 + 0.00
No regularization
-57.38 + 3.53
29.577
: 6.86
-599.0 + 1.6
9.16 0.80
2.35 + 0.14
Training with true reward
-0.93 + 0.11
-2.65 + 0.83
43.4 0.8
8.54 + 0.12
Table I: We compare using various types regularization prevent reward hacking the five environments form Figure 2. The median true reward and standard deviation across 5 random seeds shown for the best regularization coefficient for each type of regularization. The bottom rows show results for the baselines: the reference policy Rud. policy trained on the proxy reward without regularization (exhibiting reward hacking). and policy trained the true reward function (impossible in practice, but included performance). find that occupancy measure regularization consistently improves on action distribution regularization, and that x" divergence often outperforms KL divergence.
upper bound
As d(s, a) increases, the optimized policy is visiting state-action pairs that are less likely under the reference policy. However, KL divergence only penalizes d(s, a) linearly, while x² penalizes exponentially, resulting in stronger regularization even with low coefficient. Figure 4 plots the functions (6) and shows how in practice x² divergence better prevents reward hacking RLHF.
EXPERIMENTS
We now show that our theoretical results- whhlth which suggest x2 occupancy measure regularization prevent reward packing-translaa to empirical success in realistic environments.
Occupancy measure regularization in practice. Occupancy measure regularization is more difficult to implement in practice compared to action distribution regularization. While AD regular- ization can be added as a loss term deep RL algorithms like proximal policy optimization (PPO) (Schulman et al., 2017), OM divergences cannot be calculated in closed form. Instead, we follow several previous works (e.g.. Ho & Ermon 2016; Kang et al. 2018) and use discriminator network to approximate OM divergences. Specifically, in Appendix B, we show the objective in (4) can be optimized via policy gradient with an adjusted reward function that depends discriminator dg:
l' a) R(s, a)
doln.a)
where
X2 =E (dl(ornae)) -
and
の = arg min Ep, log(1 + dd
Epper
log + d Arlsaa))
(7)
That is, optimizing a discriminator network de to minimize the given loss can be used to estimate and optimize x2 OM divergence. We alternately train de via gradient descent on (7) and the policy π via PPO based on the adjusted reward. We call this algorithm Dccupancy-Regularizz Policy Optimization (ORPO). See Appendix B for full derivation of these approximations and Algorithm I for a formal description; we also describe how to regularize based on OM KL divergence.
Experimental setup. In each of the five environments shown in Figure with four types of regularization towards the reference policy: AD KL, AD x2. OM KL, and OM x². In Appendix A.S, we show that Theorem 5.1 also holds for state-only occupancy measures the environment's reward function does not depend on the action; thus, we experiment with regularizing based on both state-action and state-only OM divergence. For each environment and type of regularization, we test number of regularization coefficients X. Theorem 5.1 suggests setting À = 0 / ~T for an r-correlated proxy, so for x" regularization we test a range of values = co it from c = 1 to 10 (10-4 for RLHF). Since is less clear theoretically how to set the coefficient À for KI. regularization, experiment with wider range of values.
train policies

---

Published as a conference paper at ICLR 2025
Traffic control
Pandemic mitigation
Glucose monitoring
Al safety gridworld
x10
x10
RLHF
KL divergence X divergence
10
17
-1.5 5
True reward J( RR
12
220
-2.
14
10 2
10
10
10
10
10 2
10
10
10
-1.0
10
10
-1.5
12
10
-2.0-
16
-2.5
14
parny
porry
10'
10
10 10
10
10-2
10a
10
10
10'
10
A/OR
State-action OM
State OM
AD
Reference policy
Figure 5: The true reward achieved by policies regularized with varying amounts of action distribution or occupancy measure regularization using x² and KL divergence. normalized by the standard deviation of proxy rewards under the reference policy. Dots indicate the median reward and the shaded area is the range over random seeds. For RLHF, AD and OM regularization are equivalent, which is why OM regularization results not shown for that column.
x-axis the regularization coefficient
We train each combination of [x2 divergence, KL divergence) x (AD, state OM, state-action OM) {A1, ) with five random seeds and measure the resulting policies' expected returns under the true reward. As a baseline, we train a policy without any regularization, which leads to reward hacking in all environments. We also train policy directly limit for performance. In RLHF, we only consider AD regularization since it is equivalent to OM regularization for LLM chatbots (see Appendix A.3). We do not train policy the true reward for RLHF as we found it could be hacked with enough optimization pressure. See Appendix D for all hyperparameter and experiment details.
the true reward function as upper
Results. The results of our experiments are shown in Table I and Figure 5. Table shows the median true reward with the best coefficient for each type of regularization. We find that OM regularization consistently outperforms AD regularization across the four non-RLHF environments. In reguularizziaa two environments (glucose monitoring and pandemic mitigationn) AD regularization fails to improve on the reference policy's 's true reward at all. Furthermore, X regularization tends to perform similarly to or better than KL regularization across all environments. In RLHF in particular, x² regularization leads to larger improvement over the reference policy compared to the industry-standar KL penalty, and is more stable across seeds as well.
In Figure 5, we show the true reward achieved when training with each type of regularization across range of X values. In addition to performing best with an optimal coefficient, x² and OM regularization seem to also perform well over larger range of coefficients compared AD regularization. In Appendix E, we present the full results of our experiments and ablations of ORPO.
7 CONCLUSION
We have introduced a new definition for reward hacking based on the correlation between a proxy reward function and the unknown true reward that breaks down when optimizing the proxy. Further- more, we leveraged this definition to show theoretically and empirically that x² occupancy measure regularization can more effectively prevent reward hacking than current approaches. Our results have implications for settings, like RLHF, where RL is used to optimize complex, hard-to-specifyy objectives. We suggest that the heuristic KL penalty used currently should be replaced by a more principled form of regularization. While OM and AD regularization are equivalent for today's 's formu- lation of RLHF as a contextual bandit, they will likely diverge in the near future when LLM- based agents optimized over multi-turn conversations or with tool use (Wang al., 2023b; Abdulhai et al., 2023: Shani et al., 2024). Thus, our results provide a principled path to continuing to ensure the safety of increasingly powerful AT systems.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like thank Cam Allen, Micah Carroll, Dibya Ghosh, Katie Kang, and Sam Toyer for helpful discussions and feedback on drafts.
This work was supported by the Semiconductor Research Corporation (SRC)'s Center for the Co- Design of Cognitive Systems (COCOSYS) as well as the NSF's Human Centered Computing (HCC) program. Cassidy Laidlaw is supported by an Open Philanthropy Al Fellowship and National Defense Science and Engineering Graduate (NDSEG) Fellowship.
REFERENCES
Marwa Abdulhai, Isadora White, Charlic Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models, November 2023. URL http:/ http:::://axxv.ooo..eeoe wtt.orgg/abb/.l.aaa 18232. arXiv:231 1.Xiv:23111.8233 [cs].
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety, July 2016. URL http:://aaaxi wttp:/iww.rg/ab//////eea1/ 06565. arXiv:1606.06565 [cs].
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, April 2022. URL hwwp:htt//.. /wwxivi.vvv/ org/abs. 2204. 62862. arXiv:2204.05862 [cs].
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. Pythia: A Suite for An- alyzing Large Language Models Across Training and Scaling. In Proceedings of the 40th International Conference Machine Learning, pp. 2397-2430. PMLR, July 2023. URL https:/ https///pooceeiiggeee malrprres
w220//bidermal
html. ISSN: 2640-3498.
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially Trained Actor Critic for Offline Reinforcement Learning, July 2022. URL http: org/ /bs///////// /2002. 02446. arXiv:2202.02446 [cs].
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzmant and Mohammad Ghavamzadeh. Lyapumov-based Safe Policy Optimization for Continuous Control, February 2019. URL
als 1901 10031. arXiv:1901.1003: [cs, stat].
Sam Corbett-Davier Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. Algorithmic decision making and the cost of fairness, June 2017. URL http::tttt//iiiee wttp://wwwwwwo/////ooooo// 08230. arXiv:1701.085 701.08230 [cs, stat].
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward Model Ensembles Help Mitigate Overoptimization, March 2024. URL /tttp://arxiv.... http//awwwv......aaeaeeae/. www.oogg//bbb////eee 02223/2233300.333... 02743. arXiv:2310.02743 [cs].
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe Exploration in Continuous Action Spaces, January 2018. URL http: org/abs/ 1801.1110 08757. arXiv:1801.0875) [cs].
Esther Derman, Matthieu Geist, and Shie Mannor. Twice regularized MDPs and the equivalence between robustness and regularization, October 2021. URL http: //ap:i//////.//////a//aa/a///m www.ii..g////.........o..ooooert org/ob/////..oo //bb//////// /2100. 16267. arXiv:21 10.06267 [cs, math].
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy S. Liang, and Tatsunori B. Hashimoto. AlpacaFarm: A
11

---

Published as a conference paper at ICLR 2025
Simulation Framework for Methods that Learn from Human Feedback. Advances in Neural Information Processing Systems, 00330039-30063 December 2023. https: http:/ppoeeeiin....eceeeee 5fc4 Sfcc7880eee550000777443333b3 att Cerasttraat-Coofeennneeeee -onferrrcceeeeeeeeee
URL
cc/paper files/ filess/paper22233/aah)
Lukas Fluri, Leon Lang, Alessandro Abate, Patrick Forré, David Krueger, and Joar Skalse. The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret, June 2024. URL http: http:::/aaa.....rggaabb///////// /5006.1155. arX1v:2406.15753 [es].
Ian Fox, Joyce Lee, Rodica Pop-Busui, and Jenna Wiens. Deep Reinforcement Learning for Closed-Loop Blood Glucose Control, September 2020. URL
2009. 09051. WXXx:2009.00005 [es, stat].
Matt Franchi, J. D. Zamfirescu-Peree Pereira, Wendy Ju. and Emma Pierson. Detecting disparities in police deployments using dashcam data. In 2023 ACM Conference Fairness, Accountability, and Transparency, pp. 534-544, June 2023. doi: 10. 10.1145/35930133359445
URL http:
//arxiv. wtt.aii..ggaab/////ooo/oaaa 2305 1 15210. arXiv:2305.152 15210 [es].
Scott Fujimoto, David Meger, and Doina Precup. Off-Policy Deep Reinforcement Learning with- out Exploration. In Proceedings of the 36th International Conference Machine Learning, pp. 2052-2062. PMLR, May 2019. URL https://procceeding mlr. www.mm.p.....ss/mm.mmmmm fujimotol 9a. html. ISSN: 2640-3498.
Uri Gadot, Esther Derman, Navdeep Kumar, Maxence Mohamed Elfatihi, Kfir Levy, and Shie Mannor. Solving Non-Rectangular Reward-Robul MDPs via Frequency Regularization, February 2024. URL hwtp::ttt://atte org/ abs /tdd///23000 2309. 01107. arXiv:2309.01 107 [es].
Leo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization, October 2022. URL http: / arxiv. wwwpiiioorg//bb/// /2abb//22110 2210 10760. 1rX1v:2210.10760 [cs, stat].
Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy
Jonathan Uesato, Po-Sen Huang, Ramona Comaneseu, Fan Yang. Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Sona Mokra, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Iason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irving. Improving alignment of dialogue agents via targeted human judgements, September 2022. URL http: //arxiv. otg/ abs / 2209 14375. arXiv:2209. 14375 [cs].
Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell, and Jan Leike. Quantifying Differences in Reward Functions. In International Conference on Learning Representations, 2021. URL
net forum? i htt.:iddwwwwwweeeeee.. www.:neettoooutteeeeeeeee
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Mare Dymet- man. Aligning Language Models with Preferences through f-divergence Minimization, June 2023.
URL pett:://arxive org/ abs/ 2302. 08215. arXiv:2302.08215 [cs].
C. A. E. Goodhart. Problems of Monetary Management: The UK Experience. In C. E. Goodhart (ed.), Monetary Theory and Practice: The UK Experience, pp. 91-121. Macmillan Education UK, London, 1984. ISBN 978-1-349-17295555 doi: 10.1007/978- 0011779978--344-.00 1-349-172995- URL 1-111999-----9---7-----------
https: doi...i....
Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse Reward Design, 2017. URL httt:://axxi..or/////a abs /bs//11111111 /111/17111111 02827. arXiv:1711.02827
[es].
Alexander Havrilla, Maksym Zhuravinskyi. Duy Phung, Aman Tiwari, Jonathan Tow, Stella Biderman, Quentin Anthony, and Louis Castricato. urX: A Framework for Large Scale Reinforcement Learn- ing from Human Feedback. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 8578-8595 Singapore, December 2023. Association for Computational Linguistics. doi: 10. (0.18633/v12000 emnlp-main.530. 530. URL https ://ac///cccc //acl antho org/ /rgg/2 22... emn 1p-ma 530.
12

---

Published as a conference paper at ICLR 2025
Elad Hazan, Sham M. Kakade, Karan Singh, and Abby Van Soest. Provably Efficient Maxi- mum Entropy Exploration, January 2019. URL arXiv:1812.02690 [es, stat].
Haoyang He. A Survey on Offline Model-Based Reinforcement Learning, May 2023. URL http:
//arxiv. .org aba/230/5 03360. arXiv:2305.03360 [es, eess].
Jonathan Ho and Stefano Ermon.
Generative Adversarial Imitation Learning. In Advances in Neural Information Processing Systems, volume 29, Curran Associates,
Inc., 2016. URL http::t///// http://aapprs.eeeee nips. cc7e22881/.6bbaa992411112 wbt.:ssatta-sbttttttt.... httml.
Stteer/fllleppppe//00// 6/hash/
Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D. Lee, Wen Sun, Akshay Krishnamurthy, and Dylan J. Foster. Correcting the Mythos of KI--Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization, July 2024. URL http::tttaaaat att/abb/2407, 13399. arl..ariivii MtXiv:2407.13399 [cs].
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in Atari. 2018. doi: 10.48550/ARXIV1888 06521. URL https:: /taps://arxivy /ttp/aaxxvvv./a//a org/ abs /atos///11... 06521. Publisher: arXiv Version Number: 1.
Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. 2024. doi: 1..4550/ARXIV2240009222222 URL
https: aarxivv.vv. org/ abs//abb//ta / 6406. 09279. Publisher: arXiv Version Number: 1.
Hong Jun Jeon, Smitha Milli, and Anea Dragan. Reward-rational (implicit) choice: A unifying org
formalism for reward learning, December 2020. URL 04833. arXiv:2002.04833 [cs].
Bingyi Kang, Zequn Jie, and Jiashi Feng. Policy Optimization with Demonstrations. In Proceedings of the 35th International Conference on Machine Learning, pp. 2469-2477 PMLR, July 2018. URL https:::/ppocceeesettttttt/
mllr wtr.peresvse... http///00//////1// 1a..ta.tttt html. ISSN: 2640-3498.
Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. The Challenge of Understanding
What Users Want: Inconsistent Preferences and Engagement Optimization, October 2023. URL
httt:://arxvv.org/// 2202 11776. arXiv:2202.1 11776 [cs].
W. Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward dMis)design for Autonomous Driving, March 2022. URL 13906. arXiv:2104.13900 [es].
Vladimir Koltchinskii. Local Rademacher Complexities and Oracle Inequalities in Risk Minimization. The Annals of Statistics, 24(6):2593-2656 2006. ISSN 0090-5364. URL :ttp:s//swwt h/wp:////www.orrrr org/ sttaae/eeeeeee23333 Publisher: Institute of Mathematical Statistics.
Varun Kompella, Roberto Capobianco, Stacy Jong, Jonathan Browne, Spencer Fox, Lauren Meyers,
Learnngg Peter Wurman, and Peter Stone. Reinforcement Learning for Optimization of COVID-19 Mitigation
policies, October 2020. URL http:: http:://axxvvveee org. abbp//////////////////////// abtpaaassso/o/// 2010.00000000 10560. arxiv.000000000001011 arXiv:2010.10550 [cs].
Tomasz Korbak, Ethan Perez, and Christopher L. Buckley. RL with KL penalties is better viewed Bayesian inference, October 2022. /Rt.itt::://rrvv..oo////aaa//// arXiv:2205.1 1275 [cs, stat].
Victoria Krakovna. Specification gaming examples in AI, April 2018. URL URL.https:///krakonnn wordpress cwm/cooco//////o 201 8/04 /2/////////// sttp:/spcciic.c f ficat inn gaw.inggeeemmmees---nae
Victoria Krakovna.
Classifying specification problems variants of Goodhart's Law,
August 2019.
http///peeeea..o////////////1/11 URL ott.::/ppoooooo.aaaaaaaaaaaeeee. oe
classi wwwciiicaaioonopo......oo
13

---

Published as a conference paper at ICLR 2025
Victoria Krakovna, Laurent Orseau, Ramana Kumar, Miljan Martic, and Shane Legg. Penalizing side effects using stepwise relative reachability, March 2019. URL htthtp::/ org. abss//b/ 1806. 01186. urXiv:1806.00 000000001100 186 [cs, stat].
Thomas Kwa, Drake Thomas, and Adrià Carriga-Alonso. Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification, November 2024, URL attp:///rxivy org/ abs 2407 14503. arXiv:2407. 14503 [es].
Cassidy Laidlaw, Stuart Russell, and Anca Dragan. Bridging RL Theory and Practice with the Effec- tive Horizon, April 2023, URL /ttp:://axxvvoooo wwtp://gggbbbbbbb..oo..o 2tt.b/////4400.533.. 09853. arXiv:2304.09853
[cs, stat].
Jongmin Lee, Cosmin Paduraru, Daniel J. Mankowitz, Nicolas Heess, Doina Precup. Kee-Eung Kim, and Arthur Guez. COptiDICE: Offline Constrained Reinforcement Learnin via Stationary Dis- tribution Correction Estimation, April 2022. URL htt.ittt///ax....rrr//bbbb arXiv:2204.0899 [es].
08957.
Kimin Lee, Laura Smith, and Pieter Abbeel. PEBBLE: Feedback-Efficien Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training, June 2021. URL http:
//ttxxvv.rg////ee..... wbs. 2106. 05091. arXiv:2106.05091 [cs].
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov, Efficient Exploration via State Marginal Matching, February 2020. URL httttt///axxx..orr abs/1906. 05274. arXiv: 1906.002675 [cs, stat].
Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A. Ortega, Tom Everitt, Andrew Lefrancq. Laurent Orseau, and Shane Legg. AI Safety Gridworlds, November 2017. URL http:://tax/ii org/abs/ / 1711.111111 09883. arXiv:1711 100883 [cs].
Jan Leike, David Krueger, Tom Everitt, Miljan Martie, Vishal Maini, and Shane Legg, Scalable agent alignment via reward modeling: a research direction, November 2018. URL http:: org/abs/ /1obs//811.. 07871. arXiv:1811 181.0781 [cs, stat].
Eric Liang, Richard Liaw, Philipp Moritz, Robert Nishihara, Roy Fox, Ken Goldberg, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. RLlib: Abstractions for Distributed Reinforcement Learning, June 2018. URL http: http::/arxxv..rg//ee/e// wwwp:/ivwwrggabb////oaoo 09381. arXiv:1712.09381 [es].
Pablo Alvarez. Lopez, Michael Behrisch, Laura Bieker- Walz, Jakob Erdmann, Yun-Pang Flötteröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamarie Wiessner. Microscopic Traffic Simulation using SUMO. In 2018 21 International Conference Intelligent Transportation Systems (ITSC), pp. 2582, November 2018. doi: 10. (199/TSSC.0018856999999 URL https: httpeeeeeeeeeeeeeeeeeeeeeeeeeeee iwwrrreeeeeee ieee org/ http///dcuueeeeeeeeee /8669938,
ISSN: 2153-0017.
Kristian Lum and William Isaac. To Predict and Serve? Significance, 13(5):14-19, October 2016. ISSN 1740-9705, doi: 10.11 (el1111/j.14409771
URL https: http:://dooo.//////o 10. 11.1111// 1740-777711111 2016 009000060... x. _eprint: http::/cccaeemiiccuuc.coo/sspppp
ht//13//5/144499106666//ggg 1ff__...dffff
Chiara Dalla Man, Francesco Micheletto, Dayu Lv. Marc Breton, Boris Kovatchev, and Claudio Cobelli. The UVA/PADOVA Type Diabetes Simulator. Journal of Diabetes Science and
Technology, 8(1):26 34, January 2014. ISSN 1932-2968. doi: 10.1177/19222988331115222 URL https:// wwww ncbi n1m nih. www.hhc.wo.occc/cccoooooommm http://pp//////..iiiisseee
Debmalya Mandal, Stelios Triantafyllou, and Goran Radanovic, Performative Reinforcement Learn ing, February 2023. URL http: //tt://aaavv.//aa//ea org/ abs 2207. 00046. 2xXiv:227.000446 [cs].
David Manheim and Scott Garrabrant. Categorizing Variants Goodhart's Law, February 2019. URL http::/tt/cce..... /ttp:////xivvi org/ abs/1 803 04585. arXiv: 1oXiv:1803.04555 803.04585 [cs, q-fin, stat].
Lev McKinney, Yawen Duan, David Krueger, and Adam Gleave. On The Fragility of Learned Reward Functions, January 2023. URL http: arxiv. org/ abs/2303 03652. arXiv:2301.0365) [es].
14

---

Published as a conference paper at ICLR 2025
Smitha Milli, Luca Belli, and Moritz Hardt. From Optimizing Engagement Measuring Value. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 714-722, March 2021. doi: 10.
URL http: http://aaxiv.oooo/ org. wwt./rrgggaoo... abs/
2008. 12623. arXiv:2008. 12623 [es, stat].
Alexander Nedergaard and Matthew Cook. k-Means Maximum Entropy Exploration, November 2023. URL /tat:://arxtt org/abs 2200. 15623. arXiv:2205. 15623 [cs].
Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from a deep learning perspective, September 2023. URL http::/arziw..rr/aabbs.t arXiv:2209.06626 [es].
00266
Andi Nika, Debmalya Mandal, Parameswaran Kamalaruban, Georgios Tzannetos, Goran Radanovic, and Adish Singla. Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences. In Proceedings of the s International Conference on Machine Learning, pp. 38 38145-38 0155-3886... PMLR, July 2024, URL https: /poceed ngs. mlr. ISSN: 2640-3498.
press/v2351 nik224 html.
Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464):447-433 doi: /tt.////cccccc.ux2344... URL https: 1 wttp://////e/ceeeeeee.mmmm
October 2019. ISSN 0036-8075, 8075, 1095-9203.
www.ww www.nce.eeeccccc.cc wrg.
science aax2342.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. 2022. doi: / /ttxxviivvv/./////////a////aaav. org/ abs 2203. 02155.
10.48550/ARXIV.22000 220.0255..... URL https: :
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The Effects of Reward Misspecificaaion:: Mapping and Mitigating Misaligned Models, February 2022. URL
03544. 4. arXiv:2201 1.3544 [cs, stat].
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach Vito,,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, Library, December 2019. URL wwt.:ttt///// //tp:v///xiii//// org/ abs 1912.11110 01703. arXiv:1912.01700 [cs, stat].
Deep Learning
Meghan E. Pauley, Kalie L. Tommerdahl, Janet K. Snell- Chell-Bergeon, and Gregory P. Forlenza. Con- tinuous Glucose Monitor, Insulin Pump, and Automated Insulin Delivery Therapies for Type Diabetes: An Update on Potential for Cardiovascular Benefits. Current Cardiology Reports, 24 (12):2043-2056, December 2022. ISSN 1534-3170. doi: 10. 1886-022-01799-x URL https: //tp:i:///ii..// 110. 1007 / 1 0200002----1 01799-x.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly Reward Model, July 2024. URL http: /ttp:i//aavvvvv//v org/ abs 2305. 18290. arXiv:2305. 18290 [cs].
Paria Rashidinejad, Banghua Zhu, Cong Ma. Jiantao Jiao, and Stuart Russell. Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism, July 2023. URL http: //arxiv..rg//ab// 220.3. 12021. arXiv:2103.0 12021 [cs, math, stat].
Siddharth Reddy, Anca Dragan, Sergey Levine, Shane Legg, and Jan Leike. Learning Hu- man Objectives by Evaluating Hypothetical Behavior. In Proceedings of the 37th Interna- tional Conference on Machine Learning, pp. 8029. PMLR, November 2020. URL https:// /ttp:///ppocceiiggsss mlr. ww...pressvilll wttees//v19//eedyyyuu0 reddy20a.aaai .tml. ISSN: 2640-3498.
Julien Roy, Roger Girgis, Joshua Romoff, Pierre-Lue Bacon, and Christopher Pal. Direct Behavior Specification via Constrained Reinforcement Learning, June 2022. URL http: / /ttp//xxvvvo////// org/ bbs/2111 12228. arXiv:21 12, 12228 [cs].
15

---

Published as a conference paper at ICLR 2025
Stuart J. Russell, Peter Norvig, and Ernest Davis. Artificial intelligence: modern approach. Prentice Hall series in artificial intelligence. Prentice Hall, Upper Saddle River, 3rd ed edition, 2010. ISBN 978-0-13-60425944
Dorsa Sadigh, Anca Dragan, Shankar Sastry, and Sanjit Seshia. Active Preference- Learning of Reward Functions. In Robolicy: Science and Systems XIII. Robotics: Science and Systems Foundation, July 2017. ISBN 978-0-9923747-3-00
doi: 21.5607/RSS.2217.XII....
URL
John Schulman. Approximating KI, Divergence, March 2020. URL http: /ttp:///o/co..///// net/
html.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Pol- URL /
icy Optimization Algorithms,
arXiv:1707.06347 [cs].
Lior Shani, Aviv Rosenberg, Asaf Cassel, Oran Lang, Daniele Calandriello, Avital Zipori, Hila Noga, Orgad Keller, Bilal Piot, and Idan Szpektor. Multi-turn Reinforcement Learning from Preference Human Feedback. arXiv preprint arXiv:2405. 14655, 2024. URL https: /ttpp:://///xvv./// org org/abs/////aaa 2405. 14655.
Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and Characterizing Reward Hacking, September 2022. URL http: http:://axxw...r///abb... wwtp/bs//ro/////oooeooooo..b.. 2209. 13085. arXiv:2209. 13085 [cs, stat].
Joar Skalse. Lucy Farnik, Sumeet Ramesh Motwani, Erik Jenner, Adam Gleave, and Alessandro Abate. STARC: A General Framework For Quantifying Differences Between Reward Functions, December 2024. URL http: http::://axxx..rrgg/// httpg//bb/////// 15257. arXiv:2309.15257 [es].
Garry M. Steil. Algorithms for Closed-Loop Artificial Pancreas: The Case for ProportionaalIntegral Derivative Control. Journal of Diabetes Science and Technology, 121:11111 November 2013. ISSN 1932-2968. URL https: //www//w.../m.////////o ncbi nlm. nih wwwphhh.gopp////aracicoomo PMC3876341/.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, September 2020. URL http: / /tt///aivvvv.////ee wwwpxi..rr//bs///00000
01 325.. arXiv:2009.01. 325 [cs].
Jonathan Stray, Alon Halevy, Parisa Assar, Dylan Hadficld-Menell, Craig Boutilier, Amar Ashar, Lex Beattie, Michael Ekstrand, Claire Leibowicz, Connie Moon Schat, Sara Johansen, Lianne Kerlin, David Vickrey, Spandana Singh, Sanne Vrijenhoek, Amy Zhang, McKane Andrus, Natali Helberger, Polina Proutskova, Tanushree Mitra, and Nina Vasan. Building Human Values into Recommender Systems: An Interdisciplinary Synthesis, July 2022. URL ttttt:://aaae /ttp::/axxivvoooooee abs 2207 10192. arXiv:2207.10192 [cs].
Jessica Taylor. Quantilizers: A Safer Alternative to Maximizers for Limited Opti- mization. March 2016. URL http::/ww..eematicceeete pttpppprrr.rppppppe////epppp
Jessica Taylor, Eliezer Yudkowsky, Patrick LaVictoire, and Andrew Critch. Alignment for Advanced Machine Learning Systems. In Ethics of Artificial Intelligence. Oxford University Press, August 2020. ISBN 778-0-19-090505.2
Google-Books--D lyT3DwAAQBAI,
Jeremy Tien, Jerry Zhi- Yang He, Zackory Erickson, Anca D. Dragan, and Daniel S. Brown. Causal Confusion and Reward Misidentification in Preference-Bane -Basedd Reward Learning, March 2023. URL http::/arxtt org/ abs 2204 06601. arXiv:2204.066 [cs].
Martin Treiber, Ansgar Hennecke, and Dirk Helbing. Congested Traffic States in Empirical Observa- tions and Microscopic Simulations. Physical Review E, 62(2):1805-18)) August 2000. ISSN 1063-651X, 1095-3787. doi: 10. 10.100//hyssRee..22...088 1805. URL http: /ttp//xxivv////////////////aatv/ www.voorgggooo owg/abs/////.ooooo.../.......... cong-mat/00022177 axXiv:oond-ma//002117..
16

---

Published as a conference paper at ICLR 2025
A. M. Turner, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. Optimal Policies Tend To Seek Power. December 2019. URL https: /www weseantiisccoolle http/paper/Opptimee waliimaaaoolliieii 46d4452eb04 e33 ro-
Swweeeeeee--eeeeeeeeeeeeee Smith/
Alexander Matt Turner, Neale Ratzlaff, and Prasad Tadepalli. Avoiding Side Effects in Complex Envi- ronments, October 2020. URL hwtpt:ttt//aatteee http://awxx...rr//////00.0.00 [cs].
htlgg/ab//2000 06547. arXiv:2066.06547
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, Sergey Levine, and Karol Hausman. Jump- Start Reinforcement Learning, July 2023. URL attt:://axivv. org.absrrgabbb. /bw.ab////2/22 02372. arXiv:2204.02372 [cs].
Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist. Leverage the Average: Analysis of KL Regularization in RL, January 2021. URL
14089. arXiv:2003. 14089 [cs, stat].
Eugene Vinitsky, Aboudy Kreidieh, Luc Le Flem, Nishant Kheterpal, Kathy Jang, Cathy Wu, Fangyu Wu, Richard Liaw, Eric Liang, and Alexandre M. Bayen. Benchmarks for reinforcement learning in mixed-autonomy traffic. In Proceedings of The 2nd Conference on Robot Learning, pp. 399-409, http://ttttttttteeeeeeeeeeeee
PMLR, October 2018. URL https:://// httpppppcceenggn.e/e mlr. www.pprssssse.... wwwppppwssssssseeeeeepeep http:////vv/viittllllel
html. ISSN: 2640-3498.
Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond Reverse KL:
Generalizing Direct Preference Optimization with Diverse Divergence Constraints, September 2023a. URL http: htLhttp:/arrxiv.co http://..rrgabb//333322
http://axxvvvvee/
abs httg://bs/330..../ 11200916624.00 16240. arXiv:2309.12440 [cs].
Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating Ilms in multi-turn interaction with tools and language feedback. arXiv preprint arXiv:2309.1069) 2023b. URL https: :ttp::/i:////v...../////a/aasmm/
www.ii.w..........o........o.o.t abs wtw.aa/.......//// 2309. 10099.00066 10691.
Cathy Wu, Aboudy Kreidieh, Kanaad Parvate, Eugene Vinitsky, and Alexandre M. Bayen. Flow: A Modular Learning Framework for Mixed Autonomy Traffic. JEEE Transactions Robotics, 38 (2):1270-1286, April 2022. ISSN 1552-3098, 1941-0468. doi: 00.1109/R00220113338777 URL http:://axxii otg. abs /ww.s//1/10 710. 05465. arXiv:1710.0544 [es].
Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent Pessimism for Offline Reinforcement Learning, October 2023. URL abs/2106. 06926. arXiv:2106.06926 [cs, stat].
Shentao Yang. Yihao Feng, Shujian Zhang, and Mingyuan Zhou. Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning, June 2022. URL 07166. arXiv:2206.07166 [cs].
Yiming Zhang, Quan Vuong, and Keith W. Ross. First Order Constrained Optimization in Policy Space, October 2020. URL http: //tpx//axvvvv//////////e///aaavv org/ abs/ 2022.000000 06506. arXiv:2002.06506 [cs. stat].
17

---

Published as a conference paper at ICLR 2025
Appendix
A PROOFS AND ADDITIONAL THEORETICAL RESULTS
A.1 PROOF OF THEOREM 5.1
Theorem 5.1. Suppose that R is an r-corvelate-proxyy for the true reward function R. and let RR and OR be defined in Definition 4.1. Then for any policy such that per & prrne (i.e.,
Man (s, a) = 0 = (x( a) = 0), we have
[ R) (Treff R)
1 [T,, R) ( R)
V(L r²)x² (He (olellaaa
(3)
RR
where x2 (urllusm)) Epprr
He (s.a) (on()
1 the x² divergence between pex and yaur
Proof. For simplicity of exposition, define
a(s,a) R(s, a) [ R)
and Ž(s, R(s, a) [ R)
Combining (1) and (2), we can see that the return of policy can be re-written as the expected reward for states and actions sampled from the occupancy measure:
J(T, R) = (Essssxa a) Ep,, [R(s, a)).
(8)
Using (8), we can rewrite (10)
Ey,, as -r 7(s S (11 (11-22) (1=x2)x² (pzll
Then, the left hand side can be rewritten as
Enp [(s,, a) - - ((x
= En,, (Z(s.a) - Z(s, a)}
Epprue
[(ssaa) - Z(s, a)
+ E
[((s,a) - - 2(s, a)
= E,,, ((s,a) - Z(s, a)]
Eppruc
?(s,a) - Z(s, a)
since by definition Ey [2(s, = Eprre to this difference gives
[[(s = 0. Applying the Cauchy- Schwartz inequality
E ((88)) a) +Z((s,) Byrest ((s) r 7(s
Σ
(((s,a) - *((s a5) (ur(s,a) - Mr (s)
(MO)ESXA
VVnae (2(1, [ (x(s,a) a)) M (8,
Σ
(8.0)ESXA
Mo (8, [2(s,a) - 7(0,,0)²² (r
Σ (x.)ESXA
(Mr a) (s,
Mo (8, a)
(s.a)ESXA
(2(s,a)- -r ((,sa)) x2 (uTlluta)
Epsrat
18

---

Published as a conference paper at ICLR 2025
The expectation can be calculated as
( (((s,a)-r((ss.0)
Ey
Var (2(s,a) 2(s,
(2(5,a)) + +r² Var (2(x,a)) 2r Cov ((s,,a);(ss,0)
-aarpaat
using the fact that both reward functions have unit variance under the reference policy and that their correlation is r. Plugging this into (9) gives the desired result.
A.1.1 NEAR-OPTIMAL REFERENCE POLICIES
While Theorem 5.1 shows that optimizing the proxy reward with regularization can improve on the reference policy's 's reward, it does not guarantee that the learned policy will be near-optimal. However, simple corollary shows that the reference policy is near-optimal, then the learned policy will also be near-optimal.
proxy Corollary A.1. Suppose that R is r-correlated proxy for the reward function R, and let
o and on be defined as Definition 4.1. Furthermore, suppose that the reference policy uee is near-optimal:
[TTzy R) 2 max J( ( R) 60 n.
for some O.
Then for any policy π such that Mr & Maap we have
max,, J(T R) [(, Se-
J(T, R) J (Tees R)
(11-12)x² (uall(aae) (10)
O
This result bounds the suboptimality gap-how how close the learned policy to optimal- in terms of the suboptimality gap of the reference policy and the increase in the regularized proxy reward. The proof is straightforward and follows from Theorem 5.
A.1.2
UNDERSTANDING THE LOWER BOUND IN THFOREM 5.1
Denote by
((x) J(T, R) - ( R)
(1- -2)= (Mx (
(11)
Of
the lower bound on increase in the true reward which is the RHS of (10). One surprising observation is that this lower bound seems to be increasing as the proxy becomes less correlated with the true reward. This would suggest that a less correlated proxy leads to better optimization of the true reward. However, the following lemma shows, L(x)) is actually decreasing in
Lemma A.2. Under the same conditions as Theorem 5. the lower bound L(x) satisfies
L(T) < VI
1
x2 (M MM
(12)
This shows that the lower bound can be at most factor of 1-VI-r times the divergence between the learned and reference policies' occupancy measures. This factor is increasing r and asymptotes r/2
19

---

Published as a conference paper at ICLR 2025
Proof. Using the same notation the proof of Theorem 5.1, rewrite the lower bound
I(x)
( E as)
v(1 x²) (M* (Mll)
Following a similar argument that in the proof of Theorem 5.1, can write
Bag (s0 a) = 1r a))
Ee ((s,a)
px a) Pa (s, a)
E
V Un (8, (,,0) (s)
/ (8,
(A.0)ESXA
(Ur(s, () Unn (8, a)) Ma (8.
Σ
Maa (8, (8,))7(8,a))
(s.
(Ha)ESxA
Rer a)) x2 (1alll1TT))
(Hr (.llT
Combining this with the definition of L(x) gives
L(x) S
Vx2 (UTTllTT v(1 - x2)=22)x2 (Jx (M=))
(Hall
which completes the proof.
A.1.3 Is THE LOWER BOUND OPTIMIZABLE?
While Theorem 5. shows that the increase in true reward over the reference policy can be lower- bounded by the the proxy reward minus regularization term, it is not clear if is actually possible to increase the lower bound ((T) as defined (11). For example, if the reference policy is already optimal with respect to the proxy reward, then clearly L(T) < 0. As another example, suppose it possible to improve Tref with respect to both the true and proxy rewards, but only by visiting a state-action pair never visited by ree In this case, it is also impossible improve the lower bound in Theorem 5.1 while obeying the requirement that (4 < P
We prove two results relating to whether L(T) can increased above zero. Lemma A.3 shows that in general it is difficult to show when the lower bound can be oher are general counterexamples where it cannot be positive. However, Lemma A.4 shows that there are MDPs with r-correlated proxies for any where the lower bound can be positive.
Furthermore, we show in our experiments, in many realistic environments it does appear that the lower bound is optimizable. Furthermore, Theorem 5.1 still allows for safe optimization of the proxy reward: even if the not possible to increase the lower bound above zero, optimizing [(xr will least prevent reward hacking.
Lemma A.3. Fix any r E 10, Then there is an MDP with a true reward function R, a proxy reward R. and a reference policy Try such that R is an r-correlated proxy that can be improved upon in both true and proxy reward by policy n²:
( R) > J (Treff R)
J( it) > J (TTres R) Mo. < Mesd
20

---

Published as a conference paper at ICLR 2025
However, for any policy such that Mx Hoi
L(T) 1 ( J(T, R) - ( R)
(1- (un p
S O.
ft
That is, Lemma A.3 shows that there is an MDP where there exists policy π² that improves on the reference policy in both true reward and proxy reward, but it is still not possible increase the lower bound [(rr) above zero. This suggests that it is difficult to specify general conditions under which [(T)) can exceed zero. Thus, we rely on the empirical evidence from our experiments to show that the lower bound is often optimizable.
Proof. We consider MDPs with discount 7 = O, such that the transition probabilities are not relevant; only the initial state distribution μo and the reward functions specify the MDP. We split the analysis into two cases depending on whether S 1/2 or 2 1/2.
Case 1: S 1/2. We define an MDP with two states 81, 82 and two actions 01 U2, with initial state distribution and rewards follows:
1
po($1)= 1+
01)
R(s1, a1) 1-r
R(81, 02) = - r
R(s1, (2); =0
Ho($2) = 1+r
r
= R(s2.))
R(s2, =-
Furthermore, we define [ar s1) 1 and (ar $1) r. and (rer(a1 (2) Based on this, simple algebra shows the following facts:
1
Maa (81, .1)
= +
Mm (81,
1
J (Trets R) = (TTretss R) =
on = 0R
1++
Ey ((ss,a)((ssa))
Itr
Based on this, it clear that R is an r-correlated proxy. Furthermore, letting ग. T'(a1 (1) = 1 and n² Ta(a; $2) 1, we have
J(TT, R) 1 +r
> [[Teet: R)
r
J( 12) 1+r r
r
[ I7)
Px < Pmtt
satisfying the conditions in the lemma.
Now, consider any policy such that fo < P We can calculate the lower bound L(T) (ignoring the prefactor of !) as
J(T, fi
(1- - ( [
(13)
Since & Mag T(aj 82) = 1, so it can only differ from The in state 81. Let = =<(a1
81) 7ee 81). Then,
write the first term of (13)
J( [
+r==
1-
21

---

Published as a conference paper at ICLR 2025
Note that the x² divergence between distributions j and can be alternatively written as x((llo)=
(u(s, a) a(s,a))²
E
v(s
3,a
Therefore, the x2 divergence between frr and ponn can be lower bounded as
2
L-n+d 1+r
l-r
x² (HIll(tm)) 2 (1 a1) - (T (81, This leads to the bound on (13):
I+r
6²
Ma Mm (11, 01)
- p2
I+r
r 1- V0- (1-
82
rl(x) S 6
18|-
Clearly if 5 this is non-positive, and if 8 it is also non- non-positive since vr/(1 r2) < 1 as long as r < 1/2. Thus, the lower bound cannot be increased above zero in this case.
Case 2: r 2 1/2. In this case, we define an MDP with three states $1, $2, 83 and two actions 01 0g. with initial state distribution and rewards as follows:
2r 2r 1
(o(s1) 72
R($1,a1)
R1)1,a1)= = r
R(81,02) =
M((11 02)1, =
(1 (1-r)2
(o((2)= = y2 -
R(s2,))=0 =0
R(s2.))= = 1 1-r
- (2r 1)
po(ss)
R(s3.)) - 1-r
R(s3,)) 1-r
72 r+
We define the reference policy oe as follows:
22
(1 r)²
(1 $1) 2r 2r + 1 (1 $2) =
(e s1)
2r 2r + 1
Trer(11 53) =
As above, we can show the following facts:
2.2
(1 (1-r)²
Pan (11, 72
MT a2)
72 +1+1
r
[TTer, R) = [[(Trel: 00
OR -R=OR =
1 -
2
Epen
= 7
Again, this shows that R is an r-correlated proxy. Letting >" (1) Ta |2) = =a1(a1
$3) = 1, we have
--r
J( R) =
> 0 = [TTer R)
72 -
r
r)
1 r
J(T", R) = ,2 -2-
> 0 = [ R)
M. < Hmm
22

---

Published as a conference paper at ICLR 2025
satisfying the conditions in the lemma.
Now, consider any policy т such that Ux < Maw We can calculate the lower bound ((T) (again
ignoring the prefactor of })) as
r+l
(TT, B)V
v(1 - r²)x² (fm (|
(14)
r
As in the previous case, let Ā(a1 181) Teer(ai
$1). Then, we can write the first term of (14) as
JTr Ri) 12 12-
2r 2r² 2r +r++l
12
2r 2---21 2r
8
x2 - +
r r
The x² divergence between je and Am can be lower bounded as
2r 6
x² (Ma)))aa)> (MI($1,,)) (1) P (31, (1);
/212 - 2r +
1
8²
Ma (81,))
72 +
-1
This leads to the bound on (14):
rl(x) < 21² - 2r +
--r
(1- (2
2rr - 2r +1
V -2-r+1 r
-+1
2 2r +1
r2 r (5VI-r- 181v 12) ,2)
The prefactor is positive, and as in the previous case, if S then the bound non-positive.
5 >> O, then the bound also non-positive since vI- v1 - ,2 for any r E (0, 1). Thus, the lower bound cannot be increased above zero in this case either, completing the proof.
O
Lemma A... Fix any (0, 1). Then there is an MDP with a true reward function R, a proxy reward R. and a reference policy Tef such that R is an r-correlated proxy and:
1. There is a policy x² such that L(T²)= > O.
2. Any optimal policy with respect to I(.)) is also optimal policy with respect to the true reward function:
arg max L(T) S arg max J(T, R).
Lemma A.4 shows that no matter how low the correlation coefficient is between the true and proxy rewards, there is at least some MDP where the lower bound ((T) can be positive. Furthermore, in this MDP, maximizing M(x) actually leads to optimal policy with respect to the true reward function.
Proof. As in the proof of Lemma A.3, we construct MDP with discount factor y 0 that has three states and two actions. The initial state distribution and reward functions are given by
1-r
Mo($2) 3+
+r r 8
(o(s1)
Mo(ss)
4
R(s1, a1) a1)
1/2
1 r
R(s2.))
V 3+r
-
R(ss.-)
2
2
1 1
R(s1, a2) = RRs1,, a2) =
2--r
1-r
R(s2.
2 1-r
R(33..)
2
2 +
1-r
Itr
+r
We define the reference policy कea as follows:
(ar | $1) ( s1) 1/2
ma 82) = 1
Ter(aa | 88) 1.
23

---

Published as a conference paper at ICLR 2025
We show the following facts:
TTtt R) = J Teets R)
1 -
1 +
1
1rr
3
--r
3+r
1 --r
2
1-r +
2
2
8
1
8
3 +r
3
=0
2 = R
1-r
+r
1-r 2
1r
3+r -
3+r
-
R
8
1-r
+r
8 3+r
E of
R(s,a) a) R(s,) a)
1-r 1+r
1-r 1+r
3+r 1 -
3+r --
2
2
8
1-
3 8
3+
Thus, R is an r-correlated proxy.
Since the rewards for both actions are identical at $2 and $3, policy can only differ meaningfully from Tel at $1. Define
1 + A
aa $1)
< | $2) < $3) 1.
2
as any such policy where A E
[-1/2, 1/2].
Then, we can calculate the lower bound L(T as
L(TA)
[(TA, R) - v(L - r2)x (1Aa (|Pssaa)
1 1- r
lrr
1 (1-) (1-)))
2A
2
8
--r
(1 + r) - -r) |له|
r1-1) + +r)
2
2
r1 r)
(A - ---A))
2
Since v1 - r < 1, clearly this is maximized at A = 1/2, where
L(TA) = 1
r1 r) VI r= 2
> 0.
Furthermore, T1/2 is optimal policy with respect to the true reward function, since R(s1.a) R()1.02) and 11/2(a1 (1) 1. Thus, letting "t" 11/2 completes the proof.
A.2 FAILURE OF ACTION DISTRIBUTION REGULARIZATION
As discussed in the main text, the OM regularization method we propose differs from the AD-based regularization found in previous work on RLHF. The following thcorem shows that almost any form of policy optimization with action distribution regularization cannot guarantee an improvement in true reward over the reference policy.
Theorem A.5. Fix 1). Consider a policy optimization objective regularized by any f- divergence between the action distributions of the learned policy and the reference policy:
L'(T) = J(T,, Rt) (TTer R2) (u-
Σ D' (((: st) And(. st)) E
maximize
1=0
where f : 10, R is convex, continuous function with f(1) = 0, g. 10, 00) (0) is
24

---

Published as a conference paper at ICLR 2025
strictly increasing function with g(0) 0, and Dy is the f-divergence:
Dy(P I1 Q) = Eura [ (2))
Then there an MDP, reward functions R, RS and reference policy ae such that R is an r-correlated proxy for R. but there is policy ã such that
L'
but J( < [ R).
Theorem A.5 concerns general form of policy optimization with action distribution regularization; it considers any f-divergence between action distributions and any way of scaling the f-divergence using the function g. For instnce, could incorporate linear scaling of the divergence in the KL regularization used in previous work, square-root scaling as we use with x2 divergence in Theorem 5.1, any other scaling. The theorem shows that no matter how the regularization formulated, it cannot guarantee improvement in true reward over the reference policy; there is policy that increases the regularized objective but decreases the true reward.
Proof.
Define the inverse g= [0, oc) - [0, xo]
9 (x) sup {y (0) g(v) s x}
Since is continuous and f(11 = 0, there must be a radius p> such that
2g 1-r
(
p
f(u) 1 -
We construct an MDP with discount factor
2g 1(22)) 1 11 (1 f2)))(2) 4tp12
max
f(2) >
max
otherwise.
There are four states and two actions. The initial state distribution and transition probabilities are
1 +
(o()1)
p(s1 81, 01)
p(8
81, a2)
4
1 1
(o(s2)
pss2 $2, a1) =
p(3
$2, 02) =1
(1 (-+)(1+))
(8(83)
p(8a 83, ai)
p(ss
83, (2)
r1 77)
(o(54)
p(84 84, ai)
p(34 84, 14,
The reward functions only depend on the state and are given by
R(s1. 1
R(s .)
R(82,)) .) =
i((2,))
R(s3. :) =
R(ss, =
R(s4.)) -1
R(s 1.
In graphical form, the MDP is follows:
25

---

Published as a conference paper at ICLR 2025
Pg 4
40
$1 R =
82
R
01, d2
H =1
R =
01 02
14 (-))((+))
[1-r)(10) (1-2)
83 ==1 1 R=-1
R = -1 R=1
02
We consider the reference policy defined by
[er(a1
81]
Tee (2
81) = 0
[e (0) [2]]
Tro (2
82)
[er [3)]
Ter (02
[8)) = 1 1-7
(44
Tee (02
14) 0.
To compute the occupancy measure of the reference policy, we first can see that clearly,
1 +
P (81, 01) M (82, a1)
For 83, the agent stays in the state until takes action 02 so we have
Pa (8a) = (1 - (8) [1 + 72 + + + ...
= (1 (1-7) (1 (1 +7)
4
1 1-72
1-r
4
This also implies that Mxx (st) 1-r since the occupancy measure sums to one. Thus, we can compute
[TTer R) = 0 OR
J( R) = 0
0 =
E Perd R(s 0) M((, a)] =
This confirms that R is an r-correlated proxy for R.
Now, we define ăे as
(ac [a]
{1] = [g)
Tea (2g
81)
Tee [1
Tee (ag 52) 0
Tee [01 8z)) = 27 1
Tea (2g 83) = 2(1 - -7)
Tre [0] (84) =
(ag 51) 0.
Note that since 7 > 1/2 by definition, the policy is well-defined. As for mee. the occupancy measure
26

---

Published as a conference paper at ICLR 2025
of के at $1 and is _ For 83, have
(*(3)) = (1 (53) (17 + 1)2 +...]
-1 7) (1 - r)+1
1- 1)
(1-r) 77)
4(1 + 27)
For 84
(*(ss)) = 1 - (+((11) (xx (s
1-r (1-r)(1+1)
2
4(1 27)
- -r)(1 +37)
4(1 + 27)
Based on this, we can compute the true reward of #:
J( R)
Eys
[R(s,a) a)]
1+r
1+r
(1-r)(1+7)
(1-r)(1 37)
4
4
4(1 ((1+22) 27)
4(1 27)
7(1 -r)
2(1 + 27)
$ 0- = [TTe, R).
This verifies the claim that J( R) < (T R).
To show the second claim, we can compute the regularized objective L'( Starting with the proxy reward term, we have
+r
1+r
(1-r)(1+1)
(1 (1-r)(1+ 3y)
J(H, R2)
4
4
4(1+29)
4(1 + 2y)
711 r)
2(1 + 27)
1-r
(15)
8
Next, we compute the regularization term, which can be written as
(1 - BB
Σ DD [(. se) Teeel 82))
1=0
g (u(ss) Dy F[- 8g)
|| Feee(- s8))
since niet and only differ in state s3. We can rewrite the above as
(1 (1-r)(1+9) 7)
a(aa $3)
7(as 53) Teer(a2 83)
-9
meer(a1 ss))f
+ Tar((22
83)f
4(1 27)
Tee((a 183)
(1-
+2)
7f (2 )) + (1 - -)) (2) )
9
2
(16)
((1+27)
27

---

Published as a conference paper at ICLR 2025
Note that since y I4p have
1
S 1 + p
7
1
2
2 11-p
7
))
2y - (1-r)
f
7
1-r r
Plugging this back into (16) along with the fact that (1 - (2) v 29 (-5))
get
(1-r)
- ()1+ 7)
2g ( (1-r)
2g (1=)))
)
<
? -
4(1 + 27)
-
--r r
4g (257)
$ 9
--
1 --r
8
Combining this with (15), have
7 "Ds [T( Ter(- s8)) )
L'(T) Jax, R)
TTd. R) -9 (1-7)), E}
Σ
1-r
1 r 0,
8
which completes the proof.
A.3 ACTION DISTRIBUTION AND OCCUPANCY MEASURE DIVERGENCES IN LLMS
As noted in the main text, in the current paradigm of using RLHF to train LLMs, we can show that action distribution divergence between two policies is equivalent to occupancy measure divergence. In particular, RLHF for LLMs is usually modeled as a contextual bandit.
In our setting, contextual bandit can be defined an MDP with 7 O; then, the return of the policy ı under a reward function R is given by
[T, Rt) = E (ax [R(s, a)]-
That is, a single state is sampled from the initial state distribution μoo and then a single action is sampled from the policy ı conditioned on that state. RLHF for LLMs follows this setting as prompt is sampled from a dataset, the LLM generates a response, and the reward is calculated based on the prompt and response.
In this setting, it is simple to show that the action distribution and occupancy measure divergences are equivalent.
be the f-divergence between two distribu- tions P and Q. Then, for any two policies ग. n' a contextual bandit, we have
Lemma A.6. Let D2 Ea- =
( (T((-1)) =()((1))))(
Dy( ll (x)) = Ba-rot)
Lemma A.6 applies any f-divergence, including the KI. and x² divergences study in this paper.
28

---

Published as a conference paper at ICLR 2025
Proof. We have
Mx (x,
Dz(wz ((x)
s
(a ur
(xpp)()) (aN f (o(s))(( ( 8)
(o(())
Bs (T (TT || x()],
which completes the proof.
A.3.1 AUTOREGRESSIVE ENVIRONMENTS
While most RLHF implementations the contextual bandit formulation above for the purposes of KL regularization, one can also model training an LLM as a sequential problem where each token generated is separate action. This formulation is longer contextual bandit, but show that the action distribution and occupancy measure KL divergences are still equivalent!
Lemma A.7. Suppose that an environment satisfies the following conditions:
It is deterministic: (o(sa) = for exactly one state 8o, and for all 8t, az E S x A. p(s++1 St, St, = 1 exactly one state St+1-
Exactly one sequence of actions leads to each state: if following ag...., at-1 leads 8, then no other sequence of actions (of any length) can also lead to 8.
Then, for any policies n, n', the action distribution and occupancy measure KL divergences between them are equal (removing the (1 -1)) prefactor on the action distribution divergence):
(KI |l (x/) = Em Σ 7 Dra((T 86) x'(. 8c))
Lemma A.7 applies to LLMs since one can treat the "state" of the environment after ł timesteps as all the tokens generated so far WoW1 ... w/-1, and the actions as the next token wy which is then appended to the state:
a, TTae 5t) <(w, WoWl Wi w.-))
p(5(+1 St. ax)) 1 W6W ww
where St = Wo W4-1 and ag Wt.
Thus, regardless of the formalism used to train an LLM via RLHF, the action distribution and occupancy measure KI. equivalent. However, the conditions of Lemma A.7 unlikely to be met by many other MDPs. Many MDPs are stochastic, violating the first assumption. Even among deterministie MDPs, it very uncommon that only single action sequence lead to each state.
Proof. Given the assumptions about the environment, we can rewrite the log-occupancy measure of state-action pair in terms of the sum of log action probabilties over the unique sequence of actions leading to that state. Suppose ao..... 04-1 is the unique action sequence leading to and that this action sequence visits states sg...., St-1, s. Then
y (1 = =sss = = 1)
log ( (8, a) = log
(1 - 7)
1=1
log
((1 (1- '1
(st
ss s ====aa)
log
(1-
II T(as 8i)
t=
-log(1 - + t llo
log maa 1 st).
==0
29

---

Published as a conference paper at ICLR 2025
Using this, we can rewrite the occupancy measure KL divergence
M (T((,,)) a)
DEL (fz ll px))= = Σ
log
(s.a)eSxd
Σ 40....dEAAt1
Pr(ao A ac)
log T(a (8)) - llg (a: 8t))
(1
Σ
II 1=0
x 1s) si)
Σ
log <(;; (8:) -
llgg lo ('a; 84)
x=0
(17)
where is the state reached by taking a:-1- We will now show inductively that
Σ
1oj) Σ loog (a; (&.) log > 8i))
ag EAt+1
1=0
Σ
(K (s)) || 7- st)).
(18)
1=0 #EES
Consider first if 0. Then
Σ क <(a 80) log ( 10)) log >'a0 (00))
RDEA
DKL(T( (o) x'(: sb))
(80) -KI( 80) r'(: so))-
30

---

Published as a conference paper at ICLR 2025
Now suppose (18) holds for 1. Then for t we have
Σ
II
T(a 83)
log ((a 8z) log a' 8i)
ap.... EAt+1
110
Σ
Ma
M(a 185)
Σ (ai (t)
log st) l'g se)
B0 Edde
1=0
0.EA
log क ((as) 8.) log (as) si))
Σ a0
II 1=0
M(a 19))
p 86) 7' st))
Σ x(an st) Σ
(og 1 |s) log
n' (as 1 ssi)
a.EA
Σ
II 1=0
I 1s))) 1)))
D 1 se) t'(. (())
log (ag 1 18:) - logglo''/ (as 8:))
2=
Σ
(L)) 86) x.(()(()))))) sc))
S:ES
Σ
Ia 18))
log (a;; 1 si) - log si)
1=0
i
Σ
PRI Bi) || x.(. 8c)) +
Σ (x) (s:) 7'(. 8:))
4.ES
i= #.ES
Σ Σ
Px) 8.) x (.18:)).
SEES
where (i) is from the inductive hypothesis.
31

---

Published as a conference paper at ICLR 2025
Now, plugging (18) into (17) gives
DKL (KT || pxx)
=(1 7)
PA (x)) st)
(s)) 7 s:))
t=0
5.ES
0
00
(1 7)
(KL o'( s:))
S,ES
8
= (1 7)
Pr DeL 8:) n'(. (-(11 s;))
+ES
= -1-11 (E
DKL (T(- si) T- s;)) (+))))
1=0
= (1 - E
(T( (s:) ll x s;))
1=0
E
(al( s:) || (1
which is the desired result.
A.4 LEARNED REWARD FUNCTIONS ARE r-CORRELATED
Proxy reward functions are often learned from data like ratings or preference comparisons, including in the case of RLHF. Here, show that a learned reward function with low mean-square error common objective in supervised learning- r-correlated with the true reward function.
Lemma A.8. Let R be the true fenctttunction and it be a learned reward function. Suppose that Egea//
(R(s, a) R(s, S co Then, the learned reward function is an r-correlated praxy with 1-6.
The assumption in Lemma A.8 is that the mean-squared error over the occupancy measure of the reference policy is small. This can be achieved, for example, by learning ft via least-squares regression over a training dataset of state-action pairs sampled from the reference policy. Many results in learning theory show that this results in a small reean-suarred error over the distribution the training data was sampled from, i.c., exactly the assumption in Lemma A.8 (Koltchinskii, 2006).
Proof. Throughout the proof, all expectations, variances, and covariances are with respect to Prw. We can rewrite the assumption using the bias-variance decomposition as
E [R(ss a) - R(s R)ss
Var [R(s,a) - ai) + E [ a)] - [ites
Var (R(s,a)) Var (R(ss,) a)] 2 Cov [R(s,a), R(s a)] (R[ -B [)(s
=o + + 2Cov [(s, a), a)) + (R[ a)] - E [(ss.a)))
< coo
Note that [E a)) - [ > 0, so we can rewrite the inequality as
2Cov [(s,,a,, R(s, 2 21 11-ceh c)o + +o 2 (1 (o- +
32

---

Published as a conference paper at ICLR 2025
Dividing both sides by 20 nO & gives
Cov R(s, a()
- E R + oz
R
2 RR ht
By the AM-GM inequality, 2 2 RRO R.
Cov
R(s; a), a))
2 - c,
RROR
which is the desired result.
A.5 STATE-ONLY OCCUPANCY MEASURES
We also consider state-only occupancy measures, defined
(T(s)= = (1 E,
Σ f1(s s}
In many environments, the reward functions only depend on the state, i.e., R(s,a) R(s) and R(s, a) R(s). In this case, Theorem 5.1 holds for state-only occupancy measures as well. The proof is identical to the proof of Theorem 5.1, but replacing expectations and sums over state-action pairs with expectations over states.
B DERIVATION OF OCCUPANCY-REGULARZZR POLICY OPTIMIZATION
In this appendix section, we show how derive the approximations used for Policy Optimization (ORPO). As reminder, we would like to optimize
JTTO R) x (Mro [mmm)))
We can rewrite its gradient as
V ( It) Av x2 (MiTe |
X (Mito ll Mataa
VOJ(nO, R)
2 2xx² (proo (
Vo
Σ
Mix (x (s, (){ss,
A
Vo
Σ
Mao Mo (s, a)
2/x² x2 (NiTT MMM
Ma (8, ())
8.a
3.a
VONTo (s, (s,
A
Vo
Moo a)2
-1
2 x2 (Moto MMat
Hand (s, a)
2p (s,a)
Σ
2xx² (Woo (l VaMao (s, (s,
VelLTOO (s,
Ma (8,a)
8,0
Vopto (s.a)
R((s,)) Vx²( (P Mam (s, a)
A
Mr (8,a)
8,0
As described the main text, policy gradient algorithms can approximate this type of gradient by using an augmented reward function
Mrg (8, a) ( ( pT (s, a)
R'(s, a) R)
(19)
33

---

Published as a conference paper at ICLR 2025
However, two terms in (19) cannot be computed directly: the current x² divergence between oceu- pancy measures, and the ratio of the occupancy measures pxo (s, a)/ (8, a). We first show how to approximate the latter using a discriminator network do(s, a), trained to optimize
Φ = arg min Ept, log(1 --ddo(aa))) Eu
log(1 + (la())
(20)
It is well-known that exactly optimizing the loss function in (20) gives
ds(s, a) = log Mro (s, a)
(21)
(8,
Furthermore, the OM x² divergence is given by
x² (HrellATa)) Eyi
MTn (3, a)
Eyno (ebl(ao) - 1 x
(22)
Me (8,
Combining (21) and (22) shows that the augmented reward (19) can rewritten } ddstia))
R'(s, R(s,a)
Putting all the steps together, the following algorithm formalizes ORPO:
Algorithm 1 Reculancy Policy Optimization (ORPO).
1: for iteration i 1, ........ do
2: Collect set of n trajectories Dz from Ty.
3: Collect set of n trajectories Dx... from Tia..
4:
Optimize via SGD to minimize
ED log(1 + d=dsss)))] ] + ED, [og(1 + Ad(o,)))
5: Calculate X2 Ep, (ddstaa) -
6: Transform D, to D', by replacing the rewards with (s( -((s.a)) (elotoia) -
7: Optimize 0 via SGD to minimize the proximal policy optimization (PPO) loss Lepo (Dz).
8: end for
similar approach can also be used to optimize the proxy reward regularized by KI. divergence
J(mo, R) A DxL. (Pr (l
by changing the augmented reward Line of Algorithm to l'(s, a) f((s,a) Xdg(s,a).
How accurate is the discriminator-bae based approximation?. To determine whether using the discriminator results in an accurate approximation of x² and KL OM divergences, we plot the output of the discriminator in the glucose environment versus the theoretically correct value in Figure 6. The results suggest that the discriminator accurately approximates the log ratio of the occupancy measures, which in turn allows for accurate approximations of the OM divergences.
34

---

Published as a conference paper at ICLR 2025
Proe
100
200
300
400
d(s))
Hen (8)
2
log
Mo (s)
100
200
300
400
5 (blood glucose level)
Figure 6: The top plot shows the state occupancy measures of the reference policy That and policy optimized with ORPO in glucose environment. The bottom plot shows the exact log ratio of the occupancy measures log prr (s) (s) versus the discriminator output ds(s), which attempts approximate it. We find the discriminator output be good approximation of the log ratio.
C ENVIRONMENT DETAILS
Here, we discuss the details of the five reward-hacking environments we study.
C.I TRAFFIC CONTROL
The traffic control environment, based on the Flow simulator (Wu et al., 2022), simulates a group of human-controlled and RL-controlled vehicles
on-ramp attempting to merge into traffic a highway. The true reward prioritizes a small mean commute time, while the proxy reward the average velocity of all cars. When reward hacking, the RL. controlled vehicle the on-ramp stops indefinitely and lets cars continue forward at high speeds on the highway, which maximizes the proxy reward but increases the commute times of cars on the on-ramp infinitely. As the reference policy for the traffic environment we used the Intelligent Driver Model (IDM), a standard approximation of human driving behavior (Treiber al., 2000). In practice, reference policies are often learned via imitation learning, so to simulate this we generate data from the IDM controller and train behavioral cloning (BC) policy using the generated data.
Here, the green cars are controlled by the human driver model IDM controller, and the blue cars are controlled by RI.
This particular frame showcases reward hacking behavior. The blue RL-controlled vehicle has stopped completely the on-ramp, blocking cars behind it. This increases the average velocity of all vehicles in the simulation, as the cars on the straightway are able to continue speeding along the road without having wait for merging cars. However, the true reward (negative average commute
35

---

Published as a conference paper at ICLR 2025
time) decreases endlessly the cars the on-ramp wait.
C.2 GLUCOSE MONITORING
The SimGlucose blood glucose monitoring environment is an extension the FDA-approved glucose monitoring simulator proposed by Man et al. (2014) for Type Diabetes patients (Fox al., 2020):
FITTON
Mudeel
The RI. agent (bottom) controls the insulin administered to a simulated patient in order to maintain healthy glucose levels. The true reward is standard measure of health risk for the patient, but the proxy reward prioritizes the monetary cost of insulin. As the safe baseline policy, we train a BC policy using data generated by PID controller with parameters tuned by the original designers of the simulator (Steil, 2013).
In Figure 2, we adapt diagram from Pauley et al. (2022) represent this environment.
C.3 PANDEMIC MITIGATION
PandemicSimulator (Kompella et al., 2020) models population's 's infection dynamics using modified SEIR model that they designed represent the COVID-19 pandemic:
Covid19 Simulator
Covid Rogulation
Update Sim Characteristics
(Location
Government
Shep on tines diill day)
Observation
Infection Model
Contact Tracing
The RL agent chooses the level of lockdown restrictions placed on the population by observing the results of testing. The proxy reward function omits the political cost associated with certain decisions. Our reference policy is trained via BC on a combination of hand-specified and real-world strategies employed by governments during the pandemic, which were also used by Kompella et al. (2020) as baselines.
C.4 RLHF
We base RLHF environment on the work of Coste al. (2024), who study overoptimization of LLM-based reward models. The proxy reward model we use is fine-tuned from Pythia- (Biderman et al., 2023) the AlpacaFarm (Dubois et al., 2023) preference dataset. As true reward
36

---

Published as a conference paper at ICLR 2025
model, the Llama 3 Tulu V2 8B RM from A12 (Ivison al., 2024). As reference policy, use Coste et al.'s SFT policy, which was fine-tuned from Pythia- 1.4ia-1.4B on the AlpacaFarm SFT data. We evaluate policies' true and proxy rewards responses sampled for held out set of prompts.
C.5 TOMATO-WATERING GRIDWORLD
The tomato environment contains sprinkler state where the agent perceives all tomatoes as being watered and thus receives high proxy reward but no true reward. We train a reference policy using the true reward function, and then add 10% chance of taking random action ensure there is room to improve upon it.
The gray squares in the environment represent walls, and the white squares represent open spaces where the agent can travel:
The sprinkler state is down narrow hallway, and on the other end a tomato down another narrow hallway.
D EXPERIMENT DETAILS
D. NON-LLM EXPERIMENTS
We implement ORPO using RLLib (Liang et al., 2018) and PyTorch (Paszke et al., 2019).
Network architectures For the pandemic, traffic, and tomato-watering environments, we use policy networks based on fully-connected networks with layers of 128 units, 4 layers of 512 units, and 4 units of 512 units, respectively. The policy model for the glucose environment is basic LSTM network with layers of units each. We made this choice since the observation of the environment contains continuous historical information about the patient's blood glucose levels and previously administered insulin.
The discriminator model for all four non-LLM environments is fully connected network with layers 256 units each. We found that the discriminator architecture did not need be tuned to each environment.
Policy initialization Initializing using an imitation learning policy has been shown to effectively speed up the learning process (Laidlaw et al., 2023; Uchendu al., 2023) and is used in practice for RLHF (Stiennon et al., 2020), so we initialize our policies using the specified ाee for the more realistic traffic, glucose, and pandemic environments.
37

---

Published as a conference paper at ICLR 2025
Hyperparameters Some hyperparameters for the traffic environment tuned by Pan al. (2022). We primarily tuned the hyperparameters listed below order to ensure that the proxy reward would be properly optimized and reward hacking would occur without regularization. This enables to see the various regularization methods actually succeed at preventing reward hacking.
Hyperparameter
Tomato
Traflic
Glucose
Pandemic
Training iterations
500
250
500
260
Batch size
3000
40000
100000
3860
SGD minibatch size
128
16384
1024
64
SGD epochs per iteration
5
Optimizer
Adam
Adam
Adam
Adam
Learning rate
le-3
5e-5
le-4
0.0003
Gradient clipping
0.1
None
10
10
Discount rate (n)
0.99
0.99
0.99
0.99
GAE coefficient (A)
0.98
0.97
0.98
0.95
Entropy coefficient (start)
0.01
0.01
0.01
0.1
Entropy coefficient (end)
0.01
0.01
0.01
0.01
Entropy schedule horizon
500000
KL target
0.001
0.02
1c-3
0.01
Value function loss clipping
10
10,000
100
20
Value function loss coefficient
0.1
0.5
0.0001
0.5
Share value function layers
T
Table 2: PPO/ORPO hyperparameters.
Hyperparameter
Tomato
Traffic
Glucose
Pandemic
Discriminator reward clipping
1000
10
1e10
0.1
Regularization coefficient (A)
Varied 0.05
Varied
Varied 0.05
Varied 0.08
0.0002
Table 3: ORPO-specific hyperparameters.
ORPO details We found that couple of tricks were useful to ensure that ORPO remained stable. First, we clip the discriminator term added to the reward functions to a range [-6, 8), since sometimes it can blow up and cause numerical issues. Second, when estimating x we use trimmed mean (trimmed by 1% in each tail) to reduce the effect that outliers have on the estimate. These are both particularly important for x2 divergence, where the output of the discriminator is exponentiated for both the reward discriminator term and for estimating x
D.2 RLHF EXPERIMENTS
We train LLMs the RLHF implementation used by Coste et al. (2024), which is based on OpenAssistant and trlX (Havrilla et al., 2023). To implement x² or KL regularization, directly add loss term to the PPO loss:
(a s)
Terraa
2 divergence:
(a s)
(a(a (a s)
A log Ta(a 18) (a 8) Teer wo (a 8) -1)
KL divergence:
where is the prompt and is the sampled response. Intuitively, both loss terms have unique minimum when (a -) = Ra s) and expectation are equivalent to the correct divergence. Schulman (2020) suggests that these are particularly low-variance estimates, and find that they
38

---

Published as a conference paper at ICLR 2025
work well in practice.
E
ADDITIONAL EXPERIMENTS AND RESULTS
this appendix, we the full results from our main experiments as well as ablations of ORPO.
E.1 WIN RATES FOR RLHF
Besides calculating the true reward for RLHF models with the Llama 3 Tulu V2 8B RM, we also caleulate the win rates for the best coefficient of KL. and x² regularization. We use AlpacaEval (Dubois et al., 2023) with GPT4o-mini to compute the win rate between the RLHF policy and the SFT policy. find that the median win rate for x" divergence higher, and it is also more consistent across random seeds. In contrast, using KL divergence leads reward hacking for one seed and lower median win rate.
Divergence
Coefficient Median win rate
Win rate range
2
0.0008
52.83
51.98 53.98
KL
0.025
51.50
11.93 - 53.75
Table Win rates for RLHF-trained models using divergence vs. x² divergence. The median rate and range of win rates are reported across five seeds.
E.2 RESULTS ALL REGULARIZATION COEFFICIENTS
Here, we present the results of training with AD and OM regularization using x2 and KL divergence across all regularization coefficients. Each table shows the median and the standard deviation of the true rewards achieved by the learned policy across 5 random seeds.
x² divergence
Coefficient
AD
State OM
State-action OM
0.000002
-60.79 + 8.40
-61.04 2.20
-59.11 + 4.53
0.000004
-62.01 + 8.58
-61.85
2.22
-58.21 + 5.52
0.00001
-60.43 + 6.61
-59.82 2.85
-54.35 + 2.81
0.00002
-62.01 + 4.54
-56.53 10.66
-1.35 30.28
0.00004
-50.84 + 6.28
::24::22000
-1.15 + 0.06
0.0001
-1.29 + 0.12
-2.18 0.42
-1.38 0.29
0.0002
-1.70 + 0.12
-2.46 + 0.65
-1.85 + 0.29
KL divergence
Coefficient
AD
State OM
State-action OM
0.000001
-56.20 + 4.02
-61.59
2.72
-61.18 + 2.87
0.0000025
-59.58 + 4.84
-54.99 3.16
-59.59 2.36
0.000005
57.24
-59.03 = 5,41
-61.24 + 2.01
0.00001
14.8443..11
-57.62 = 3.13
-58.85 2.73
0.000025
55.10
-59.32 = 1.37
-56.86
0.00005
4.04
-59.96 = 1.65
23.79
0.0001
45.72
-1.34 25.30
-1.25 0.07
0.00025
+
-1.47 = 0.20
-1.51 + 0.10
0.0005
-1.52
-1.76 = 0.22
-1.99 0.35
0.001
-.073
-1.76 = 0.26
-2.30 + 1.14
0.0025
-1.98 1.98+00 0.07 0.07
-1.76 = 0.51
-1.94 0.30
0.005
-2.15 0.05
-1.90 0.83
-2.08 + 0.61
0.01
-2.11 + 0.05
-2.12 = 1.00
-2.14 + 0.56
Table 5: All traffic control results (x (x10²).
39

---

Published as a conference paper at ICLR 2025
divergence
Coefficient
AD
State OM
State-action OM
0,0008
-17.60 + 1.78
-12.59 1.63
-27.16 12.83
0.0016
-11.25 = 10.08
-11.17 = 0.19
0.004
-34.16 12.90
-10.88 + 0.91
-11.65 = 2.12
0.008
-12.45 0.25
-10.99 3.13
-12.02 = 0.18
0.016
-12.31 0.08
-10.68 0.17
-12.18 = 0.46
0,04
-12.29 0.05
-10.78 0.12
-12.34 = 0.10
0.08
-12.39 = 0.04
-10.73 0.84
-12.20 = 0.11
KL divergence
Coefficient
AD
State OM
State-action OM
0.00006
-21.23 + 9.74
-33.59 + 9.89
-30.96 22.42
0.00012
-30.39 + 22.16
-41.96 = 12.70
-19.67 = 6.43
0.0003
-23.10 5.63
-35.29 = 10.32
-27.56 = 7.50
0,0006
-21.85 19.03
-34.56 11.97
-22.40 = 9.84
0.0012
25.17 10.24
-31.28 + 7.83
-31.77 = 6.01
0.003
-23.51
-35.76 = 10.53
223900 12.81
0.006
-12.26 11.82
-58.08 46.98
299.42 25.37
0.012
-12.30 9.27
-10.60 0.87
-11.88 = 0.81
0,03
-12.28 0.14
-1.033 6.86
-11.73 0.21
0.06
+
-10.71
0.18
-12.23 14.25
0.12
-12.33 0.04
-10.24 0.61
-12.09 0.38
0,3
1 0.04
0.57
-12.11 = 0.25
0.6
-12.40 1 0.04
-10.61 0.36
-12.11 = 0.28
1.2
-12.33 + 0.03
-10.50 + 0.26
-12.02 = 0.28
Table 6: All pandemic mitigation results.
40

---

Published as a conference paper at ICLR 2025
divergence
Coefficient
AD
State OM
State-action OM
0.0005
-580.7 73.8
-48.2 56.6
0.001
-94.2 + 14.0
-263.9 = 19.2
-127.0 = 5.9
0.0025
-97.1 8.2
-146.4 = 18.1
-93.3 = 9.3
0.005
-76.6 10.5
-109.7 = 10.3
-55.2 = 0.7
0.01
8.5
-72.8 8.8
47.5 = 0.6
0.025
-85.6 7.9
-54.3 + 1.3
-50.9 = 2.3
0.05
-74.8 = 13.1
-57.1 3.5
-13.33 32.8
KL divergence
Coefficient
AD
State OM
State-action OM
0.00003
-598.4 + 39.7
-60.11 + 10.7
-583.8 = 61.2
0.00006
-600.6 12.4
-88.00 =
-594.7
0.00015
+
-607.9 11.1
-577.1 = 11.22
0,0003
-593.6 6.0
-592.0 293
-497.9 = 11.2
0.0006
7.5
-593.1
+3.4.3
0,0015
114.1
21.
7.5
0.003
39.7
-332.3
5.0
0.006
5.5
61.9
0,015
-84.1
5.6
0.5
0.03
+8.
+
49.6
0.06
-88.6 12.8
-59.0 7.2
-78.3 10.2
0.15
-82.1 11.6
-75.9 4.6
-06.66 = 19.6
0.3
-73.4 + 9.2
-98.1 16.1
-127.3 24.7
0.6
-88.6 + 5.6
-112.5 + 16.6
-118.5 = 10.7
Table 7: All glucose monitoring results (x (x10)).
Coefficient
AD x2
AD KL
0.00008
9.20 + 0.68
8.80 + 2.24
0,00025
14.05 3.27
9.48 - 1.02
0.0008
1..94
8.84
0.0025
4 1.08
0.008
1.11
12.73 + 2.75
0.025
1..59
16.81 + 0.27
0.08
16.43::000
16.52 + 0.08
0.25
16.22 0.10
16.33
+ 0.04
0.8
16.13 + 0.10
16.25 + 0.13
Table 8: All results RLHE
41

---

Published as a conference paper at ICLR 2025
x" divergence
Coefficient
AD
State OM
State-action OM
0.0005
2.65 + 0.67
0.64 + 0.19
0.53 0.29
0,001
3.87 + 0.25
0.65 + 4.63
0.60 = 0.16
0.0025
6.24 0.10
9.06 + 0.17
9.04 + 4.71
0.005
6.17 0.03
9.06 0.11
9.16 0.09
0.01
6.16 0.04
9.07 0.07
9.17
0.025
6.19 0.04
8.64
8.61 = 0.18
0.05
6.14 + 0.00
7.95 = 0.05
7.99 = 0.22
KL divergence
Coefficient
AD
State OM
State-action OM
0.0008
2.52 + 0.18
2.32 + 0.86
2.31 + 0.08
0.0016
2.98 0.33
2.31 = 0.07
1.11 0.89
0.004
+
2.23 + 0.06
2.01 + 0.23
0.008
6.10
1.30 0.19
1.25 0.32
0.016
6.33
0.82 + 0.22
0.84 0.21
0.04
6.26
1.17 2.92
0.08
6.26
7.62 0.06
0.16
6.21
0.12
0.4
6.16
6.89 0.14
6.84
0.8
4
7.07 + 0.12
6.86 0.19
1.6
6.14+0004
6.90 + 0.13
6.61 + 0.31
4
6,13
6,80 + 0.16
6.79 +
8
6.13 + 0.01
6.81 1 0.28
6.80 0.06
16
6.13 + 0.00
6.94 + 0.10
6.83 + 0.25
Table 9: All results for the tomato-watering gridworld.
E.3 ABLATIONS
Here, we present the results of two ablations of ORPO. For each the non-RLHF environments, we fix the optimal coefficient for nate-action OM x² regularization and modify other hyperparameters. do not ablate the RLHF experiments because RLHF is a contextual bandit (Appendix A.3) and so it isn't actually necessary to ORPO for RLHE. The results shown in Table 10 below.
Order training policy and discriminator networks. First, we experiment with modifying ORPO to train the discriminator after the policy. Algorithm 1, the discriminator dg is optimized, then the rewards are updated with the discriminator outputs, and then the policy is trained with the updated rewards. An alternative is to wait to train the discriminator until after the policy has been updated, i.e., put Line after Line 7. We experimented with this and found that in in most environments there is not too much difference between the two orders, although training the discriminator first gives slightly better results. However, the pandemic mitigation environment, we found that it training the discriminator second gave results with much higher variance. This suggests that it is best to train the discriminator before augmenting the rewards to train the policy.
Discriminator reward clipping. Second, we experiment with modifying the discriminator reward clipping parameter of ORPO. We found that removing the clipping parameter entirely led to NaN emors and training could not complete, to this parameter, we tried training with a clipping parameter 10x larger and 10x smaller in each environment. We found that the results did not vary by much across different clipping parameters. This suggests that ORPO is relatively robust to the hyperparameter, so does not need to be tuned precisely.
do not report those results. However, to test sensitivity
42

---

Published as a conference paper at ICLR 2025
Environment
Traffic control
Pandemic mitigation
Glucose monitoring
AI safety
Method
gridworld
(x10*)
(x10²)
Default parameters
-1.15 0.06
-00668
0.15.
-47.5 + 0.6
+ 0.12
Train policy before discriminator
-1.24 + 0.07
-11.51 = 0.40
-48.2 + 0.7
8.97 = 0.13
Discriminator reward clipping x0.1
-1.18 0.14
-10.77 + 0.16
-47.7 0.8
9.21 0.18
Discriminator reward clipping x 10
-1.24 + 0.03
-10.73
+ 0.37
-47.9 + 0.7
9.20 + 0.14
Table 10: Results our ablations of ORPO. report the median and standard deviation of true reward across five random seeds for the four non-RLHE environments. The top row shows results using each environment' most effective form of occupancy measure regularization: state-action OM for traffic, glucose, and tomato and state OM for pandemic, with the optimal x² divergence coefficients. The other rows show ablations with the coefficient. We find that ORPO mostly robust different hyperparameters but that it is probably best train the discriminator network before the policy network.
43


Reviews:
[{"review_id": "T1g1mByGam", "reviewer": "Reviewer_qUra", "summary": "The authors propose a new definition for reward hacking, based on the idea that we should only care about hacking of proxy reward functions which are correlated (in terms of state-action occupancy) with the true reward function under some \"reasonable\" base policy.\n\nThey then show theoretically and empirically that regularizing reward using an occupancy measure term as opposed to the standard KL divergence leads to less reward hacking across several simple environments.", "questions": "## Typos\nthere is an issue with the text in Definition 4.2: I think your $\\tilde{\\mathcal{R}}$ ate something after \"policy\" and another thing before \"lower\".\n\n## Suggestions\n* line 24: \"superior\" feels a bit strong\n* line 164: the SEIR and glucose environments diagrams are too small to read/understand. just a comic of the environment would be better. the gridworld is on the edge of too small but ok\n* line 217: I would avoid using the word \"misaligned\", since that has all kinds of connotations. Consider just \"different\", \"not identical to\", etc. This is a nit: if you feel strongly about \"misaligned\", I think it's probably ok.\n* line 312: the SEIR model has been around for a long time (a quick search found stuff from 2013 but I think it's been around for significantly longer); please try to find an original reference OR change the sentence to talk specifically about that one from 2020 which you're using.\n* 369: RHS = right hand side? some readers might not know this; please define on first use\n* 402: \"widely used to prevent reward hacking in RLHF\" well, I don't think this is quite true. KL divergence, to my understanding, is primarily used to avoid instabilities in the training procedure (important in when using PPO in general, not just RLHF!). Now, it seems likely that there is a side-effect of this regularization which helps prevent hacking, but that's not the main reason (in particular: if you remove KL divergence, you don't get hacking, you just get a failure to train). Please update this sentence! (or please explain if I'm missing something).\n\n## Questions\n* copying from above, my biggest concern is this one: even if a natural base policy does exist, it's not obvious that the state-occupancy distribution induced by it is the one we actually care about: the optimal policy wrt the true reward might be quite different from what we think of as the \"natural\" base policy. The issue I see here is that if the \"actually really good\" (much better than \"reasonable base\") policies have a way different state-occupancy distribution than the base policy, then the \"check\" that the proxy reward is \"similar enough\" to the true reward over the \"reasonable\" distribution just doesn't matter: what matters is what happens near optimality. Am I missing something here?\n* why do you only consider the *optimal* policy wrt the proxy in Definition 4.2? This seems too restrictive (and you apply the regularization at all points during training, so unless I'm missing something you don't rely on this optimality?)\n* how do you expect your technique to scale?\n\n## Closing note\nI gave the paper a 5 because I think it's important that some things be changed before it's ready for publication. But I will be happy to increase the score to a 6 if my minor points are met, and possibly more if my greater fears about the utility of the technique in capturing reward hacking in different but relevant parts of state-action occupancy space are allayed. Indeed, it's possible that just acknowledging and writing about this issue would be enough: so long as it is clearly framed what this paper is and is not doing and to what extent it \"covers all the bases\" wrt reward hacking, I think it's worth publishing. It's really nicely presented and a nice idea with experiments to back it.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The paper is extremely well-written.\n\nThe paper builds on past literature in a coherent and helpful way.\n\nThe paper shows experiments on interesting environments.", "weaknesses": "While the new definition can be defended as an improvement over past definitions, it still leaves some things to be desired. Specifically:\n\n* it requires a \"natural base policy\", which might not exist in some real-world relevant settings\n* even if a natural base policy does exist, it's not obvious that the state-occupancy distribution induced by it is the one we actually care about: the optimal policy wrt the true reward might be quite different from what we think of as the \"natural\" base policy.\n* the definition only considers hacking to have occurred when looking at the *optimal* policy with respect to the proxy; in practice, we never find the optimal policy, yet hacking occurs nonetheless with intermediate sub-optimal (or locally optimal) policies\n* the paper doesn't say how to know whether, and what to do if, the (eg, learned) proxy reward function is not r-correlated with the true one.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "taC2HD0J8m", "reviewer": "Reviewer_zddp", "summary": "The paper addresses the issue of reward hacking in RL, where policies optimized with proxy rewards. To address this, the authors propose a novel, formal definition of reward hacking, which is based on the correlation between proxy and true rewards observed by a base policy. The paper also propose a regularization method based on $\\chi^{2}$ divergence between the policies' occupancy measures.", "questions": "## Questions about underlying assumptions\n1. Does the paper assume that both the base policy $\\pi_{base}$ and the optimized policy $\\pi$ are able to acess the same observation states?\n2. **Handling of Unobserved Variables**: In the traffic domain, several potentially impactful variables may be unobserved, as noted in [1][2]. For example, factors like the driver’s expertise level [1] or weather conditions [1] are often missing in driving datasets. How does your framework address scenarios where such unobserved variables exist?\n3. During the training phase, does the agent have access to the ground truth reward value?\n4. What are major assumptions behind your framework?\n\n## Questions about metrics\n1. **Unknown True Reward**: In many practical applications, such as driving, it is challenging to properly define a true reward function [1][2]. In RLHF, the system often relies on preference pairs rather than a true reward. When the true reward is unavailable, what alternative metrics would you recommend for evaluation?\n2. **Alternative Metrics for RLHF**: For RLHF, additional metrics might be helpful, such as the winning rate (preferred vs. rejected) or a safety score. How could these be considered in your framework?\n\n## Other concerns\n1. **Similarity to Prior Work**: This paper is similar to [3], particularly in the theoretical sections and some experiments. It would be beneficial for the authors to clarify the distinctions between their proposed approach and that of [3].\n2. **$\\chi^{2}$ vs. KL Divergence**: Does $\\chi^{2}$ divergence consistently outperform KL divergence in all scenarios? Are there cases where KL divergence yields better results? If so, what trade-offs are involved? If not, could the authors outline situations where KL divergence might be more suitable?\n\n----------\n[1] Ruan, Kangrui, and Xuan Di. \"Learning human driving behaviors with sequential causal imitation learning.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 4. 2022.\n\n[2] De Haan, Pim, Dinesh Jayaraman, and Sergey Levine. \"Causal confusion in imitation learning.\" Advances in neural information processing systems 32 (2019).\n\n[3] Laidlaw, Cassidy, Shivam Singhal, and Anca Dragan. \"Preventing Reward Hacking with Occupancy Measure Regularization.\" ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The concept presented, while not very complex, demonstrates impressive effectiveness.The authors support their claims with a solid mix of theoretical proofs and comprehensive experiments across various domains. Specific comparison between $\\chi^{2}$ vs. KL divergence has been provided. Intuitive examples are also provided. Additionally, intuition about \"optimizing the occupancy measure (OM) divergence between policies\" might be helpful to the whole community. Overall, this is an excellent paper with good results and insightful contributions.", "weaknesses": "However, my primary concern lies in the assumptions underpinning their framework and the choice of metrics. Please see the specific questions below for further details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XdEIZKq3Y6", "reviewer": "Reviewer_4snh", "summary": "The authors provide a new definition of reward hacking based on a new definition of correlated policies. Reward hacking is a phenomenon that occurs when optimizing a proxy reward for a true reward function yields worse performance over the true reward. The definition of correlated captures proxy reward functions that match the true reward function in visited state action pairs by a base policy. In addition, the authors propose occupancy measure regularization with respect to a base policy as a method to overcome reward hacking. They show theoretically and empirically that the proposed method mitigates reward hacking.", "questions": "- The definition of correlated rewards allows for the correlated reward to deviate arbitrarily in states with zero occupancy by the base policy.  Doesn't this allow for policies that can be dangerous to still be correlated with the true reward? For example, if the base policy avoids a state with a large negative true reward but the proxy reward function assigns a very high positive reward for that state. Wouldn't this reward incentivize policies that try to visit this state and get a large negative true reward? In addition, wouldn't these kinds of proxy create policies that break the assumption in Theorem 5.1?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is very well written with an astounding flow of information. \n- The authors provide a lot of great intuitive examples for the arguments they make.\n- The authors provide a wide range of experiments that show both the soundness and robustness of the claims.", "weaknesses": "- In Theorem 5.1, there is the assumption that the state-occupancy measure of the policy is absolutely continuous with respect to the base policy. However, during training, this is not something that can be guaranteed. I suggest adding an extra discussion on how regularization also pushes toward the learned policy satisfying this assumption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yaPbDCrZXl", "reviewer": "Reviewer_icTG", "summary": "This paper critically formalizes a notion of exploitability in RL-based\nfine-tuning when the reward function is difficult or impossible to\nspecify exactly, and is instead replaced by a \"proxy\" reward. It argues\nthat reward hacking is characterized by the proxy reward admitting\noptimal policies that are in fact *not* aligned with our desired\nbehavior, and their characterization is validated with several intuitive\ncase studies. Moreover, under this characterization, the authors propose\na RL fine-tuning algorithm that prevents reward hacking by regularizing\nthe *occupancy measure* of the fine-tuned policy towards a base policy\nwith respect to $\\chi^2$ divergence, under weaker assumptions on the\nproxy reward than existing results. Moreover, the paper introduces a\npractical implementation using techniques from inverse RL (particularly\nGAIL), and demonstrate that their method performs better than baselines\nacross several fine-tuning tasks.", "questions": "In the introduction, you claim that the formalization of a \"good proxy\"\nhas been elusive. Hasn't this issue been addressed by the EPIC distance\n\\[1\\]? What does EPIC leave to be desired?\n\nIs there a mistake in the Table 1 cell for traffic control with state\noccupancy KL? Particularly, the standard deviation of $22.6$ is\nalarming.\n\nTheorem 5.1 is nice in the sense that it tells us that occupancy measure\nregularization (particularly w.r.t $\\chi^2$) can prevent reward hacking\neven when the proxy is correlated with the true reward. However, there\nis no claim about whether or not reward hacking can (at least in\nprinciple) be prevented with action distribution regularization. So,\nnaturally, I'm curious whether or not a similar result can be achieved\nwith action distribution regularization. The empirical results suggest\nnot, but perhaps we just haven't found the right way to do it yet. Do\nyou think it's impossible?\n\nIn the experiments, RLHF is treated as a contextual bandit problem, in\nwhich case there is no difference between occupancy measure\nregularization and action distribution regularization (because state\noccupancy is independent of actions). However, especially given the\nautoregressive nature of many existing LLMs, RLHF can also be viewed as\na sequential problem where the policy sequentially chooses individual\ntokens, and a reward is presented after the last token is chosen. How\nwould OM under this formulation of RLHF compare to AD in the \"atomic\"\ncontextual bandit case?\n\nTheorem 5.1 looks nice because intuitovely, it tells us that by\ncontrolling the $\\chi^2$ between the learned policy and the base policy,\nwe can prevent reward hacking. But(!), it is actually not entirely clear\nto me if this lower bound admits the possibility that we can actually\n*improve* upon the base policy using this regularization. Clearly, as\nthe $\\chi^2$ divergence tends to $0$, the lower bound tends to $0$. But\ncan the lower bound ever be greater than $0$?\n\n## References\n\n1.  Gleave, Dennis & Legg et al. (2021) Quantifying Differences in\n    Reward Functions, arXiv.\n2.  Fujimoto, Meger & Precup (2019) Off-Policy Deep Reinforcement\n    Learning Without Exploration, ICML.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper studies a very hot topic in the field, and in my opinion,\nprovides some very interesting insight. Most notably, I really enjoyed\nthe following entangled results in the paper:\n\n1.  Even with a proxy reward that is highly correlated with the ground\n    truth reward, reward hacking is possible;\n2.  But under such correlated rewards, it is (probably) possible to\n    prevent reward hacking by regularizing the fine-tuned policie's\n    occupancy measure (wrt $\\chi^2$).\n\nIt is also nice that their proposed framework fits nicely with that of\nIRL, allowing the authors to leverage performant IRL algorithms for RL\nfine-tuning.", "weaknesses": "My main concerns with the paper regard lack of clarity in several\nrespects. I will outline these below, and further examples are given in\n\"Minor Issues\" and \"Questions\".\n\n1.  I find the narrative of the paper to be a little misguided. The\n    paper goes on to try to essentially axiomatize reward hacking,\n    leading to many fairly philosophical or imprecise heuristic\n    discussions (e.g. \"it's not reward hacking if the proxy is just\n    bad!\"). I think this is, in a sense, actually overcomplicating\n    things. Under the hood, I think there is a very clear and\n    uncontroversial message here:\n    1.  When fine-tuning, we want to learn a policy that is better than\n        some base policy at maximizing some ground truth reward (this is\n        hard to argue with);\n    2.  Proxy rewards that are highly correlated to the ground truth\n        reward are still \"hackable\", in the sense that policy\n        optimization with such proxies prohibits improvement over the\n        base policy (the authors show this);\n    3.  But hackable correlated proxies are not useless—it is possible\n        to successfully fine-tune a base policy with such a proxy using\n        $\\chi^2$ regularization (the authors basically show this, see\n        Questions).\n2.  While interesting, I think some of the theoretical results are\n    actually fairly weak (see Questions). For instance, while the\n    theoretical results suggest regularization of the fine-tuned\n    policy's occupancy measure, they do not actually suggest that this\n    should be an improvement over action-distribution regularization.\n    Moreover, it is not actually clear that the lower bound of Theorem\n    5.1, which is the main theorem of the paper, can actually be\n    positive (in other words, Theorem 5.1 does not explicitly show that\n    fine-tuning with occupancy measure regularization can actually lead\n    to improved alignment). I suspect it is possible for the authors to\n    correct these, and that would strengthen the paper.\n3.  Overall, there are several issues with writing, such as discussion\n    of related work, redundancy in intro/background sections, and some\n    terminology, which I outlined below.\n\nMy score is mostly a reflection of weakness 2 here. I trust that the authors can fix weaknesses 1 and 3 fairly easily. Should the authors address weakness 2 (as well as the questions below), I would be happy to increase my score.\n\n## Minor Issues\n\nStiennon et al. (2020) is definitely not the first example of using KL\nregularization to an \"offline policy\". This is very commonly done in all\nof maximum entropy / entropy-regularized reinforcement learning, dating\nback at least fifteen years before this reference.\n\nWith regard to prior work in offline RL that enforce that the learned\npolicy does not veer too far away from the data distribution, the work\nof \\[2\\] is the most appropriate reference (see list of references in\nQuestions section below).\n\nThe paper starts off much too slow. The first four pages are fairly\nredundant, the isssue of formalizing the notion of reward hacking is\naddressed several times, and only very heuristic high-level ideas are\nsuggested for addressing the issue. I believe this pages can be\ncondensed a lot, which would make the paper easier to read.\n\nThe section about the desiderata for a definition of reward hacking can\nbe clarified a lot. In particular, it is not clear to me exactly what\nthe desiderata are, they should be stated very explicitly (especially\nsince the paper later refers to them in some order, e.g. \"the second\ndesideratum\").\n\nAt the bottom of page 5, $\\sigma_{\\tilde{R}}$ and $\\sigma_R$ are really\nvariances, not standard deviations (so they should probably be written\nas $\\sigma_{\\bullet}^2$).\n\nIn definition 4.2, there is a missing word — \"*reward hacking* occurs\nwhen an optimal policy $\\tilde{R}$ **lower** true reward…\".\n\nIn equation (8), you use $\\phi$ on both sides of the equation (you\nshould use e.g. $\\phi'$ as the variable being minimized).\n\nWhile definition 4.2 is interesting, I don't think \"reward hacking\" is\nthe right name for the phenomenon being described in this definition.\nParticularly, this definition presents a property on a proxy reward, but\nnot on the process of fine-tuning a policy with this proxy. Indeed, one\nof the contributions presented later in the work is an algorithm that\nprevents reward hacking under proxies satisfying definition 4.2! Rather,\nthis definition presents conditions under which reward hacking *may*\noccur. As such, I believe \"reward exploitability\" is an example of a\nmore appropriate name: standard policy optimizations *can* exploit/hack\nthe reward function if you're not careful.\n\nOn line 313, \"polulation\" should be \"population\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zyIQXsOB47", "reviewer": "Reviewer_Lc7U", "summary": "The paper studies reward hacking, including in the context of reinforcement learning from human feedback. It proposes a new definition of what constitutes reward hacking and shows that a specific form of regularization can help with preventing reward hacking. In the context of RLHF, the results suggest a different regularization scheme than a standard KL penalty. The authors further argue that this type of regularization better mitigates reward hacking in practice.", "questions": "Please see my comments and questions *Weaknesses*. I'd appreciate if the authors could respond to these comments and questions.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written, enjoyable to read, and relatively easy to follow. It provides an interesting perspective on reward hacking through the lens of regularized RL objectives. The approach is based on a new definition of reward hacking, which assumes access to a base policy-a reference used to determine whether a proxy reward is hackable by a learning agent. This definition is intuitive, although I have some questions and comments about its expressiveness.\n\nI find the connection to the KL-regularized RLHF objective, and the corresponding result (Theorem 3.1), particularly interesting. To the best of my knowledge, this result is novel. The experimental results are extensive and, in my opinion, demonstrate the effectiveness of the proposed approach.", "weaknesses": "Regularization to a base policy in RLHF for language models is somewhat natural, as we aim to preserve the language generation capabilities of our policy. However, as shown in Table 1, it can also lead to considerable performance loss in some environments (e.g., Pandemic Mitigation), where the base policy does not perform well. I wonder how preference-based RL (without regularization) would fare in such environments. It seems that the proposed approach is somewhat constrained, as it requires humans to provide a reliable reference point (base policy), which may be challenging in complex environments.\n \nWhile I am generally positive about this work, I think I didn’t fully understand how it compares to some other well-known approaches to mitigating similar problems (e.g., techniques based on inverse reward design or risk-averse RL). I understand that these approaches do not necessarily address the same problem, but the experimental setup does not seem to include baselines that would illustrate the importance of Definition 4.2. For example, since $\\pi_{base}$ is given, why isn’t IRL an adequate baseline?\n \nThe related work section covers most of the important references, but a more comprehensive literature review on RLHF could be beneficial. For example, the paper mentions:\n\n> Our regularized objective in (5) differs in two main ways from the KL regularization used in RLHF. First, our results suggest optimizing the occupancy measure (OM) divergence between policies, whereas RLHF uses the action distribution (AD) divergence. \n\nhowever, it does not explain how this connects to recent results that propose OM-based regularizers. For example, it would be useful if the authors could comment on the MDP formulation of RLHF from\n\n> Nika et al., Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences, ICML 2024.  \n\nwhich in Section 7.2 introduces the KL-regularized RHF objective based on occupancy measures. \n\nOne potential drawback of the proposed approach is that it is based on estimating occupancy measures, which may be challenging to estimate for high-dimensional state spaces. It would be helpful if the authors could comment on the level of difficulty of the current environments for an RL agent. Furthermore, how accurate are the occupancy measures obtained in the experiments? It seems that it should be possible to estimate this for some of the environments.\n\nSome minor details in the experimental setup were not clear to me. The appendix states:\n\n> We primarily tuned the hyperparameters listed below in order to ensure that the proxy reward would be properly optimized and reward hacking would occur without regularization.\n\nDoes this mean that we typically don't observe reward hacking in these environments?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)