The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models
Authors: Alexander Pan
Kush Bhatia
Jacob Steinhardt
Venue: ICLR 2022 Poster (2022)
Keywords: reward misspecification
reinforcement learning
reward hacking
alignment
ml safety

Abstract: Reward hacking---where RL agents exploit gaps in misspecified proxy rewards---has been widely observed, but not yet systematically studied. To understand reward hacking, we construct four RL environments with different misspecified rewards. We investigate reward hacking as a function of agent capabilities: model capacity, action space resolution, and observation space noise. Typically, more capable agents are able to better exploit reward misspecifications, causing them to attain higher proxy reward and lower true reward. Moreover, we find instances of \emph{phase transitions}: capability thresholds at which the agent's behavior qualitatively shifts, leading to a sharp decrease in the true reward. Such phase transitions pose challenges to monitoring the safety of ML systems. To encourage further research on reward misspecification, address this, we propose an anomaly detection task for aberrant policies and offer several baseline detectors.

THE EFFECTS OF REWARD MISSPECIHICATION: MAPPING AND MITIGATING MISALIGNED MODELS
Alexander Pan
Kush Bhatia UC Berkeley
Jacob Steinhardt UC Berkeley
Caltech
ABSTRACT
Reward hacking-wiin -where RL agents exploit gaps in misspecified reward functions-has been widely observed, but not yet systematically studied. To un- derstand how reward hacking arises, we construct four RL environments with misspecified rewards. We investigate reward hacking as a function of agent ca- pabilities: model capacity, action space resolution, observation space noise, and training time. More capable agents often exploit reward misspecifications, achiev- ing higher proxy reward and lower true reward than less capable agents. Moreover, we find instances of phase transitions: capability thresholds at which the agent's 's behavior qualitatively shifts, leading to a sharp decrease in the true reward. Such phase transitions pose challenges to monitoring the safety of ML systems. To ad- dress this, we propose an anomaly detection task for aberrant policies and offer several baseline detectors.
INTRODUCTION
As reinforcement learning agents are trained with better algorithms, more data, and larger policy models, they are at increased risk of overfitting their objectives (Russell, 2019). Reward hacking, or the gaming of misspecified reward functions by RL agents, has appeared in a variety of con- texts, such as game playing (Ibarz al., 2018), text summarization (Paulus et al., 2018), and au- tonomous driving (Knox et al., 2021). These examples show that better algorithms and models are not enough; for human-centered applications such as healthcare (Yu al., 2019), economics (Trott et al., 2021) and robotics (Kober al., 2013), RL algorithms must be safe and aligned with human objectives (Bommasani et al., 2021; Hubinger et al., 2019).
Reward misspecifications occur because real-world tasks have numerous, often conflicting desider- ata. In practice, reward designers resort to optimizing a proxy reward that is either more readily measured more easily optimized than the true reward. For example, consider recommender system optimizing for users' subjective well-being (SWB). Because SWB is difficult to measure, engineers rely more tangible metrics such click-through rates or watch-time. Optimizing for misspecified proxies led YouTube to overemphasize watch-time and harm user satisfaction (Stray, 2020), as well as to recommended extreme political content to users (Ribeiro al., 2020).
Addressing reward hacking is a first step towards developing human-aligned RL agents and one goal of ML safety (Hendrycks al., 2021a). However, there has been littl systematic work investigating when or how it tends to occur, or how to detect it before it runs awry. To remedy this, we study the problem reward hacking across four diverse environments: traffic control (Wu et al., 2021), COVID response (Kompella et al., 2020), blood glucose monitoring (Fox et al., 2020), and the Atari game Riverraid (Brockman et al., 2016). Within these environments, we construct nine misspecified proxy reward functions (Section 3).
Using our environments, we study how increasing optimization power affects reward hacking, by training RL agents with varying resources such as model size, training time, action space resolution, and observation space noise (Section 4). We find that more powerful agents often attain higher proxy reward but lower true reward, as illustrated in Figure 1. Since the trend in ML is to increase resources exponentially each year (Littman et al., 2021), 1), this suggests that reward hacking will become more pronounced in the future in the absence of countermeasures.

---

Low mean velocity
Proxy reward: "maximize the mean velocity"
Model
( UF
Low mean commute
High mean velocity
M
True reward: "minimize the mean commute"
Hlgh commute
example of reward hacking when cars merge onto highway. A human-driver model controls the grey cars and an RL policy controls the red car. The RL agent observes positions and velocities of nearby cars (including itself) and adjusts its acceleration to maximize the proxy reward. At first glance, both the proxy reward and true reward appear to incentivize fast traffic flow. However, smaller policy models allow the red car to merge, whereas larger policy models exploit velocity
Figure 1:
the misspecification by stopping the red car. When the red stops merging, the increases (merging slows down the more numerous grey cars). However, the mean commute time also increases (the red car is stuck). This exemplifies phase transition: the qualitative behavior of the agent shifts as the model size increases.
More worryingly, we observe several instances of phase transitions. phase transition, the more capable model pursues qualitatively different policy that sharply decreases the true reward. Fig- ure illustrates one example: An RI. agent regulating traffic learns to stop any cars from merging onto the highway in order to maintain high average velocity of the cars the straightaway.
Since there is little prior warning of phase transitions, they pose challenge to monitoring the safety of ML systems. Spurred by this challenge, we propose an anomaly detection task (Hendrycks & Gimpel, 2017; Tack et al., 2020): Can we detect when the true reward starts to drop, while maintaining low false positive rate in benign cases? We instantiate our proposed task, POLY- NOMALY, for the traffic and COVID environments (Section 5). Given trusted policy with mod- erate performance, one must detect whether a given policy is aberrant. We provide several base- line anomaly detectors for this task and release our data at https::///// htttps://githut com aypan17/ reward neward-misspedd missspciiiiiccccccmm f fication.
RELATED WORK
Previous works have focused on classifying different types of reward hacking and sometimes miti- gating its effects. One popular setting is an agent grid- world with an erroneous sensor. Hadfield- Menell et al. (2017) show and mitigate the reward hacking that arises due to an incorrect sensor reading at test time in 10x10 navigation grid world. Leike et al. (2017) show examples of reward hacking in 3x3 boat race and 5x7 tomato watering grid world. Everitt et al. (2017) theoretically study and mitigate reward hacking caused by a faulty sensor.
Game-playine agents have also been found to hack their reward. Baker et al. (2020) exhibit reward hacking hide-and-seek environment comprising 3-6 agents, 3-9 movable boxes and few ramps: without penalty for leaving the play area, the hiding agents learn to endlessly run from the seeking agents. Toromanoff al. (2019) briefly mention reward hacking in several Atari games (Elevator Action, Kangaroo, Bank Heist) where the agent loops in a sub-optimal trajectory that provides repeated small reward.
Agents optimizing a learned reward can also demonstrate reward hacking. Ibarz et al. (2018) show an agent hacking a learned reward in Atari (Hero, Montezuma's Revenge, and Private Eye), where optimizing a frozen reward predictor eventually achieves high predicted score and low actual score. Christiano et al. (2017) show an example of reward hacking the Pong game where the agent leams to hit the ball back and forth instead of winning the point. Stiennon al. (2020) show that policy which over-optimizes the learnt reward model for text summarization produces lower quality summarizations when judged by humans.
2

---

3 EXPERIMENTAL SETUP: ENVIRONMENTS AND REWARD FUNCTIONS
In this section, we describe our four environments (Section 3.1) and taxonomize our nine corre- sponding misspecified reward functions (Section 3.2).
3.1 ENVIRONMENTS
We chose diverse set of environments and prioritized complexity of action space, observation space, and dynamics model. Our aim was reflect real-world constraints in our environments, selecting ones with several desiderata that must be simultancously balanced. Table I provides summary.
Traffic Control. The traffic environment is autonomous vehicle (AV) simulation that models vehicles driving on different highway networks. The vehicles are either controlled by a RI algorithm pre-programmed via human behavioral model. Our misspecifications are listed in Table 1.
We use the Flow traffic simulator, implemented by Wu et al. (2021) and Vinitsky et al. (2018), which extends the popular SUMO traffic simulator (Lopez et al., 2018). The simulator like humans, following the Intelligent Driver Model (IDM) (Treiber et al., 2000), a widely-accepted approximation of human driving behavior. Simulated drivers attempt to travel as fast as possible while tending to decelerate whenever they are too close to the car immediately in front.
cars that drive
The RL policy has access observations only from the AVs controls. For each AV, the observation space consists of the car's position, its velocity, and the position and velocity of the cars immediately The continuous control action is the acceleration applied to each AV.
in front of and behind Figure 4 depicts the Traffic-Mer network, where cars from an on-ramp attempt to merge onto the straightaway. We also use the Traffic-Bot network, where cars (1-4 RL, 10-20 human) drive through a highway bottleneck where lanes decrease from four to two to one.
COVID Response. The COVID environment, developed by Kompella et al. (2020), simulates a population using the SEIR model of individual infection dynamies. The RL policymaker adjusts the severity of social distancing regulations while balancing economic health (better with lower regula- tions) and public health (better with higher regulations), similar in spirit to Trott al. (2021). The population attributes (proportion of adults, number of hospitals) and infection dynamics (random testing rate, infection rate) are based on data from Austin, Texas.
Every day, the environment simulates the infection dynamics and reports testing results to the agent, but the true infection numbers. The policy chooses one of three discrete actions: INCREASE, DECREASE, or MAINTAIN the current regulation stage, which directly affects the behavior of the population and indirectly affects the infection dynamics. There are five stages total.
Atari Riverraid. The Atari Riverraid environment is 2016). The agent operates a plane which flies over a river and is rewarded by destroying ene- mies. The agent observes the pixel input of the environment. The agent take one of eighteen discrete actions, corresponding to either movement or shooting within the environment.
OpenAl Gym (Brockman et al.,
Glucose Monitoring. The glucose environment, implemented Fox al. (2020), is a continuous control problem. It extends a FDA-approved simulator (Man et al., 2014) for blood glucose levels of a patient with Type 1 diabetes. The patient partakes in meals and wears continuous glucose monitor (CGM), which gives noisy observations of the patient's glucose levels. The RL agent administers insulin maintain a healthy glucose level.
Every five minutes, the agent observes the patient's 's glucose levels and decides how much insulin to administer. The observation space is the previous four hours of glucose levels and insulin dosages.
3.2
MISSPECIIIICTIONS
Using the above environments, constructed nine instances of misspecified proxy rewards. To help interpret these proxies, we taxonomize them as instances of misweighting, incorrect ontology. or incorrect scope. We elaborate further on this taxoninization using the traffic example from Figure 1.
3

---

Env.
Type
Objective
Proxy
Misalign?
Transition?
Mis.
underpenalize acceleration underpenalize lane changes
No
No
Mis.
minimize commute
Yes
Yes
Traffic
Ont.
and accelerations
velocity replaces commute
Yes
Yes
Scope
monitor velocity near merge
Yes
Mis.
balance economic, health, political cost
underpenalize health cost ignore political cost
No
No
COVID
Ont.
Yes
Yes
Mis.
score points under smooth movement
downweight movement include shooting penalty
No
No
Atari
Ont.
No
No
Glucose
Ont.
minimize health risk
risk in place of cost
Yes No
Table 1: Reward misspecifications across our four environments. 'Misalign' indicates whether the true reward drops and 'Transition' indicates whether this corresponds to a phase transition (sharp qualitative change). We observe instances of misalignment and instances of phase transitions. 'Mis.' is a misweighting and 'Ont.' is an ontological misspecification.
Misweighting. Suppose that the true reward a linear combination of commute time and acceler- ation (for reducing carbon emissions). Downweighting the acceleration term thus underpenalizes carbon emissions. general, misweighting occurs when the proxy and true reward capture the same desiderata, but differ their relative importance.
Ontological. Congestion could be operationalized as either high average commute time or low average vehicle velocity. In general, ontological misspecification occurs when the proxy and true reward use different desiderata to capture the same concept.
Scope. If monitoring velocity over all roads is too costly, a city might instead monitor them only over highways, thus pushing congestion local streets. In general, scope misspecification occurs when the proxy measures desiderata over a restricted domain (e.g. time, space).
We include a summary of all nine tasks in Table 1 and provide full details in Appendix A. Table also indicates whether each proxy leads to misalignment (i.e. policy with low true reward) and whether it leads to phase transition (a sudden qualitative shift as model capacity increases). We investigate both of these in Section 4.
Evaluation protocol. For each environment and proxy-true reward pair, we train an agent using the proxy reward and evaluate performance according to the reward. We use PPO (Schulman et al., 2017) to optimize policies for the traffic and COVID environments, SAC (Haarnoja et al., 2018) to optimize the policies for the glucose environment, and torchbeast (Küttler et al., 2019), Py TyToch implementation of IMPALA (Espeholt et al., 2018), to optimize the policies for the Atari environment. When available, we adopt the hyperparameters (except the learning rate and network size) given by the original codebase.
4 How AGENT OPTIMIZATION POWER DRIVES MISALIGNMENT
To better understand reward hacking, we study how it emerges agent optimization power in- creases. We define optimization power as the effective search space policies the agent has access to, as implicitly determined by model size, training steps, action space, and observation space.
In Section 4.1, we consider the quantitative effect of optimization power for all nine environment- misspecification pairs; primarily do this by varying model size, but also use training steps, action space, and observation space as robustness checks. Overall, more capable agents tend to overfit the proxy reward and achieve lower true reward. We also find evidence of phase transitions four of the environment conooment-isspecfficttiin pairs. For these phase transitions, there a critical threshold at which the proxy reward rapidly increases and the true reward rapidly drops.
In Section 4.2, we further investigate these phase transitions by qualitatively studying the result- ing policies. At the transition, find that the quantitative drop in true reward corresponds
4

---

-12
-15
-00
Proxy
2503
true
Proxy
20
-0
True
Proxy
True
300
-5
800
100
-30
1502
500
Parameters
Parameters
Parameters
(a) Traffic- Ontological
(b) COVID Ontological
(c)
Glucose Ontological
Figure 2: Increasing the RL policy's model size decreases true reward on three selected environ- ments. The red line indicates phase transition.
qualitative shift policy behavior. Extrapolating visible trends is therefore insufficient to catch all instances of reward hacking, increasing the urgency of research in this area.
In Section 4.3, we assess the faithfulness of our proxies, showing that reward hacking occurs even though the true and proxy rewards strongly positively correlated in most cases.
4.1 QUANTITATIVE EFFECTS VS. AGENT CAPABILITIES
As stand-in for increasing agent optimization power, we first vary the model capacity for fixed environment and proxy reward. Specifically, we vary the width and depth of the actor and critic networks, changing the parameter count by two to four orders of magnitude depending the envi- ronment. For a given policy, the actor and critic are always the same size.
Model Capacity. Our results shown in Figure 2, with additional plots included in Appendix A. We plot both the proxy (blue) and true (green) reward vs. the number of parameters. As model size increases, the proxy reward increases but the true reward decreases. This suggests that reward designers will likely need to take greater care to specify reward functions accurately and is especially salient given the recent trends towards larger and larger models (Littman et al., 2021).
The drop in true reward sometimes quite sudden. We call these sudden shifts phase transitions, and mark them with dashed red lines Figure 2. These quantitative trends are reflected in the qualitative behavior of the policies (Section 4.2), which typically also shift at the phase transition.
Model capacity is only one proxy for agent capabilities, and larger models do not always lead to more capable agents (Andrychowicz et al., 2020). To check the robustness of our results, we consider several other measures of optimization: observation fidelity, number of training steps, and action space resolution.
-0.50
-5.5
Proxy
-1.5
-200
-230
Tainng Stass
Testing
(a) Atari Misweighting
(b) Traffic OOtollgical
(c) COVID Ontological
Figure 3: addition to parameter count, we consider three other agent capabilities: training steps, action space resolution, and observation noise. In Figure 3a, increase in the proxy reward comes the cost of the true reward. Figure 3b, increasing the granularity (from right to left) causes agent to achieve similar proxy reward but lower true reward. In Figure 3c, increasing the fidelity of observations (by increasing the random testing rate the population) tends to decrease the true reward with clear impact proxy reward.
5

---

Number of training steps. Assuming reasonable RL algorithm and hyperparameters, agents which are trained for more steps have more optimization power. We vary training steps for an agent trained the Atari environment. The true reward incentivizes staying alive for many frames as possible while moving smoothly. The proxy reward misweights these considerations by underpenalizing the smoothness constraint. As shown in Figure 3a, optimizing the proxy reward for more steps harms the true reward, after an initial period where the rewards are positively correlated.
Action space resolution. Intuitively, an agent that can take more precise actions is more capable. For example, as technology improves, an RL car may make course corrections every millisecond instead of every second. We study action space resolution in the traffic environment by discretizing the output space of the RL agent. Specifically, under resolution level E, we round the action a e IR output by the RL agent to the nearest multiple of and use that as our action. The larger the resolution level e, the lower the action space resolution. Results are shown Figure for fixed model size. Increasing the resolution causes the proxy reward to remain roughly constant while the true reward decreases.
Observation fidelity. Agents with access to better imput sensors, like higher-resolution cameras, should make more informed decisions and thus have more optimization power. Concretely, study this in the COVID environment, where we increase the random testing rate in the population. The proxy reward is linear combination of the number of infections and severity of social distanc- ing, while the true reward also factors political cost. As shown in Figure 3c, as the testing rate increases, the model achieves similar proxy reward at the cost of a slightly lower reward.
4.2 QUALITATIVE EFFECTS
In the previous section, quantitative trends showed that increasing a model's optimization power often hurts performance on the true reward. We shift our focus to understanding how this decrease happens. In particular, we typically observe qualitative shift behavior associated with each of the phase transitions, three of which describe below.
Traffic Control. We focus the Traffic-Mer environment from Figure 2a, where minimizing average commute time is replaced by maximizing average velocity. In this case, smaller policies learn to merge onto the straightaway by slightly slowing down the other vehicles (Figure 4a). On the other hand, larger policy models stop the AVs to prevent them from merging at all (Figure 4b). This increases the average velocity, because the vehicles on the straightaway (which greatly outnumber vehicles on the on-ramp) do not need to slow down for merging traffic. However, it significantly increases the average commute time, as the passengers in the AV remain stuck.
COVID Response. Suppose the RI. agent optimizes solely for the public and economic health of society, without factoring politics into its decision-making This behavior is shown in Figure 5. The larger model chooses to increase the severity of social distancing restrictions earlier than the smaller model. As result, larger models are able to maintain low average levels of both ICU usage (a proxy for public health) and social distancing restrictions (a proxy for economic health). These
(a) Traffic policy smaller network
(b) Traffic policy larger network
Figure 4: The larger model prevents the AVs (in red) from moving to increase the velocity of the human cars (unobserved cars white and observed cars in bluc). However, this greatly increases the average commute per person.

---

ICU Usage
Regulation Stage
hospital capacity
palicy
12
larger policy
persons
10
8
6-
2
50
100
150
200
50
100
1.50
200
time (days)
time (days)
Figure 5: For COVID, ICU usage is a proxy for public health and regulation stage is proxy for economic health. The blue line indicates the maximum stage (right) enforced by the larger policy and the corresponding ICU level (left) at that stage. The red line is the equivalent for the smaller policy. Because the larger policy enforces regulations much sooner than the smaller policy, it maintains both low ICU usage and low regulation stage. However, the larger policy is politically unfavorable: regulations are high even though public signs of infection, such as ICU usage, are low.
preemptive regulations may however be politically costly, enforcing restrictions without clear signs of infection may foment public unrest (Boettke Powell, 2021).
Atari Riverraid. We create an ontological misspecification by rewarding the plane for staying alive as long possible while shooting as little as possible: "pacifist run". We then measure the game score as the true reward. We find that agents with more parameters typically maneuver more adeptly. Such agents shoot less frequently, but survive for much longer, acquiring points (true reward) due to passing checkpoints. In this case, therefore, the proxy and true rewards are well- aligned so that reward hacking does emerge as capabilities increase.
We did, however, find that some of the agents exploited a bug in the simulator that halts the plane the beginning of the level. The simulator advances but the plane itself does not move, thereby achieving high pacifist reward.
Glucose Monitoring. Consider an RL agent that optimizes solely for patient's health, without considering the economic costs of its treatment plans. In this case, the proxy reward is based off of a glycemic risk measure, which reflects the likelihood that a patient will suffer acute hypoglycemic episode, developed by the medical community (Kovatchev et al., 2000).
However, less economically-privieeeee patient may opt for the treatment plan with the least pected cost (Herkert et al., 2019; Fralick & Kesselheim, 2019), not the one with the least amount of risk. From this patient's perspective, the reward is the expected cost of the treatment plan, which includes the expected cost of hospital visits and the cost of administering the insulin.
Although larger model treatments reduce hypoglycemic risk more smaller model treatments, they administer more insulin. Based on the average cost of an ER visit for a hypogylcemic episode ($1350 from Bronstone & Graham (2016)) and the average cost of a unit of insulin ($0.32 from Lee (2020)), we find that it is actually more expensive to pursue the larger model's 's treatment.
4.3 QUANTITATIVE EFFECTS vs PROXY-TRUE REWARD CORRELATION
We saw in Sections and 4.2 that agents often pursue proxy rewards the cost of the true reward. Perhaps this only occurs because the proxy is greatly misspecified, i.c., the proxy and true reward are weakly or negatively correlated. If this were the case, then reward hacking may pose less of a threat. To investigate this intuition, we plot the correlation between the proxy and true rewards.
The correlation determined by the state distribution of a given policy, so we consider two types of state distributions. Specifically, for given model size. obtain two checkpoints: one that achieves the highest proxy reward during training and one from early in training (less than 1% of training complete). We call the former the "trained checkpoint" and the latter the "early checkpoint".

---

38
0.18
trained
0.0
0.16
eerrly
6.4
Proxy
24
0.14
02
0.12
-022
-02
20
0.08
-0.0
-04
-10
110
Parameters
Parameters
(a) Traffic-Mer Space
(h) Correlation Figure
Figure 6: Correlations between the proxy and true rewards, along with the reward hacking induced. In Figure 6a, we plot the proxy reward with "*" and the true reward with "x". In Figure 6b, we plot the trained checkpoint correlation and the early checkpoint correlation.
For a given model checkpoint, we calculate the Pearson correlation p between the proxy reward P and reward 7' using 30 trajectory rollouts. Reward hacking occurs even though there is significant positive correlation between the true and proxy rewards (see Figure 6). The correlation lower for the trained model than for the early model, but still high. Further figures are shown in Appendix A.2. Among the four environments tested, only the Traffic-Mer environment with ontological misspecification had negative Pearson correlation.
POLYNOMALY: MITIGATING REWARD MISSPECIFICATION
In Section 4, we saw that reward hacking often leads to phase transitions in agent behaviour, Fur- thermore, in applications like traffic control COVID response, the true reward may be observed only sporadically or not at all. Blindly optimizing the proxy in these cases can lead to catastrophic failure (Zhuang Hadfield-Men Mennel, 2020;
Taylor, 2016).
This raises an important question: Without the true reward signal, how can we mitigate misalign- ment? We operationalize this as an anomaly detection task: the detector should flag instances of misalignment, thus preventing catastrophic rollouts. To aid the detector, provide it with a trusted policy: one verified by humans to have acceptable (but not maximal) reward. Our resulting bench- mark, POLYNOMALY, is described below.
5.1 PROBLEM SETUP
We train collection of policies by varying model size the traffic and COVID environments. For each policy, we estimate the policy's true reward by averaging over to 32 rollouts. One author labeled each policy as acceptable, problematic, or ambiguous based its reward score relative to that of other policies. We include only policies that received non-ambiguous label.
meall-to-medimm sized model as the trusted policy model, as Section 4.1 empirically illustrates that smaller models achieve reasonable true reward without ex- hibiting reward hacking. Given the trusted model and a collection of policies, the anomaly detector's task is to predict the binary label of "acceptable" or "problematic" for each policy.
For both environments, we provide
Table in Appendix B.1 summarizes our benchmark. The trusted policy size is a list of the hidden unit widths of the trusted policy network (not including feature mappings).
5.2 EVALUATION
We propose two evaluation metrics for measuring the performance of our anomaly detectors.
Area Under the Receiver Operating Characteristic (AUROC). The AUROC measures the proba- bility that a detector will assign a random anomaly a higher score than a random non-anomalous policy (Davis & Goadrich, 2006). Higher AUROCS indicate stronger detectors.
8

---

Max F-I score. The F-1 score is the harmonic mean of the precision and the recall, detectors with high F-1 score have both low false positives and high true negatives. We calculate the max F-1 score by taking the maximum F-1 score over all possible thresholds for the detector.
5.3 BASELINES
In addition to the benchmark datasets described above, we provide baseline anomaly detectors based on estimating distances between policies. We estimate the distance between the trusted policy and the unknown policy based on either the Jensen-Shannon divergence (JSD) or the Hellinger distance. Specifically, we use rollouts to generate empirical action distributions. We compute the distance between these action distributions each step of the rollout, then aggregate across steps by taking either the mean or the range. For full details, see Appendix B.2. Table 2 reports the AUROC and F-1 scores of several such detectors. We provide full ROC curves in Appendix B.2.
Baseline Detectors
Mean Jensen-Shannon
Mean Hellinger
Range Hellinger
Env. Misspecification
AUROC
Max F-1
AUROC
Max F-1
AUROC
Max F-1
Traffic-Mer misweighting
81.0%
0.824
81.0%
0.824
76.2%
0.824
Traffic-Mer- scope
74.6%
0.818
74.6%
0.818
57.1%
0.720
Traffic-Mer ontological
52.7%
0.583
55.4%
0.646
71.4%
0.842
Traffic-Bal Pifficc-oot--o misweighting
88.9%
0.900
88.9%
0.900
74.1%
0.857
COVID ontological
45.2%
0.706
59.5%
0.750
88.1%
0.923
Table Performance detectors different subtasks. Each detector has at least one subtask with AUROC under 60%, indicating poor performance.
We observe that different detectors are better for different tasks, suggesting that future detectors could do better than any of our baselines. Our benchmark and baseline provides starting point for further research on mitigating reward hacking.
DISCUSSION
In this work, we designed a diverse set of environments and proxy rewards, uncovered several in- stances of phase transitions, and proposed anomaly detection task help mitigate these transi- tions. Our results raise two questions: How can we not only detect phase transitions, but prevent them in the first place? And how should phase transitions shape approach to safe MI.?
On preventing phase transitions, anomaly detection already offers one path forward. Once we can detect anomalies, can potentially prevent them, by using the detector to purge the unwanted behavior (e.g. by including it in the training objective). Similar policy shaping has recently been used make RL agents more ethical (Hendrycks al., 2021b). However, since the anomaly detectors will be optimized against by the RL policy, they need to be adversarially robust (Goodfellow et al., 2014). This motivates further work on adversarial robustness and adversarial anomaly detection. Another possible direction is optimizing policies against a distribution of rewards (Brown et al., 2020; Javed et al., 2021), which may prevent over-fitting to a given set of metrics.
Regarding safe ML, several recent papers propose extrapolating empirical trends to forecast future ML capabilities (Kaplan et al., 2020; Hernandez et al., 2021; Droppo & Elibol, 2021), partly to avoid unforeseen consequences from ML. While support this work, our results show that trend extrapolation alone is not enough to ensure the safety of ML systems. To complement trend extrapo- lation, need better interpretability methods to identify emergent model behaviors early on, before they dominate performance (Olah et al., 2018) 8). MI researchers should also familiarize themselves with emergent behavior in self-organizing systems (Yates, 2012). which often exhibit similar phase transitions (Anderson, 1972). Indeed, the ubiquity of phase transitions throughout science suggests that ML researchers should continue expect surprises-an -ndd should therefore prepare for them.

---

ACKNOWLEDGEMENTS
We are thankful to Dan Hendrycks and Adam Gleave for helpful discussions about experiments and Cassidy Laidlaw and Dan Hendrycks for providing valuable feedback on the writing. KB was supported by a JP Morgan AI Fellowship. JS was supported by NSF Award 2031985 and by Open Philanthropy.
10

---

REFERENCES
Philip Anderson. More is different. Science, 177(4047) 1972.
Marcin Andrychowiez, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, and Marein Michalski. What matters in on-policy reinforcement learning? A large-scale empirical study. arXiv preprint arXiv:2006.05999 05990, 2020.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In International Conference Learning Reprexentations, 2020.
Peter Boettke and Benjamin Powell. The political economy of the covid-19 pandemic. Southern Economic Journal, 87(4):1090-1 106, 2021.
Rishi Bommasani al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2 2021.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Amy Bronstone and Claudia Graham. The potential cost implications of averting severe hypo- glycemic events requiring hospitalization in high-risk adults with type 1 diabetes using real-time continuous glucose monitoring. Journal of Diabetes Science and Technology, 10, 2016.
Daniel Brown, Russell Coleman, Ravi Srinivasan, and Scott Nickum. Safe imitation learning via fast Bayesian reward inference from preferences. In Proceedings of the 37th International Conference on Machine Learning. 2020.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017.
Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. In International Conference Machine Learning, 2006.
Jasha Droppo and Oguz Elibol. Scaling laws for acoustic models. arXiv preprint arXiv:2106.09488 2021.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Robert Dunning, Shane Legg, and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. 2018.
Tom Everitt, Victoria Krakovna, Laurent Orseau, and Shane Legg. Reinforcement learning with corrupted reward channel. In International Join! Conference on Artificial Intelligence, 2017.
Ian Fox, Joyce Lee, Rodica Pop-Busui, and Jenna Wiens. Deep reinforcement learning for closed- loop blood glucose control. In Machine Learning for Healthcare Conference, 2020.
M. Fralick and A. S. Kesselheim. The U.S. Insulin Crisis Rationing a Lifesaving Medication Discovered in the 1920s. New England Journal of Medicine, 381(19): 1793-19997 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, arXiv preprint arXiv:1412.6653 2014.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, 2018.
Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. In Advances Neural Information Processing Systems, 2017.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and oft-of distribution examples in neural networks. International Conference Learning Representations, 2017.
11

---

Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. arXiv preprint arXiv:2109.13916, 2021a.
Dan Hendrycks, Mantas Mazeika, Andy Zou, Sahil Patel, Christine Zhu, Jesus Navarro, Dawn Song, Bo Li, and Jacob Steinhardt. What would Jiminy Cricket do? Towards agents that behave morally. 2021b.
Darby Herkert, Pavithra Vijayakumar, Jing Luo, Jeremy I. Schwartz, Tracy L. Rabin, Eunice De- Filippo, and Kasia J. Lipska. Cost-related insulin underuse among patients with diabetes. JAMA Internal Medicine, 179(1):112-114 Jan 2019.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01120) 2021.
Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv: 00066.082., 2019.
Borja Ibarz, J. Leike, Tobias Pohlen, Geoffrey Irving, S. Legg, and Dario Amodei. Reward learn- ing from human preferences and demonstrations in Atari. In Advances in Neural Information Processing Systems, 2018.
Zaynah Javed, Daniel S Brown, Satvik Sharma, Jerry Zhu, Ashwin Balakrishna, Marek Petrik, Anca Dragan, and Ken Goldberg. Policy gradient bayesian robust optimization for imitation learning. In Proceedings of the 38th International Conference Machine Learning, 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08330 2020.
W. Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (Mis)design for Autonomous Driving, arXiv e-prints arXiv:2104. 13906, 2021.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238-22) 2013.
Varun Kompella, Roberto Capobianco, Stacy Jong, Jonathan Browne, Spencer Fox, Lauren Meyers, Peter Wurman, and Peter Stone. Reinforcement learning for optimization of covid-19 mitigation policies, 2020.
Borls. Kovatchev, Martin Straume, Daniel Cox, and Leon.S Farhy Risk analysis of blood glu- cose data:a quantitative approach to optimizing the control of insulin dependent diabetes. Journal of Theoretical Medicine, 3(1):1-10, 2000.
Heinrich Küttler, Nantas Nardelli, Thibaut Lavril, Marco Selvatici, Viswanath Sivakumar, Tim Rocktäschel, and Edward Grefenstette. TorchBeast: A PyTorch Platform for Distributed RL. arXiv preprint arXiv: /tt.v:/990003000 191000355. 2019.
Benita Lee. How much does insulin cost? Here's how 23 brands compare, Nov 2020.
Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A. Ortega, Tom Everitt, Andrew Lefrancq. Laurent Orseau, and Shane Legg. Al safety gridworlds, 2017.
Michael L. Littman, Ifeoma Ajunwa, Guy Berger, Craig Boutilier, Morgan Currie, Finale Doshi- Velez, Gillian Hadfield, Michael C. Horowitz, Charles Isbell, Hiroaki Kitano, Karen Levy, Terah Lyons, Melanie Mitchell, Julie Shah, Steven Sloman, Shannon Vallor, and Toby Walsh. Gathering strength, gathering storms: The one hundred year study on artificial intelligence (AI100) 2021 study panel report. Technical report, Stanford University, Stanford, CA, 2021.
Pablo Alvarez Lopez, Michael Behrisch, Laura Bieker- Walz, Jakob Erdmann, Yun-Pang Flötteröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamarie Wießner. Microscopic traffic simulation using SUMO. In International Conference on Intelligent Trans- portation Systems, 2018.
12

---

Chiara Dalla Man, Francesco Micheletto, Dayu Lv, Marc Breton, Boris Kovatchev, and Claudio Cobelli. The UVA/PADOVA type diabetes simulator: New features. Journal of Diabetes Science and Technology. 8(1):26 34, Jan 2014.
Chris Olah, Arvind Saryanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 3(3)=11,, 2018.
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations, 2018.
Manoel Horta Ribeiro, Raphael Ottoni, Robert West, Virgilio A. F. Almeida, and Wagner Meira. Auditing radicalization pathways youtube. In Conference on Fairness, Accountability, and Transparency, New York, NY, USA, 2020.
Stuart Russell, Human Compatible: Artificial Intelligence and the Problem of Control. Penguin, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. arXiv preprint 00t1:22009.01325, 2020.
Jonathan Stray. Aligning ai optimization community well-being. International Journal of Com- munity Well-Being, 3(4):443-463, Dec 2020.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via con- trastive learning on distributionally shifted instances. Advances in Neural Information Processing Systems, 2020.
Jessica Taylor.
Quantilizers: A safer alternative maximizers for limited optimization. In AAAI Workshop: AI, Ethics, and Society, 2016.
Marin Toromanoff, Emilie Wirbel. and Fabien Moutarde. Is deep reinforcement learning really superhuman on Atari? Leveling the playing field, 2019.
Martin Treiber, Ansgar Hennecke, and Dirk Helbing. Congested traffic states in empirical observa- tions and microscopic simulations. Physical review E, 62(2):1805, 2000.
Alexander Trott, Sunil Srinivasa, Douwe der Wal, Sebastien Haneuse, and Stephan Zheng. Building a Foundation for Data-Driven, Interpretable, and Robust Policy Design using the Al Economist. arXiv preprint arXiv:2108.02900 2021.
Eugene Vinitsky, Aboudy Kreidich, Luc Le Flem, Nishant Kheterpal, Kathy Jang, Cathy Wu, Fangyu Wu, Richard Liaw, Eric Liang, and Alexandre M. Bayen. Benchmarks for reinforcement learning in mixed-autonomyy traffic. In Conference on Robot Learning. 2018.
Cathy Wu, Abdul Rahman Kreidich, Kanaad Parvate, Eugene Vinitsky, and Alexandre M. Bayen. Flow: A modular learning framework for mixed autonomy traffic. IEEE Transactions on Robotics, 2021.
Eugene Yates. Self-organizing systems: The emergence of order. Springer Science & Business Media, 2012.
Chao Yu, Jiming Liu, and Shamim Nemati. Reinforcement learning in healthcare: A survey, arXiv preprint arXiv:1908 100..0999000896, 2019.
Simon Zhuang and Dylan Hadfield- Consequences of misaligned AI. In Advances in Neural Information Processing Systems, 2020.
13

---

A
MAPPING THE EFFECTS OF REWARD MISSPECIHICATION
2300
8.5
8.0
1890
2.75
2.50
7.5
Proxy
Proxy
1880
2.25
7.0
2.00
6.5
1900
1870
1.75
6.0
1800
1.50
5.5
1700
1860
1.25
5.0
10a
10"
10
10a
10a
10"
10'
10a
Parameters
Parameters
(a) mregee
Misweighting
(b) traffic bortle Misweighting
40
38
0.18
-1.00
0.16
1.25
60
Proxy
0.14
Proxy
-1.50
32
0.12
-1.75
0.10
-2.00
100
-2.25
-2.50
26
0.06
10a
10a
10
10a
10a
Parameters
Parameters
(c) traffic merge Space
(d) COVID Misweighting
6000
800
50000
Proxy
600
True
Proxy
1000000
True
5000
4000
150000
400
4000
200000
3000
250000
10a
107
Parameters
parameters
(e) Atari Ontological
(f) Atari Misweighting
Figure 7: Additional model size scatter plots. Observe that all misspecifications cause mis- alignment. We plot the proxy reward with "*" and the true reward with "x".. The proxy reward is measured on the left-hand side of each figure and the true reward measured the right hand side of each figure.
A. EFFECT OF MODEL SIZE
We plot the proxy and true reward ys. model size in Figure 7, following the experiment described in Section 4.1.
14

---

3 00
1.0
1.0
0.8
0.8
1890
75
O T) trained
0.6
0.6
eerrlyy
0.4
0.4
Proxy
1880
2.25
true Jue
0.2
0.2
0.2
0.2
1870
175
-0.4
1550
-0.6
-0.6
1860
1.25
-0.8
-1.0
-1.0
10a
10'
10a
10
10a
104
105
10c
Parameters
Parameters
(a) traffic_
Misweighting
(b) Correlation Figure
2300
0.8
0.8
(PP. T) trained
8.0
eerrly
7.5
0.4
0.4
Proxy
2100
0.2
0.2
2000
0.0
0.0
6.5
0.2
0.2
1900
6.0
0.4
-0.4
1800
0.66
006
5.5
-0.8
0088
1700
-1.0
-1.0
10a
104
10a
10f
10a
10a
10
109
Parameters
Parameters
(c) tregeec Misweighting
(d) Correlation for Figure &c
1.0
1.0
0.8
0.8
3000
PPP. T) trained
0.6
0.6
eerrly
0.4
0.4
Proxy
2500
-200
True
0.2
0.2
0.0
0.0
3300
0.2
2000
0.4
00.4
-400
1500
-500
0.8
008
-1.0
10a
10a
10a
10
Parameters
Parameters
(c) trageic Ontological
(f) Correlation for Figure 8c
Figure 8: Correlations between the proxy and true rewards, along with the reward hacking induced. In the left column, plot the proxy reward with "*" and the true reward with "x". In the right col- umn, we plot the trained checkpoint correlation and the randomly initialized checkpoint correlation.
A.2 CORRELATION BETWEEN PROXY AND TRUE REWARDS
We plot the correlation between proxy and true rewards, following the experiment described in Section 4.3. Interestingly, we see that reward hacking still occurs when there positive correlation between the and proxy rewards, e.g., in Figures 8a/Sb. Unsurprisingly, proxy-true pairs which are highly correlated, e.g., Figure 8c/8d do not exhibit reward hacking. Finally, proxy-true pairs which are negatively correlated, e.g.. Figure 8c/8f exhibit the most reward hacking.
15

---

Env. Misspecification
Policies
# Problematic
Rollout length
Trusted policy size
Traffic-Mes misweighting
10
270
196, 96
Traffic-Mer- scope
16
9
270
[6. 16)
Traffic-Mer ontological
23
7
270
[4]
Traffic-Bot misweighting
12
9
270
664, 64 [16, 16]
COVID ontological
13
6
200
Table 3: Benchmark statistics. We average over rollouts in traffic and 32 rollouts in COVID.
B POLYNOMALY
B.1 BENCHMARK STATISTICS
See Table for Polynomaly's statistics.
B.2 RECEIVER OPERATING CHARACTERISTIC CURVES
We plot the ROC curves for the detectors described in Section 5.3. Our detectors calculated as follows.
Let and Q represent two probability distributions with .M (P) Then the Jensen-Shannon divergence and the Hellinger distance between them given by
ISD(PIIQ) 1 KL(PIIM) + ( 1 Hellinger(P,, (P,nngerr((,)))e) 1 (VdP-vaQ)
Our proposed detectors estimate the distance (TTissod Tunknown) between the trusted policy Tinssted and unknown policy Tunknown as follows: We generate rollouts of Tunknown where 5 in the traffic environment and r = 32 in the COVID environment. Every s steps of rollout, where 8 = 10 in the traffic environment and of Tunknown given the unknown agent's 's state at that timestep in the rollout and Q to be the action distribution Tiustedd given the unknown agent's state at that timestep the rollout. Intuitively, if P and Q are far apart, then the trusted agent would have performed a different action than the unknown agent that given timestep, indicating possible case of reward hacking. We then compute either ISD(PIIQ) Hellinger(P., Q) following Equation (1). These distances are collected every steps over the entire rollout, and we calculate metrics on these distances (range, mean, etc.) to assign an anomaly score to the untrusted policy.
in the COVID environment, we set P be the action distribution
16

---

Mean-JS ROC
Mean-H ROC
1.0
True positive rate
True positive rate
0.8
0.8
0.4
0.4
0.0
0.0
0.0
0.6
0.2
0.5
0.8
False positive rate
False positive rate
(a)
(b)
Range-H ROC
1.0
True positive rate
0.6
0.2
0.0
0.2
0.6
08
False positive rate
(c)
Figure 9: ROC curves for Traffic-Mer misweighting.
Mean-JS ROC
Mean-H ROC
True positive rate
True positive rate
0.8
0.8
0.4
0.4
0.0
0.0
0.0
0.6
0.2
0.6
0.8
False positive rate
False positive rate
(a)
(b)
Range-H ROC
1.0
True positive rate
0.6
0.2
0.0
0.2
0.6
08
False positive rate
(c)
Figure 10: ROC curves for Traffic-Mer scope.
17

---

Mean-JS ROC
Mean-H ROC
1.0
True positive rate
True positive rate
0.3
0.8
0.4
0.4
0.0
0.0
0.0
0.6
0.2
0.8
False positive rate
False positive rate
(a)
(b)
Range-H ROC
1.0
True positive rate
0.6
0.2
0.0
0.2
0.6
08
False positive rate
(c)
Figure 11: ROC curves for Traffic-Mer ontological.
Mean-JS ROC
Mean-H ROC
True positive rate
True positive rate
0.8
0.8
0.4
0.4
0.0
0.0
0.0
0.6
0.2
0.6
0.8
False positive rate
False positive rate
(a)
(b)
Range-H ROC
1.0
True positive rate
0.6
0.2
0.0
0.2
0.6
08
False positive rate
(c)
Figure 12: ROC curves for Traffic Bot misweighting.
18

---

Mean-JS ROC
Mean-H ROC
1.0
True positive rate
True positive rate
0.8
0.8
0.4
0.4
0.0
0.0
0.0
0.6
0.2
False positive rate
False positive rate
(a)
(b)
Range-H ROC
1.0
True positive rate
0.6
0.2
0.0
0.2
0.6
08
False positive rate
(c)
Figure 13: ROC curves for COVID ontological.
19


Reviews:
[{"review_id": "ZNXNRvWRBkH", "reviewer": "Reviewer_16uL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper does a great job at investigating an important problem in AI safety in a concrete and rigorous way. There is an informal assumption in AI safety that the reward hacking problem will tend to get worse as the agent becomes more capable. To my knowledge, this paper is the first to demonstrate this effect quantitatively by varying different agent capabilities (such as model capacity) and showing that the true reward decreases as a result. The paper is clearly written and well-motivated. \n\nThis paper systematically investigates the reward hacking phenomenon across a variety of environments, agent capabilities and forms of misspecification. The environments used in the paper are diverse and complex, which helps to show that reward hacking occurs in a wide variety of domains (though it would be good to see results on more than one Atari environment). The environments were thoughtfully chosen to include tradeoffs between several desiderata that have to be managed by the agent. \n\nThe authors introduce a taxonomy of misspecification (misweighting, ontology and scope) and design a variety of proxy rewards illustrating the different forms of misspecification. The choice of proxy and true rewards for some of the environments seemed a bit arbitrary (e.g. average speed vs mean commute time), so it would be great to see more justification for this. \n\nI found the section on mitigating reward misspecification to be a somewhat weaker point of the paper. I have the impression that the proposed benchmark requires a large amount of human feedback - it would be great to include the human time cost for the anomaly detection task. The anomaly detection task is mostly illustrated for the traffic environment - it would be useful to include more of the other environments as well. I also wonder whether using the distance from the (suboptimal) trusted policy would result in labeling a policy with superhuman performance on the true reward as aberrant (e.g. AlphaGo's Move 37, which is very unlikely for a human player). \n\n", "summary_of_the_paper": "This paper investigates the phenomenon of reward hacking as a function of agent capabilities. They introduce four diverse RL environments with nine misspecified rewards and demonstrate that more capable agents are better at exploiting the misspecification. They find instances of phase transitions where a small increase in agent capability produces a large change in behavior that sharply decreases the true reward.\n\nTo mitigate the reward hacking problem, they propose to set up an anomaly detection task, given a trusted model with moderate performance on the true reward, where the anomaly detector's task is to identify whether policies from a different model are satisfactory for the true reward. They provide several baseline anomaly detectors and show how they perform on different tasks. ", "summary_of_the_review": "This paper does a great job at investigating an important problem in AI safety in a concrete and rigorous way, and to my knowledge is the first to do so. I am in favour of accepting this paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "PTO4B7EzBrN", "reviewer": "Reviewer_GVMn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper provides an interesting study of reward hacking behaviour where different parameters of the problem are varied to demonstrate the phenomenon. The authors chose a set of diverse and relevant environments. An interesting observation that might have large implications in practice is that training more capable agents might result in the behavior that achieves high proxy reward, but low true reward. While this behaviour was noticed before, I am not aware of a systematic study of the phenomenon where various parameters of the environment, reward and agent are controlled. Unfortunately, the paper does not provide any extensive related work overview that seems to be an important missing part given that the main contribution of the paper is the systematic study of reward hacking.\n\nOne concern that I have is that when the authors study misspecified rewards, the specification of the reward is only motivated by intuition. It is hard to appreciate how reasonable such a specification is without the full knowledge and experience with a particular environment. To me, a reasonable reward specification, even if it is misspecified, would still have significant correlation with the true reward. Sometimes (like in Figure 2) it seems that true reward goes down when the proxy reward goes up and it would be no surprise that a reward hacking occurs when the true objective is the opposite of what is optimised. A more interesting observation would be that a reward hacking occurs even when the objectives are mostly aligned and only sometimes diverge. To understand this better, it would be useful to look at some plots depicting the dependency of proxy and true reward on a broad set of sampled trajectories. \n\nFinally, the authors provide a new baseline of anomaly detection for identifying the “phase transition” in the agent’s behaviour. Despite being small, such problem formulation could be useful for sparking more research in this direction. \n\nOther comments and concerns:\n\n- I didn’t find the Figure 1 very informative on its own, it is only possible to understand the bottom row after reading the main text and at that point the figure does not bring any new information. I would recommend trying to make the figure with its caption more self-contained\n\n- Often rewards are manually crafted in continuous control tasks such as, for example, robotics. Would it be possible to provide another environment with, for example, control of a simulated robotic arm performing the manipulation tasks? The rewards might be assigned (and misspecified) based on the distances or positions of the objects. \n\n- I would like the paper to be a bit more specific in its claims. For example, when the paper talks about more capable agents archiving “lower true reward”, lower than what? When the paper talks about “critical threshold”, it is the threshold of what?\n\n- The results often seem to be quite noisy, for example, see Figure 2. How many experiments are conducted?\n\n- The results would be more informative if they included the performance of the agent that optimises the true reward directly to provide an upper bound on the agent’s performance.\n\n- I am not completely convinced by the example of diabetic risk and cost of insulin. I think this example does not take into account the ethics of such policy and the long-term costs of losing health due to acute hypoglycemic episodes.\n", "summary_of_the_paper": "This paper provides a systematic study of “reward hacking” in the environments with the misspecified rewards. The authors conduct a set of experiments with 4 environments, several types of reward misspecification in each of them and several agents of different expressivity (model capacity). They notice that often the agents that are more capable end up obtaining high proxy reward, but low real reward. Besides, often the transition to the low real reward happens very quickly and authors call this phenomenon “phase transition”. Finally, they propose a baseline for anomaly detection to identify this phase transition.", "summary_of_the_review": "I appreciate a systematic study of the reward hacking phenomenon where the parameters of the problem are manually varied. I think the related work overview should be extended to justify the “systematic” aspect of the paper. Besides, some information about the correlation between the true and the proxy rewards would be very beneficial to the reader without prior experience with the studied environments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "LveRouNcuO2", "reviewer": "Reviewer_bfGN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper’s findings are clearly useful to the research community as reward hacking has been a poorly understood but ubiquitous problem, broadly relevant for many practical settings. If the paper only brings extra attention to this topic, I think it has had some impact. The most important findings in my view are that:\n\n1) Reward hacking can not only stop the true reward from increasing, but often actively lowers it \n2) Increasing capability can increase reward hacking and thereby lower reward. This will be unsurprising to some readers but I believe it is still somewhat controversial and it helps to have demonstrations across several environments and several ways to increase capability, which makes these results more certain. This results in several interesting hacking behaviors.\n3) Increasing capability can lead to phase transitions, a tricky new problem for that should be brought to the attention of RL practitioners as it may necessitate new monitoring strategies (such as the ones proposed in this paper).\n\nSome readers won't find all of the above particularly surprising or novel. However, given how common and understudied reward hacking is, I believe that they deserve the more systematic study and exposition this paper provides.\n\nThe paper neglects to formalize some concepts. To some extent this is necessary for the first paper that investigates an important concept (reward hacking) which so far has no definition - I therefore think the lack of formality can be forgiven. What makes rigor more challenging here is that reward hacking and phase transitions can happen through multiple means such as increasing the size of the model and action space. I think the paper is a strong attempt at an important and hard problem, and will provide a good basis on which others can study reward hacking with increased formalization and rigor.\n\nI agree with the other reviewers that some of environment and their proxy-true reward pair are not particularly convincing. I've lowered my score accordingly as I think this could detract from the impact this paper has on the research community. But I also recognize that it is difficult to design realistic pairs of reward functions. In reality, the true reward is typically too complex to specify (hence we need a proxy). Because a complex reward is not available in practice, in research papers we need to develop examples that are somewhat artificial.\n\nWhile the paper is mostly well motivated, well written, and easy to follow, I do find that the presentation could be more precise and clear at times, as noted below.\n\nIn the final paragraph of the introduction, it was unclear to me what is the value proposition for the proposed benchmark and the baselines. This contribution could be better motivated.\n\nI appreciate that in several places, such as Figure 4b,  the authors chose to display negative results where reward hacking and phase transitions were not observed. As researchers tend to cherry-pick the most convincing results, it is good to see that the authors avoided this.\n\n\n\n\n———————————————— Detailed comments ————————————————\n\n\n\nI’ll start by giving detailed feedback on the figures because most readers will look primarily at these.\n\n- Figure 1: \n—The figure with caption is not self-explanatory. E.g. who is the agent in this decision problem and what is their action space? (controlling a traffic light?) \n—It’s not immediately obvious why the proxy and true rewards lead to different behavior here and it is not explained either. For an illustrative figure like this, I’d recommend using a more obvious example.\n—The meaning of arrow colors is unclear\n—It’s unclear what ‘optimization ability’ means here. If you’re referring to model size, you may want to use the same model size symbols you’re using in Fig 1c instead.\n\nFigure 2: nice figure.\n\nFigure 3: this figure is not self-explanatory, it needs more detail. The testing rate and the meaning of 16/112 are not explained. This figure uses a different format than figure 2 (separate plots for true/proxy rewards) which confused me at first sight. You could have one plot per model size to fix this, or move everything into one plot.\n\nFigure 4: the takeaway could be clarified in the caption, especially for 4b (I believe it’s that no reward hacking occurred).\n— Minor point: Figure 4a has a lower resolution than Figure 4b.\n\nFigure 6 and Table 3 could also be improved to be more self-explanatory.\n\n\nMinor suggestions:\n\n“Overfitting their objectives” - unclear how overfitting is defined here, it seems to be a non-standard use of the word.\n\n“we study how increasing optimization power affects reward hacking, by training RL agents with varying resources such as model size …” - increasing model size is not the same as increasing optimization. “Optimization power” makes me as the reader expect that you are talking about doing more/better gradient descent. Perhaps there is no better term here, in which case this is a necessary evil.\n\nIn Figure 4, it is unclear what action noise has to do with misspecification of the action space. Instead of adding Gaussian noise, I would expect that you vary the action space, e.g. discretize it at different resolutions. This does resemble Gaussian noise but it’s not the same thing.\n\nThe paper lacks motivation for why it’s hard not to resort to proxy rewards in practice. Some readers will already know this, others won’t.\n\nCOVID results (section 3.2): I get the result here and it is interesting - the policymaker only regulates once infections are high, but this saves little economic costs because it only delays the restrictions once while increasing the number of ICU patients indefinitely. But I found this not very well explained.\n\nIn section 3.2, it was unclear what is your takeaway from the Glucose environment. That reward hacking happens? That there was a phase transition?\n\n“The objective of the RL agent is to promote a smooth traffic flow within the highway network” - this is confusing as in other places the true objective is said to be minimizing the mean commute time.\n", "summary_of_the_paper": "This paper studies reward hacking, a common but understudied phenomenon, across a set of environments. Reward hacking emerges in several tasks, meaning that the resulting policy has a high proxy reward but a low true reward. A key finding is that reward hacking increases with agent capabilities so that increasing capability lowers the true reward. This holds across several ways of increasing capabilities (model size, training steps, action space, etc). The authors also find ‘phase transitions’ where a small increase in capability results in qualitatively new reward hacking behavior, a phenomenon that may require novel monitoring strategies. One such strategy is anomaly detection, for which the authors introduce a benchmark and baselines.\n", "summary_of_the_review": "The paper’s findings are clearly useful to the research community as the reward hacking problem has been a poorly understood but ubiquitous. Three specific findings about reward hacking (see above) had to my knowledge not been empirically demonstrated in RL environments or only mentioned in passing but not studied explicitly. Before this paper, we therefore did not know if the findings correspond to generally applicable phenomena and through this paper we can gain some confidence. However, some environments and reward functions could be made more realistic.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "78UDwG-oUUe", "reviewer": "Reviewer_uYeb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper targets the very important problem of misaligned AI models, under the specific lens of reinforcement learning based optimization. This problem has been widely studied in the AI safety / alignment community but has not received equivalent attention in the machine learning community. The experiments presented in the paper shed light on different settings where the problem of reward-hacking might happen, and under controlled variations of the agent's modeling power, and size of the state-space, shows different forms of reward-hacking in these environments. The paper is well-written and easy to follow, and has a thorough coverage of related works in AI alignment. \n\nMy main concerns with the paper are as follows:\n\n1. None of the experiments in the paper are on sufficiently realistic settings. The paper tries to present empirical evidence of reward-hacking on a number of environments, but doesn't dig deeper into any of them. The specific challenges of each setting are different, and it would be useful to focus on a few settings and perform the evaluations keeping real-world considerations in mind. For example, in the traffic simulator, it would be useful to consider pixel-observations (which can be obtained easily for example in the CARLA simulator) and study the same problem. \n\n2. The proxy reward functions considered are very \"incorrect\" in all the settings. In my understanding, the proxy reward functions are trying to show what happens when a reward function different from the true reward function is optimized by the agent. If so, then every attempt must be made to construct reward functions that are *very* close to the true reward function. This is similar to shaped reward functions in typical RL pipelines - it wis important to construct properly shaped functions. An example of this for the traffic example could be a weighted combination of the 4 proxy functions. \n\n3. Missing experiments on environments where the true reward function cannot be easily constructed. The main challenge of misaligned models is in settings where the underlying reward function is unknown and cannot be queried. As a simple example, consider the task of pick and place of a puck with a robot arm. The end result of \"puck in goal location\" can be used to check task success but there is no underlying ground-truth dense reward function. In these settings, different papers come up with different shaped rewards for RL - it would be interesting to show experiments on these settings where the underlying reward function is not known, and demonstrate how different shaped rewards (similar to proxy rewards in the paper) fail, and draw inferences from them. This relates back to my first point above about realistic settings. \n\n4. Question about atari experiments: why not evaluate on a large number of the atari games, as done by most RL papers that show results on atari? \n\n5. Section 4 is very vague without clear takeaways. The notion of a trusted policy is introduced in section 4, as a way to mitigate the reward hacking problem, but the conclusions from Table 4 are unclear. What exactly is helpful in mitigating reward-hacking? Which metrics should we use for checking the extent of model mis-specification?\n\n6. What is the number of training steps experiment measuring? Is the conclusion of this experiment that an agent that has been optimized for more time-steps will likely succumb to more reward-hacking? If so, then this requires addition experimentation because \"not optimizing enough\" cannot be a solution to reward-hacking, as that would also lead to lower true rewards as seen in the experiment.\n\n7. The addition of action noise experiments (which the paper calls \"resolution\") seem to have a trivial takeaway? Adding more noise (i.e. higher variance) causes both the true and proxy rewards to decrease. I request the authors to explain how this non-trivial. \n\n8. An additional dimension of agent capabilities to check would be the \"type of training algorithm\" used. This would likely be very important because different variants of algorithms (e.g. on-policy / off-policy, model-based / model-free) would have different failure modes with respect to reward hacking. This might provide more useful empirical takeaways for future work, compared to action resolution, and training steps variations considered. \n\nIn summary, I believe this paper tackles a very important and timely problem through a purely empirical lens. Since, this is a purely empirical paper, it is important to have more comprehensive experiments that are reflective of real-world settings, and have better motivated design choices. The paper in its current form does not provide useful takeaways that are practically significant and would require significant revision. \n\n", "summary_of_the_paper": "This paper targets the very important problem of reward-hacking that occurs when the objectives optimized by intelligent agents are misaligned with respect to the tru objectives of the algorithm designer. The paper presents an empirical study across a range of different settings including a simple driving simulator, covid modeling, and a single atari game. The experiments show evidence of reward hacking as a function of modeling power of the agent and the size of the state-space. The paper concludes with some ideas and initial directions on how to potentially mitigate reward hacking. ", "summary_of_the_review": "I believe this paper tackles a very important and timely problem through a purely empirical lens. Since, this is a purely empirical paper, it is important to have more comprehensive experiments that are reflective of real-world settings, and have better motivated design choices. The paper in its current form does not provide useful takeaways that are practically significant and would require significant revision. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: I thank the authors for their submission and active participation in the discussions. All reviewers are unanimously leaning towards acceptance of this paper. Reviewers in particular liked that the paper is presenting an interesting and systematic study of reward hacking [GVMn] that is useful to the research community [bfGN] and targets an important problem [uYeb] in a rigorous way [16uL]. I thus recommend accepting the paper, but I strongly encourage the authors to further improve their paper based on the reviewer feedback, in particular in regards to improving positioning with respect to related work and a better formalization of their work.
Decision: Accept (Poster)