Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides a systematic study of “reward hacking” in the environments with the misspecified rewards. The authors conduct a set of experiments with 4 environments, several types of reward misspecification in each of them and several agents of different expressivity (model capacity). They notice that often the agents that are more capable end up obtaining high proxy reward, but low real reward. Besides, often the transition to the low real reward happens very quickly and authors call this phenomenon “phase transition”. Finally, they propose a baseline for anomaly detection to identify this phase transition.
main_review: The paper provides an interesting study of reward hacking behaviour where different parameters of the problem are varied to demonstrate the phenomenon. The authors chose a set of diverse and relevant environments. An interesting observation that might have large implications in practice is that training more capable agents might result in the behavior that achieves high proxy reward, but low true reward. While this behaviour was noticed before, I am not aware of a systematic study of the phenomenon where various parameters of the environment, reward and agent are controlled. Unfortunately, the paper does not provide any extensive related work overview that seems to be an important missing part given that the main contribution of the paper is the systematic study of reward hacking.

One concern that I have is that when the authors study misspecified rewards, the specification of the reward is only motivated by intuition. It is hard to appreciate how reasonable such a specification is without the full knowledge and experience with a particular environment. To me, a reasonable reward specification, even if it is misspecified, would still have significant correlation with the true reward. Sometimes (like in Figure 2) it seems that true reward goes down when the proxy reward goes up and it would be no surprise that a reward hacking occurs when the true objective is the opposite of what is optimised. A more interesting observation would be that a reward hacking occurs even when the objectives are mostly aligned and only sometimes diverge. To understand this better, it would be useful to look at some plots depicting the dependency of proxy and true reward on a broad set of sampled trajectories. 

Finally, the authors provide a new baseline of anomaly detection for identifying the “phase transition” in the agent’s behaviour. Despite being small, such problem formulation could be useful for sparking more research in this direction. 

Other comments and concerns:

- I didn’t find the Figure 1 very informative on its own, it is only possible to understand the bottom row after reading the main text and at that point the figure does not bring any new information. I would recommend trying to make the figure with its caption more self-contained

- Often rewards are manually crafted in continuous control tasks such as, for example, robotics. Would it be possible to provide another environment with, for example, control of a simulated robotic arm performing the manipulation tasks? The rewards might be assigned (and misspecified) based on the distances or positions of the objects. 

- I would like the paper to be a bit more specific in its claims. For example, when the paper talks about more capable agents archiving “lower true reward”, lower than what? When the paper talks about “critical threshold”, it is the threshold of what?

- The results often seem to be quite noisy, for example, see Figure 2. How many experiments are conducted?

- The results would be more informative if they included the performance of the agent that optimises the true reward directly to provide an upper bound on the agent’s performance.

- I am not completely convinced by the example of diabetic risk and cost of insulin. I think this example does not take into account the ethics of such policy and the long-term costs of losing health due to acute hypoglycemic episodes.

summary_of_the_review: I appreciate a systematic study of the reward hacking phenomenon where the parameters of the problem are manually varied. I think the related work overview should be extended to justify the “systematic” aspect of the paper. Besides, some information about the correlation between the true and the proxy rewards would be very beneficial to the reader without prior experience with the studied environments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies reward hacking, a common but understudied phenomenon, across a set of environments. Reward hacking emerges in several tasks, meaning that the resulting policy has a high proxy reward but a low true reward. A key finding is that reward hacking increases with agent capabilities so that increasing capability lowers the true reward. This holds across several ways of increasing capabilities (model size, training steps, action space, etc). The authors also find ‘phase transitions’ where a small increase in capability results in qualitatively new reward hacking behavior, a phenomenon that may require novel monitoring strategies. One such strategy is anomaly detection, for which the authors introduce a benchmark and baselines.

main_review: The paper’s findings are clearly useful to the research community as reward hacking has been a poorly understood but ubiquitous problem, broadly relevant for many practical settings. If the paper only brings extra attention to this topic, I think it has had some impact. The most important findings in my view are that:

1) Reward hacking can not only stop the true reward from increasing, but often actively lowers it 
2) Increasing capability can increase reward hacking and thereby lower reward. This will be unsurprising to some readers but I believe it is still somewhat controversial and it helps to have demonstrations across several environments and several ways to increase capability, which makes these results more certain. This results in several interesting hacking behaviors.
3) Increasing capability can lead to phase transitions, a tricky new problem for that should be brought to the attention of RL practitioners as it may necessitate new monitoring strategies (such as the ones proposed in this paper).

Some readers won't find all of the above particularly surprising or novel. However, given how common and understudied reward hacking is, I believe that they deserve the more systematic study and exposition this paper provides.

The paper neglects to formalize some concepts. To some extent this is necessary for the first paper that investigates an important concept (reward hacking) which so far has no definition - I therefore think the lack of formality can be forgiven. What makes rigor more challenging here is that reward hacking and phase transitions can happen through multiple means such as increasing the size of the model and action space. I think the paper is a strong attempt at an important and hard problem, and will provide a good basis on which others can study reward hacking with increased formalization and rigor.

I agree with the other reviewers that some of environment and their proxy-true reward pair are not particularly convincing. I've lowered my score accordingly as I think this could detract from the impact this paper has on the research community. But I also recognize that it is difficult to design realistic pairs of reward functions. In reality, the true reward is typically too complex to specify (hence we need a proxy). Because a complex reward is not available in practice, in research papers we need to develop examples that are somewhat artificial.

While the paper is mostly well motivated, well written, and easy to follow, I do find that the presentation could be more precise and clear at times, as noted below.

In the final paragraph of the introduction, it was unclear to me what is the value proposition for the proposed benchmark and the baselines. This contribution could be better motivated.

I appreciate that in several places, such as Figure 4b,  the authors chose to display negative results where reward hacking and phase transitions were not observed. As researchers tend to cherry-pick the most convincing results, it is good to see that the authors avoided this.




———————————————— Detailed comments ————————————————



I’ll start by giving detailed feedback on the figures because most readers will look primarily at these.

- Figure 1: 
—The figure with caption is not self-explanatory. E.g. who is the agent in this decision problem and what is their action space? (controlling a traffic light?) 
—It’s not immediately obvious why the proxy and true rewards lead to different behavior here and it is not explained either. For an illustrative figure like this, I’d recommend using a more obvious example.
—The meaning of arrow colors is unclear
—It’s unclear what ‘optimization ability’ means here. If you’re referring to model size, you may want to use the same model size symbols you’re using in Fig 1c instead.

Figure 2: nice figure.

Figure 3: this figure is not self-explanatory, it needs more detail. The testing rate and the meaning of 16/112 are not explained. This figure uses a different format than figure 2 (separate plots for true/proxy rewards) which confused me at first sight. You could have one plot per model size to fix this, or move everything into one plot.

Figure 4: the takeaway could be clarified in the caption, especially for 4b (I believe it’s that no reward hacking occurred).
— Minor point: Figure 4a has a lower resolution than Figure 4b.

Figure 6 and Table 3 could also be improved to be more self-explanatory.


Minor suggestions:

“Overfitting their objectives” - unclear how overfitting is defined here, it seems to be a non-standard use of the word.

“we study how increasing optimization power affects reward hacking, by training RL agents with varying resources such as model size …” - increasing model size is not the same as increasing optimization. “Optimization power” makes me as the reader expect that you are talking about doing more/better gradient descent. Perhaps there is no better term here, in which case this is a necessary evil.

In Figure 4, it is unclear what action noise has to do with misspecification of the action space. Instead of adding Gaussian noise, I would expect that you vary the action space, e.g. discretize it at different resolutions. This does resemble Gaussian noise but it’s not the same thing.

The paper lacks motivation for why it’s hard not to resort to proxy rewards in practice. Some readers will already know this, others won’t.

COVID results (section 3.2): I get the result here and it is interesting - the policymaker only regulates once infections are high, but this saves little economic costs because it only delays the restrictions once while increasing the number of ICU patients indefinitely. But I found this not very well explained.

In section 3.2, it was unclear what is your takeaway from the Glucose environment. That reward hacking happens? That there was a phase transition?

“The objective of the RL agent is to promote a smooth traffic flow within the highway network” - this is confusing as in other places the true objective is said to be minimizing the mean commute time.

summary_of_the_review: The paper’s findings are clearly useful to the research community as the reward hacking problem has been a poorly understood but ubiquitous. Three specific findings about reward hacking (see above) had to my knowledge not been empirically demonstrated in RL environments or only mentioned in passing but not studied explicitly. Before this paper, we therefore did not know if the findings correspond to generally applicable phenomena and through this paper we can gain some confidence. However, some environments and reward functions could be made more realistic.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper targets the very important problem of reward-hacking that occurs when the objectives optimized by intelligent agents are misaligned with respect to the tru objectives of the algorithm designer. The paper presents an empirical study across a range of different settings including a simple driving simulator, covid modeling, and a single atari game. The experiments show evidence of reward hacking as a function of modeling power of the agent and the size of the state-space. The paper concludes with some ideas and initial directions on how to potentially mitigate reward hacking. 
main_review: This paper targets the very important problem of misaligned AI models, under the specific lens of reinforcement learning based optimization. This problem has been widely studied in the AI safety / alignment community but has not received equivalent attention in the machine learning community. The experiments presented in the paper shed light on different settings where the problem of reward-hacking might happen, and under controlled variations of the agent's modeling power, and size of the state-space, shows different forms of reward-hacking in these environments. The paper is well-written and easy to follow, and has a thorough coverage of related works in AI alignment. 

My main concerns with the paper are as follows:

1. None of the experiments in the paper are on sufficiently realistic settings. The paper tries to present empirical evidence of reward-hacking on a number of environments, but doesn't dig deeper into any of them. The specific challenges of each setting are different, and it would be useful to focus on a few settings and perform the evaluations keeping real-world considerations in mind. For example, in the traffic simulator, it would be useful to consider pixel-observations (which can be obtained easily for example in the CARLA simulator) and study the same problem. 

2. The proxy reward functions considered are very "incorrect" in all the settings. In my understanding, the proxy reward functions are trying to show what happens when a reward function different from the true reward function is optimized by the agent. If so, then every attempt must be made to construct reward functions that are *very* close to the true reward function. This is similar to shaped reward functions in typical RL pipelines - it wis important to construct properly shaped functions. An example of this for the traffic example could be a weighted combination of the 4 proxy functions. 

3. Missing experiments on environments where the true reward function cannot be easily constructed. The main challenge of misaligned models is in settings where the underlying reward function is unknown and cannot be queried. As a simple example, consider the task of pick and place of a puck with a robot arm. The end result of "puck in goal location" can be used to check task success but there is no underlying ground-truth dense reward function. In these settings, different papers come up with different shaped rewards for RL - it would be interesting to show experiments on these settings where the underlying reward function is not known, and demonstrate how different shaped rewards (similar to proxy rewards in the paper) fail, and draw inferences from them. This relates back to my first point above about realistic settings. 

4. Question about atari experiments: why not evaluate on a large number of the atari games, as done by most RL papers that show results on atari? 

5. Section 4 is very vague without clear takeaways. The notion of a trusted policy is introduced in section 4, as a way to mitigate the reward hacking problem, but the conclusions from Table 4 are unclear. What exactly is helpful in mitigating reward-hacking? Which metrics should we use for checking the extent of model mis-specification?

6. What is the number of training steps experiment measuring? Is the conclusion of this experiment that an agent that has been optimized for more time-steps will likely succumb to more reward-hacking? If so, then this requires addition experimentation because "not optimizing enough" cannot be a solution to reward-hacking, as that would also lead to lower true rewards as seen in the experiment.

7. The addition of action noise experiments (which the paper calls "resolution") seem to have a trivial takeaway? Adding more noise (i.e. higher variance) causes both the true and proxy rewards to decrease. I request the authors to explain how this non-trivial. 

8. An additional dimension of agent capabilities to check would be the "type of training algorithm" used. This would likely be very important because different variants of algorithms (e.g. on-policy / off-policy, model-based / model-free) would have different failure modes with respect to reward hacking. This might provide more useful empirical takeaways for future work, compared to action resolution, and training steps variations considered. 

In summary, I believe this paper tackles a very important and timely problem through a purely empirical lens. Since, this is a purely empirical paper, it is important to have more comprehensive experiments that are reflective of real-world settings, and have better motivated design choices. The paper in its current form does not provide useful takeaways that are practically significant and would require significant revision. 


summary_of_the_review: I believe this paper tackles a very important and timely problem through a purely empirical lens. Since, this is a purely empirical paper, it is important to have more comprehensive experiments that are reflective of real-world settings, and have better motivated design choices. The paper in its current form does not provide useful takeaways that are practically significant and would require significant revision.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates the phenomenon of reward hacking as a function of agent capabilities. They introduce four diverse RL environments with nine misspecified rewards and demonstrate that more capable agents are better at exploiting the misspecification. They find instances of phase transitions where a small increase in agent capability produces a large change in behavior that sharply decreases the true reward.

To mitigate the reward hacking problem, they propose to set up an anomaly detection task, given a trusted model with moderate performance on the true reward, where the anomaly detector's task is to identify whether policies from a different model are satisfactory for the true reward. They provide several baseline anomaly detectors and show how they perform on different tasks. 
main_review: This paper does a great job at investigating an important problem in AI safety in a concrete and rigorous way. There is an informal assumption in AI safety that the reward hacking problem will tend to get worse as the agent becomes more capable. To my knowledge, this paper is the first to demonstrate this effect quantitatively by varying different agent capabilities (such as model capacity) and showing that the true reward decreases as a result. The paper is clearly written and well-motivated. 

This paper systematically investigates the reward hacking phenomenon across a variety of environments, agent capabilities and forms of misspecification. The environments used in the paper are diverse and complex, which helps to show that reward hacking occurs in a wide variety of domains (though it would be good to see results on more than one Atari environment). The environments were thoughtfully chosen to include tradeoffs between several desiderata that have to be managed by the agent. 

The authors introduce a taxonomy of misspecification (misweighting, ontology and scope) and design a variety of proxy rewards illustrating the different forms of misspecification. The choice of proxy and true rewards for some of the environments seemed a bit arbitrary (e.g. average speed vs mean commute time), so it would be great to see more justification for this. 

I found the section on mitigating reward misspecification to be a somewhat weaker point of the paper. I have the impression that the proposed benchmark requires a large amount of human feedback - it would be great to include the human time cost for the anomaly detection task. The anomaly detection task is mostly illustrated for the traffic environment - it would be useful to include more of the other environments as well. I also wonder whether using the distance from the (suboptimal) trusted policy would result in labeling a policy with superhuman performance on the true reward as aberrant (e.g. AlphaGo's Move 37, which is very unlikely for a human player). 


summary_of_the_review: This paper does a great job at investigating an important problem in AI safety in a concrete and rigorous way, and to my knowledge is the first to do so. I am in favour of accepting this paper.
review_title: null

Meta-review: I thank the authors for their submission and active participation in the discussions. All reviewers are unanimously leaning towards acceptance of this paper. Reviewers in particular liked that the paper is presenting an interesting and systematic study of reward hacking [GVMn] that is useful to the research community [bfGN] and targets an important problem [uYeb] in a rigorous way [16uL]. I thus recommend accepting the paper, but I strongly encourage the authors to further improve their paper based on the reviewer feedback, in particular in regards to improving positioning with respect to related work and a better formalization of their work.
Decision: Accept (Poster)