Modification-Considering Value Learning for Reward Hacking Mitigation in RL

Authors: Evgenii Opryshko, Umangi Jain, Igor Gilitschenski
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Reward Hacking, AI Safety, Alignment, Reinforcement Learning, Deep Reinforcement Learning, Reward Tampering, Sensor Tampering, Reinforcement Learning with General Utilities

Abstract:
Reinforcement learning (RL) agents can exploit unintended strategies to achieve high rewards without fulfilling the desired objectives, a phenomenon known as reward hacking. In this work, we examine reward hacking through the lens of General Utility RL, which generalizes RL by considering utility functions over entire trajectories rather than state-based rewards. From this perspective, many instances of reward hacking can be seen as inconsistencies between current and updated utility functions, where the behavior optimized for an updated utility function is poorly evaluated by the original one. Our main contribution is Modification-Considering Value Learning (MC-VL), a novel algorithm designed to address this inconsistency during learning. Starting with a coarse yet value-aligned initial utility function, the MC-VL agent iteratively refines this function based on past observations while considering the potential consequences of updates. This approach enables the agent to anticipate and reject modifications that may lead to undesired behavior. To validate our approach, we implement MC-VL agents based on the Double Deep Q-Network (DDQN) and Twin Delayed Deep Deterministic Policy Gradients (TD3), demonstrating their effectiveness in preventing reward hacking in diverse environments, including those from AI Safety Gridworlds and the MuJoCo gym.

Decision: reject
Reviewer scores: 5, 1, 5, 6

Reviews and discussion:
<thread>
creation date:
1729356105735

thread_type:
official_review

<thread_object>
creator: reviewer_olm1

<content>
summary:
The paper introduces Modification-Considering Value Learning (MC-VL) to address reward hacking in reinforcement learning, where agents exploit unintended strategies to maximize rewards without achieving the desired outcomes.
Using the General Utility RL (GU-RL) framework, MC-VL refines the agent's utility function by evaluating policy updates across entire trajectories, helping to avoid short-term reward maximization that harms long-term objectives.
Implemented with Double Deep Q-Networks (DDQN), the method prevents reward hacking in safety-critical environments, offering a novel solution for achieving long-term alignment in RL.

strengths:
- The paper is well written.
- Reward hacking is one of the main reasons why reinforcement learning is not widely deployed in real-world applications; any contribution that tackles this issue is important for the community.
- Using the utility function to avoid short-term reward maximization that harms long-term objectives is an elegant way to avoid value hacking.

weaknesses:
- The evaluations are solely on toy environments; it would be interesting to test this approach on more complex dynamics and reward functions, such as in the continuous control suite MuJoCo, where reward hacking and undesired behaviors could be more prevalent.

questions:
- Could MC-VL be considered not only as an approach for avoiding reward hacking, but also as a potential method for improving exploration?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729657527198

thread_type:
official_review

<thread_object>
creator: reviewer_blns

<content>
summary:
This paper designs a novel algorithm Modification-Considering Value Learning (MC-VL) to address a phenomenon known as reward hacking in RL. MC-VL allows an agent to optimize its current utility function while considering the future consequences of utility updates, where the concept of current utility optimization is formalized using the General Utility RL framework. To the best of the authors’ knowledge, MC-VL is the first implementation of an agent that optimizes its utility function while considering the potential consequences of modifying it. The authors demonstrate the effectiveness of MC-VL in preventing reward hacking across various grid-world tasks, including benchmarks from the AI Safety Gridworlds suite.

strengths:
1. The motivation is well justified, the method is reasonable
2. The algorithm addressed reward hacking through the lens of General Utility RL is novel and interesting
3. The empirical results show the effectiveness of the proposed algorithm, including benchmarks from the AI Safety Gridworlds suite.

weaknesses:
1. MC-VL is based on the General Utility RL framework and the basic RL method for implementing MC-VL is only DDQN. Why did authors not try to implement MC-VL based on other RL methods, such as PPO or DDPG?
2. As stated in L65-L72, the robot grasping task is a typical case, why did authors not choose this environment to evaluate the performance? In addition, how does the proposed algorithm scale with a larger number of agents and more complex environments?
3. The comparison baseline method is only the basic RL method DDQN, it is better to compare more methods addressing reward hacking, such as the methods authors introduce in Related Work

questions:
I have asked my questions in the Weaknesses section. I have no further questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730780355046

thread_type:
official_review

<thread_object>
creator: reviewer_rzje

<content>
summary:
This paper focuses on preventing reward hacking, in which optimizing a misspecified reward function causes undesired behavior. In particular, the authors focus on reward tampering, in which an agent can modify its reward function to receive higher reward with undesirable side effects. The authors propose a modification to DQN-like algorithms which decides whether to add newly collected transitions to the replay buffer depending on whether it seems like they will improve the current policy. They test the method, MC-DDQN, on some toy environments and find that it seems to prevent agents from reward hacking.

strengths:
The paper focuses on an important problem and takes what appears to be a novel approach.

weaknesses:
* **Mathematical correctness:** while the MC-DDQN algorithm is claimed to be grounded in General Utility RL, there are a number of issues with the presented algorithm that make it difficult to understand what it is actually doing. Consider Algorithm 2, for example. The ⇝ notation is already non-standard (e.g., it does not seem to be any of the uses [here](https://math.stackexchange.com/questions/680786/the-meaning-of-rightsquigarrow-in-math)). On line 2, the algorithm seems to suggest setting $U_{t+1}$ to $U^{\pi\_t}\_{VL\_t}$; however, on the first iteration of the for loop, $\pi\_t$ is not yet defined. On line 3, the algorithm refers to $\tau^\pi\_\rho$, which is not defined anywhere. In line 4, $\pi\_t$ takes as input the previous *transition* $T\_{t-1}$, despite policies being defined as taking states as input on line 133. The first part of line 5 should be $D \gets D \cup \\{ T\_{t - 1} \\}$. On the second part of line 5, $U\_{RL}$ has not been defined, only $F_{RL}$. Thus, *the core algorithm is completely unclear from the given description, given that most of the notation is undefined*.

 * **Limited experiments and evidence of usefulness:** given that the mathematical basis for the MC-DDQN algorithm is unclear, the remaining contribution of the paper lies in the experimental results. However, the experiments only focus on four gridworlds, and furthermore, in most of the experiments the proposed algorithm simply prevents the policy performance from dropping, which could just as easily be accomplished by freezing the policy when switching to the full environment. 

Out of these two weaknesses, the first—that the main algorithm is ill defined—is the primary reason I rate the paper with a low score.

questions:
How can Section 4 be reworked such that the concepts and algorithms presented are well-defined?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730881040670

thread_type:
official_review

<thread_object>
creator: reviewer_vgry

<content>
summary:
This paper explores how modifying the utility function’s design could mitigate the issue of reward hacking in RL. It introduces a new algorithm called MC-VL, aimed at addressing cases where RL agents maximize reward signals by exploiting incomplete or ambiguous reward functions, potentially misaligning with the designer's actual goals.

strengths:
1. This paper proposes a novel approach to address inconsistencies during the learning process.  
2. It discusses reward hacking within the framework of GU-RL, providing in-depth insights and theoretical support for tackling this issue.

weaknesses:
- The paper notes that MC-VL requires predictions about future behaviors, which could entail higher computational costs and may limit its applicability in more complex environments.  
- The proposed method assumes that the agent can learn the correct utility function from observed rewards. In more complex or uncertain environments, the performance of the algorithm could suffer if the reward model fails to generalize accurately to new trajectories.  
- The experiments are primarily conducted in relatively simple grid-world environments, leaving the algorithm’s effectiveness in more complex, high-dimensional environments still unverified.

questions:
1. Were all experiments conducted by training on the "Safe version" before transitioning to the "Full version"? If so, how would the proposed algorithm generalize to typical environments?  
2. The authors mention that hyperparameters significantly influence algorithm performance, listing many in Table 1 of Appendix E, such as "Inconsistency check training steps l," "Inconsistency check rollout steps h," "Number of inconsistency check rollouts k," and "Predicted reward difference threshold." However, it’s unclear how these parameters specifically impact the proposed algorithm or how they should be selected.  
3. Reward hacking is common across various environments, as mentioned in the robotic context at the paper's beginning. Could this algorithm potentially be applied to continuous action versions, like TD3 for DDQN? If extended to continuous environments, would the accuracy of the utility function become more critical to the algorithm's performance?  
4. Could the authors validate the algorithm's effectiveness in more complex environments, such as those with high-dimensional inputs (e.g., continuous or image-based inputs)?
5. check the definnation of $\lambda(\tau)$
</content>
</thread_object>

</thread>
<thread>
creation date:
1731800949882

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response

comment:
We thank the reviewers for their thoughtful feedback! We are encouraged that reviewers find our work novel (**VGrY**, **rZJE**, **bLnS**) and recognize that it tackles an important problem  (**rZJE**, **bLnS**, **oLm1**). We are grateful for your praise of it as interesting (**bLnS**), insightful (**VGrY**), and elegant (**oLm1**). As we are working on improving the paper and running additional experiments, we would like to use this initial response to addressing the reviewer’s points and ask for some clarifications or additional feedback. The responses provided here and the additional experiments will also be incorporated in the paper. We understand the reviewers' concern about the fact that our algorithm is evaluated only in grid-world environments. Reviewers also asked if our algorithm can be applied to continuous action spaces (**VGrY**, **oLm1**) and to implement our MC-VL algorithm based on other RL methods, such as TD3 or DDPG (**VGrY**, **bLnS**).

We decided to make our best effort to address the reviewers’ concerns within our computational constraints. We are currently implementing a version of our algorithm based on **TD3**, which will be applicable to continuous action spaces. We are planning to evaluate it on the environment based on *MuJoCo Reacher-v5*. The environment contains a robotic arm, and the goal is to reach a randomly selected goal position. The reward is a sum of negative distance to the goal and negative torques applied to the joints. To study reward hacking, we modify the environment by introducing an invisible button which provides a maximum possible reward if the arm holds it for several consecutive frames. We hope to address several concerns with this experiment:
- Show that MC-VL can be implemented based on a different RL method (TD3).
- Verify that our algorithm can work with continuous action spaces.
- Provide evaluation in a more complex environment.

We want to note that our work, to the best of our knowledge, is first to show empirical evidence for the possibility of mitigation of reward hacking without constraining the policy to be close to a known safe policy or requiring a prior over manually specified utility functions, even in the grid-world setting. Our primary objective is not providing a practical and ready-to-use solution to reward hacking in all environments, but showing that such a solution is possible in cases when reward hacking is caused by an inconsistent update to the utility function. Our experiments were designed to illustrate that such inconsistency may arise in several environments discussed in the literature, and avoiding it prevents the agent from reward hacking.

However, we agree that more extensive empirical evaluation is always a good thing. We would love to hear the reviewer’s feedback on this additional experiment and hope that it will address any concerns along with the clarifications in the replies below.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732319648620

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response #2 [1/2]

comment:
We are excited to share that we were able to complete the additional experiment requested in the review process. The new agent based on TD3, which we call MC-TD3, successfully trained in the modified MuJoCo Reacher-v5 environment while avoiding reward hacking. This environment involves a reaching task with a robotic arm following the suggestions by reviewers bLnS and oLm1. We sincerely thank the reviewers for their valuable feedback, which motivated this experiment and has significantly strengthened our empirical evaluation. This experiment demonstrates that the MC-VL algorithm is both general enough to be integrated with other RL algorithms and robust enough to perform effectively in complex environments with continuous action and observation spaces. Additionally, it shows that our approach works even when the initial utility function is learned from transitions collected during random exploration, as long as those transitions do not include reward hacking behaviors.

To reflect this new contribution, we have updated the paper. To simplify the reviewers' work, we provide all information on the new experiment below, and we also attached a LaTeXdiff version of the paper showing the differences to the initially submitted draft at the end of the updated manuscript. The material related to the newly added experiment contains the following key additions and revisions:

- In Section 5.1 (Environments) we added a paragraph describing the Reacher environment:

> **Reacher** This environment is based on the Reacher-v5 environment from the MuJoCo subset of the Gymnasium benchmark (Towers et al., 2024). It involves a robotic arm tasked with reaching a randomly placed target, starting from random joint rotations and velocities. We modified the environment by adding an invisible button that provides a one-time reward of +50 when held for 15 consecutive steps, simulating reward tampering. The performance metric does not include this reward. Further details are provided in Appendix G.
- We added Figure 3.e. which demonstrates the performance of MC-TD3 in the Reacher environment. The plot shows that the performance of MC-TD3 steadily improves throughout the training. The returns of the TD3 baseline grow, but the performance metric drops, indicating reward hacking.
- We added Appendix E.3, which contains the hyperparameters of MC-TD3.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper aims to create a method to reduce the prevalence of reward hacking, which makes designing easy-to-optimize rewards significantly challenging. The paper's proposed method is to leverage a utility function (a reward for a whole trajectory) and a corresponding value function, then augment the action space with an additional action to modify the utility function. This modification allows the agent to learn if changing the utility functions leads to better learning. The hope of this method is that agent can learn when learning from certain transitions lead to reward hacking. 

As noted by the reviewers, the paper lacks mathematical precision, which makes the method unclear and prevents a sufficient understanding of the paper's contribution. The reviewers also note concern about the limit of using four gridworld environments in the experiments. I do not share this same concern by default, but the paper's motivation is to alleviate reward hacking in more complex environments. As such, the paper should make it clear what is learned from these experiments in pursuit of this direction.