ICML 2025

MONA: Myopic Optimization with Non-myopic Approval Can Mitigate
Multi-step Reward Hacking
Sebastian Farquhar * 1 Vikrant Varma * 1 David Lindner * 1 David Elson * 1
Caleb Biddulph 1 Ian Goodfellow 1 Rohin Shah a 1
Abstract
Future advanced AI systems may learn sophisti-
cated strategies through reinforcement learning
(RL) that humans cannot understand well enough
to safely evaluate. We propose a training method
which avoids agents learning undesired multi-step
plans that receive high reward (multi-step “reward
hacks”) even if humans are not able to detect that
the behaviour is undesired. The method, My-
opic Optimization with Non-myopic Approval
(MONA), works by combining short-sighted op-
timization with far-sighted reward. We demon-
strate that MONA can prevent multi-step reward
hacking that ordinary RL causes, even without
being able to detect the reward hacking and with-
out any extra information that ordinary RL does
not get access to. We study MONA empirically
in three settings which model different misalign-
ment failure modes including 2-step environments
with LLMs representing delegated oversight and
encoded reasoning and longer-horizon gridworld
environments representing sensor tampering.
1. Introduction
When training an agent with reinforcement learning (RL)
and imperfectly-specified rewards, the agent may engage
in “reward hacking”, where its behaviour is undesired but
achieves a high reward (Clark & Amodei, 2016; Amodei
et al., 2016). For example, large language models (LLMs)
trained with RL from human feedback (Christiano et al.,
2017) can become sycophantic, where an agent says what
users likely want to hear (Sharma et al., 2023).
Currently, most reward hacking is addressed via “patching”:
noticing bad behavior and changing the reward to stop in-
1Google DeepMind ∗Core contributor aSenior author.
Correspondence to: dlindner@google.com, sebfar@google.com.
See arxiv.org/abs/2501.13011 for an extended version.
Proceedings of the 42 st International Confer...