<figure><img src="image_1.png" /></figure>

HAL
open science

# Reward Hacking: Do You Fully Know What Your AI Agent Can Do?

Deabrota Basu, Julien Teigny

<figure><img src="image_2.png" /></figure>

## To cite this version:

Deabrota Basu, Julien Teigny. Reward Hacking: Do You Fully Know What Your AI Agent Can Do?. 2026. hal-05337548

HAL Id: hal-05337548

https://hal.science/hal-05337548v1

Preprint submitted on 6 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Reward Hacking: Do You Fully Know What Your AI Agent Can Do?

July 17, 2006

Deabrota Basu

Univ. Lille, Inria
CNRS, Centrale Lille
UMR 9189 - CRISTAL, Lille, France
debabrota.basu@inria.fr

Julien Teigny

Univ. Lille, Inria
julien.teigny@inria.fr

**Subtitle:** Can an AI designed to do what it is told, end up doing something undesirable?

**CoastRunner:** In a 2016 video, we see an “AI agent”, developed by OpenAI, controlling a boat in the game “CoastRunner”. The objective for developers was to have an AI agent capable of winning the boat racing game. In CoastRunner, a player has to finish a boat race as rapidly as possible to score more points than the other players. In this case, the AI agent can perform the same actions, and observe the same information, as a human player. To train an AI agent, developers need to give it an objective to follow, often in the form of a numerical value that it seeks to maximize. This objective allows us to estimate how good the AI agent is doing, and to automatically reward/penalize it according to its behavior. A well-designed objective reinforces the “positive” choices, and diminishes the “negative” ones.

In the case of CoastRunner, the AI agent sequentially interacts with an environment to understand its dynamics. Depending on its actions and the state of the environment, the agent receives a reward (positive or negative) indicating how good its actions are. The agent tries to maximize a function of rewards that quantifies the utility of a sequence of actions taken. This approach of training an AI agent is called “Reinforcement Learning”.

While training the AI agent for “CoastRunner”, the developers assumed that a good reward function that encourages to “finish the race quickly” would be the number of obtained points observable on the screen.

You can find here the video with the AI agent playing CoastRunner! (More information is available [here]())

As you can see in the video, the result is not what we expected: the boat does not finish the race...
What happens? Did the training fail? Not really...

Unfortunately for them, the AI has found a place to go round and round on bonuses, which in the end yielded far more points than finishing the race!

## Reward Hacking: What is It?

This phenomenon is known as **"reward hacking"**. In simple terms, "reward hacking" is when an AI agent finds out a creative way of maximizing its rewards, at the expense of the objective **anticipated**
by its developers. Let us reflect on what this can lead to: reward hacking can turn powerful tools into unpredictable entities, capable of making decisions beyond our control.

**Q*bert:** Another outstanding example of reward hacking is the AI "Canonical ES" trained to play the Atari game "Q*bert": "The objective of each level in the game is to change every cube in a pyramid to a target color by making Q*bert, the on-screen character, hop on top of the cube while avoiding obstacles and enemies." (wikipedia). When the character changes the color of a cell, or finish the level, the player gains points.

In some of the training sessions, the AI agent discovered a bug in the game, which was previously unknown to humans. This enables the AI to lock the game between two levels, and accumulate a huge amount of points. So the AI agent was able to “hack” the game, to increase its reward without doing what the developers anticipated.

Video: https://youtu.be/uVHAAiA3F_U?feature=shared

The natural question that comes to our mind is whether this is a problem specific to AI or it has a broader origin?

## A Universal Issue of Utility-driven Decision Making

Though reward hacking may seem ingenious to AI and even fascinating, it generally raises crucial questions about the security and reliability of algorithmic decision making systems designed with a utility function in mind.

Firstly, *we note that reward hacking is not an AI-specific behavior.* Rather, reward hacking can be seen as a manifestation of the *Goodhart's Law* stating "when a measure becomes a target, it ceases to be a good measure". The measure can be a proxy for the target, but if we seek to optimize the measure rather than the target, we can find loopholes that take us far from the desired objective.

We can find manifestation of reward hacking in classical marks-driven examinations. The aim of the examination is to check that the students have understood the subject, but they can get higher marks by: - understanding the lesson :+1: - rote learning :-1: - cheating :-1:

The last two defeats the purpose of learning as scoring marks become the main goal. This phenomenon is often studied as the "Campbell's law".

Another concerning and relevant manifestation of this phenomenon is the rebound or take-back effect in energy consumption. With increasing awareness about climate change and the role of greenhouse gases, we want to replace the fossil fuel-driven energy sources by new 'green' energy sources. But if we look into the last century, we never replaced an old source of energy by a new one. In this case, our desired goal is to find new sources of energy which should be good proxies for "reducing fossil fuel resources", i.e. if we have a greener energy sources than the older ones, we will replace the older ones with the new ones. Instead of achieving the desired objective, we include the new sources with the older ones to consume more and more energy (Wikipedia). In economics terminology, this phenomenon is related to the Jevon's paradox.

There are multiple other examples in modern history like the "four pests management campaign", increasing bin vandalism in Netherlands as a response to incentivizing return of cans, preemptive habitat destruction (draining swamps or cutting down trees that might host valuable species) by landowners who fear losing the lucrative development-friendliness due to the United States Endangered Species Act of 1973, and many others. This is popularly known as the "Cobra effect" or "Perverse Incentive".

These socioeconomic examples demonstratfullye that “reward hacking” is not a problem specific to AI but it is a problem of the traditional decision theoretic approach. This traditional approach advocates a utility-driven design of a decision making system where we do not consider the impacts of actions or options available to the algorithm outside what we modelled. Psychologists and economists have tried to address this phenomenon by coining “wireheading” and “radical uncertainty”, respectively. Still, it remains understudied in other domains like AI and less understood from computer science perspective.
## Why is It Dangerous?

The common belief while developing AI has been that an AI agent will efficiently do what we really want it to do. But reward hacking shows that the AI agent will do **what we tell it to do** rather than **what we really want it to do**.

Stuart Russel echoed this in his quote:

> "The primary concern is not spooky emergent consciousness but simply the ability to make high-quality decisions."

If an AI agent becomes really good at completing its objectives, and is indifferent to anything that we value highly as human, then we have a problem. The AI system will pursue its goals without caring about us. We already see such problems with today's AIs:

* **In the attention economy with social networks:** Recommendation AIs, whose goal is to retain users on the platform, often recommend polarizing and sensationalist contents because that leads to user engagement, without taking into account veracity, or how it affects users' lives.

* **In Large Language Models (LLMs):** The goodness of the answers yielded by LLMs (e.g. ChatGPT, Llama) are evaluated in terms of the feedback of the humans employed to interact with it during the training. Thus, these tools often do not seek to give the right answer, but gives the answer that will yield the best satisfaction to the people who provide the feedback.

## Fundamental Difficulties to Resolve Reward Hacking

We can see at least two reasons behind this reward hacking phenomenon.

The first issue is **Outer Alignment**, which is related to the contextual nature of AI design. The designer specifies an objective that seems very close to the desired one but is not exactly it. There may be several reasons for this:

* The objective is difficult to formulate (as in the CoastRunner). The more complex is an objective, the more difficult it is for the developer to correctly and quantitatively formulate it for the AI agent.

* The developer might not know exactly what the objective is, which can happen in multi-objective problems, like socioeconomic decision-making, and might not have a vision of all the possible ways of fulfilling a certain objective. For example, if an AI agent's objective is to satisfactorily respond to a human (e.g. a chatbot), it is difficult to perfectly specify this objective as all the humans do not have the same values and expectations regarding what constitutes a "satisfying response".

The second issue is **Inner Alignment**, which is related to the myopic nature of AI design. The designer specifies an objective for AI that corresponds *exactly* to what (s)he wants, but the AI has significant side effects on other elements that are not taken into account while training. For example, if we ask an AI robot to bring us a coffee and all the training scenarios include only "not moving" obstacles, it could learn to avoid these obstacles... But If in the real environment some obstacles start moving (as a puppy), it could not be seen as an obstacle and the AI robot might go in a straight line between the coffee machine and us, in order to obtain its reward as quickly as possible. To make it trustworthy, we would have to add in its utility function and training data everything that we expect a civil person to obey.

In reality, **both of the alignment problems can occur together.** For example, in Q*Bert, the bug that the AI agent leverages was not known to humans before, and it appears in the AI agent's behavior as the specified objective was to score high and not to finish the game.

An AI agent does not need to be particularly 'intelligent' to do reward hacking. Due to its exploration-based behavior, its fast execution speed, and often access to huge databases, an AI agent is particularly effective at testing lots of different options, and finding loopholes that might better satisfy its predefined objectives in ways unexpected to its developers.
<figure><img src="image_4.png" /></figure>

<< They told me to bring coffee, but they didn't say do not stomp on any puppies that stand in your way! >>

Figure 1: image page 20 diapo
To avoid these issues of reward hacking by an AI agent, it is necessary not only to specify an objective
that very precisely matches the **desired** one, but also to specify **all the undesired cases** (e.g. what the
AI agent must not do, to achieve the objective) and having a training data or mechanism that covers all
possible situations. The problem is that it is difficult (even impossible?) to define all possible situations
if we want to deploy an AI agent in the real-world.

Specifically, we see that these problems are difficult to solve and sometimes inherent to utility-driven
decision making itself. Still, as researchers and developers of AI, we wonder is it an unavoidable tragedy,
or are there ways to resolve, limit or bypass them?