# Bounding Goodhart's Law

[Goodhart's law](https://wiki.lesswrong.com/wiki/Goodhart's_law) seems to suggest that errors in utility or reward function specification are necessarily bad in sense that an optimal policy for the incorrect reward function would result in low return according to the true reward. But how strong is this effect?

Suppose the reward function were only slightly wrong. Can the resulting policy be arbitrarily bad according to the true reward or is it only slightly worse? It turns out the answer is "only slightly worse" (for the appropriate definition of "slightly wrong").

Definitions
-----------

Consider a Markov Decision Process (MDP) < refresh to render LaTeX > where

*   < refresh to render LaTeX > is the set of states,
*   < refresh to render LaTeX > is the set of actions,
*   < refresh to render LaTeX > are the conditional transition probabilities, and
*   < refresh to render LaTeX > is the reward function. (Note: "reward" is standard terminology for MDPs but it's fine to think of this as "utility")

A policy < refresh to render LaTeX > is a mapping from states to distributions over actions with < refresh to render LaTeX > .

Any given policy < refresh to render LaTeX > induces a distribution < refresh to render LaTeX > over states in this MDP. If we are concerned about average reward we can take < refresh to render LaTeX > to be the stationary distribution or, if the environment is episodic, we can take < refresh to render LaTeX > to be the distribution of states visited during the episode. The exact definition is not particularly important for us.

Define the return of policy < refresh to render LaTeX > according to reward function < refresh to render LaTeX > to be

<refresh to render LaTeX>

Goodhart Regret
---------------

Suppose we have an approximate reward signal < refresh to render LaTeX > and we use it to specify a policy < refresh to render LaTeX > . How bad is < refresh to render LaTeX > according to the true reward < refresh to render LaTeX > ?

More specifically, what is the regret of using < refresh to render LaTeX > compared to the optimal policy < refresh to render LaTeX > ? Formally,

<refresh to render LaTeX>

We can expand this as

<refresh to render LaTeX>

Let < refresh to render LaTeX > then < refresh to render LaTeX > if the following conditions are satisfied by < refresh to render LaTeX > and < refresh to render LaTeX > :

1\. < refresh to render LaTeX >

2\. < refresh to render LaTeX >

3\. < refresh to render LaTeX >

Condition 2 says that < refresh to render LaTeX > is not much worse than < refresh to render LaTeX > when measured against < refresh to render LaTeX > . That is what we expect if we designed < refresh to render LaTeX > to be specifically good at < refresh to render LaTeX > , so condition 2 is just a formalization of the notion that < refresh to render LaTeX > is tailored to < refresh to render LaTeX > .

Conditions 1 and 3 compare a fixed policy against two different reward functions. In general for policy < refresh to render LaTeX > and reward functions < refresh to render LaTeX > and < refresh to render LaTeX > ,

<refresh to render LaTeX>

Result: Uniformly Bounded Error
-------------------------------

Assume that we have a reward approximation < refresh to render LaTeX > with uniformly bounded error. That is, < refresh to render LaTeX > . Take < refresh to render LaTeX > .

Then < refresh to render LaTeX > . (Condition 2 has bound 0 in this case).

Result: One-sided Error Bounds
------------------------------

A uniform bound on the error is a stronger condition than we really need. The conditions on < refresh to render LaTeX > can be re-written:

1\. < refresh to render LaTeX > ; < refresh to render LaTeX > does not substantially underestimate the reward in the regions of state-space that are frequently visited by < refresh to render LaTeX > .

3\. < refresh to render LaTeX > ; < refresh to render LaTeX > does not substantially overestimate the reward in the regions of state-space that are frequently visited by < refresh to render LaTeX > .

In other words, it doesn't matter if the reward estimate is too low for states that < refresh to render LaTeX > doesn't want to visit anyways. This tells us that we should prefer biasing our reward approximation to be low in the absence of more information. We do need to be careful about not overestimating where < refresh to render LaTeX > does visit, which is made difficult by the fact that condition 2 pushes < refresh to render LaTeX > to visit states that < refresh to render LaTeX > assigns high reward.

If we don't know what < refresh to render LaTeX > is then it might be difficult to ensure we don't underestimate the reward over < refresh to render LaTeX > . We probably have access to < refresh to render LaTeX > so we might expect to have better reward approximations over < refresh to render LaTeX > and therefore have an easier time satisfying condition 3 than 1.

Non-Optimal < refresh to render LaTeX >
---------------------------------------

The bound on < refresh to render LaTeX > does not actually require < refresh to render LaTeX > to be an optimal policy for < refresh to render LaTeX > . We can take < refresh to render LaTeX > to be any policy and if we can satisfy the 3 bounds then < refresh to render LaTeX > will be not much worse than < refresh to render LaTeX > (and could be better). Using a known policy for < refresh to render LaTeX > likely makes it easier to satisfy condition 1, which is an expectation over the state-action distribution of < refresh to render LaTeX > .

**Example: Human Reward Learning**

Since < refresh to render LaTeX > need not be optimal, we can take < refresh to render LaTeX > to be the policy of a human and try to learn our reward / utility function. _Note: this is a very flawed proposal, its purpose is to demonstrate how one might think about using these bounds in reward learning, not to be a concrete example of safe reward learning._

Suppose that there is some ideal reward function < refresh to render LaTeX > that we'd like to approximate. We don't know < refresh to render LaTeX > in general but imagine we can evaluate it for state-action pairs that are performed by a human. Let < refresh to render LaTeX > be a collection of human demonstrations with labelled reward.

Consider the following algorithm:

1\. Fit < refresh to render LaTeX > to < refresh to render LaTeX > so that it does not underestimate.

2\. Set < refresh to render LaTeX > .

3\. Collect a batch < refresh to render LaTeX > of demonstrations from < refresh to render LaTeX > (no reward labels).

4\. Augment < refresh to render LaTeX > with some additional human demonstrations.

5\. Modify < refresh to render LaTeX > to assign lower reward to all of < refresh to render LaTeX > while not underestimating on < refresh to render LaTeX >

6\. Repeat from 2.

Assuming < refresh to render LaTeX > is sufficiently expressive, this algorithm pushes < refresh to render LaTeX > and < refresh to render LaTeX > towards satisfying all three conditions: < refresh to render LaTeX > does not underestimate on the distribution of human-visited states, < refresh to render LaTeX > is otherwise as low as possible on states < refresh to render LaTeX > visits, and < refresh to render LaTeX > is optimal for < refresh to render LaTeX > . If these are all met, the resulting policy would be no worse than the human at maximizing < refresh to render LaTeX > and possibly much better.

Comments and Takeaways
----------------------

Goodhart's law is more impactful in the context of the sparse reward setting, in which approximation means deciding _what_ to value, not how much to value. Consider a state space that is the real line and suppose that the true reward function is < refresh to render LaTeX > where < refresh to render LaTeX > is the indicator function.

If we estimate

<refresh to render LaTeX>

then Goodhart's law applies and we can expect that a policy optimized for < refresh to render LaTeX > will have high regret on < refresh to render LaTeX > .

On the other hand, if we estimate

<refresh to render LaTeX>

then the regret bounds apply and a policy optimized for < refresh to render LaTeX > will do very well on < refresh to render LaTeX > .

*   Assigning high value to the wrong things is bad.
*   Assigning the wrong value to the right things is not too bad.
*   Throwing a large amount of optimization power against an incorrect utility function is not always bad.