Goodhart's Law in Reinforcement Learning
Authors: Jacek Karwowski
Oliver Hayman
Xingjian Bai
Klaus Kiendlhofer
Charlie Griffin
Joar Max Viktor Skalse
Venue: ICLR 2024 poster (2024)
Keywords: reinforcement learning
goodhart's law
misspecification
reward learning
Primary area: reinforcement learning
TL;DR: We study Goodhart's law in RL empirically, provide a theoretical explanation for why it occurs, and use these theoretical insights to derive two methods for avoiding Goodharting.

Abstract: Implementing a reward function that perfectly captures a complex task in the real world is impractical. As a result, it is often appropriate to think of the reward function as a *proxy* for the true objective rather than as its definition. We study this phenomenon through the lens of *Goodhart’s law*, which predicts that increasing optimisation of an imperfect proxy beyond some critical point decreases performance on the true objective. First, we propose a way to *quantify* the magnitude of this effect and *show empirically* that optimising an imperfect proxy reward often leads to the behaviour predicted by Goodhart’s law for a wide range of environments and reward functions. We then provide a *geometric explanation* for why Goodhart's law occurs in Markov decision processes. We use these theoretical insights to propose an *optimal early stopping method* that provably avoids the aforementioned pitfall and derive theoretical *regret bounds* for this method. Moreover, we derive a training method that maximises worst-case reward, for the setting where there is uncertainty about the true reward function. Finally, we evaluate our early stopping method experimentally. Our results support a foundation for a theoretically-principled study of reinforcement learning under reward misspecification.

Published as a conference paper at ICLR 2024
GOODHART'S S LAW IN REINFORCEMENT LEARNING
Jacek Karwowski Oliver Hayman'
Xingjian Bai"
Klaus Kiendlhofer
Charlie Griflin'
Joar Skalse 3 Futurre of Humanity Institute
University of Oxford
"Ideependent
ABSTRACT
Implementing a reward function that perfectly captures a complex task in the real world is impractical. As result, it is often appropriate to think of the reward function as a proxy for the true objective rather than as its definition. We study this phenomenon through the lens of Goodhart's law, which predicts that increasing optimisation of an imperfect proxy beyond some critical point decreases performance on the true objective. First, we propose a way to quantify imperfect
the magnitude of this effect and show empirically that optimising proxy reward often leads to the behaviour predicted by Goodhart's law for a wide range of environments and reward functions. We then provide geometric explanation for why Goodhart's 's law occurs in Markov decision processes. We these theoretical insights propose optimal early stopping method that provably avoids the aforementioned pitfall and derive theoretical regret bounds for this method. Moreover, we derive training method that maximises worst-case reward, for the setting where there is uncertainty about the true reward function. Finally, we evaluate our early stopping method experimentally. Our results support foundation for pearetccallyprinciphhee study of reinforcement learning under reward misspecification.
INTRODUCTION
To solve problem using Reinforcement Learning (RL), is necessary first to formalise that problem using a reward function (Sutton & Barto 2018). However, due to the complexity of many real- world tasks, exceedingly difficult to directly specify a reward function that fully captures the task in the intended way. However, misspecified reward functions will often lead to undesirable behaviour (Paulus et al. 2018 Ibarz et al. 2018) Knox al. 2023 Pan al. 202T). This makes designing good reward functions major obstacle to using RL in practice, especially for safety-critical applications.
An increasingly popular solution is to learn reward functions from mechanisms such as human or automated feedback (e.g. Christiano al., 2017 Ng & Russell 2000). However, this approach comes with its own set of challenges: the right data can be difficult to collect (e.g. Paulus et al. 2018), and it is often challenging interpret it correctly (e.g. Mindermann & Armstrong 2018 Skalse & Abate 2023). Moreover, optimising a policy against a learned reward model effectively constitutes distributional shift (Gao et al., 2023); i.e., even if a reward function is accurate under the training distribution, it may fail to induce desirable behaviour from the RL agent.
Therefore in practice it often more appropriate think of the reward function as proxy for the true objective rather than being the true objective. This means that we need a more principled understanding of what happens when proxy reward is maximised, in order know how we should expect RL systems to behave, and in order to design better algorithms. For example, we aim to answer questions such as: When is proxy safe to maximise without constraint? What is the best way to maximise misspecified proxy? What types of failure modes should we expect from misspecified proxy? Currently, the field of RI. largely lacks rigorous answers to these types of questions.
In this paper, we study the effects of proxy misspecification through the lens of Goodhart's 's law, an informal principle often stated as "any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes" (Goodhart) 1984), more simply: "when measure becomes a target, it ceases to be a good measure". For example, a students' knowledge of a subject
a *Correspondence jacek. ka rwowski Ecs. ackukk

---

Published as a conference paper at ICLR 2024
may be correlated with their ability to pass exams that subject by default. However, students who have sufficiently strong incentives to do well in exams may also include strategies such as cheating for increasing their test score without increasing their understanding. In the context of RL, we can think of a misspecified proxy reward as a measure correlated, but not robustly aligned, with the true objective across some distribution of policies. Goodhart's law then says, informally, that we should expect optimisation of the proxy to initially lead to improvements on the true objectíve, up until point where the correlation between the proxy reward and the true objective breaks down, after which further optimisation should lead to worse performance according to the true objective (Figure I.
In this paper, we present several novel contributions. First, we show that "Goodharting" occurs with high probability for wide range of environments and pairs of true and proxy reward functions. Next, we provide a mechanistic explanation of why Goodhart's 's law emerges in RL. We use this to derive two new policy optimisation methods and show that they provably avoid Goodharting. Finally, evaluate these methods empirically. We thus contribute towards building a better understanding of the dynamics of optimising towards imperfect proxy reward functions, and show that these insights may be used to design new algorithms.
True reward
Proxy reward
Optimisation pressure
Figure 1: A cartoon of Goodharting.
1.1 RELATED WORK
Goodhart's law was first introduced by Goodhart (984)), and has later been elaborated upon by works such as Manheim & Garrabrant (2119) Goodhart's s law has also previously been studied in the context of machine learning. In particular, Hennessy & Goodhart (2023) investigate Goodhart's law analytically in the context where a machine learning model is used to evaluate an agent's 's actions unlike them, we specifically consider the RL setting. [Ashton] (202T) shows by example that RL systems be susceptible Goodharting in certain situations. In contrast, we show that Goodhart's law is a robust phenomenon across a wide range of environments, explain why it occurs in RL, and use it to devise new solution methods.
In the context of RL, Goodhart's law is closely related to reward gaming. Specifically, reward gaming means an agent finding an unintended way increase its reward, then Goodharting is an instance of reward gaming where optimisation of the proxy initially leads to desirable behaviour, followed by a decrease after some threshold. Krakovna al. (2020) list illustrative examples of reward hacking, while Pan et al. (2021) manually construct proxy rewards for several environments and then demonstrate that most of them lead to reward hacking. Zhuang & Madfield-Menell. (2020) consider proxy rewards that depend on a strict subset of the features which are relevant to the true reward and then show that optimising such proxy in some cases may be arbitrarily bad, given certain assumptions. Skalse et al. (2022) introduce theoretical framework for analysing reward hacking. They then demonstrate that, in any environment and for any true reward function, it is impossible to create a non-trivial proxy reward that is guaranteed to be unhackable. Also relevant, [Everitt et al. (217)) study the related problem of reward corruption, Song et al. (2019) investigate overfitting in model-free RL due to faulty implications from correlations in the environment, and Pang al. (2022) examine reward gaming in language models. Unlike these works, we analyse reward hacking through the lens of Goodhart's law and show that this perspective provides novel insights.
(elevant,EEveiittt
Gao et al. 2023) consider the setting where a large language model is optimised against a reward model that has been trained on "gold standard" reward function, and investigate how the performance of the language model according to the gold standard reward scales in the size of the language model, the amount of training data, and the size of the reward model. They find that the performance of the policy follows Goodhart curve, where the slope gets less prominent for larger reward models and larger amounts of training data. Unlike them, we do not only focus on language, but rather, aim to establish to what extent Goodhart dynamics occur for a wide range of RI. environments. Moreover, we also aim to explain Goodhart's law, and use it as a starting point for developing new algorithms.
PRELIMINARIES
A Markov Decision Process (MDP) is a tuple (S, A, μ,, R. where is a set of states, A is a set of actions, SSA ASS) is transition function describing the outcomes of taking actions
2

---

Published as a conference paper at ICLR 2024
at certain states, Pز A(S) is the distribution of the initial state, R E R|SxAl gives the reward for taking actions at each state, and y E [0, 1] is a time discount factor. In the remainder of the paper, we consider A and S to be finite. Our work will mostly be concerned with rewardless MDPs, denoted by MDPAR (5, A, J4, 7), where the true reward R is unknown. A trajectory is a sequence (SD, do; $1, (1,-..-) such that 0; A, 8, S for all i. We denote the space of all trajectories by E. A policy is a function : S - A(A). We say that the policy π is deterministic if for each state there is some a A such that (s) 8g- We denote the space of all policies by II and the set of all deterministic policies by Hg. Each policy и on an MDPRR induces a probability distribution over trajectories ()] drawing trajectory (Su. du, $1, 01,...) from a policy æ means that So drawn from μr, each a; is drawn from i (8;), and 8;+1 is drawn from r(s;, ai) for each i.
For given MDP, the return of trajectory& is defined to be G(E) == > 7 R(st, st. ae) and the expected return of policy æ to be [(x) Eg-- [G{E)] An optimal policy is one that maximizes expected return; the set of optimal policies is denoted by π.. There might be more than one optimal policy, but the set æ, always contains at least one deterministic policy (Sutton & Barto 2018). We define the value- calue-function V S R such that [<[s] Ee-- [G[E]]su Q-function QT Sx A - R to be (" a) = E [G(E)] = 8, dg = a]. V*, ," are the value and Q functions under an optimal policy. Given an MDPIR each reward R defines separate Vit Qh, and (T). In the remainder of this section, we fix a particular MDPIR (S, T,
s/. and define the
2.1 THE CONVEX PERSPECTIVE
In this section, we introduce some theoretical constructs that needed to express many of our results. We first need to familiarise ourselves with the occupancy measures of policies:
Definition 1 (State-action occupancy measure). We define function 1) II - RISxAl, assigning, to each ग E II, vector of occupancy measure describing the discounted frequency that a policy takes each action in each state. Formally,
30 Σ PP((s 8,a aal E m) 1=0
17 (s,
We can recover π from 1y" on all visited states by x(s. = (1-
(s, (ss, If
Exre 17" (s, a') 0, we can set (T,, a) arbitrarily. This means that we often can decide work with the set of possible occupancy measures, rather than the set of all policies. Moreover:
Proposition 1.
The ser Ω {17 : π E III is the convex hull of the fintt set of points corresponding the deterministic policies {17" ग Mo]. It lies in an affine subspace of dimension |S|([A) 1).
Note that JR(T)= 17" = R, meaning that each reward R induces a linear function on the convex polytope Q which reduces finding the optimal policy to solving linear programming problem in Q. Many of our results crucially rely on this insight. We denote the orthogonal projection map from || to s?an(S?) by My. which means JR(TT) = M, R. The proof of Proposition | and all other proofs, are given the appendix.
2.2 QUANTIFYING GOODHART'S ' LAW
Our work is concerned with quantifying the Goodhart effect. To do this, need way to quantify the distance between rewards. We do this using the projected angle between reward vectors.
Definition 2 (Projected angle). Given two reward functions Ro, R1, we define arg (Ro, R1) to be the angle between M, Ro and M, R1.
The projected angle distance is an instance of STARC metric, introduced by (Skalse al. (2023a) Such metries enjoy strong theoretical guarantees and satisfy many desirable desiderata for reward function metrics. For details, see Skalse et al. (2023a). In particular:
Proposition 2. We have arg (Ro, R1) = 0 and only if Ro, R4 induce the same ordering of policies, or, in other words, Ino (T) & Jn. (T')
Jn, (T) & Jn, (5n) all policies π, T².
In their terminology, the canonicalisation function is Mr, and measuring the angle between the resulting vectors (bilipschitz) equivalent normalising and measuring the distance with the l² z'-nomm
3

---

Published as a conference paper at ICLR 2024
Falane 4
Corss3 manry
Opaniarsi ponsus
Figure 2: Depiction of Goodharting in RandomMDP. Compare to Figure here we only show the Irue reward obtained by policy trained each proxy. Darker color means more distant proxy.
We also need way to quantify optimisation pressure. We do this using two different training methods. Both are parametrised by regularisation strength a (0, o0): Given reward R, they output regularised policy Ta For ease of discussion and plotting, is often more appropriate to refer to the (bounded) inverse of the regularisation strength: the optimisation pressure Xa = eee e As the optimisation pressure increases, J (Ta) also increases.
Definition 3 (Maximal Causal Entropy). We denote by To the optimal policy according to the regularised objective R(s a) == R(s,a) + o (x())) where H(x(s)) is the Shannon entropy.
Definition 4 (Boltzmann Rationality). The Boltzmann rational policy is defined as P (s) = a. oce (*.a)) where Q" is the optimal Q-function.
We perform experiments to verify that our key results hold for either way of quantifying optimisation pressure. In both cases, the optimisation algorithm is Value Iteration (see e.g. [Stttnn & Barto 2018).
Finally, we need a way to quantify the magnitude of the Goodhart effect. Assume that we have true reward Rg and a proxy reward Ri, that R₁ is optimised according to one of the methods in Definition 3 and that is the policy that is obtained at optimisation pressure A. Suppose also that Ro, R₁ are normalised, so that min, J (xr)
and max, J(r) 1 for both Rg and R1.
Definition 5 (Normalised drop height). We define the normalised drop height (NDH) as ma JR. (TX) JR, (TT), i.c. as the loss of true reward throughout the optimisation process.
For an illustration of the above definition, see the grey dashed line in Figure 1 NDH is non-zero if and only if, over increasing optimisation pressure, the proxy and true rewards initially correlated, and then become anti-correlated (we will see later that as long as the angle distance less than n/2, their returns will almost always be initially correlated). In the Appendix C. introduce more complex measures which quantify Goodhart's law differently. Since our experiments indicate that they are all are strongly correlated, we decided focus on NDH as the simplest one.
We observe that
GOODHARTING IS PERVASIVE IN REINFORCEMENT LEARNING
In this section, we empirically demonstrate that Goodharting occurs pervasively across varied environments by showing that, for given true reward Ro and a proxy reward R1, beyond a certain optimisation threshold, the performance on Rg decreases when the agent is trained towards Rt We test this claim different kinds of environments (varying number of states, actions, terminal states and 7), reward functions (varying rewards' types and sparsity) and optimisation pressure definitions.
3.1 ENVIRONMENT AND REWARD TYPES
Gridworld a deterministic, gasd environment, with the state space size n x n for parameter N+, with fixed set of five actions: J, and WAIT. The upper-left and lower-right corners are designated as terminal states. Attempting an illegal action a in state s does not change the state. Cliff' (Sutton & Barto 2018 Example 6.6) is Gridworld variant where agent aims reach the lower right terminal state, avoiding the cliff formed by the bottom row't 's cells. Any cliff-adie adjacent move has slipping probability p of falling into the cliff.
4

---

Published as a conference paper at ICLR 2024
RandomMDP is an environment in which, for fixed number of states |S], actions |A], and terminal states k, the transition matrix T is sampled uniformly across all stochastic matrices of shape |SxAA] x |S|, satisfying the property of having exactly l terminal states.
TreeMDP is an environment corresponding to nodes of a rooted tree with branching factor b = |A] and depth d. The root is the initial state and each action from non-leaf node results in states corresponding to the node's children. Half of the leaf nodes are terminal states and the other half loop back to the root, which makes it isomorphic an infinite self-similar tree.
In our experiments, we only use reward functions that depend on the next state R(x, a) = R(s). In Terminal, the rewards are sampled iid from (10,) 1) for terminal states and from U(-1,0) for non-terminal states. In Cliff, where the rewards are sampled iid from ((-5.0) 0) for cliff states, from U(-1,0) 0) for non-terminal states, and from (O, 1) for the goal state. In Path, where we first sample random walk P moving only - and between the upper-left and lower-right terminal state, and then the rewards are constantly 0 on the path P, sampled from U--1, 0) for the non-terminal states, and from (0, 1) for the terminal state.
3.2 ESTIMATING THE PREVALENCE OF GOODHARTING
To get an estimate of how prevalent Goodharting is, we run an experiment where we vary all hyperparameters of MDPs in grid search manner. Specifically, we sample: Gridworld for grid lengths n E {2. 32...... 14) and either Terminal or Path rewards; Cliff with tripping probability p 0.5 and grid lengths E (2.3..... 9} and Cliff rewards; RandomMDP with number of states |S] E [2, 4, 8, 16, 512), number of actions |A| {2, 3, 3,4, a fixed number of terminal states 2, and Terminal rewards; TreMMDP with branching factor 2 and depth d E [2, ..,.., IJ, for two different kinds of trees: (1) where the first half of the leaves are terminal states, and (2) where every second leaf is terminal state, both using Terminal rewards.
For each of those, we also vary temporal discount factor E {0.5, 0.7, 0.9, 0.99). sparsity factor e f0.1, 0.3, 0.5, 0.7, 0.9), optimisation pressure log(z)) for values of evenly spaced on [0.01, 0.75] and 20 values evenly spaced on [0.8, 0.99].
After sampling an MDPAR, we randomly sample a pair of reward functions Rg and Rı from a chosen distribution. These are then sparsified (random fraction of values are zeroed) and linearly interpolated, creating a sequence of proxy reward functions R. = (1 - Ro + tR for t E [0. 1]. Note that for every environment, reward sampling scheme and fixed choice of parameters considered in Section 3.1 the sample space of rewards is convex. In high dimensions, two random vectors are approximately orthogonal with high probability, so the sequence RG spans range of distances.
Each run consists of 10 proxy rewards; we use threshold = 0.001 for value iteration. We get a total of 30400 data points. An initial increase, followed by a decline in value with increasing optimisation pressure, indicates Goodharting behaviour. Overall, we find that a Goodhart drop occurs (meaning that the NDH > 0) for 19.3 % of all experiments sampled over the parameter ranges given above. This suggests that Goodharting is a common (albeit not universal) phenomenon in RI. and occurs in various environments and for various reward functions. We present additional empirical insights, such as that training myopic agents makes Goodharting less severe, in Appendix
For illustrative purposes, we present single run of the above experiment in Figure 2 We can see that, as the proxy R3 is maximised, the true reward Ro will typically either increase monotonically or increase and then decrease. This is in accordance with the predictions of Goodhart's 's law.
4 EXPLAINING GOODHART'S S LAW IN REINFORCEMEN LEARNING
In this section, we provide an intuitive, mechanistic account of why Goodharting happens in MDPs, that explains some of the results in Section 3 An extended discussion is also given in Appendix A
First, recall that Jn(m)= T7* R, where 17 is the occupancy measure of л. Recall also that Ω is convex polytope. Therefore, the problem of finding an optimal policy can be viewed as maximising a linear function R within a convex polytope Ω which is a linear programming problem.
Steepest ascent is the process that changes 7j in the direction that most rapidly increases 7 R (for formal definition, see Chang & Murty (989) or Denel et a (1981)). The path of steepest ascent
5

---

Published as a conference paper at ICLR 2024
405
(eitance 0.53c
paint Prodktte 200g:ing point
Thu 1.0.0e
3.52
0.35
2.5
00
0.4
0.5
(b) Training runs for each of the reward functions embedded in the state- action occupancy measure space. Even though the full frequency space is |S|| A| = 4-dimensional, the image of the policy space occupics only |S| (|A) 1) 2- 2-dimensional linear subspace. Goodharting occurs when the cosine distance between rewards passes the critical threshold and the policy snaps different endpoint.
Optinmudaon procame
(a) Goodharting behavior in M222 over three reward functions. Our method is able to predict the optimal stopping time (in blue).
Figure 3: Visualisation of Goodhart's 's law in case of M2,2.
forms a piecewise linear curve whose linear segments lic on the boundary of (except the first segment, which may lie in the interior). Due to its similarity to gradient-based optimisation methods, we expect most policy optimisation algorithms to follow a path that roughly approximates steepest ascent. Steepest ascent also has the following property:
Proposition (Concavity of Steepest Ascent). Iff: := [ -1-+1-m]]] for π7 produced by steepest ascent
1-11-%
on reward vector R, then E. is decreasing.
We can now explain Goodhart's ' law in MDPs. Assume we have a true reward Rg and a proxy reward R1, that we optimise Ri through steepest ascent, and that this produces sequence of occupancy measures {m... Recall that this sequence forms piecewise linear path along the boundary of convex polytope S2 and that JR, and JR, correspond to linear functions on 52 (whose directions of steepest ascent are given by M, Ru and M1) First, if the angle between M, and M₁ is less than x/2, and the initial policy To lies in the interior of f2 then it is guaranteed that 17- Rg will increase along the first segment of {m.}. However, when (лx) reaches the boundary of Ω. steepest ascent continues in the direction of the projection of M, R2 onto this boundary. this projection is far enough from Ro optimising in the direction of M, R, would lead to a decrease in JR. (c.f. Figure 3b). This corresponds to Goodharting.
Ro may continue to increase, even after another boundary region has been hit. However, each time (nd) hits a new boundary, it changes direction, and there is a risk that n Ry will decrease. In general, this is more likely if the angle between that boundary and {nd} is close to T/2, and less likely if the angle between M, Ro and M, R, is small. This explains why Goodharting is less likely when the angle between M, Ro and M, Rú is small. Next, note that Proposition 3 immplie that the angle between {n,] and the boundary of Ω will increase over time along {m.}. This explains why Goodharting becomes more likely when more optimisation pressure is applied.
Let us consider an example to make our explanation of Goodhart's 's law more intuitive. Let M2. be an MDP with states and actions, and let Rg R1, R2g be three reward functions in M2 The full specifications for M2.2 and Ro. R1, Rs are given in Appendix E We will refer to Ru as the true reward. The angle between Rg and R1 is larger than the angle between Rg and Rg. Using Maximal Causal Entropy, we can train a policy over each of the reward functions, using varying degrees of optimisation pressure, and record the performance of the resulting policy with respect to the true reward. Zero optimisation pressure results in the uniformly random policy, and maximal optimisation pressure results in the optimal policy for the given proxy (see Figure Ba). As we can see, we get Goodharting for R2 increasing R2 initially increases Rg, but there is critical point after which further optimisation leads to worse performance under Ro.
To understand what is happening, we embed the policies produced during each training run in Ω, together with the projections of Ro. R, R2 (see Figure(3b, We can now see that Goodharting must occur precisely when the angle between the true reward and the proxy reward passes the critical

---

Published as a conference paper at ICLR 2024
Angle addene..0e7
procedure EARLYSTOPPINGCS,
traiing (an
0
F < M,R
U
To < 17"
to argmax 8
while (t, * 0) and R E & sin(0)] do
À max(A m XF, 2)
7++ n Af:
8- - argmax TeT L) R
i - i+1
(a) y-embedded training run for steepest ascent. The training curve is split into two linear segments: the first is parallel the proxy reward, while the second is parallel the proxy reward projected onto some boundary plane P. Goodharting only occurs along P. (Compare to the MCE approximation of Steepest Ascent Figure 3b
end while return (174)-1 end procedure
(b) Farly stopping pseudocode for Steepest Ascent. Given the correct 6, the algorithm would stop at the point where the training run hits the boundary of the convex hull. The cone of tangents, T(r) defined in Denel et al. (1981).
Figure 4: Early stopping algorithm and its behaviour.
threshold, such that the training run deflects upon stumbling on the border of Q, and the optimal deterministic policy changes from the lower-left to the upper-left corner. This is the underlying mechanism that produces Goodhart behaviour in reinforcement learning!
We thus have an explanation for why the Goodhart curves are so common. Moreover, this insight also explains why Goodharting does not always happen and why smaller distance between the true reward and the proxy reward is associated with less Goodharting. We can also see that Goodharting will be more likely when the angle between {n:} and the boundary of Ω is close to T/2 - this is why Proposition iimplies that Goodharting becomes more likely with more optimisation pressure.
PREVENTING GOODHARTING BEHAVIOUR
We have seen that when proxy reward RS optimised, is common for the true reward Rg first increase, and then decrease. If we can stop the optimisation process before Ro starts decrease, then we can avoid Goodharting. Our next result shows that we can provably prevent Goodharting, given that we have bound 0 the distance between R1 and Ru:
Let Rı be any reward function, let O E [0, x1 be any angle, and let n be any two policies. Then there exists a reward function Ro with arg (Rg, R,) and JRu (T JRo (T)) iff
Theorem 1.
[R,(TT) (TB) (TA) 1|7 - T
sin(f)]|M,RIIl
Corollary (Optimal Stopping). Let R₁ be a proxy reward, and let {m²} be sequence of policies produced by optimisation algorithm. Suppose the optimisation algorithm is concave with respect the policy, in the sense that 3n, -nn,))_nnnn (m.) decreasing. Then, stopping at minimal with
In 111
JR. (T+1) JR. (T.) 11
sin(f)]l
1751|
gives the policy T; E {T²) that maximizes min (TT), where FQ is the set of rewards given by [Ro arg RR, R1) & 0. M|,, 0}.
Let us unpack the statement of this result. we have proxy reward R₁, and we believe that the angle between R; and the true reward Ro is at most 6, then F is the set of all possible true reward functions with a given magnitude m. Note that no generality is lost by assuming that Rg has magnitude m, since rescale any reward function without affecting its policy order. Now, if
7

---

Published as a conference paper at ICLR 2024
optimise R1, and want to provably avoid Goodharting, then must stop the optimisation process a point where there is no Goodharting for any reward function in F Theorem provides us with such stopping point. Moreover, if the policy optimisation process concave, then Corollary tells us that this stopping point, in a certain sense, is worst-case optimal. By Proposition 3 we should expect most optimisation algorithms be approximately concave.
Theorem I derives an optimal stopping point among a single optimisation curve. Our next result finds the optimum among all policies through maximising a regularised objective function.
Proposition 4. Given proxy reward RA, let FR be the set of possible true rewards R such that arg (R, R1) & and is normalized so that ||M = || ||, Rill. Then, a policy a maximises min REFR, Jn(œ)) if and only it maximises JR, (T) K||nn sin (arg (n²², RR)), where tan 0) M- Moreover, each local maximum of this objective is a global maximum when restricted to Ω giving that this function can be practically optimised for.
The above objective can be rewritten as || x|lbaall where 77|1 77 are the components of 1y" parallel and perpendicular to M,R1.
Stopping early clearly loses proxy reward, but is important to note that it may also lose true reward. Since the algorithm is pessimistic, the optimisation stops before any reward in F decreases. If we continued ascent past this stopping point, exactly one reward function in F would decrease (almost surely), but most other reward function would increase. If the true reward function is in this latter set, then early stopping loses some true reward. Our next result gives an upper bound on this quantity:
Proposition 5. Let Ro be true reward and R₁ proxy reward such that Roll [Rill = and arg (Ro, R1)= = 0, and assume that the steepest ascent algorithm applied to Rí produces a sequence of policies no, Ti,... .y. If T, is optimal for Ro, we have that
In. (Tn) & diameter(S)) - -1117 17 cos(0).
would be interesting to develop policy optimisation algorithms that start with an initial estimate R1 of the true reward Rg and then refine R₁ over time as the ambiguity in R₁ becomes relevant. Band then be used to check when more information about the true reward needed. While we mostly leave this for future work, we carry out some initial exploration in Appendix F
5.1 EXPERIMENTAL EVALUATION OF EARLY STOPPING
We evaluate the early stopping algorithm experimentally. One problem is that Algorithm 4b involves the projection onto Ω, which is infeasible to compute exactly due to the number of deterministic policies being exponential in |.S). Instead, we observe that using MCE and BR approximates the steepest ascent trajectory.
Using the exact setup described in Section 3.2 we verify that the early stopping procedure prevents Goodharting in all cases, that is, employing the criterion from Corollary always results in NDH = 0. Because early stopping is pessimistic, some reward will usually be lost. We are interested in whether the choice of (1) operationalisation of optimisation pressure, (2) the type of environment or (3) the angle distance 0 impacts the performance of early stopping. A priori, we expected the answer the first question to be negative and the answer to the third be positive. Figure [aj shows that, as expected, the choice between MCE and Boltzmann Rationality has little effect on the performance. Unfortunately, and somewhat surprisingly, the early stopping procedure can, in general, lose out on a lot of reward; in experiments, this is on average between 10% and 44%, depending on the size and the type of environment. The relationship between the distance and the lost reward seems to indicate that for small values of f, the loss of reward is less significant (c.f. Figure 5b).
DISCUSSION
Computing 1) in high dimensions: Our early stopping method requires computing the occupancy measure 77. Occupancy measures can be approximated via rollouts, though this approximation may be expensive and noisy. Another option is to solve for 1) 17t via oj (I IIT) -IIz where T is the transition matrix, μ is the initial state distribution, and II, (e..)..)) = P(((s) = a). This solution could be approximated in large environments.
"II
8

---

Published as a conference paper at ICLR 2024
34-
Nr
0.5
Seowarld
0.-
0.4
-04
(b)
Figure 5: (a) Reward % lost due to the early stopping (o show groups' medians). (b) The relationship between and the lost reward (shaded area between 25th-75th quantiles), aggregated into 25 buckets.
Approximating 0: Our early stopping method requires an upper bound 0 on the angle between the true reward and the proxy reward. In practice, this should be seen mcasure of how accurate believe the proxy to be. If the proxy reward is obtained through reward learning, then we may be able to estimate / based the learning algorithm, the amount of training data, and on. Moreover, if we have a (potentially expensive) method to evaluate the true reward, such as expert judgement, then we can estimate directly (even in large environments). For details, see Skalse et al. (2023a).
Key assumptions: An important consideration when employing any optimisation algorithm its behaviour when its key assumptions are not met. For our early stopping method, if the provided does not upper-bound the angle between the proxy and the true reward, then the learnt policy may, in the worst case, result in much Goodharting as a policy produced by naive optimisation other hand, if the optimisation algorithm not concave, then this can only cause the carly-stopping procedure to stop at sub-optimal point; Goodharting is still guaranteed to be avoided. This is also true the upper bound is not tight.
P On the
Significance and Implications: Our work has several direct implications. In Section 3 we show that Goodharting occurs for wide range of environments and reward functions. This means that we should expect to see Goodharting often when optimising for misspecified proxy rewards. In Section[4] we provide mechanistic explanation for why Goodharting occurs. We expect this to be helpful for further progress in the study of reward misspecification. In Section we provide early stopping methods that provably avoid Goodharting, and show that these methods, in a certain sense, are worst-case optimal. However, these methods can lead to less true reward than naïve optimisation, This means that they are most applicable when it is essential to avoid Goodharting.
We do not have a comprehensive understanding of the dynamics play when misspecified reward function maximised, and our work does not exhaust this area of study. An important question is what types of failure modes can occur in this setting, and how they may detected and mitigated. Our work studies one important fallure mode (i.e. Goodharting). but there may be other distinctive failure modes that could be described and studied as well. A related important question precisely how a proxy reward R, may differ from the true reward Ro. before maximising RG might be bad according to Ro- There are several existing results pertaining the
Limitations and Future Work:
to this question (Ng et al. 1999. Gleave al., 20200) Skalss et al. 2022 20236), but there moment comprehensive answer. Another interesting direction is to our results to develop policy optimisation algorithms that collect more data about the reward function over time, as this information is needed. We discuss this direction in Appendix [F] Finally, it would be interesting to try to find principled relaxations of the methods in Section []] that attain better practical performance while retaining desirable theoretical guarantees.
However, it might still be possible to bound the worst-case performance further using the norm of the transition matrix (defining the geometry of the polytope Q). This will be interesting topic for future work.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
The authors would like to thank Oliver Sourbut, Bogdan Ionut Cirstea, Sam Staton and anonymous reviewers for their detailed feedback on the draft this paper. This research was conducted and funded as part of Oxford Al Safety Hub Labs. The first author was supported by separate grant from Open Philanthropy.
REFERENCES
Hal Ashton. Causal Gondhell the 13th International Conference Agents and Artificial Intelligence, pp. 67-73, 2021, doi: 10.5220001019730600073.
Goodhart's Law and Reinforcement Learning.. Proceedings of
Soo Y. Chang and Katta G. Murty. The steepest descent gravitational method for linear programming. Discrete Applied Mathematics, 25(3):211-255 1989. ISSN 0166-218X. doi: 10. 11100660066-2188199990022244
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 4302- 4310, Red Hook, NY, USA, December 2017. Curran Associates Inc. ISBN 978- 178-1-5108-6096-4
J. Denel, J. C. Fiorot, and P. Huard. The steepest-ascent method for the linear programming problem. In RAIRO. Analyse Numérique, volume 15, pp. 195-200 1981. doi: 10.1051/m2an/ 1981150301951
Tom Everitt, Victoria Krakovna, Laurent Orseau, and Shane Legg. Reinforcement learning with corrupted reward channel. Proceedings of the 26th International Joint Conference on Artificial Intelligence. IJCAI'17, pp. 4705-4713 Melbourne, Australia, August 2017. AAAI Press. ISBN 978-0-9992411-044
Eugene A. Feinberg and Uriel G. Rothblum. Splitting Randomized Stationary Policies in Total- Reward Markov Decision Processes. Mathematics of Operations Research, 37(1): 15399-11360000 2012. ISSN 2364-765X
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10888 PMLR, 2023.
Adam Gleave, Michael Dennis, Shane Legg, Staart Russell, and Jan Leike. Quantifying Differences in Reward Functions. In International Conference Learning Representations, October 2020.
C. A. E. Goodhart. Problems of Monetary Management: The UK Experience. In A. Goodhart (ed.), Monetary Theory and Practice: The UK Experience, pp- 91-121. Macmillan Education UK, London, 1984. ISBN 978 378-1-349-17295-2
doi: 10. 1001007/978- 1-349- 17295-54..
Christopher A. Hennessy and Charles A. E. Goodhart. Goodhart's Law and Machine Learning: Structural Perspective. International Economic Review, 64(3) 2023. ISSN 1468-2354. doi: 10. 1/iere. 12633.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in Atari. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 8022-8034, Red Hook, NY, USA, December 2018. Curran Associates Inc.
W. Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (Mis)design for autonomous driving. Artificial Intelligence, 316:103829, March 2023. ISSN 0004-3702, doi: 10.10066).artim..22..0000
Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike. and Shane Legg. Specification gaming: the flip side of AI ingenuity, 2020. URL https:: /ww.: dwwpmind. ctt.cloo//oooooeoo sseeii snecificam fiatificaaiiio on-tioongamil ng- the- foa-fflppaae de-id---fffa -tt.--aiiee-fff--iieee-----
10

---

Published as a conference paper at ICLR 2024
David Manheim and Scott Garrabrant. Categorizing Variants of Goodhart's Law, February 2019.
Soren Mindermann and Stuart Armstrong. Occam's razor is insufficient to infer the preferences of irrational agents. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 5603-5614, Red Hook, NY, USA, December 2018. Curran Associates Inc.
Andrew Y. Ng and Stuart J. Russell. Algorithms for Inverse Reinforcement Learning. In Proceedings of the Seventeenth International Conference on Machine Learning, ICML '00, pp. 663-670, San Francisco, CA, USA, June 2000. Morgan Kaufmann Publishers Inc. ISBN 978-1-55860-707-1.
Andrew Y. Ng. Daishi Harada, and Jtuart Russell. Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. In Proceedings of the Sixteenth International Conference Machine Learning, ICML '99, pp. 278-287, San Francisco, CA, USA, June 1999. Morgan Kaufmann Publishers Inc. ISBN 978-1 778-1-55860-612-8.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models. In International Conference on Learning Representations, October 2021.
Richard Yuanzhe Pang, Vishakh Padmakumar, Thibault Sellam, Ankur P. Parikh, and He He. Reward Gaming in Conditional Text Generation. arXiv, 2022. doi: 10.48550/ARXVV22211 1.000111
Romain Paulus, Caiming Xiong, and Richard Socher. A Deep Reinforced Model for Abstractive Summarization. In International Conference Learning Representations, February 2018.
Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., USA, 1st edition, 1994. ISBN 978-0-471-61977-22
Joar Skalse and Alessandro Abate. Misspecification in inverse reinforcement learning. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference Innovative Applications of Artificial Intelligence and Thirteenth Symposium Educational Advances in Artificial Intelligence, volume 37 of
pp. 15136-16 15143. doi: 10. 10.1609/aaai. 1tt......711222666..
AAAI Press, September 2023. ISBN 978-1-57735-880-0.
Joar Skalse, Lucy Farnik, Sumeet Ramesh Motwani, Erik Jenner, Adam Gleave, and Alessandro Abate. STARC: A General Framework For Quantifying Differences Between Reward Functions, September 2023a.
Joar Max Viktor Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and Characterizing Reward Gaming, In Advances in Neural Information Processing Systems, May 2022.
Joar Max Viktor Skalse, Matthew Farrugia- Stuart Russell, Alessandro Abate, and Adam Gleave. Invariance in policy optimisation and partial identifiability in reward learning. In International Conference on Machine Learning, pp. 32033-32058. PMLR, 2023b.
Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du and Behnam Neyshabur. Observational Overfitting Reinforcement Learning. In International Conference on Learning Representations, September 2019.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, October 2018. ISBN 978-0-262-039224-
Simon Zhuang and Dylan Hadfield-Menell. Consequences of misaligned AI. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS' 20, pp. 15763- 15773, Red Hook, NY, USA, December 2020, Curran Associates Inc ISBN 978- 978-1-71382-954-6
11

---

Published as a conference paper at ICLR 2024
A A MORE DETAILED EXPLANATION OF GOODHART'S LAW
In this section, provide intuitive explanation of why Goodharting occurs in MDPs, that will be more detailed an clear than the explanation provided in Section 4
First of all, in Section recall that JR(TT "/" R, where 7f" is the occupancy measure of This means that we can decompose JR into two steps, the first of which is independent of R. and maps II to S2, and the second of which is linear function. Recall also that 92 is a convex polytope. Therefore, the problem of finding an optimal policy can be viewed as maximising a linear function within convex polytope Ω. If Rí is the reward function optimising, then visualise this as follows:
R
Ω
Here the red arrow denotes the direction of RS within Ω. Note that this direction corresponds to M, Rt, rather than Ri, since Ω lies in a lower-dimensional affine subspace. Similarly, the red lines correspond to the level sets of R₁, i.c. the directions we can move in without changing Ri.
Now, if Rí is proxy reward, then we may assume that there is also some (unknown) true reward function Rg. This reward also induces a linear function on Q:
R
R
Ω
Suppose we pick a random point 1y" in Ω and then move in a direction that increases 17 Rg. This corresponds to picking random policy and then modifying it in direction that increases JR, (TT) In particular, let us consider what happens to the true reward function Ro. as we move the direction that most rapidly increases the proxy reward Ri.
To start with, if we are in the interior of (i.e., not close to any constraints), then the direction that most rapidly increases R₁ is to move parallel to M, R1. Moreover, if the angle between M, R and M, Ro is no more than π /2, then this is guaranteed to also increase the value Rg. To see this, simply consider the following diagram:
12

---

Published as a conference paper at ICLR 2024
R1 Ro
However, we move parallel to M,R,, R1, we will eventually hit the boundary of f2. When we do this, the direction that most rapidly increases f& will no longer be parallel to M, Instead, it will be parallel to the projection of Rú onto the boundary of Ω that we just hit. Moreover, if keep moving in this direction, then we might no longer be increasing the true reward Rg- To see this, consider the following diagram:
Boundary of Q
Ro
R1
The dashed green line corresponds to the path that most rapidly increases R1. As we move along this path, Ro initially increases. However, after the path hits the boundary of Q2 and changes direction, Ro will instead start to decrease. Thus, if we were to plot JR, (T) and JR, (T) over time, we would get plot that looks roughly like this:
R1
Ro
Steps
Next, it is important to note that Rg is not guaranteed to decrease after we hit the boundary of Ω2 To
this, consider the following diagram:
13

---

Published as a conference paper at ICLR 2024
Boundary of Ω.
Ro R
The dashed green line again corresponds to the path that most rapidly increases R4. As we move along this path, Rg will increase both before and after the path has hit the boundary of S2. If we were to plot JR. (T) and Jno (T)) over time, we would get plot that looks roughly like this:
R1
Reward
Ro
Steps
The next thing to note that we will not just hit the boundary of f2 once. If we pick a random point 17" in Ω2 and keep moving in the direction that most rapidly increases 17t R₁ until we have found the maximal value of R, in Ω then we will hit the boundary of Ω over and over again. Each time we hit this boundary we will change the direction that are moving in, and each time this happens, there a risk that we will start moving a direction that decreases Ro-
Note that Goodharting corresponds to the case where we follow path through Ω along which Ro initially increases, but eventually starts to decrease. As we have seen, this must be caused by the boundaries of Ω. We may now ask; under what conditions do these boundaries force the path of steepest ascent (of R1) to move in a direction that decreases Ro? By inspecting the above diagrams, that this depends on the angle between the normal vector of that boundary and M, R1, and the angle between M, Rí and M, Ro- In particular, in order for Ro to start decreasing, it has to be the case that the angle between Ri and Ro larger than the angle between M, and the normal vector of the boundary of Ω. This immediately tells us that if the angle between M, R2 and M, Ro is small (i.e., if arg Ro, R1) is small), then Goodharting will be less likely occur.
Moreover, as the angle between M, R₁ and the normal vector of the boundary of Ω becomes smaller, Goodharting should be correspondingly more likely to occur. Next, recall that Proposition tells us that this angle will decrease monotonically along the path of steepest ascent (of R1). As such, Goodharting will get more and more likely, the further we move along the path of steepest ascent. This explains why Goodharting becomes more likely when more optimisation pressure is applied.
14

---

Published as a conference paper at ICLR 2024
B PROOFS
Proposition 1. The set S2 = {nj π E IT] is the convex hull of the finite set points corresponding to the deterministic policies Q2 = {17 π Ho). It lies in linear subspace of dimension |S[(|A] 1).
Proof. Proof of the second half of this proposition, which says that the dimension of the affine space containing Q has at most |S[[||] - 1) dimensions, can be found in (Skalse & Abate 2023
Lemma A.2). Here, we will prove the first half (that Ω 12a) using the linear program outlined in
Puterman (994 Equation 6.9.2).
maximise: R-
subject to: Σ (ns.aa) 0) -7
Σ eES,aEA
T(s,a.s') (5,,,.5)_7(s.o)) =(ss)
V
(15 a) >
Vs S A
Puterman [1994] Theorem 6.9.1) proves that (i) for any π E II, 17" satisfies this linear program and (ii) for any feasible solution to this linear program 7,, there is a policy such that 1 = 17" In other words, f {n 17 |||44||
An 1,, 0. where A is [S] by [S|[_|] matrix.
Denote the convex hull of a finite set X as conv(X) We first show that & = (Qa). The fact that cone (Ra) Ω follows straight from the fact that Z2g Ω and from the fact that Z must be convex since it is the set of solutions a set of linear equations.
We show that Ω S (Sa) by strong induction
k(n)
Σ SES
m0 1(a A A:n(s,)) > 0}| 1)
Intuitively, k()) 0 if and only if there is deterministic policy corresponding to n and k(n) increases with the number of potential actions available in visited states. The base case of the induction is simple, if i<(n) O, then there is a deterministic policy Fu such that = 17 and therefore 1) P2 S (Sa).
For the inductive step, suppose r E con for all ry' e Q with k(n) < K and consider any n with k(n) = K. We will use the following lemma, which is closely related to (Feinberg & Rothblum 2012
Lemma 6.3).
Lemma 1. For any occupancy measure with k(n) > O, let occupancy measure x be deterministic reduction of y if and only if k(x)) O and, for all s, a, if xxss a) > then (1ss, a) 0. If x deterministic reduction of a, then there exists some a E (0. 1) and Ω such that rj = oz (1 a)y and k(y) k(()))
Intuitively, since a deterministic reduction always exists, lemma I says that any occupancy measure corresponding stochastic policy can be split into an occupancy measure corresponding deterministic policy, and an occupancy measure with a smaller & number. Proof of lemma [ is easy, choose a be the maximum value such that (1 - (s,) a) 0 for all s and a, then set I (1-azz) az). For at least one we will have (s, a) = 0 and therefore k(y) < k(n). It remains to show that E Q2 but this follows straightforwardly from (Puterman 1994 Theorem 6.9.1)
and the fact that y > O, and Ay - A(n ax) T- ab) b.
If k(n))* < K, then by lemma 1 = az + (1 - a)y with k(z) 0 and k(y) K. By inductive hypothesis, since k(v) < K, E conv(S2.)) and therefore y is convex combination of vectors in Qz Since k(xx) O, we know that x Qd and therefore orr (1 - a)y is also a convex combination of vectors in 22 This suffices to show 77 E conv(Sl,)
By induction 7 conw(Ra), for all values of k())), and therefore Ω S cone(Qa.
We have arg (Rg R1) = 0 if and only if Ro. R₁ induce the same ordering of policies, or, in other words, In (T) & Jno(TT)) In, (T) In, (7") for all policies n, n².
Proposition 2.
15

---

Published as a conference paper at ICLR 2024
Theorem 2.6). Recall that arg RR. R1) is the angle between M, Ru and M, R1, where Mr projects vectors onto Ω. Now, note that two reward functions Rg. R₁ induce different policy orderings if and only if the corresponding policy evaluation functions To. T1 induce different policy orderings. Moreover, recall that for each Ti can be viewed as the linear function Rg 1f" for 1j S2. Two linear functions fo, & defined over a domain D which contains an open set induce different orderings if and only if to and & have non-zero angle after being projected onto D. Finally, 52 does contain a set that is open in the smallest affine space which contains 52, as per Proposition [ This means that Ro and R1 induce the same ordering of policies if and only if the angle between M, and M, R1 is 0 (meaning that arg (Ro R1) 0). This completes the proof.
Proof. We show that arg (Ro, R1) satisfies the conditions of (Skalse & Abate 2023
Proposition (Concavity of Steepest Ascent). Iff; :=
17 11 -1- for 17th produced by steepest
ascent on reward vector R, f R is nonincreasing.
1981 & will be the unit vector in Proof. By the definition of steepest ascent given Denel the "cone of tangents"
T(7") = (2: || r, 0, n + AEE 92}
that maximizes E R. This is what it formally means to go in the direction that leads to the fastest increase in reward.
For sake of contradiction, assume 4+1 R> > [4: R, and let 17 12 - Then
1+2 -1-
RR= [1+11117=112 17 || E 17 1771 ) R
||7 17
17 1y Ts+
+ [. 1
17 -1,
R> E R
||1 - 17 ||7 - 11 -
where the former inequality follows from triangle inequality and the latter follows the expression is weighted average of T2+1 R and & R. We also have for À ||, But then E, E ), contradicting that & 7r R.
17th ||, 17 A! = 17 =1 Ω.
Theorem 1 (Optimal Stopping). Let R₁ be any reward function, let 0 [0, let T TB be any two policies. Then there exists a reward function Ro with arg (Ro, R1) & 0 and ) if and only if
π be any angle, and
Tng (TA) (T))
Jn, (TB) Jn, (TT) < (in())]|M,RIIl
7 TTTA|l
Proof. Let di = TBT a denote the difference in occupancy measures. The inequality can be rewritten as
ER such that arg (R, R1) & 0 and d. R < 0 cos arg RR, d)
sin(O)
To show one direction, if d R < we have d' M,R < 0 as d is parallel to Q. This gives arg R2 d
and
arg R, d & arg R1, + arg arg Ri) & arg RR, a) + 0.
follows that arg R1, d) > _ - 0, and thus cos (arg (R) < sin(
To choose R, there will be two vectors R E Fit that lie at the intersection of the plane span(1?. M1) with the cone arg (R, R1) = 0. One will satisfy arg (R, = arg (R, Ri) arg (R1, (informally, when R
instead cos arg R1. (B.dd) < s0n. we have arg R, d > _/4
lies between 17t and RR. Then this R gives
arg (R,d) arg (R, R2) arg RR > 0+
π 2
2
so R. d 0.
16

---

Published as a conference paper at ICLR 2024
Proposition 4. Given proxy reward R1, let FR be the of possible true rewards R such that arg R1) & 0 and normalized so that ||, = RiM,
Then we have that a policy π maximises ninnerg JR(mT) if and only it maximises JR. (T) x|| (|||n"] sin (arg (n7" R,)), where (an(
Moreover, each local maximum of this objective is global maximun when restricted to Q, giving that this function can be practically optimised for.
Proof. Note that
min Jn(T) 17" M,R R = || Ill RE
min cos (arg (11", R)) (Re7%
as | = ||M, Roll for all Ro. Now we claim
min cos (arg (R, r)) = cos (arg (1), + 0).
REFR
To show this, we can take Re e Fh with arg (1) arg (R1. 177) + (such an described in appendix B.. This then gives
min cos (arg (R, & cos (arg (R, + 0)
REFR
We also have
cos (arg R, )))) x cos (arg (R1,1")) + arg (R1, R)) cos (arg (Ri,"") + 8)
for any R. Then
min cos (arg (R, cos (arg (Ri, .7")) 0) ReF R
cos( cos (arg 17)) - sin(O) sin (arg (R1, ()"))
Rearranging gives
min JR(TT) R1 17" tan 0||7 M|| sin (arg (R1n7")
REFI
which is equivalent to the given objective.
To show that all local maxima global maxima, note that ming JR(TT) min 17" R in P is minimum over linear functions, and is therefore convex. This then gives that each local maximum of min Jn(TT) a global maximum, so the same holds for the given objective function.
Proposition 5. Let Ro be true reward and Rí a proxy reward such that [|oll [R and arg (Ro, Ri) = 0, and assume that the steepest ascent algorithm applied to Rí produces a sequence of policies Fu, #1,... Fy If T. is optimal for Ro, we have that
(T)) JRo (T.)) | & diameter(S)) 1|17 17 cos(O).
Proof. The bound is composed of two terms: (1) how much total reward R is there to gain, and (2) how much did we gain already. Since the reward vector is normalised, the range of reward over the Ω is its diameter. The gains that had already been made by the Steepest Ascent algorithm equal ||7] - |. but this has be scaled by the (pessimistic) factor of cos ()), since this the alignment of the true and proxy reward.
The bound can be difficult to compute exactly. A simple but crude approximation of the diameter is
2
max [| 122la & 2 max
|//2
nu mref
nest
I-
17

---

Published as a conference paper at ICLR 2024
C MEASURING GOODHARTING
While Goodhart's law is qualitatively well-described, a quantitative measure is needed to systematically analyse it. We propose a number of different metries do that. Below, implicitly assuming that all rewards are normalised (as in the Section 2). we denote by f [0, 1J - [o, 1J the true reward obtained by a policy trained on a proxy reward, as function of optimisation pressure X, similarly by fo the true reward obtained by policy trained on the true reward, and X arg maxxxe(o.l (A)
Normalised drop height:
NDH(f) = f(1) (LL)-f(M) f(x*)
Simple integration:
SI(f) ((, (S.. s(ajaa)
Weighted correlation-anticorrlattto
CACW()) max(po. 0) max(p1, V) X*)
where pf p([(I)) I.) are the Pearson correlation coefficients for to Uni X*).
In Unif[>",1]
Regression angle:
LR(f) = - 8
where 3o, 31 are the angles of the linear regression of f on [0] and [X*, respectively. Relative weighted integration:
RWI(f) (1 (1- /// (f()) (o(-))())a))
(1-x If(A) - fo(A)laa)
The metries were independently designed to capture the intuition of a sudden drop in reward with an increased optimisation pressure. We then generated dataset of 40000 varied environments:
Gridworld, Terminal reward, with |S| ~ Pooss(100), N=1000
Cliff. Cliff reward, with ||| N=500 N=500
RandomMDP, Terminal reward, |S] Poiss (100), |A| ~ Poiss(6), N=500
RandomMDP, Terminal reward, |S] Unif(16, 64), |A| Unif((, 16), N=500
RandomMDP, Uniform reward, [S] Unif(16, 64), |A| Unif(2, N=500
CyclicMDP, Terminal reward, depth P3iss(3), N=1000
We have manually verified that all metrics seem to activate strongly on graphs that we would intuitively assign high degree of Goodharting. In Figure 7 show, for each metric, the top three training curves from the dataset that each metric assigns the highest score.
We find that all of the metrics highly correlated Figure 6 Because of this, believe that it is meaningful to talk about quantitative Goodharting score. Since normalised drop height is the simplest metrie, we the proxy for Goodharting the rest of the paper.
18

---

Published as a conference paper at ICLR 2024
UR
0.17
0.47
0.28
0.5
CACW
0.177
0.72
0.22
0.46
Si
0.28
0.22
0.5
0.59
NDH
IR
CACW
R00
VOH
Figure 6: Correlations between different Goodharting metrics, computed over examples where the drop occurs for À 0.3, to avoid selecting adversarial examples.
TOP 1. NDII 0.1565
TOP 2. NOV 03313
TOP NBI 0.0700
100
kg
(a) Top curves according to NDH metric.
OF Wibs CADW 0.932
oe Bjes1 (A CACN 0.704
T0"
CACW 22250
(b) Top curves according to CACW metric.
ToR 1 q 88009
ToP3 a 201a
Tee 8/11
(c) Top curves according to metric.
TOP LB 0.0155
TOP LB 2507
TOP 3. 18 8.5078
61228
(d) Top curves according to LR metric.
T0P 0.3248
10P 2: RMB 2:209
HM 0.2224
8.3
(c) Top curves according to RWI metric.
Figure 7: Examples of training curves that obtain high Goodharting scores according each metric.
19

---

Published as a conference paper at ICLR 2024
D EXPERIMENTAL EVALUATION OF THE EARLY STOPPING ALGORITHM
To sanity-check the experiments, we present an additional graph of the relationship between NDH and early stopping algorithm reward loss in Figure and the full numerical data for Figure 5a We also show example runs of the experiment in all environments in Figure 9
0.75
0,50
Lost reward
0.00
-0.25
-0.75
0.0
0.1
0.2
0.4
0.5 0.6
NDH
Figure 8: The relationship between the amount of Goodharting, measured by NDH, and the amount of reward that is lost due pessimistic stopping. As Goodharting increases (measured by an increase in NDH), the potential for gaining reward by early stopping increases (fitted linear regression shown as the red line. [0.380, 0.396]. R 0.23, 1e 10). Only the points with NDH O are shown in the plot.
1.863x + 0.388, 95% CI for slope: [- 1.955,, 1.775]. 95% CI for intercept:
Environment
count
mean
std
min
25%
50%
75%
max
Cliff
BR
2600.0
43.82
32.89
-4.09
12.22
42.34
71.42
100.00
MCE
2600.0
44.49
32.88
-14.42
12.89
44.38
71.84
100.00
Gridworld
BR
2600.0
30.95
30.25
19.69
3.23
21.74
53.54
100.00
MCE
2600.0
28.83
28.23
-20.57
3.70
21.06
46.45
99.99
Path
BR
2600.0
40.52
34.25
-60.02
7.12
37.17
67.93
100.00
MCE
2600.0
41.17
35.01
-62.37
7.64
36.09
70.00
100.00
RandomMDP
BR
5320.0
10.09
19.47
-4.17
0.00
0.10
14.32
99.84
MCE
5320.0
10.36
19.67
-42.96
0.00
0.13
14.94
99.84
TreeMDP
BR 1920.0
21.16
26.83
-44.59
0.11
9.14
38.92
100.00
MCE
1920.0
20.11
25.95
-48.22
0.08
9.56
35.61
100.00
Table 1: A full breakdown of the true reward lost due early stopping, with respect the type of environment and training method used. See Section 3 for the descriptions of environments and reward sampling methods. 320 missing datapoints cases where numerical instability in our early stopping algorithm implementation resulted in NaN values.
20

---

Published as a conference paper at ICLR 2024
ATge 5hnan5om0-40 407
8.0
ang
outirsatonn LAUE
Optnal soos pERe
7.
04-
65-
6.0-
44-
(a) Gridworld of size 4x4.
detarce-2
Aggl dkaine-St.
Argic Chlerecm0.200 200
Ancle Sharrcewd
0.5
Pndd and
00
Opt mike pronup
Angle deraace
Optl Sptiiaatt DIIDULEE
(b) Path environment size 4x4.
Angle Satarce-2.00
Arch dtanee-s..1
Anglo 2etereee0:000
Ancla dhtarreewd.3
andd
0.
05
80
Opt milua proviupe
A123-d Mancc
Optitiuttoon precuse
(c) Cliff environment size 4x4, with a probability of slipping-0.5.
Figure (Cont. below)
21

---

Published as a conference paper at ICLR 2024
Angle ExtNTOw
16
350
codtee natls pemeaa
Opomaatt anglt
Optiaal siac2 poit
medited mopprrg part
33
0.
00
0.6
1.0
(d) TreeMDP of depth=3 and width=2, where terminal states the 1st and 2nd leaves.
0.75
6.50
030
04
Lager cotonnnn
Angis 3ianco-0.133
ingl etanee
550
64
Outirel stoppregaant
030
525
02
007
00-
balis
Angle
Optimeation
(c) RandomMPD of sizc=16, with terminal states, and 3 actions.
Figure 9: Example runs of the Early Stopping algorithm different kinds of environments. The left column shows the true reward obtained by training a policy on different proxy rewards, under increasing optimisation pressures. The middle column depicts the same plot under different spatial projection, which makes easier to see how much the optimal stopping point differs from the pessimistic recommended the Early Stopping algorithm. The right column shows how the optimisation angle (cosine similarity) changes over increasing optimisation pressure for each proxy reward (for detailed explanation of this type of plot, see Appendix
22

---

Published as a conference paper at ICLR 2024
E A SIMPLE EXAMPLE OF GOODHARTING
In Section 4 we motivated our explanation of Goodhart's law using a simple MDP M2,2, with states and actions, which is depicted in Figure I0 We assumed 7 0.9, and uniform initial state distribution jc.
So
S
10
10 1O
10
10
Figure 10: MDP M2.2 with 2 states and 2 actions. Edges corresponding the action dg orange and solid, and edges corresponding to a are purple and dotted.
We have sampled three rewards Ro, R1, Rs SxA - I, implicitly assuming that R(s, a, 8') = R(s, 8") for all s', s" S.
Ro
40
a1
R
Gg
Q1
Sn
0.170
0.228
So
0.248
0.196
S1
0.538
0.064
S1
0.467
0.089
(a) Reward 0
(b) Reward
R2
4g
So
0.325
0.165
S1
0.396
0.114
(c) Reward 2
Figure 11: Reward tables for Ro, R1, Ra
We used 30 equidistant optimisation pressures in [0.01, 0.99] for numerical stability. The hyperparameter for the value iteration algorithm (used in the implementation of MCE) was set to 0.0001.
23

---

Published as a conference paper at ICLR 2024
F ITERATIVE IMPROVEMENT
One potential application of Theorem iss that when we have computationally expensive method of evaluating the true reward Ro. we can design practical training regimes that provably avoid Goodharting. Typically training regimes for such reward functions involve iteratively training on low-cost proxy reward function and fine-tuning on true reward using human feedback (Paulus et al. 2018). We can use true reward function to approximate arg (Ro. R1). and then optimal stopping gives the optimal amount of time before a "branching point" where possible reward training curves diverge (thus, creating Goodharting).
Specifically, let us assume that we have access to an oracle ORACLE R,, which produces increasingly accurate approximations of some true reward R" when called with a proxy reward R2 and a bound 0, > arg (It", R.), it returns R&+1 and 0:+1 such that 0,+1 > arg (It", R(+1) and lim; 0; O. Then algorithm]i [ is an iterative feedback algorithm that avoids Goodharting and terminates at the optimal policy for R*
Algorithm Iterative improvement algorithm
I: procedure IYERATTVELAPRRVEMENTTS.
2: It Unif R
3: Uni [ SxA
4: 1 tiu- 17
5:
6. to - (e
7: while i, * do
8: while f & do
9:
R OR
10: R-R R
11:
end while
12:
À max(A m ATa ?2
7 th, At;
14:
4+1 argmax F. R
15:
i- 1
16: end while
17:
return n7th
18:
end procedure
Proposition 6. Algorithm 7 is valid optimisation procedure, that is, it terminates at the policy n² which is optimal for the true reward R
Proof. By Theorem 1 the inner loop of the algorithm maintains that JR, (T++1) JRo (T:) If the algorithm terminates, then it must be that f 0, and the only point that this happen is in point 17"
Since steepest ascent terminates, showing algorithm I terminates reduces to showing we only make finitely many calls to ORACLER.. can be shown that for any f. produced by steepest ascent on R, T, = proje(R) for some linear subspace P on the boundary of Ω formed by subset of boundary conditions. Since there are finitely many such P. there is some so that for all Rt R with arg (R, R') < e, arg (proje(B) (atpp(())ppppppppp projp(R")) < for all P.
Because have assumed that lim,-x 0, 0, lim, arg (R,, Rt) and ORACLEg will only be called until arg (R,, R+) = 0. Then the number of calls finite and so the algorithm terminates.
We expect this algorithm forms theoretical basis for a training regime that avoids Goodharting and can be used in practice.
24

---

Published as a conference paper at ICLR 2024
G FURTHER EMPIRICAL INVESTIGATION OF GOODHARTING
G.1 ADDITIONAL PLOTS FOR THE GOODHARTING PREVALENCE EXPERIMENT
Optiricetion 1A77 Berwoniity
VCC
0.50
10%
035
0.00
10-7
0.32
10-4
0.50
Hith
HondomuDD
ThANDA
Endkormeett type
MDH
(a) The relationship between the type of environment and the probability of Goodharting.
(b) Log-scale histogram of the distribution of NDH metric in the dataset.
(a) The choice of operationalisation of Figure 12: Summary of the experiment described Section 3 optimisation pressure does not seem to change results in any significant way. (b) NDH metrie follows roughly exponential distribution when restricted to cases when NDH
G.2 EXAMINING THE IMPACT OF KEY ENVIRONMENT PARAMETERS ON GOODIAATTIN
To further understand the conditions that produce Goodharting, between key parameters of the MDP, such as the number of states or temporal discount factor 7 and NDH. Doing directly the dataset described in Section 3 does not yield good results, as there is not enough data points for each dimension, and there are confounding cross-correlations between different parameters changing at the same time.
investigate the correlations
To address those issues, we opted to replace the grid-search method that produced the datasets for Section 3 and Section [.]] We have first picked a base distribution over representative environments, and then created a separate dataset for each of the key parameters, where only that parameter is varied
Specifically, the base distributions is given over RandomMDP with [S] sampled uniformly between and 64, |A] sampled uniformly between and 16, and the number of terminal states sampled uniformly between 1 and 4, where 7 = 0.9, = 1, and where we use 25 X's spaced equally (on a log scale) between 0.01 and 0.99. Then, in each of the runs we modify this base distribution of environments along single axis, such as sampling 7 uniformly across (0, 1) shown in Figure 16c
Key findings (positive): The number of actions |A| seems to have a suprisingly significant impact on NDH (Figure [5]. The further away is the proxy reward function from the true reward, the more Goodharting is observed (Figure 16a). which corroborates the explanation given at the end of Section We note that in many examples (for example in Figure in any of graphs in Figure 9 the closer the proxy reward is, the later the Goodhart "hump" appeared this positive correlation presented in Figure 16b] We also find that Goodharting is less significant the case of myopic agents, that is, there is positive correlation between and NDH (Figure [6c). Type of environment seems have impact on observed NDH, but note that this might be a spurious correlation.
Key findings (negative): The number of states S|| does not seem to significantly impact the amount of Goodharting, as measured by NDH (Figures 13 and [4]. This suggests that having proper methods of addressing Goodhart's 's law is important as we scale to more realistic scenarios, and also partially explains the existence of the wide variety of literature on reward gaming (see Section I.II. The determinism of the environment (as measured by the Shannon entropy of the transition matrix 7) does not seem to play any role.
SSnne this is impossible when comparing between environment types, use the original dataset. from Section 3 in Figure T6f
25

---

Published as a conference paper at ICLR 2024
0:45
W My Mr /0 1/ /
0.33
0.23
04:3
0.00
Figure 13: Small negative correlation (not statistically sifnificant) between ISI and NDH: y - 5e 05xr + 0.02852, 95% CI for slope: [-0.00018, 7.23e - 05], 95% CI for intercept: (0.01892, 0.03813]. Pearson's 's -0.0119, R 0.0, 0.4058. On the left: scatter plot, with the least-squares regression fitted line shown in red. On the right: mean NDH per ISI, smoothed with rolling average, window size=10. Below, we have repeated the experiment for larger ISI to investigate asymptotic behaviour. N=1000.
0228
M/ Myp
0425
0.28-
0.00
370
P4
Figure 14: Small negative correlation (not statistically significant) between the number of states in the
environment and NDH: y = -0.0z + 0.02047, 95% CI for slope: [- 8.2529e - 06, 2.2416c 06]. 95% CI for intercept: f0.017, 0.0239], Pearson's 's - -0.0116, R 0.0, p 0.261. On the left: scatter plot, with the least-squares regression fitted line shown in red. On the right: mean NDH per ISI, smoothed with rolling average, window size=10. N=1000.
30
12
32
Figure 15: Correlation between IAl and NDH: -0.00011x + 0.00871, 95% CI for slope:
[00.00115 11:6.-7..86 -7.28e 05]. 95% CI for intercept: [0.00723, 0.01019]. Pearson's -0.0604, R 0.0, p < le - 08. On the left: scatter plot, with the least-squares regression fitted line shown in red. On the right: mean NDH per IAI. N=1000.
26

---

Published as a conference paper at ICLR 2024
10-
04-
Locaton of Coochart hamoo
0.
02-
03-
0.0
0.5 2.0
2.5
2.0
Anglle dislance
0.00
0.25
0:30
0.75
1.00
1.55
1.30
3.93
2.00
Angle distenc
(a) Correlation between the angle distance to the proxy reward and NDH metric: 0. 0.02389r 0.00776, 95% CI for slope: [0.02232, 0.02546] 95% CI for intereept: 1-0.00883, -0.00670].
Correlation between the distance the proxy reward, and location the Goodhart's hump:
0.10169z + 0.99048, 95% CI for slope: =
0.11 0. 0933331]. 95% CI for intercept: 0.98360, 0.99736]. Pearson's
Pearson's 's r 0.2895, R² = 0.08, 1.2853e - 187.
-0.3422,
R 0.12, p 2.7400e
118.
05
0.4
920
0.1
0.-
1.0
Spanciy
(c) Correlation between and the NDH metric:
(d) Correlation between the sparsity of the reward,
y
0.00696z: + 0.00326, 95% CI for [0.00504, 0.00888]. 95% for intercept:
and the NDH metric: y - 0.007012 +0.l22++0000000 0.00937. 95% CI for slope: [-0.00852, -0.00550) 95%
slope:
[0.00217, 0.00436]. Pearson's r 0.07137,
CI for intercept: [0.00850, 0.01024), Pearson's
R²
0.01, 1.2581e 12.
0.09090, 0.01, p < 9.5146e 20.
0.
0.
322
3.3
(c) Correlation between the determinism of the environment. and the NDH metric: 0.77221x 0.01727, 95% CI for slope: [0.31058,, 1.23384]. 95% CI for intercept: [0.01570, 0.01884], Pearson's r = 0.03278,
(I) Distribution NDH metric for different kinds of environments. Note that other parameters are not kept constant across environments, which might introduce coreelaorss,.onss
y
R² = 0.01, 0.0010.
Figure 16: Correlation plots for different parameters of MDPs. N=1000 for all graphs above except for Figure 16f which uses the dataset from Section 3 where N=30400.
27

---

Published as a conference paper at ICLR 2024
H IMPLEMENTING THE EXPERIMENTS
H.1 COMPUTING THE PROJECTION MATRIX
For reward R. we want find its projection M, onto the |SK[A] - 1) -i-mmnnnsi hyperplane H span([2) containing all valid policies. H is defined by the linear equation AT = b corresponding the constraints defined in appendia giving M, I A'(AA') A by standard linear algebra results. However, this is too computationally expensive to compute for environments with a high number states.
There is another potential method that we designed but did not implement. It can be shown that the subspace vectors orthogonal to H corresponds exactly to expected reward vectors generated by potential functions that is, the set of vectors orthogonal to H is exactly the vectors
R(s, a) (o)
o(s)
for potential function Φ 5 R. Note this also gives that all vectors of shaped rewards have the same projection, so aim shape rewards to be orthogonal to all vectors described above.
To do this, initialise two potential functions 6, 6 and consider the expected reward vectors of (s, = R(s.a) + E (x-r(s))((((s')) o(s)
and
R (s, a) (G.a)) (6)) (())-
We optimise á to maximize the dot product between these vectors and Ø minimize it. á converges so that Ri is orthogonal all reward vectors Ri, and will thus be It's projection onto H.
H.2 COMPUTE RESOURCES
We performed our large-scale experiments on AWS. Overall, the process took about 100 hours of a c5a. 16xlarge instance with 64 cores and 128 GB RAM, as well as about 100 hours of t2. 222xxee instance with 8 cores and GB RAM.
28

---

Published as a conference paper at ICLR 2024
I AN ADDITIONAL EXAMPLE OF THE PHASE SHIFT DYNAMICS
Figure and Appendix we have explored an example of 2-state, 2-action MDP. The image space being 2-dimensional makes the visualisation of the run easy, but the disadvantage is that we do not get to see multiple state transitions. Here, we show an example of a 3-state, 2-action MDP, which does exhibit multiple changes in the direction of the optimisation angle.
In the Figure 4
We use an an MDP M3.2 defined by the following transition matrix T:
8g
Q1
Qg
Q1
0g
Q1
So
0.9
0.1
So
0.1
0.9
So
0.0
0.0
S1
0.1
0.9
S1
0.9
0.1
S
0.0
0.0
2
0.0
0.0
S2
0.0
0.0
S2
1.0
1.0
(a) Starting from So
(b) Starting from S1
(c) Starting from S2
with N different proxy functions interpolated linearly between Ro and R1. We use 30 optimisation strengths spaced equally between 0.01 and 0.99.
ag
41
Qg
Q1
S
0.290
0.020
So
0,263
0.195
81
0.191
0.202
S1
0.110
0.090
52
0.263
0.034
52
0.161
0.181
(a) Reward Ro
(b) Reward R:
Figure 18:
Reward Tables
The rest of the hyperparameters are set as in the Appendix with the difference that we are now using exactly steepest ascent with early stopping, as described in Figure 4b) instead of MCE approximation to it.
29

---

Published as a conference paper at ICLR 2024
24
Angle dellacee
Angle dbsancpe0:133
Optimaa suapanng part
stopping paine
0.0
36
0.5
10
0.0
0.1
0.2
0.5
(a) Goodharting behavior for M3.2 five reward functions. Observe that the training method is concave, with Proposition B Compare to the theoretical explanation in Appendix [A] in particular to the figures showing piece-wise linear plots of obtained reward over increasing optimisation pressure.
(b) The same plot under different spatial projection, which makes easier how much the optimal stopping point differs from the pessimistic one recommended by the Early Stopping algorithm.
accordance
10
Angl detence-0.000 Citical anglem) 000
Angls: dotan:=0..00, Criical engic=
angl distacecuu 203, (ftiraa angieso 309
Angln ditamcsen:66 Eolical
Uritiaal
angle
04
0.0
0.4
0.3
Opemicobon pressure
(c) A visualisation of how the optimisation angle (cosine similarity) changes over increasing optimisation pressure for each proxy reward. This is the angle between the current direction of optimisation in Ω, i.e. (1"0111 - 17 "7"*1). ), and the proxy reward function projection M, (defined cos (arg [,', in the proof of Thcorem M Once the angle crosses the critical threshold, the algorithm stops. The eritical threshold depends on the distance between the proxy and the true reward, and it is drawn in as dotted line, with a color corresponding to the color of the proxy reward. Compare this plot to Figure 20 we can see exact places where the phase transition happens, the training Figure [9a] where it be how the algorithm stops (in blue) immediately after the training
meets the boundary of the convex space. Also, compare
the corresponding critical angle value (in case fo last two proxy rewards), or continues the end (in case the first two).
Figure 19: Summary plots for the Steepest Ascent training algorithm over five proxy reward functions.
30

---

Published as a conference paper at ICLR 2024
Angli disiance
Angle distance C.083
Opteristirr premure
69
05
n1
Angle destncce 0.203
Angle distence 0.364
Angle
Figure 20: Trajectories of optimisations for using different proxy rewards. Note that the occupancy measure space [ 1) 3-dimensional in this example, and the convex hull is over |A| |S deterministic policies. We hide the true/proxy reward vectors for presentation clarity.
31


Reviews:
[{"review_id": "iwI3XrWEnS", "reviewer": "Reviewer_Y2A1", "summary": "This paper presents an analysis of Goodhart’s law in reinforcement learning. The paper starts with a formalization of the Goodhart problem in terms of misalignment of reward functions on finite MDPs: one proxy reward that the policy optimizes when we really wish to optimize the other. The paper justifies that the problem occurs in small scale experiments demonstrating that increasing the optimization pressure on the proxy eventually leads to a decrease in the true reward. A theoretical analysis is given with some examples about why this occurs in finite MDPs. Finally an early-stopping algorithm is proposed to mitigate this issue along with some preliminary experiments on the algorithm.", "questions": "How do you think the theoretical results generalize to the setting where the reward function is considerably more sophisticated than simple finite MDPs? For example, high dimensional, continuous state-action, long-horizon problems?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The problem is clearly very important and a better understanding of proxy rewards, overoptimization, and Goodhart’s law are definitely needed in the community.\n\nThe paper is presented fairly clearly, except in some areas which I point out later.\n\nThe paper provides insights from multiple frontiers to help shape this understanding (empirical, theoretical, and conceptual).\n\nThe theoretical findings are useful, but not entirely surprising given what is known already in the literature (see below). However, I do believe it’s useful to have this formalized and characterized when specifically talking about Goodhart’s law.", "weaknesses": "My primary complaint is that, although this is a solid analysis, I do not believe it strikes the heart of the Goodhart problem. The position of the paper is that misalignment can be characterized by the worst-case angle between reward functions. This is a fairly well-understood setting (e.g. see ‘simulation lemma’ by Kearns & Singh or any number of classical RL papers). However, it’s unclear how this maps into problems that (1) are beyond the finite case, or (2) are classical examples of Goodhart’s law like the snake bounty. While one could model (2) in the framework studied here, I am not sure this would be an informative model in those settings as the ‘theta’ is just so large.\n\nThe above is more of a conceptual disagreement about the premise. For the rest of the review, I give the benefit of the doubt and simply accept the premise is true.\n\nUnfortunately most of the important empirical results have been relegated to the appendix, leaving the main paper with vague / difficult-to-verify statement such as ‘a Goodhart drop occurs for x% of all experiments. Without figures or tables, it’s difficult to understand what this means, such as what the criteria of a ‘Goodhart drop’ is (any non-zero drop, some negligible drop, etc). It would be helpful to make room in the main paper for results that present a more comprehensive picture of the findings.\n\nThe early stopping proposal is natural, but also seems very conservative. This appears to be consistent with the empirical findings. Furthermore it requires knowledge of $\\theta$, which is just assumed to be known. While it’s hard to imagine anything can be down without some knowledge of the true reward or structure, this seems quite coarse.\n\nFigure 5 is difficult to appreciate in absolute terms as one cannot tell if, for example, 0.4 is a large value relative to the reward achievable. I think this plot would be better replaced with a typical plot showing how the true and proxy rewards change as the policy is optimized and when the algorithm decides to stop, as well as the counterfactual of what would happen if it does not stop.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I4MT8FLLfE", "reviewer": "Reviewer_YkV9", "summary": "This paper studies the problem of reward misspecification. The authors point out that over-optimizing an incorrect reward function can lead to the wrong behaviour for the true (\"test\") reward function, and dub this phenomenon Goodharting. The authors propose a quantitative way to evaluate this phenomenon (cf. Definition 5), and perform an experimental case study on some simple MDPs to establish that Goodharting is a common phenomenon in RL. The authors then provide an intuitive geometric explanation for this phenomenon and propose an early stopping method to avoid overfitting. Further experimental evaluations are performed on the early stopping method to", "questions": "See weaknesses section above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper investigates an interesting, albeit not entirely surprising phenomenon, and investigates it thoroughly and carefully. The problem of reward misspecification is quite relevant for practical considerings of RL, so gaining some understanding of this problem is appreciated. The paper is well-written and the messages are conveyed clearly. The theoretical contributions, while not exactly practical, are a nice step towards preventing this problem from affecting performance.", "weaknesses": "While I am overall positive about the paper, I have a few comments and suggestions for possible improvement. \n\n- The definition of optimization pressure is a bit strange. Why should we not define it as simply the distance from the optimal policy? For instance, we can say that the optimization pressure is epsilon if we obtain a policy $\\hat{\\pi}$ such that $J_R(\\pi^\\star) - J_R(\\hat{\\pi}) \\leq \\varepsilon$. I feel that tying the optimization pressure to a certain regularization scheme detracts from the fundamental aspect of the problem, and furthermore that regularization is only used here as a proxy for \"how close to optimal are we\", which can be defined more directly as above.\n- The environments that have been used to establish that Goodharting is pervasive (Section 3) are somewhat simple. I understand that it is difficult to measure the NDH metric in environments where we cannot solve for the optimal policy, but it would have been nice to understand how pervasive this is in \"real\" problems, or at least in popular RL benchmark environments. As a side note, the fact that the NDH metric is inherently difficult to measure can be considered as a drawback of the proposed methodology -- can the authors comment?\n- It would also have been interesting to more systematically study which properties of environments imply that Goodharting is more likely to take place, do the dynamics of the MDP (e.g. a bottleneck structure) have any role?\n- The proposed optimal stopping algorithm is very pessimistic since it tries to avoid overfitting to any possible reward function in a certain set (is this pessimism unavoidable?), and as the authors point out it is computationally infeasible. In addition, if I understand correctly, it requires knowing the transition dynamics and knowing the distance between the proxy reward and the true reward function, which is fairly unpractical.\n\n- Incorrect/unclear sentences: \n1. \"We observe that NDH is non-zero if and only if, over increasing optimisation pressure, the proxy and true rewards are initially correlated, and then become anti-correlated\". I believe the authors meant the NDH is non-negative, not non-zero.\n2. \"Note that this scheme is valid because for any environment, reward sampling scheme and fixed parameters, the sample space of rewards is convex. In high dimensions, two random vectors are approximately orthogonal with high probability, so the sequence R_t spans a range of distances.\". It is not clear what point the first sentence is attempting to communicate (what does \"valid\" mean?), and the second sentence is incorrect as stated (what distribution is one sampling from? I can imagine many distributions where this is untrue, say a deterministic one.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CGXUwDNoh9", "reviewer": "Reviewer_eHjA", "summary": "Most reinforcement learning algorithms are designed for accurate reward feedback. However, in practice, accurate reward feedback may not be available. In the presence of inaccurate reward feedback, it is possible to observe a phenomenon that the performance of the training policy first increases and then decreases after passing a threshold point. This paper addresses this interesting phenomenon and names it “Goodhart’s Law in RL”. To solve this problem, this paper quantifies the magnitude of this effect and how it exists in a wide range of environments and reward functions. It provides a geometric explanation and an optimal early stopping method with theoretical regret bounds. They then empirically showed the performance of their early stopping method.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper is quite novel because it raises an interesting and important observation – the performance of a policy increases first and then decreases. Such observation is caused by inaccurate reward feedback, which indeed exists in real RL applications.\n\n2. This paper quantifies the magnitude of such phenomena and provides a clear geometric explanation.\n\n3. With these insights, this paper proposes an optimal early stopping method with theoretical regret bound analysis.\n\n4. The experimental results supported the authors' claim.\n\n5. This paper is well-written. Concepts are conveyed efficiently. The analysis is detailed while keeping a clear line of high-level logic.", "weaknesses": "1. The optimal early stopping rule relies on the knowledge of the occupancy measure and the upper bound $\\theta$ of the angle between the true reward and the proxy reward. Methods to approximate the occupancy measure are well-researched. My concern is on the approximation of $\\theta$, which is a relatively new concept and requires some knowledge of the true reward feedback or true reward samples. When such estimation is not accurate, the stopping method could exhibit negative performance. It would be better if the author could show empirical results with approximated $\\theta$.\n\n2. This paper is preliminary because it only considers finite state and action space. The empirical results are also only on small grid world environments. It is not clear whether such a phenomenon exists in more broad continuous settings and what would be the practical way to solve it in these settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yfeAnO2dgn", "reviewer": "Reviewer_MUg9", "summary": "This paper studies the interplay between reward misspecification and optimisation in RL as an instance of Goodhart's law. The authors provice geometric explanations of how optimisation of a misspecified reward function can lead to worse performance beyond some threshold, and show in experiments that several environments and reward functions are prone to Goodhart's law and optimisation of a reward proxy eventually leads to worse performance. The authors also propose an early stopping algorithm to address this problem.", "questions": "- Your work seems to be tailored to the specific choice of difference metric between two reward functions (their angle). I guess that the main reason for choosing this distance metric is that it is a STARC metric. \n\t- However, can you provide further justification for why the \"angle\" is a good choice or even the *right* choice? \n\t- What could another reasonable metric be?  \n\t- And, how would choosing a different metric impact your results?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- First of all, the studied topic is, in my opinion, important and could be of interest to many in the ICLR community. \n- The paper is a good attempt at extending prior work on reward misspecification and reward gaming (e.g., Skalse et al. 2022) to the question of what role optimisation plays and whether we can characterize reward misspecification from a policy optimisation standpoint as well. I am not very well acquainted with the related work, but the contributions and many of the ideas in this paper seem novel to me.\n- The results are very interesting and provide some nice intuition about the interplay of reward distance, optimisation and MDP model. While I don't think that one should overinterpret the results as they are either based on empirical studies of a some specific set of environments or on theoretical insights with idealised assumptions, I think that the findings of this paper are overall very interesting.", "weaknesses": "- The evidence on the \"Goodharting\" effect are only circumstantial. Experiments on some specific set of environments such as grid worlds do not necessarily allow us to extrapolate. After all, the Goodharting effect can only be \"explained\" but not characterised. Nevertheless, these experiments and the geometric explanations provide good intuition which I think is very interesting and could inspire future lines of work. \n- A minor weakness is that the proposed early stopping algorithm might not perform well due to large reward losses from stopping early, which is somewhat expected due to its pessimistic nature. The algorithm is also fairly impractical bcause it assumes prior knowledge of $\\theta$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)