Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification

Authors: thomas  kwa
drake  thomas
  anthropic
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 03:27:58

Abstract: When applying reinforcement learning from human feedback (RLHF), the reward is learned from data and, therefore, always has some error. It is common to mitigate this by regularizing the policy with KL divergence from a base model, with the hope that balancing reward with regularization will achieve desirable outcomes despite this reward misspecification. We show that when the reward function has light-tailed error, optimal policies under less restrictive KL penalties achieve arbitrarily high utility. However, if error is heavy-tailed, some policies obtain arbitrarily high reward despite achieving no more utility than the base model--a phenomenon we call catastrophic Goodhart. We adapt a discrete optimization method to measure the tails of reward models, finding that they are consistent with light-tailed error. However, the pervasiveness of heavy-tailed distributions in many real-world applications indicates that future sources of RL reward could have heavy-tailed error, increasing the likelihood of reward hacking even with KL regularization.

Main paper: 
 introduction: 
 Kullback-Leibler (KL) divergence constraints in reinforcement learning (RL) are employed to stay in regimes where the objective is accurate enough. Some on-policy (Schulman et al., 2015 (Schulman et al., , 2017) ) and many off-policy (Abdolmaleki et al., 2018; Jaques et al., 2019) policy gradient algorithms employ KL constraints or penalties during optimization to prevent the policy from deviating too much from the data collection distribution. This ensures that estimates of each action's advantage are reliable enough to update the policy in a helpful way.Reinforcement learning from human feedback (Christiano et al., 2017; Ziegler et al., 2020, RLHF) is a very popular method to induce desirable behavior in language models. RLHF starts with a base pre-trained model, then learns a reward function from human annotator data. Next, it trains an RL policy to maximize this reward, while penalizing high KL divergence from the policy to the base model. RLHF uses an on-policy algorithm and has accurate advantages, but the reward function is always somewhat misspecified compared to desired behavior, due to insufficient data, human biases, and other factors.The main purpose of the KL penalty in RLHF is to limit the consequence of reward modeling errors by keeping the policy within a distribution similar to that on which it was trained. Ideally, in the low-KL regime the reward model's errors are small enough that it provides correct updates to the base model. Gao et al. (2023) empirically supports this view: if the KL divergence in RLHF is allowed to grow too much, with a misspecified reward, the model's performance on the true utility starts to decrease.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The abstract lists the important claims: the relationship between Goodhart's Law and whether the error in a misspecified reward is heavy-tailed. The main limitation of independence assumptions is clearly stated in the introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
kl divergence and kl regularization: 
 Recall that KL divergence between two distributions P and Q is defined asD KL (P ∥Q) = x∈X P (x) log P (x) Q(x) .If we have two policies π, π 0 , we define D KL (π∥π 0 ) as the KL divergence between the distributions of actions taken on the states in trajectories reached by π. That is, if T r(π) is the distribution of trajectories taken by π, we penalize D KL (π∥π 0 ) ≜ E s∈T,T ∼T r(π) [D KL (π(s)∥π 0 (s))].In RLHF, it is common to use the regularization term βD KL (π∥π 0 ) to prevent the learned policy from deviating too much from the base policy, which can prevent unstable behavior or overfitting to the reward model. If our reward model gives reward U , then the optimal policy for RLHF with a KL penalty is arg max π E[U (π)] -βD KL (π∥π 0 ) .Often the regularization parameter β is dynamically adjusted to keep the D KL near some target value (Ziegler et al., 2020) .A distribution P over R with cumulative distribution function (CDF) F P is heavy-tailed if its tail function FP (x) ≜ 1 -F P (x) satisfies lim x→∞ e tx F (x) = ∞ for allt > 0.Heavy-tailed distributions are well-known in statistics to have a higher probability of producing a single extreme value. For example, if the sum of two independent variables from heavy-tailed distributions is large, it is most likely due to one extreme sample rather than two equally large samples. (Wierman, 2013) Reward misspecification has caused low-utility outcomes in practice; for example, in (Clark and Amodei, 2016) , an RL agent trained to play a racing videogame according to a misspecified reward function achieves a high score while failing to complete the course. Gao et al. (2023) introduce the concept of "overoptimization": optimizing for a proxy objective decreases performance according to the true objective. This raises the question: in general, when RLHF reward is misspecified, when does the optimal policy produce high utility?By applying the proxy reward and true reward functions to a distribution over text (generated by an LLM), we get two scalar random variables, which we call U for proxy reward and V for true reward / utility. Then we can define the error in the proxy reward as X ≜ U -V , so that U = X + V . Framed this way, optimization for a proxy reward U is a mix of desirable optimization for V and undesirable optimization for X. The joint distribution of V and X determines the limiting value of V as we apply more optimization. When we say that reward misspecification can have negative effects, we mean that too much variance in X can "redirect" the optimization pressure from V to X, and prevent utility gain from optimization.Reward misspecification is also studied by (Lambert and Calandra, 2024) , (Laidlaw et al., 2024) , and others. Laidlaw et al show that a KL penalty between action distributions can be ineffective, and propose instead regularizing state occupancy measure. Our results show an inherent weakness of KL divergence, including when applied to state occupancy measure.We prove that in many cases, V → 0 in the limit of optimization for some proxy U . We call this phenomenon "catastrophic Goodhart", after Goodhart's law: "when a measure becomes a target, it ceases to be a good measure" (Strathern, 1997) . In these cases, the end result of optimizing for a proxy of V is no better than not optimizing at all. However, in other cases, V → ∞ despite some reward misspecification; in these cases the reward misspecification is not severe enough to prevent good outcomes.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Section 7 lists the limitations, which we have combined with the discussion section due to heavy overlap.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theoretical results: 
 When applying KL regularization, the trained model is regularized towards some base policy π 0 . One would hope that a KL penalty can produce good outcomes even in the case of reward misspecification; that is, if the reward U is the sum of true utility V and an error term X, we would hope that optimal policies under a KL penalty achieve high V even if the magnitude of X is large. We show that this is not always the case: Corollary 1 of Theorems 1, 3, and 2 establishes that when X(π 0 ) is heavy-tailed, there are arbitrarily well-performing policies π withE π [V ] ≈ E π0 [V ]. However, Theorem 4 shows that when error is light-tailed and independent of V , the optimal policy under a KL penalty results in V > 0, and V can be made arbitrarily large. Thus, the tails of the error distribution are crucial in determining how much utility will result from optimization towards an imperfect proxy.Theorems 5 and 6 (Section 3.4) show that the relationship of catastrophic Goodhart to heavy-tailed error is not just a quirk of KL divergence by using a different model of optimization based on conditioning on high reward values. Under this model (and given additional regularity conditions), it is also true that heavy-tailed error results in catastrophic Goodhart and light-tailed error plus independence results in arbitrarily large utility. All proofs are in the appendix.Theorem 1. Given any heavy-tailed reference distribution Q over R with mean µ Q , and any M, ϵ > 0, there is a distribution P with mean µ P > M and D KL (P ∥Q) < ϵ.Outline of proof (see appendix for full proof): WLOG take µ Q = 0. If we set P t to upweight the probability mass of P r Pt (X > t) to c/t for some c, t, then the mean of P t will be approximately at least c. As t → ∞, the KL divergence D KL (P t ∥Q) will shrink to zero.Intuitively, in a heavy-tailed distribution, events with extremely high x are not very rare, so you don't pay much of a KL penalty to upweight them so they happen about 1/x of the time.Theorem 2. However, if the distribution Q is light-tailed and d = D KL (P ∥Q) is bounded, then µ P is bounded, and µ P -µ Q → 0 as d → 0.We now adapt our result to the case where the policy is a language model and we are training it using RLHF. We are now applying KL divergence over the policies rather than the return distributions.We first formally define the properties of RLHF on language models that cause the result to hold: namely, when when considered as a Markov decision process (MDP), environmental transitions are deterministic and return depends only on the final state reached.Definition: A deterministic-transition MDP with Markovian returns (DMRMDP) is an MDP (S, A, P, R) such that:• The transition function P : S × A → S is deterministic, i.e., for each state s ∈ S and action a ∈ A, there exists a unique state s ′ ∈ S such that P (s ′ |s, a) = 1.In RLHF: the transition is appending the generated token a to the context s.• There is a set of sink states E ⊆ S that terminate every trajectory, which is disjoint from the set of start states.In RLHF: The sink states are sequences ending in <EOS> or above a certain length.• Returns are Markovian; that is, for any two trajectories τ= (s 1 , a 1 , . . . , s n ), τ ′ = (s ′ 1 , a ′ 1 , . . . , s ′ n ), if s n = s ′ n, then τ and τ ′ have identical return distributions. Equivalently, for the trajectory random variable T = (S 1 , A 1 , . . . ) distributed according to any policy, with return G, G⊥ ⊥(S <i , A <i ) | S i for any i ≥ 1.In RLHF: the return only depends on the full generated string, which is the final state.The language model stochastically outputs the next token a given s, and corresponds to the policy. A DMRMDP is therefore a good model of RLHF. Theorem 3. Let W = (S, A, P, R) be a deterministic-transition MDP with Markovian returns. Given W we define the function that takes policies to trajectories T r : (S → ∆A) → ∆(S × A) * , and the average return function g : (S × A) * → R, which induces a function G : ∆(S × A) * → ∆R. Let π 0 : S → ∆A be some base policy. If G • T r(π 0 ) is heavy-tailed with finite mean µ Q , then for any M, ϵ > 0, there is a policy π with mean returnE[U |U ∼ G • T r(π)] > M and E s∈T,T ∼T r(π) [D KL (π(s)∥π 0 (s))] < ϵ.Corollary 1. Theorems 2 and 3 imply that when utility is light-tailed, reward modeling errors make the proxy reward heavy-tailed, and a policy π is regularized severely enough to have KL divergence values approaching zero, the reward E[U (π)] can go to infinity while utility E[V (π)] approaches a value no higher than the base policy.Theorem 4. If U = X+V with X and V both light-tailed and V unbounded, and the distribution of U is continuous, andπ * (β) ≜ arg max π E[U (π)] -βD KL (π, π 0 ), then lim β→0 + E[V (π * (β))] = ∞.Although we think a KL divergence penalty or cap is the most realistic setting for RLHF, it is not the only model of optimization where heavy-tailedness of the error determines whether catastrophic Goodhart occurs. Consider another model of optimization where U = X + V as before, but we simply condition on U being higher than some threshold t.2 Then we are interested in the quantitylim t→∞ E[V |X + V ≥ t].If we slightly strengthen the heavy-tailedness and light-tailedness assumptions, heavy-tailed error results in catastrophic Goodhart, while light-tailed error results in arbitrarily high expected utility.Theorem 5. Let X and V be two independent random variables with CDFs F X and F V and tail functionsFV ≜ 1 -F V , FX ≜ 1 -F X such that • V has a finite mean. • X is subexponential; that is, lim x→∞ Pr(X1+X2>x) Pr(X>x)= 2 if X 1 , X 2 are two independent samples from X. This is a slightly stronger property than being heavy-tailed.• The tail of V is sufficiently lighter than the tail of X that lim t→∞ t p FV (t) FX (t) = 0 for some p > 1.Then lim t→∞ E[V |X + V ≥ t] = E[V ]; that is, catastrophic Goodhart occurs in the limit of optimization for U = X + V .The proof is included in the appendix. It requires expressing the conditional expectation in question as Pr(X>t-v) , then partitioning the interval (-∞, ∞) into four regions and bounding the integrand in the numerator above by a different quantity in each region.∞ -∞ vf V (v)Pr(X>t-v) ∞ -∞ f V (v)3.4.2 Conditioning with light-tailed error produces arbitrarily high utility Theorem 6. Let X, V be independent random variables such that lim t→∞ FX (t+1) FX (t) = 0. (This implies that X has tails that are dominated by e -cx for any c, though it's a slightly stronger claim because it requires that X not have large jumps in the decay of its tails.) Then for any V with a finite mean which has no upper bound,lim t→∞ E[V |X + V > t] = ∞.Theorem 6 generalizes a consequence of the "Regressional Goodhart Identity" in (Gao et al., 2023) .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: All proofs are given inline or in the appendix, except for Theorem 5 which appears in the supplemental material.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 Our theoretical results now raise the question of whether the error in reward models is heavy-tailed or light-tailed in practice. 3 If we observe the reward distribution to be light-tailed, this is a strong indication that error is light-tailed. 4To empirically test whether the reward is heavy-tailed, we consider two lines of evidence: examining the distributions directly through random sampling and temperature-1 sampling, and finding adversarial token sequences that get high rewards. We examine one small and one medium reward model that performed reasonably well on RewardBench (Lambert et al., 2023) . The small model is an OpenAssistant model based on Pythia 1.4B, and the medium model is Starling 7B-alpha (Zhu et al., 2023) 5 .For random sampling, we sample 30000 length-1024 sequences of uniformly random tokens and observe the distribution of rewards assigned by both Pythia 1.4B and Llama 7B-chat. We also use Llama 7B-chat to generate 16000 length-133 sequences at temperature 1 and observe the distribution of rewards assigned by Starling 7B-alpha.Because sampling is inefficient at probing the extreme tail, we also find token sequences that optimize Starling 7B-alpha for reward. We considered Greedy Coordinate Gradient (GCG) from (Zou et al., 2023) , a method used to find adversarial suffixes that circumvent jailbreaking, but decided on a faster version of GCG called Accelerated Coordinate Gradient (ACG) from (Haize Labs, 2024) . See Table 4 for ACG hyperparameters.Generating plots took about 5 GPU-hours on 1x Nvidia H100, and running ACG took a further 8 hours.When sampling token sequences, both the Pythia model on random inputs (Figure B.1) and Starling 7B-alpha on Llama-generated inputs (Figure 2 ) appear approximately normal and, therefore, lighttailed. Starling on random inputs (Figure 1 light-tailed distribution. Because Llama-7B-chat is a more reasonable base model than a completely random policy, we believe that Starling 7B-alpha is more likely to be light-tailed for the purposes of our theoretical results.The ACG results need some interpretation. The KL divergence between two distributions P and Q if P is the same as Q a fraction 1 -α of the time, but is some value x a fraction α of the time is given by D KL (P ∥Q) = [(1 -α)q(x) + α] log (1-α)q(x)+α q(x)+ (1 -α) log(1 -α)(1 -q(x)).When α is small but much larger than q(x), we approximate this to first order as D KL (P ∥Q) ≈ α log α q(x) . In Theorems 1 and 3, we prove that when the error is sufficiently heavy-tailed, a policy that gets extremely large reward a small fraction of the time will achieve high expected reward with low KL divergence. This is not the case here because the rewards achieved through ACG were small and the log-probabilities extremely negative. For example, a policy that matches Llama 2-chat's base reward 99% of the time and uses the highest-reward input generated by ACG α =1% of the time will have KL divergence from Llama 2-chat of α(log(α) -1339.70) = 13.35 nats, but reward only about α * (2.2377 -0.3329) = 0.02571 greater than the base model, far less than can be obtained with the same KL divergence by conditioning.5 Discussion and LimitationsQuestion: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Sampling rewards requires no hyperparameters, and hyperparameters are provided for ACG.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
how likely is catastrophic goodhart?: 
 The low-KL policies that result in catastrophic Goodhart are not a unique optimal policy, just one family of high-performing policies. When optimizing E[U (π)]-βD KL (π, π 0 ), the outcome depends on RL training dynamics; it could be that D KL → 0 causing catastrophic Goodhart, but more likely both terms will go to infinity, potentially allowing V → ∞. Catastrophic Goodhart can be prevented by using a light-tailed or bounded reward function.Even so, catastrophic Goodhart is likely to occur in many scenarios where KL regularization is naively employed in an attempt to avoid Goodhart's Law:• If we maximize σ(E[U ]) + D KL (T r(π)∥T r(π 0 )), where σ is a bounded function (e.g. sigmoid), all near-optimal policies will have V ≈ 0. Since we can only obtain so much reward from σ(E[U ]), it pays to make the KL (and thus V) go to zero. • If we cap KL to a finite value (or dynamically adjust the KL penalty to target a finite KL, as done in Ziegler et al. (2020) , then E[V ] is also upper bounded by a finite value (see Theorem 2), and we think it is likely that E[V ] ≈ 0. Consider a toy model where an AI can adjust three parameters: true quality V of responses, frequency of reward hacking (producing actions with extremely high X), and severity of hacking (value of X on those actions). Adjusting the policy to increase E[U ] without increasing KL increase the severity of hacking while decreasing either frequency of hacking or quality of responses. When E[U ] is already large, decreasing quality has much better returns than decreasing frequency. This is similar to Theorems 5, 6 about hard-threshold optimization.• Any way we maximizeE[U (π)] -βD KL (π, π 0 ) results in very large values of E[U (π)],and there are a number of arguments that extreme optimization for an imperfect proxy can result in decreased utility due to tradeoffs between X and V ; e.g., the constrained resource scenario in (Zhuang and Hadfield-Menell, 2021) .Theorems 1-3 do not require any independence assumption, but Theorems 4, 5, and 6 require that error X and utility V are independent, which seems to be violated in practice. Future work could weaken this assumption, although intuitively obvious ways to weaken it result in the statement being false.6 We did not search the entire space of token sequences, so we cannot rule out that the reward is heavy-tailed enough to cause catastrophic Goodhart in some situations. While it is intractable to search the more than 10 2000 possible token sequences, future work could get more evidence through more powerful optimization methods.In some cases, a heavy-tailed reward can be reparameterized to make it light-tailed and avoid catastrophic Goodhart; however, in settings where the true reward is heavy-tailed, making reward artificially light-tailed or bounded can result in unintended behavior.For example, a stock-trading agent should be rewarded by profit, but financial returns are known to be heavy-tailed. If we clip or otherwise transform rewards into a bounded interval, it will have no incentive to take into account huge gains or losses. Since RLHF rewards as implemented in Ziegler et al are unbounded, clipping or transforming rewards could itself cause reward misspecification.In some cases, e.g. when the reward is not the true intended one, it is possible to reparameterize the reward without adverse effects. In the RL literature for Atari games, rewards are changes in score clipped to [-1, 1] (Machado et al., 2018) . et al. (2023) found that optimizing the reward of small reward models causes overoptimization: a decrease in utility with increasing optimization. However, we observed that reward models are lighttailed, and (Theorem 4) that independence combined with light-tailed error prevents overoptimization.We think this discrepancy is explained by dependence between error and utility. Policies optimized for high error may activate features in the proxy reward models that are undesirable according to the true utility function. 7 More research is needed to understand why high-error completions have low utility and to design reward models that do not suffer from this problem; perhaps it is possible to construct reward models whose errors are in directions orthogonal to human preferences, so that the large-reward completions do not have lower utility.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Code will be provided in the supplemental material.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 We have argued that the purpose of the KL divergence regularization in RLHF is to mitigate reward misspecification. However, we have also proven that when errors in the reward function are heavytailed, it cannot serve this purpose: even with zero KL divergence, there are policies that achieve very high misspecified reward and no actual reward.When errors are light-tailed and independent, the KL divergence can mitigate misspecification, but when they are dependent, this may not be possible. Thus, we must look to places other than the KL objective to explain the current success of RLHF and ensure its continued success in the future. In this case, high values of X are upweighted to 1/t 0.8 ; upweighting them to 1/t would cause E[X] to converge to 1 while KL divergence goes to zero faster.Proof. WLOG let µQ = 0. We construct a sequence of distributions {Pt} such that limt→∞ EP t [X] ≥ c for any constant c, and limt→∞ DKL(Pt∥Q) = 0. We define Pt for any t > c thusly. Writing FP t (x) for the CDF P rX∼P t (X ≤ x) and FP t (x) for 1 -FP t (x), we letF _{P _t}(x) = 1 -1-c/t F Q (t) FQ(x) x ≤ t c/t FQ (t) FQ(x) x > tIntuitively, we rescale the part of the distribution to the right of t evenly to have total probability c/t, which is less than 1 because t > c.We must check that limt→∞ EP t [X] = c. We can writeEP t [X] = FP t (t)EP t [X|X ≤ t] + FP t (t)EP t [X|X > t] = FP t (t)EQ[X|X ≤ t] + FP t (t)EQ[X|X > t] = FQ(t)EQ[X|X ≤ t] + FQ(t)EQ[X|X > t]+ (FP t (t) -FQ(t))EQ[X|X ≤ t] + ( FP t (t) -FQ(t))EQ[X|X > t] = EQ[X] + ( FP t (t)-FQ(t))(EQ[X|X > t] -EQ[X|X ≤ t])We know that EQ[X|X > t] > t because it is an integral of values strictly greater than t. Because EQ[X] = 0 is a weighted average of EQ[X|X > t] and EQ[X|X ≤ t], and EQ[X|X > t] > 0, we know EQ[X|X ≤ t] < 0.So EQ[X|X > t] -EQ[X|X ≤ t] > t. We also know that for sufficiently large t, (FP t (t) -FQ(t)) > 0.Intuitively, starting from Q, which has mean 0, Pt moves a probability mass approaching c t from mean <0 to mean >t.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Other than hyperparameters, there are no details required to understand the results.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: The only error bars are inter-run variability of ACG. The standard deviation was reported rather than error bars due to the small number of runs. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: The experiments took minimal compute resources except H100 hours for ACG, and we report the number of GPU-hours used.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: The research conforms to all data-related concerns. No human subjects were involved, and we think the risk of harmful societal impact is minimal.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The immediate societal impacts are limited, but we discuss some potential applications to long-term safety.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: We do not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: No human subjects are involved.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: No human subjects are involved.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: lim t→∞ EP t [X] > lim t→∞ EQ[X] + ( FP t (t) -FQ(t))(t -0) = lim t→∞ c t -FQ(t) t = lim t→∞ c -t FQ(t)Because Q has a finite mean, limt→∞ t FQ(t) = 0, and so limt→∞EP t [X] ≥ c. Now we check that limt→∞ DKL(Pt∥Q) = 0:DKL(Pt∥Q) = R log Pt(dx) Q(dx) Pt(dx) = x≤t log Pt(dx) Q(dx) Pt(dx) + x>t log Pt(dx) Q(dx) Pt(dx) = FP t (t) log FP t (t) FQ(t) + FP t (t) log FP t (t) FQ(t)since both ratios are constant= FP t (t) log 1 -c/t FQ(t) + FP t (t) log FP t (t) FQ(t)Since both 1 -c/t and FQ(t) go to 1 as t → ∞, the left term goes to 0, and solim t→∞ DKL(Pt∥Q) ≤ 0 + lim t→∞ FP t (t) log FP t (t) FQ(t) = lim t→∞ c t log c t FQ(t) ≤ lim t→∞ c t log 1 FQ(t) = lim t→∞ - c t log FQ(t) since t>cQ is heavy-tailed, so by definition limt→∞ e at FQ(t) = ∞ for all a > 0. This implies that for every a > 0 there is a sufficiently large tc so that for all t > tc, FQ(x) > e -at , which means that log FQ(t) > -at.Therefore for every a > 0, limt→∞DKL(Pt∥Q) ≤ limt→∞ -c/t log FQ(t) < limt→∞ --act t = ac, which since KL divergence is nonnegative means thatlimt→∞ DKL(Pt∥Q) = 0 as desired. ■ A.2 Theorem 2 Restatement of Theorem 2. If V is light-tailed, EQ[V ] is zero, and d = DKL(P ∥Q) is bounded, then EP [V ] is bounded, and EP [V ] → 0 as d → 0.Proof. Using Lagrange multipliers, we find that when KL divergence is minimized, we have P (V )[λ1 log P (V ) Q(V ) + λ2 -X] = 0 for some constants λ1, λ2, soEQUATIONP (V ) = Q(V ) exp V -λ2 λ1 = Q(V ) (2) e V /λ_1 e -λ_2/λ_1 = CQ(V )e V /λ_1(3)That is, the new PDF is an exponential tilting of the old PDF. Now, what is Proof: We will exhibit a distribution of trajectories ρ such that DKL(ρ∥T r(π0)) < ϵ and E[G(ρ)] > M , and then construct a policy π with T r(π) = ρ. Note that this proof applies for continuous action spaces if trajectories are replaced with measurable sets, but this would make it harder to read.EP [V ]? It's just ∞ -∞ CV e V /λ 1 Q(X) dV . If the distribution of V is heavy-tailed distribution, this is ∞; if it is light-tailed,Let ρπ 0 = T r(π0). We have a heavy-tailed distribution of return Q ≜ G(ρπ 0 ) over R, so we can apply Theorem 1. But to define ρ, we can construct Pt in the proof of Theorem 1 in a particular way. For any t > c, we need a Pt that uniformly upweights values of mean return such that FP t (t) = c/t. We can define ρt such that any trajectory τ is upweighted by a factor depending only on its mean return:ρt(τ ) = 1-c/t F Q (t) ρπ 0 (τ ) g(τ ) ≤ t c/t FQ (t) ρπ 0 (τ ) g(τ ) > tThen we can let Pt ≜ G • ρt and the rest of the proof of Theorem 1 applies. Therefore, applying the theorem, we can let ρ = ρt for sufficiently large t, and then µG•ρ > M and DKL(G • ρ, G • ρπ 0 ) < ϵ. By the chain rule for KL divergence, DKL(ρ, ρπ 0 ) = DKL(G•ρ, G•ρπ 0 )+Eγ∼G•ρ[DKL(ρ(T )|G(T ) = γ ∥ ρπ 0 (T )|G(T ) = γ)].Since we constructed ρ so that the probabilities of each τ conditional on its return being γ are equal, the second term is zero, and we also have DKL(ρ, ρπ 0 ) < ϵ.Finally, since the KL divergence between trajectory distributions is the sum of KL divergence between policies at each action in the trajectory, and each trajectory has at least one action, E s∈T,T ∼T r(π) [DKL(π(s)∥π0(s))] ≤ E T ∼T r(π) s∈T [DKL(π(s)∥π0(s))] = DKL(ρ∥ρπ 0 ) < ϵ as desired.To define π such that T r(π) = ρ, we let π(s, a) = P r(ai = a|τ = (..., s, ai, ...) ∼ ρ).Then, the probability that any trajectory τ = (s1, a1, . . . , an) is sampled is:EQUATIONIn ( 2), returns are Markovian, so all trajectory prefixes ending in state s have the same distribution of returns under any policy. In the construction of ρ, all trajectories with the same mean return have equal measure. Therefore, conditioning on earlier states and actions of τ does not change the measure, so we can write (3). So T r(π) = ρ as desired. ■A.4 Theorem 4Restatement of Theorem 4. If U = X + V with X and V both light-tailed and V unbounded, and the distribution of U is continuous, andπ * (β) ≜ arg maxπ E[U (π)] -βDKL(π, π0), then lim β→0 + E[V (π * (β))] = ∞.Proof. Fix some β. Using Lagrange multipliers, we find that for any event S, Prπ(S) = Prπ 0 (S)e λU (S) . Let c(β) be the median value of U under the policy π * (β); that is,P r(U > c(β)|U ∼ G • T r(π * (β))) = 1 2. This exists because U has a continuous distribution. Then:E[V |π] = 1 2 E[V |π, U < c] + 1 2 E[V |π, U ≥ c] ≥ 1 2 E[V |π, U < c] + 1 2 E[V |π] lim β→0 + E[V |π] ≥ lim β→0 + 1 2 E[V |π, U < c] + lim β→0 + 1 2 E[V |π]The left term is c, while the right term is ∞, so the overall limit is ∞.A.5 Theorem 5Restatement of theorem 5. Let X and V be two independent random variables with CDFs FX and FV and tail functions FV ≜ 1 -FV , FX ≜ 1 -FX such that• V has a finite mean.• X is subexponential; that is, limx→∞ Pr(X 1 +X 2 >x) Pr(X>x)= 2 if X1, X2 are two independent samples from X. This is a slightly stronger property than being heavy-tailed.• The tail of V is sufficiently lighter than the tail of X that limt→∞ t p FV (t) FX (t) = 0 for some p > 1.Thenlimt→∞ E[V |X + V ≥ t] = E[V ]; that is, catastrophic Goodhart occurs in the limit of optimization for U = X + V .The proof requires expressing the conditional expectation in question as∞ -∞ vf V (v)Pr(X>t-v) ∞ -∞ f V (v)Pr(X>t-v), then partitioning the interval (-∞, ∞) into four regions and bounding the integrand in the numerator above by a different quantity in each region.In addition to the works cited in the main paper, we make reference to the textbook (Foss et al., 2013) throughout the proof. Many similar results about random variables are present in the textbook.Why its effect on E[V |c] is small Explanationr 1 = (-∞, -h(t)] P[V ∈ r 1 |c] is too lowIn this region, |V | > h(t) and X > t + h(t), both of which are unlikely.r 2 = (-h(t), h(t)) E[V |V ∈ r 2 , c] ≈ E[V |V ∈ r 2 ]The tail distribution of X is too flat to change the shape of V 's distribution within this region.r 3 = [h(t), t-h(t)] P [V ∈ r 3 | c]is low, and V < t. There are increasing returns to each bit of optimization for X, so it's unlikely that both X and V have moderate values. 9r 4 = (t -h(t), ∞) P[V ∈ r 4 | c] is too low X is heavier-tailed than V, so the condition that V > t -h(t) is much less likely than X > t -h(t)in r 2 . Table A .1: A summary of the proof strategy for Theorem 5. 8 and we divide the integral in the numerator into 4 regions, showing that each region's effect on the conditional expectation of V is similar to that of the corresponding region in the unconditional expectation E[V ].[V |X + V > t] is given by ∞ -∞ vf V (v)Pr(X>t-v) ∞ -∞ f V (v)Pr(X>t-v) ,The regions are defined in terms of a slow-growing function h(t) : R → R ≥0 such that the fiddly bounds on different pieces of the proof work out. Roughly, we want it to go to infinity so that |V | is likely to be less than h(t) in the limit, but grow slowly enough that the shape of V 's distribution within the interval [-h(t), h(t)] doesn't change much after conditioning.In Table A .5.1, we abbreviate the condition X + V > t as c.Note that up to a constant vertical shift of normalization, the green curve is the pointwise sum of the blue and orange curves.To be more precise, we're going to make the following definitions and assumptions:Let fV (v) be the PDF of V at the value v. We assume for convenience that fV exists, is integrable, etc, though we suspect that this isn't necessary, and that one could work through a similar proof just referring to the tails of V . We won't make this assumption for X. Let FX (x) = Pr(X ≤ x) and FX (x) = Pr(X > x), similarly for FV and FV . Assume that • V has a finite mean:∞ -∞ vfV (v) dv converges absolutely. • X is subexponential.Formally, this means that limx→∞ Pr(X 1 +X 2 >x)Pr(X>x)= 2. This occurs roughly whenever X has tails that are heavier than e -cx for any c and is reasonably well-behaved; counterexamples to the claim "long-tailed implies subexponential" exist, but they're nontrivial to exhibit. Examples of subexponential distributions include lognormal distributions, anything that decays like a power law, the Pareto distribution,and distributions with tails asymptotic to e -x a for any 0 < a < 1.We require for V that its tail function is substantially lighter than X's, namely that limt→∞ t p FV (t) FX (t) = 0 for some p > 1. (This implies that FV (t) = O( FX (t)/t).)With these definitions and assumptions, we can move on to the proof.The unnormalized PDF of V conditioned on X + V ≥ t is given by fV (v) FX (t -v). Its expectation is given by∞ -∞ vf V (v) FX (t-v) ∞ -∞ f V (v) FX (t-v) .Meanwhile, the unconditional expectation of V is given by ∞ -∞ vfV (v). We'd like to show that these two expectations are equal in the limit for large t. To do this, we'll introduceQ(v) = FX (t-v)FX (t) . (More pedantically, this should really be Qt(v), which we'll occasionally use where it's helpful to remember that this is a function of t.)For a given value of t, Q(v) is just a scaled version of FX (t -v), so the conditional expectation of V is given by ∞ -∞ vf V (v)Q(v) ∞ -∞ f V (v)Q(v). But because Q(0) = 1, the numerator and denominator of this fraction are (for small v ) close to the unconditional expectation and 1, respectively. We'll aim to show that for all ϵ > 0, we have for sufficiently large t that∞ -∞ vfV (v)Qt(v) - ∞ -∞ vfV (v) < ϵ and ∞ -∞ fV (v)Qt(v) ∈ [1 -ϵ, 1 + ϵ],which implies (exercise) that the two expectations have limiting difference zero. But first we need some lemmas.Lemma 1. There is h(t) depending on FX such that:(a) limx→∞ h(t) = ∞ (b) limt→∞ t -h(t) = ∞ (c) limt→∞ FX (t-h(t)) FX (t) = 1 (d) limt→∞ sup |v|≤h(t) |Q(v, t) -1| = 0.Proof. Lemma 2.19 from (Foss et al., 2013) implies that if X is long-tailed (which it is, because subexponential implies long-tailed), then there is h(t) such that condition (a) holds and FX is h-insensitive; by Proposition 2.20 we can take h such that h(t) ≤ t/2 for sufficiently large t, implying condition (b). Conditions (c) and (d) follow from being h-insensitive.Lemma 2. Suppose that FX is whole-line subexponential and h is chosen as in Lemma 1. Also suppose thatFV (t) = O( FX (t)/t). Then P r[X + V > t, V > h(t), X > h(t)] = o( FX (t)/t).Proof. This is a slight variation on lemma 3.8 from (Foss et al., 2013) , and follows from the proof of Lemma 2.37. Lemma 2.37 states that Lemma 2.37. Let h be any increasing function on R + such that h(x) → ∞. Then, for any distributions F1, F2, G1, and G2 on R,lim sup x→∞ P {ξ1 + η1 > x, ξ1 > h(x), η1 > h(x)} P {ξ2 + η2 > x, ξ2 > h(x), η2 > h(x)} ≤ lim sup x→∞ F1(x) F2(x) • lim sup x→∞ G1(x) G2(x) ,where ξ1, ξ2, η1, and η2 are independent random variables with respective distributions F1, F2, G1 and G2.but it is actually proved thatP {ξ1 + η1 > x, ξ1 > h(x), η1 > h(x)} ≤ sup z>h(x) F1(z) F2(z) • sup z>h(x) G1(z) G2(z) • P {ξ2 + η2 > x, ξ2 > h(x), η2 > h(x)}. (8)If we let F1 = FV , F2 = G1 = G2 = FX , then we getEQUATIONwhere X, X ′ ∼ FX . Multiplying by t, we haveEQUATIONand because h(t) → ∞ as t → ∞ and FV (t) = O( FX (t)/t), we can say that for some c < ∞, limt→∞ sup z>h(t)t FV (z) FX (z) < c. Therefore for sufficiently large tP {X + V > t, X > h(t), V > h(t)} ≤ c t P {X +X ′ > t, X > h(t), X ′ > h(t)}.By Theorem 3.6, P {X +X ′ > t, X > h(t), X ′ > h(t)} is o( FX (t)), so the LHS is o( FX (t)/t) as desired.We want to show, for arbitrary ϵ > 0, that∞ -∞ vfV (v)Q(v) - ∞ -∞ vfV (v) < ϵ in the limit as t → ∞. Since ∞ -∞ vfV (v)Q(v) - ∞ -∞ vfV (v) ≤ ∞ -∞ |vfV (v)(Q(v) -1)| = ∞ -∞ |v| • fV (v) • |Q(v)-1| it will suffice to show that the latter quantity is less than ϵ for large t.We're going to show that∞ -∞ |v| • fV (v) • |Q(v) -1|is small by showing that the integral gets arbitrarily small on each of four pieces: (-∞, -h(t)], (-h(t), h(t)), [h(t), t -h(t)], and (t -h(t), ∞).We'll handle these case by case (they'll get monotonically trickier).Region 1: (-∞, -h(t)] Since ∞ -∞ vfV (v) is absolutely convergent, for sufficiently large t we will have-h(t) -∞ |v|fV (v) < ϵ, since h(t) goes to infinity by Lemma 1(a). Since Q(v) is monotonically increasing and Q(0) = 1, we know that in this interval |Q(v) -1| = 1 -Q(v).-h(t) -∞ |v| • fV (v) • |Q(v) -1| = -h(t) -∞ |v|fV (v)(1 -Q(v)) < -h(t) -∞ |v|fV (v) < ϵ as desired.Region 2: (-h(t), h(t)) By lemma 1(d), h is such that for sufficiently large t,|Q(v) -1| < ϵ ∞ -∞ |v|f V (v)on the interval [-h(t), h(t)]. (Note that the value of this upper bound depends only on V and ϵ, not on t or h.) So we haveh(t) -h(t) |v|fV (v)|Q(v) -1| < ϵ ∞ -∞ |v|f V (v) h(t) -h(t) |v|fV (v) < ϵ ∞ -∞ |v|f V (v) ∞ -∞ |v|fV (v) = ϵ.Region 3: [h(t), t -h(t)] For the third part, we'd like to show that t-h(t)h(t) vfV (v)(Q(v) -1) < ϵ. Since t-h(t) h(t) vfV (v)(Q(v) -1) < t-h(t) h(t) tfV (v)Q(v) = t FX (t) t-h(t) h(t)fV (v) FX (t -v) it would suffice to show that the latter expression becomes less than ϵ for large t, or equivalently thatt-h(t) h(t) fV (v) FX (t -v) = o FX (t) t .The LHS in this expression is the unconditional probability that X + V > t and h(t) < V < t -h(t), but this event implies X + V > t, V > h(t), and X > h(t). So we can writet-h(t) h(t) fV (v) FX (t -v) = P r[X + V > t, h(t) < V < t -h(t)] < P r[X + V > t, V > h(t), X > h(t)] = o( FX (t)/t) by Lemma 2. Region 4: (t -h(t), ∞) For the fourth part, we'd like to show that ∞ t-h(t) vfV (v)Q(v) → 0 forlarge t. Since Q(v) = FX (t-v) FX (t) < 1 FX (t) , it would suffice to show ∞ t-h(t) vfV (v) = o( FX (t)). But note that since limt→∞ FX (t-h(t)) FX (t) = 1 by Lemma 1(c), this is equivalent to ∞ t-h(t) vfV (v) = o( FX (t -h(t))), which (by Lemma 1(b)) is equivalent to ∞ t vfV (v) = o( FX (t)). Note that ∞ t vfV (v) = t ∞ t fV (v) + ∞ t (v -t)fV (v) = t FV (t) + ∞ tFV (v), so it will suffice to show that both terms in this sum are o( FX (t)).The first term t FV (t) is o( FX (t)) because we assumed limt→∞ t p FV (t) FX (t) = 0 for some p > 1.For the second term, we have for the same reason∞ t FV (v) < ∞ t FX (v) v p = FX (t) ∞ t v -p = t 1-p p-1 FX (t) = o( FX (t)).For the denominator, we want to show that limt→∞∞ -∞ fV (v)Qt(v) = 1 = ∞ -∞ fV (v), so it'll suffice to show | ∞ -∞ fV (v)(Qt(v) -1)| = o(1) as t → ∞.Again, we'll break up this integral into pieces, though they'll be more straightforward than last time. We'll look at (-∞, -h(t)), [-h(t) , h(t)], and (h(t), ∞).• | -h(t) -∞ fV (v)(Q(v) -1)| = -h(t) -∞ fV (v)(1 -Q(v)) < -h(t) -∞ fV (v).-But since h(t) goes to infinity, this left tail of the integral will contain less and less of V 's probability mass as t increases.•| h(t) -h(t) fV (v)(Q(v) -1)| ≤ h(t) -h(t) fV (v)|Q(v) -1| • ≤ sup |v|≤h(t) |Q(v, t) -1| h(t)-h(t) fV (v) ≤ sup |v|≤h(t) |Q(v, t) -1| -By Lemma 1(d) we know that this goes to zero for large t.• | ∞ h(t) fV (v)(Q(v) -1)| = ∞ h(t) fV (v)(Q(v) -1) < ∞ h(t) fV (v)Q(v).But for sufficiently large t we have h(t) > 1, so we obtain 1) by the results of the previous section. This completes the proof.∞ h(t) fV (v)Q(v) < ∞ h(t) vfV (v)Q(v) < ∞ -∞ vfV (v)Q(v) = o(Restatement of theorem 6. Let X, V be independent random variables such that limt→∞ FX (t+1) FX (t) = 0. (This implies that X has tails that are dominated by e -cx for any c, though it's a slightly stronger claim because it requires that X not have large jumps in the decay of its tails.) Then for any V with a finite mean which has no upper bound, limt→∞E[V |X + V > t] = ∞.Theorem 6 generalizes a consequence of the "Regressional Goodhart Identity" in (Gao et al., 2023) .Proof. Let Pr(V > c + 1) = p > 0, which exists by our assumption that V is unbounded.Let E[V |V < c] = q. (If this is undefined because the conditional has probability 0, we'll have the desired result anyway since then V would always be at least c.)Observe that for all t, E[V |V < c, X + V > t] ≥ q (assuming it is defined), because we're conditioning (V |V < c) on an event which is more likely for larger v (since X and V are independent). First, let's see that limt→∞ P (V <c|X+V ≥t) P (V >c+1|X+V ≥t) = 0. This ratio of probabilities is equal to c-∞ f V (v) FX (t-v) ∞ c+1 f V (v) FX (t-v) ≤ c -∞ f V (v) FX (t-c) ∞ c+1 f V (v) FX (t-c-1) = FX (t-c) FX (t-c-1) • c -∞ f V (v) ∞ c+1 f V (v)= FX (t-c) FX (t-c-1) • Pr(V <c) Pr(V >c+1) ≤ FX (t-c) FX (t-c-1) • 1 p which, by our assumption that limt→∞ FX (t+1) FX (t) = 0, will get arbitrarily small as t increases for any positive p.Now, consider E[V |X + V ≥ t].We can break this up as the sum across outcomesZ of E[V |Z, X + V ≥ t] • Pr(Z|X + V ≥ t)for the three disjoint outcomes V < c, c ≤ V ≤ c + 1, and V > c + 1. Note that we can lower bound these expectations by q, c, c + 1 respectively. But then once t is large enough that Pr(V <c|X+V ≥t)Pr(V >c+1|X+V ≥t) < 1 c-q , this weighted sum of conditional expectations will add to more than c. We created a synthetic experiment by letting reward U = X + V , where X and V are independent and sampled from different probability distributions, consistent with our theoretical assumptions. We vary N from 1 to 65536, do 100 trials of taking the best-of-N sample with highest U , and note whether V goes towards 0 (overoptimization) or not.Possible distributions for V are normal and t-distribution with df=10. Possible distributions for X are normal, t with df=3, t with df=5, lognormal, and Levy. (All of these heavy-tailed except for the normal distribution.) V is scaled to a standard deviation of 2 and X has s.d. of 1 (except for the Levy distribution, which has infinite variance), representing that in ordinary regimes most of the variance comes from utility rather than error.When the error X is normal and thus light-tailed, V increases monotonically with N , consistent with our Theorem 6. However, when both X and V are heavy-tailed, we see results consistent with theorem 5. In 5 of 6 cases when X is lognormal or student-t, V first increases then starts to decline around N = 10 2 or 10 3 . When X is (t, df=5) and V is (t, df=10), V instead peaks around N = 10 5 (but declines afterwards). Finally, when X is Levy-distributed, utility never goes significantly above zero (optimization completely fails) because the Levy distribution is too heavy-tailed.The results are shown in Figure B .3. Briefly, the results are consistent with the asymptotic results in theorems 5 and 6, showing that overoptimizationIn this experiment, we examined PPO with artificially heavy-tailed rewards to see if catastrophic Goodhart could be observed.OpenRLHF (Hu et al., 2024) was used to train Pythia 1B with a reward model derived also from Pythia 1B, on the default OpenRLHF prompt dataset. We used the reward model to represent true utility, and a heavy-tailed error term based on the number of "the" tokens was added to get the proxy reward. The kl_target=0.5 option was used to dynamically adjust KL penalty, as we mention is done in Ziegler et al. (2020) . Rewards were not clipped. (Reward clipping can be useful to prevent overoptimization, but is not always used in PPO.) Response length was limited to 256.Two example completions are in figure B.4. Midway through training, the model generates reasonable text with a higher number of "the" tokens than usual, but as training progresses the model starts to generate completions with a very high number of "the" tokens while proxy reward reaches values > 10 9 , demonstrating that the basic pattern of catastrophic Goodhart can occur in RLHF under conditions of heavy-tailed error.The preparation of polyarylethylene which is either employed for the purpose of applications in the petroleum refining or in the production of the well-known automobile oil lubes is carried out by the so-called preparation by the salt bath method described... in 1980 by D. W. Perkins in the United States. This method presupposes the usage of the salt bath as the medium for the preparation of the polyarylediene-the method of preparation of the crude oil base by the process which consists in letting it be saturated up with a particular the A mirror, the essence of the understanding of the given the conclusion that the manifestation in the form of the gender and the identification the solutions related to different fields of the modern utilization of the the functionality of the the the the modification of the the the the the the state of the the the the the the the the the the the employment and the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the and the the the the the the the the the the the the the the the the the the the the by the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the and the the the the the the the the the the the the the the the the the The result depends on hyperparameters (e.g. reward clipping would prevent this), so our observation should not be taken as a claim that catastrophic Goodhart is inevitable in all RLHF settings with heavy-tailed error.We use three models for our experiments: Starling 7B-alpha, Llama 2 7B-chat, and Pythia-1.4B. Starling was developed by Berkeley, and Pythia by EleutherAI. Starling and Pythia models are licensed under Apache-2.0.10 11 Llama 2 models were developed by Meta and licensed under a license published by Meta. 12The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:• You should answer [Yes] , [No] , or [NA] .• [NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.• Please provide a short (1-2 sentence) justification right after your answer (even for NA).The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.IMPORTANT, please:• Delete this instruction block, but keep the section heading "NeurIPS paper checklist",• Keep the checklist subsection headings, questions/answers and guidelines below.• Do not modify the questions and only use the provided macros for your answers.• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.This could model a satisficing agent that takes random acceptable actions.Note that distributions over a finite set are bounded and cannot be heavy-tailed in a technical sense, and models with a finite context window have a finite input space. For our purposes, a distribution of reward or error is effectively heavy-tailed if it takes on sufficiently large values and is well-modeled by a heavy-tailed distribution on its support.It is possible for U to be light-tailed while X and V are both heavy-tailed, but this is unusual and we do not expect it to happen in practice.https://huggingface.co/berkeley-nest/Starling-RM-7B-alphaSuppose that error X is light-tailed conditional on any value of V , but our proxy is merely unbiased (E[X|V = v] = 0 for all v). Then the limit of V under optimization for X + V still depends on the relationship between X and V . If they are independent, Theorem 6 says that limt→∞E[V |X + V ≥ t] = ∞. But if V ∼ N (0, 1), and X|V ∼ N (0, 4) when V ∈ [-1, 1], otherwise X = 0, then limt→∞ E[V |X + V ≥ t] = 0.There are other explanations possible. Perhaps better optimization methods would find heavy-tailed reward in open reward models; or OpenAI's reward models have heavy-tailed error (and their results are straightforwardly explained by our Theorem 1), while open reward models have light-tailed error.We'll generally omit dx and dv terms in the interests of compactness and conciseness; the implied differentials should be pretty clear.https://huggingface.co/berkeley-nest/Starling-RM-7B-alphahttps://huggingface.co/EleutherAI/pythia-1.4bhttps://ai.meta.com/llama/license/ 


Decision: This paper studies whether KL regularization is sufficient to mitigate the (potentially-catastrophic) errors due to reward-misspecification in RL.
They prove that KL regularization works under a fairly natural assumption on the type of reward mis-specification (independent noise with light-tails). Conversely, they prove that even KL-regularization can be catastrophic if this assumption is broken (ie if noise is heavy-tailed).

Reviewers agreed that the work presents a clean and novel theoretical insight to the important problem of reward misspecification.
There were several concerns about clarity and experimental support, which the authors addressed in their rebuttal. After the rebuttal, all reviewers recommended acceptance. I thus recommend acceptance.

Reviews: review: paper_summary: paper_summary: The paper first considers a theoretical stylized model of jointly distributed utility and (mis-specified) rewards and proves a novel result that for any heavy tailed reference distribution, it is possible to find another distribution that arbitrarily approximates it in terms of the KL divergence, yet has an unbounded reward. Beginning with this basic result, the authors then attempt to apply these insights to the popular technique of regularizing the policy optimization procedure in RLHF using a learned reward model with the KL divergence to the base SFT policy. Experiments on open source LLM reward models are conducted to investigate how closely the assumptions made in the theory match with current state of the art models.
strength_weakness: strength_weakness: - The clarity in some sections can be improved -- see questions below.
- The notion of heavy tailed versus light tailed, while theoretically appealing and convenient, seems quite sensitive to parameterization and/or scale in practice. While the authors do mention this briefly in footnote 2 (Page 6), it appears that simple reward reparametrization can transform the resulting distributions in a way that is not robust to the notion of heavy/light tailedness.
questions: questions: - The definition of DMRMDP in Section 4 is unclear. Is there any precedent to this particular definition? For instance, are you always required to end a trajectory in one of the sink states? If not, are rewards also defined for partial trajectories?
- Is the converse of Theorem 1 also true? i.e. if Q is not heavy tailed, does the maximum achievable mean stay bounded as $\epsilon \rightarrow 0$?
- In the experimental results section, random sampling and adversarial sampling are used to generate the reward samples which are estimated from another model. However, it is hard to know the true utility, unlike in the case of Gao et. al. (2023), who perform a synthetic study with known controlled true utilities. Since the paper's insights and results are more conceptual than practical at the moment,  wouldn't it be interesting to study a synthetic experimental setup similar to Gao et. al. where both true utilities and proxy utilities can be measured explicitly?
- When you speak of a "heavy/light tail under a given policy $\pi$, one possibility is to consider the action conditioned expected utility, or the unconditional version where action is simply a latent variable that determines the final reward. Is this distinction significant for any of the results? If so, which one is being assumed, and what are the implications for the alternate version on the basic result?   


Nits: 

- Figure reference in L110 seems like a typo.
- Figure 1 has fonts which are too small to read.
limitations: limitations: Yes
paper_summary: paper_summary: This paper investigates the effectiveness of using KL divergence for regularization in reinforcement learning from human feedback (RLHF), particularly when dealing with reward functions that have misspecified errors. It introduces the concept of "catastrophic Goodhart," a scenario where policies can achieve extremely high rewards with heavy-tailed errors, but without actual improvement in utility.
strength_weakness: strength_weakness: [writing / presentation] The paper contributes to the field of LLM safety/alignment, yet the writing could be made more engaging and accessible. Improving clarity, enhancing the depth of discussions, and ensuring visual and structural appeal could greatly increase its impact and readability. These changes would not only cater to experts in the field but also to readers who may be new to this area of study.

[Experiments] the experimental evidence could be further bolstered to validate the presented claims comprehensively. Specifically:

- The experiments primarily focus on simulated environments. Including real-world applications could demonstrate the practical implications and limitations of the findings more effectively.

- Exploring a wider variety of reward functions, particularly those commonly used in commercial or industrial settings, could help ascertain the generalizability of the catastrophic Goodhart phenomenon across different contexts.

- Additional experiments comparing KL divergence with other regularization or penalty methods could offer a clearer understanding of its relative efficacy and limitations.

- How about other optimization methods like Best-of-N where KL divergence term is not used? 

- Will the implementation like LoRA affect the proposed effect?
questions: questions: Please see the weakness section
limitations: limitations: please see weakness
paper_summary: paper_summary: This paper analyzes a phenomenon called "catastrophic Goodhart": in RL training, suppose the learned reward function is the sum of the true utility function and some noise, then if (1) the true utility and the noise are independent and light-tailed, maximizing reward with KL regularization can also give high utility, while if (2) the noise is heavy-tailed, then there exists a policy with arbitrarily high reward and arbitrarily small KL penalty, but its true utility is close to the initial policy. Empirical results suggest that open-source reward models could be light-tailed.
strength_weakness: strength_weakness: One missing piece is the implicit regularization effects of the optimization algorithms: although there exists a policy with low utility even under a KL penalty, probably the training algorithm can avoid those bad policies and still find a good one. It will be nice if we can understand whether certain optimization techniques can solve this issue.

Some additional comments:
1. The notation $U$ and $V$ are used inconsistently in the paper: on line 34, $U$ denotes the true utility and $V$ denotes the proxy reward, while on line 71 their meanings are reversed.
questions: questions: I have one question regarding Theorem 4: the final conclusion is that the expected utility can be arbitrarily large, but if the utility function is bounded, I don't understand why its expectation can be arbitrarily large.

I also have a more open-ended question: if we make the proxy reward function bounded (e.g., by clipping), do we have similar results?
limitations: limitations: N/A
paper_summary: paper_summary: The authors presents a study on the RM in RLHF. Theoretical results of the paper assume that the reward from the RM consists of a true reward plus a noise term which is modeled as a random variable. Under this model the authors show that

- If the noise terms are heavy-tailed there exists policies with vanishing KL-divergence but infinite proxy reward.
- If the noise terms are IID and light-tailed, then KL divergence suffices to guarantee good outcomes

The authors also investigate the actual rewards given by real RMs. They find that the rewards on random sequences are roughly
strength_weakness: strength_weakness: - there is probably some discrepancy between the theory and practice. The results are asymptotic, it would be better if the statements could hold in non asymptotic regimes. 
- The experimental results seems to show that the rewards are not heavy tailed, and thus the theory seems to suggest that KL-divergence should work well. We already know that it does work well from practice, so there is no novel take-home message.
questions: questions: - The theoretical results are related to asymptotically heavy tails. What would happen in I take my RM and clip the output to be in [-10000, +10000]. I’m guessing in practice nothing would happen, but would the theoretical results would be invalid since the tails disappear?

- In practice the output of the RM isn’t a random variable, it’s deterministic. Can you comment on the validity of the assumption that the rewards are random?
limitations: limitations: na

Comments: title: PPO results
Clarification on bounded case
Best-of-N results
Acknowledgement of rebuttal
thanks for clarify
comment: In addition to the best-of-N results in the comment above, we examined PPO with artificially heavy-tailed rewards.
* We used OpenRLHF to train Pythia 1B with a reward model derived also from Pythia 1B, on the default OpenRLHF prompt dataset.
* We used the reward model to represent true utility, and a heavy-tailed error term based on the number of "the" tokens was added to get the proxy reward.
* The kl_target=0.5 option was used to dynamically adjust KL penalty, as we mention is done in Ziegler et al (2020).
* Rewards were not clipped. (Reward clipping can be useful to prevent overoptimization, but is not always used in PPO, see point 1 of our rebuttal to G3eM)
* Response length was limited to 256. Other hyperparameters were unremarkable but can be provided if needed.

Results:
* Initially the policy balanced maximizing error and utility, generating completions like `the suggestions are shown in the images of the Progress Spinner of the completed list view.` (It's only 1B so not very coherent)
* Later in training the policy achieved extremely large values of reward with similar or lower KL divergence by generating normal text followed by long strings of repeated "the" tokens. (Due to the negative correlation, utility became negative. Note theorems 2 and 3 are still relevant because they have no independence assumptions)

We think this validates that the basic pattern of catastrophic Goodhart can occur in RLHF under conditions of heavy-tailed error. Limitations include the artificial nature of the reward and small size of the models, but because the theorems are very generalizable facts, we expect the same to hold in other conditions, and we hope this has addressed your concerns.
We wanted to add some information about the bounded proxy reward case:

* A correction: above, we said the optimal policy was Boltzmann rational, but in addition to the exponential term it still has the base likelihood.
* The best-of-N results from the comment to ngHC look very similar when reward is clipped to [-100,100], though in that case utility will increase again for very large N, and also when noise is Levy-distributed. When reward is clipped to [-10, 10], we no longer see overoptimization if utility is heavy-tailed.
The best-of-n experiment results are as follows:

We created a synthetic experiment by letting reward U = X + V, where X and V are independent and sampled from different probability distributions, consistent with our theoretical assumptions. We vary N from 1 to 65536, do 100 trials of taking the best-of-N sample with highest U, and note whether V goes towards 0 (overoptimization) or not.
* Possible distributions for V are normal and t-distribution with df=10.
* Possible distributions for X are normal, t with df=3, t with df=5, lognormal, and Levy. All of these are heavy-tailed except for normal.
* V is scaled to a standard deviation of 2 and X has s.d. of 1 (except for the Levy distribution, which has infinite variance), representing that in ordinary regimes most of the variance comes from utility rather than error.

Results (See image at: https://imgur.com/a/mo02KET )
* When the error X is normal and thus light-tailed, V increases monotonically with N, consistent with our Theorem 6.
* In 5 of 6 cases when X is lognormal or student-t, V first increases then starts to decline around N=10^2 or 10^3. When X is (t, df=5) and V is (t, df=10), V instead peaks around N=65536 but declines by N=1048576. This is consistent with Theorem 5.
* When X is Levy-distributed, utility never goes significantly above zero because Levy distribution is too heavy-tailed. In this scenario optimization completely fails.

RLHF experiments are still in progress and we will share results soon. We will be using Llama3 on OpenRLHF and adding noise to an open reward model which we deem "true".
Thank you for the response and the clarifications. I have update my initial review's score accordingly.
I appreciate the authors' detailed response. Can the authors please also provide the results they promised to provide?