Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification
Authors: Thomas Kwa
Drake Thomas
Adrià Garriga-Alonso
Venue: NeurIPS 2024 poster (2024)
Keywords: RLHF
extreme value theory
reward misspecification
Primary area: safety_in_machine_learning
TL;DR: KL regularization doesn't guarantee good outcomes in RLHF.

Abstract: When applying reinforcement learning from human feedback (RLHF), the reward is learned from data and, therefore, always has some error. It is common to mitigate this by regularizing the policy with KL divergence from a base model, with the hope that balancing reward with regularization will achieve desirable outcomes despite this reward misspecification. We show that when the reward function has light-tailed error, optimal policies under less restrictive KL penalties achieve arbitrarily high utility. However, if error is heavy-tailed, some policies obtain arbitrarily high reward despite achieving no more utility than the base model--a phenomenon we call catastrophic Goodhart. We adapt a discrete optimization method to measure the tails of reward models, finding that they are consistent with light-tailed error. However, the pervasiveness of heavy-tailed distributions in many real-world applications indicates that future sources of RL reward could have heavy-tailed error, increasing the likelihood of reward hacking even with KL regularization.

Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification
Thomas Kwa Independent FAR Labs kwath kwathomasO0gmail. com
Drake Thomas* Anthropic wrakeeanttropicc.om
Adrià Garriga-Alonso
FAR AI adria@far.al
Abstract
When applying reinforcement learning from human feedback (RLHF), the reward is learned from data and, therefore, always has some error. It is common mitigate this by regularizing the policy with KI. divergence from base model, with the hope that balancing reward with regularization will achieve desirable outcomes despite this reward misspecification. We show that when the reward function has light-tailed error, optimal policies under less restrictive KL penalties achieve arbitrarily high utility. However, error is heavy-tailed, some policies obtain arbitrarily high reward despite achieving more utility than the base model phenomenon we call catastrophic Goodhart. We adapt a discrete optimization method to measure the tails of reward models, finding that they consistent with light-tailed error. However, the pervasiveness of heavy-tailed distributions in many real-world applications indicates that future sources of RI reward could have heavy-tailed error, increasing the likelihood of reward hacking even with KI. regularization.
Introduction
Kullback-Leibe (KL) divergence constraints in reinforcement learning (RL) are employed to stay in regimes where the objective accurate enough. Some on on-policy (Schulman al. 2015 2017) and many off-policy (Abdolmaleki et al., 2018 Jaques et al., 2019) policy gradient a algorithms employ KI. constraints or penalties during optimization prevent the policy from deviating too much from the data collection distribution. This ensures that estimates of each action's 's advantage are reliable enough to update the policy in a helpful way.
Reinforcement learning from human feedback (Christiano et al. 2017 Ziegler et al. 2020 RLHF) is very popular method induce desirable behavior in language models. RLHF starts with a base pre-trained model, then learns reward function from human annotator data. Next, it trains an RL policy to maximize this reward, while penalizing high KL divergence from the policy the base model. RLHF uses on on-policy algorithm and has accurate advantages, but the reward function is always somewhat misspecified compared to desired behavior, due insufficient data, human biases, and other factors.
The main purpose of the KL penalty in RLHF is limit the consequence of reward modeling errors by keeping the policy within a distribution similar to that on which it was trained. Ideally, in the low-KL regime the reward model's errors are small enough that it provides correct updates the base model. Gao et al. (2023) empirically supports this view: if the KL divergence in RLHF is allowed grow too much, with misspecified reward, the model's performance on the true utility starts to decrease.
Work was done while not at Anthropic
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

We ask: can we obtain good outcomes from misspecified reward in RLHF by controlling the KL divergence? That is, if there is some error between the reward V and the proxy reward U, can the KI. help us to still optimize V2? Using mathematical proof, we answer the question in the negative for heavy-tailed errors: there exist policies which have infinite proxy reward U, but whose KI. with the base model vanishes (these have undetermined V). We term this phenomenon "catastrophic Goodhart", after Goodhart's law.
If the misspecification errors are independent and light-tailed, the KL divergence does suffice to guarantee good outcomes. There may also be guarantees under weaker assumptions, but assumptions that intuitively seem sufficient are often not (see Section 5)
Possibly, other regularization schemes would guarantee good outcomes for heavy-tailed errors, but this is not just problem of KI.. We show that optimizing by conditioning on large reward T has similar outcomes in light- and heavy-tailed regimes.
Empirically, open-source language reward models seem to be light-tailed, which does not imply light-tailed errors but suggests it (Section 4.1). However, the errors likely not independent and, given the prevalence of heavy-tailed distributions the real world, error in future reward models may also be heavy-tailed. In any case, the present success of RLHF with misspecified rewards cannot be explained solely by the KL regularization in its objective.
Background
2.1 KL divergence and KL regularization
Recall that KL divergence between two distributions P and Q is defined as
P(x)
DKL(P)1Q) = Σ P(x) log
Q(x)
TEX
If we have two policies т, To, we define DxL (T To) as the KL divergence between the distributions of actions taken on the states in trajectories reached by æ. That is, Tr(T)) the distribution of trajectories taken by æ, we penalize (TT(TT & EseT, DDr. (T ([)()))))
In RLHF, it is common to the regularization term BDKL (=|[ to prevent the learned policy from deviating too much from the base policy, which can prevent unstable behavior or overfitting to the reward model. If our reward model gives reward U, then the optimal policy for RLHF with KI. penalty is
arg max [[U( 3D (T||=T).
Often the regularization parameter is dynamically adjusted to keep the DEL near target value (Ziegler et al. 2020).
2.2 Heavy-tailed distributions
A distribution P over R with cumulative distribution function (CDF) Fp is heavy-tailed if its tail function Fr(x) A 1 Fz(z) satisfies
lim e ((=((x)= 00 for allt > 0.
Heavy-tailed distributions are well-known in statistics to have higher probability of producing a single extreme value. For example, if the sum of two independent variables from heavy-tailes distributions is large, it is most likely due to one extreme sample rather than two equally large samples. (Wierman 2013)
2.3 Reward misspecification and Goodhart's Law
Reward misspecification has caused low-utility outcomes in practice; for example, in (Clark and Amodei 2016), an RI. agent trained to play a racing videogame according to a misspecified reward function achieves high score while failing complete the course.

---

Gao et al. (2023) introduce the concept of "overoptimization". optimizing for a proxy objective decreases performance according the true objective. This raises the question: in general, when RLHF reward is misspecified, when does the optimal policy produce high utility?
By applying the proxy reward and true reward functions to distribution LLM), we get two scalar random variables, which we call U for proxy reward and V for true reward / utility. Then we can define the error in the proxy reward as X - U - VV so that U = X + V. Framed this way, optimization for proxy reward U/ is mix of desirable optimization for V and undesirable optimization for X. The joint distribution of V and X determines the limiting value of V as we apply more optimization. When we say that reward misspecification can have negative effects, we mean that too much variance in X can "redirect" the optimization pressure from V to X, and prevent utility gain from optimization.
text (generated by an
Reward misspecification is also studied by (Lambert and Calandra 2024), Laidlaw et al. 2024). and others. Laidlaw al show that a KL penalty between action distributions can be ineffective, and propose instead regularizing state occupancy measure. Our results show an inherent weakness of KL divergence, including when applied state occupancy measure.
We prove that in many cases, V - in the limit of optimization for some proxy U. We call this phenomenon "catastrophic Goodhart", after Goodhart's 's law: "when a measure becomes a target, it ceases to be a good measure" (Strathern) 1997). In these cases, the end result of optimizing for a proxy of V is no better than not optimizing at all. However, in other cases, V + oo despite some reward misspecification; in these cases the reward misspecification is not severe enough to prevent good outcomes.
3 Theoretical results
When applying KL regularization, the trained model is regularized towards some base policy xo- One would hope that a KL penalty produce good outcomes even in the case of reward misspecification; that is, if the reward U is the sum of true utility V and an error term X, we would hope that optimal policies under KL penalty achieve high V even if the magnitude of X is large. We show that this is not always the case: Corollary of Theorems 3 and 2 establishes that when X (mo) is heavy-tailed there are arbitrarily well-performing policies ग with E, [V] Rs E.. [V] However, Theorem 4 shows that when error is light-tailed and independent of V, the optimal policy under a KI. penalty results in V >>> 0, and V can be made arbitrarily large. Thus, the tails of the error distribution are crucial in determining how much utility will result from optimization towards an imperfect proxy.
Theorems 5 6 (Section 3.4) show that the relationship of catastrophic Goodhart heavy-tailed error is not just a quirk of KL divergence by using a different model of optimization based on conditioning high reward values. Under this model (and given additional regularity conditions), is also true that heavy-tailes error results in catastrophic Goodhart and light-tailed error plus independence results in arbitrarily large utility. All proofs in the appendix.
3.1 KL divergence on heavy- and light-tailed distributions
Theorem Given any heavy-tailed reference distribution Q over R. with mean pq. and any M, > there is distribution P with mean p& > M and D D (Pl() <c
Outline of proof (see appendix for full proof): WLOG take μı 0. we set P. to upweight the probability mass of Prp, (X > t) to c/t for some c, t, then the mean of Pt will be approximately at least c. As oo, the KL divergence DrL (P.llQ) will shrink to zero.
Intuitively, in a heavy-tailed distribution, events with extremely high x are not very rare, so you don't pay much of a KL penalty to upweight them so they happen about 1/x of the time.
Theorem 2. However, if the distribution Q is light-tailed and = Dx (PlIQ) is bounded, then p.p bounded, and μp - pq
d O.
3.2 RLHF with KL penalty under heavy-tailed return distribution
We now adapt our result to the case where the policy is language model and we are training it using RLHF. We are now applying KL divergence over the policies rather than the return distributions.
3

---

We first formally define the properties of RLHF on language models that cause the result to hold: namely, when when considered as a Markov decision process (MDP), environmental transitions are deterministic and return depends only on the final state reached.
Definition: A deterministic-transitie MDP with Markovian returns (DMRMDPP) is MDP (S,A, Rt) such that:
The transition function P: 8 x A A
+ S deterministic, i.e., for each state s E S and action a E A. there exists a unique state s' E S such that P(s'|s, a) 1.
In RLHF: the transition is appending the generated token to the context
There is set of sink states E' S S that terminate every trajectory, which is disjoint from the set of start states.
In RLHF: The sink states are sequences ending in <EDS> or above a certain length.
Returns are Markovian; that is, for any two trajectories = (8,, 01,............ "n),7' (s1, (..-..,,%)) if 8n Sy2, then т and 7' have identical return distributions. Equiva- lently, for the trajectory random variable T (Si, ......) distributed according any policy, with return G, GJL (S Aci) S; for any 2 1.
In RLHFF: the return only depends the full generated string, which is the final state.
The language model stochastically outputs the next token given and corresponds to the policy. A DMRMDP is therefore good model RLHF.
Theorem Let W = AS, P, R) be deterministicc-ransiittii MDP with Markovian returns. Given W we define the function that takes policies trajectories Tr (S - AA) A x A)", and the average return function g (S A) - R. which induces a function
A x A) + AR. Let To S - AA be some base policy. If G a Tr(Te) is heavy-tailed with finite mean uQ. then for any M, > O, there is a policy ι with mean return TUU ~ Go G > M and (TT(TT-T((s)) DKL(TTS (=(s)][=o(s))
Corollary 1. Theorems Dande iiply that when utility is light-tailed, reward modeling errors make the proxy reward heavy-tailed, and a policy a is regularized severely enough to have KL. divergence values approaching zero, the reward [EU(() (rr) can go to infinity while utility E approaches value no higher than the base policy.
3.3
Light-tailed + independence imply [[[] + 20
Theorem 4. IFU X-V with X and V both light-tailed and unbounded, and the distribution of continuous, and xt xB) A arg max, EEU (T) (D Ro), then lim E[V (T* (3))] oc.
3.4 Conditioning as alternate model of optimization
Although we think KL divergence penalty or cap is the most realistic setting for RLHF, it is not the only model of optimization where heavy-tailedness of the error determines whether catastrophic Goodhart occurs. Consider another model of optimization where U X + V as before, but we simply condition on being higher than some threshold ( Then we are interested in the quantity lim,-.o E[V]X V 2 t). If we slightly strengthen the haivy-tailedness and light- highl-tailedness assumptions, heavy- error results in catastrophic Goodhart, while light-tailed error results in arbitrarily high expected utility.
3.4.1 Conditioning with heavy-tailed error produces catastrophic Goodhart
Theorem 5. Let X and V be two independent random variables with CDFs Fx and Fy and tail functions Fy - Fv. Fx & Fx such that
has a finite mean.
PriX X2 X is subexponential; that is, limg-400 Pr. 24) samples from X. This is a slightly stronger property than being heavy-tailed.
2 if X1, X2 are two independent
This could model satisficing agent that takes random acceptable actions.

---

The tail of V is sufficiently lighter than the tail of X that lime p p>1 1.
(" Fx( 0 for some
Then lim X + 2 t = EV; that is, catastrophic Goodhart occurs in the limit of optimization for U X
The proof is included in the appendix. It requires expressing the conditional expectation in question _ (fv(()PP(X2t--) as J_ [x then partitioning the interval (-o0, oo) into four regions and bounding the integrand in the numerator above by different quantity in each region.
3.4.2 Conditioning with light-tailed error produces arbitrarily high utility
Theorem Let X,V be independent random variables such that lime-roo Px
(This
(x)
implies that X has tails that dominated by e e- for any though it's slightly stronger claim because requires that X not have large jumps in the decay of its tails.) Then for any V with a finite
mean which has upper bound, lim B V > DC.
Theorem 6 generalizes a consequence of the "Regressional Goodhart Identity" in (ao et al. 2023)
4 Experiments
Our theoretical results now raise the question of whether the error in reward models heavy-tailed light-tailed in practice. F] If we observe the reward distribution be light-tailed, this is strong indication that error is light-tailed.
To empirically test whether the reward is heavy-tailed, consider two lines of evidence: exam- ining the distributions directly through random sampling and temperature-1 sampling, and finding adversarial token sequences that get high rewards. We examine one small and one medium reward
model that performed reasonably well on RewardBench (Lambert al. 2023). The small model an Ossnssstttant model based Pythia 1.4B, and the medium model is Starling 7B-alpha (Zhu
ct al. 2023
For random sampling, sample 30000 length- 1024 sequences of uniformly random tokens and
observe the distribution of rewards assigned by both Pythia 1.4B and Llama 7B-chat. We also use Llama 7B-chat to generate 16000 length- 133 sequences at temperature and observe the distribution of rewards assigned by Starling 7B-alpha.
Because sampling is inefficient at probing the extreme tail, we also find token sequences that optimize Starling 7B-alpha for reward. We considered Greedy Coordinate Gradient (GCG) from (Zou et al.) 2023), a method used to find adversarial suffixes that cireumvent jailbreaking, but decided on a faster version of GCG called Accelerated Coordinate Gradient (ACG) from (Haize Labs 2024). See Table 4 for ACG hyperparameters.
Generating plots took about 5 GPU-hours on 1x Nvidia H100, and running ACG took further 8 hours.
4.1 Results
When sampling token sequences, both the Pythia model random inputs (Figure B. and Starling 7B-alpha on Jaama-geeerated inputs (Figure 2 appear approximately normal and, therefore, light- tailed. Starling on random inputs (Figure is ambiguous, with the exponential Q-Q plot having an outlier that could indicate heavy-tailed distribution, but the Hill estimator is consistent with
Nooe that distributions models finite context window have finite input space. our purposes, a distribution of reward or error is effectively heavy-tailed if it takes on sufficiently large values and is well-modeled by a heavy- distribution on its support.
finite
bounded and cannot he heavy-tailed in technical sense, and
U to be light-tailed while and V are both heavy-tailed, but this unusual we not expect to happen in practice.
'It is possible
hhtps://uugiiggfcc http:://huggigggeee.co/oeeeeeeet Raaes//tttarin--M---B-- RM-7B-alpha
5

---

Parameter
Value
Context length
133
Iterations
1000
Candidates per seq- position (k)
3
Annealing starting value
9
Annealing ending value
2
Table I: Hyperparameters for ACG
Plots reward from 30000 random token sequences to Starling alpha reward model
Histogram
Normal probability plot
2000
1750
frequency
1250
reward
-
1000
750
-3
500
-4-
-5
-5
-4
Reward
Exponential probability plot
Hill Estimator for Heavy-tailed Distribution
0.40
0.5
0.0
0.30
Hill estimator
-100
0.20
-15
-20
0.05
0
10
25
50
100
125
150
175
Exponential quantile
Number upper order statistics
Figure 1: Plots of the distribution of reward from 30000 random length- 1024 token sequences to Starling 7B 7B-alph.. Clockwise from top left: The histogram shows a unimodal distribution with slight right skew. The normal probability plot indicates the data are heavier- than normal. The Hill estimator (error bars are standard error) appears to 0.20 for higher values but fluctuates for lower values. The exponential probability plot of the right half of the distribution is consistent with either light or heavy tails (under heavy tails, the slope would go to infinity).
light-tailed distribution. Because Llama-7B-chat a more reasonable base model than a completely random policy, we believe that Starling 7B-alpha is more likely be light-tailed for the purposes of our theoretical results.
The ACG results need some interpretation. The KL divergence between two distributions P and Q if P is the same as fraction a of the time, but some value x fraction a of the time given
by DKL (PI[Q) [[1 ()((x) + a] log (1=a)g(z)+a) + (1 a)) log(1 a) - q(()).
((z)
When a is small but much larger than g(x), we approximate this to first order as DxL (Pl))) x
log
In Theorems and we prove that when the error sufficiently heavy-tailed, a policy

---

Plots of reward from 16000 Llama-7B-chat generated inputs to Starling 7B alpha reward model
Histogram
Normal probability plot
500
Frequency
300
reward
0-
-2-
100
-5
-4
Reward
Normal
Exponential probability plot
Hill Estimator for Heavy-tailed Distribution
0.16
0.14
0.12
Hill extimatoo
0.30
reward
0.06
0.04
0.02
0.00
200 250
300
Exponential quantile
Number of upper order statistics (k)
Figure 2: Plots of the reward distribution from 16000 token sequences generated by Llama 7B-chat of length < 133, starting with five random tokens. Clockwise from top left: A histogram shows the reward distribution has a left skew. The normal probability plot suggests reward is approximately normal and thus laght-tailed. The Hill estimator plot should stabilize if the distribution heavy-tailds but it does not; thus, there is no evidence the distribution is heavy-tailed. The exponential probability plot also indicates light tails, because the curve is bending downwards.
that gets extremely large reward small fraction of the time will achieve high expected reward with low KL divergence. This is not the case here because the rewards achieved through ACG were small and the tog-probabilities extremely negative. For example, a policy that matches Llama 2-chat's base reward 99% of the time and uses the highest-reward input generated by ACG a =1 % of the time will have KL divergence from Llama 2-chat of alog(aa) 1339.701 13.35 nats, but reward only about * (2.2377 - 0.3329) = 0.02571 greater than the base model, far less than can be obtained with the same KL divergence by conditioning.
5 Discussion and Limitations
5.1 How likely is catastrophic Goodhart?
The Kow-KL policies that result in catastrophic Goodhart are not unique optimal policy, just one family of high-performing policies. When optimizing -3)]=33Drr (T,, mo). the outcome depends on RL training dynamics; it could be that DRL. 0 causing catastrophic Goodhart, but more likely both terms will go to infinity, potentially allowing V - xo. Catastrophic Goodhart can be prevented by using a light-tald tailed or bounded reward function.
7

---

Even so, catastrophic Goodhart is likely to occur in many scenarios where KL regularization is naively employed in an attempt to avoid Goodhart's Law:
If we maximize ( + (x where is a bounded function (e.g. sigmoid), all near-optimal policies will have V zo O. Since we can only obtain so much reward from o (B(U)), it pays to make the KL (and thus V) go zero.
If we cap KI. finite value (or dynamically adjust the KI. penalty target finite KI., as done Ziegler et al. 2020). then B[V] is also upper bounded by finite value (see Theorem 2). and we think is likely that E[[] O. Consider a toy model where an Al can adjust three parameters: true quality V of responses, frequency of reward hacking (producing actions with extremely high X), and severity of hacking (value of X on those actions). Adjusting the policy to increase EEU] without increasing KL increase the severity of hacking while decreasing either frequency of hacking or quality of responses. When E[U] is already large, decreasing quality has much better returns than decreasing frequency. This similar Theorems 5 6 about hard-threshole optimization.
Any way maximize [[U(T)] 3D (T, mo) results in very large values of [E(((T) (T)), and there are a number of arguments that extreme optimization for an imperfect proxy can result in decreased utility due to tradeoffs between X and V; e.g., the constrained resource scenario in (Zhuang and Hadfield- Mannield-Menell] 2021).
5.2 Independence assumptions
Theorems 1-3 do not require any independence assumption, but Theorems 4 5 and 6 requure that error X and utility V are independent, which seems to be violated in practice. Future work could weaken this assumption, although intuitively obvious ways weaken it result in the statement being false. g
5.3 Stronger optimization methods
We did not search the entire space of token sequences, cannot rule out that the reward is heavy-tailed enough to cause catastrophic Goodhart in some situations. While it is intractable to search the more than 102000 possible token sequences, future work could get more evidence through more powerful optimization methods.
5.4 Reparameterizing reward
In some cases, heavy-tailed reward can be reparameterized to make it light-tailed and avoid catastrophic Goodhart; however, in settings where the true reward is heavy-tailed, making reward artificially hight-tailed or bounded can result in unintended behavior.
For example, a stock-trading agent should be rewarded by profit, but financial returns are known to be heavy-tailed. If we clip or otherwise transform rewards into bounded interval, it will have no incentive to take into account huge gains or losses. Since RLHF rewards as implemented in Ziegler et al are unbounded, clipping or transforming rewards could itself cause reward misspecification.
In some cases, e.g. when the reward is not the true intended one, it is possible reparameterize the reward without adverse effects. In the RL literature for Atari games, rewards are changes in score clipped to [-1, 1] (Machado et al. 2018).
5.5
Relation to previous overoptimization work
Gao et al. 2023) found that optimizing the reward of small reward models causes overoptimization: a decrease utility with increasing optimization. However, we observed that reward models are light- tailed, and (Theorem 4 that independence combined with light-tailed error prevents overoptimization. We think this discrepancy explained by dependence between error and utility. Policies optimized
"Suppose that error X light-tailed conditional on any value of but our proxy merely unbiased [B[ = v] = 0 for all v). Then the limit of V under optimization for X + still depends on the relationship between X and V. If they are independent, Theorem 6 says that lim BV V 2 = co. But if
~ N (0, 1). and X|V ~ (0, 4) when V [-1, 1]. otherwise X = 0, then lim E[V X + =

---

for high error may activate features in the proxy reward models that are undesirable according to the true utility function More research needed understand why high-error completions have low utility and to design reward models that do not suffer from this problem; perhaps it is possible to construct reward models whose errors are in directions orthogonal to human preferences, so that the large-reward completions do not have lower utility.
6
Conclusion
We have argued that the purpose of the KL divergence regularization in RLHF is to mitigate reward misspecification. However, we have also proven that when errors in the reward function are heavy- tailed, it cannot serve this purpose: even with zero KL divergence, there are policies that achieve very high misspecified reward and no actual reward.
When errors are light-tailed and independent, the KL divergence can mitigate misspecification, but when they are dependent, this may not be possible. Thus, we must look places other than the KL objective to explain the current success of RLHF and ensure its continued success in the future.
Acknowledgments and Disclosure of Funding
This work was supported by the Long-Term Future Fund (LTFF). We also thank the anonymous reviewers for their valuable feedback and constructive suggestions.
Impact Statement
As this work aims to improve the safety of future ML systems by characterizing possible failure mode reward misspecification RLHF, we hope the social impact is positive. We see no particular ethical issues discuss.
References
Abdolmaleki. A., Springenberg, J. T., Tassa, Y., Munos, R., Heess, N., and Riedmiller, M. (2018). Maximum posteriori policy optimisation. CoRR.
Christiano, P. F., Leike, J., Brown, T.. Martic, M., Legg. S., and Amodei. D. (2017). Deep reinforcement learning from human preferences. In Guyon, L., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances Neural Information Processing Systems, volume Curran Associates, Inc.
Clark, J. and Amodei, D. (2016). Faulty reward functions in the wild. Accessed: 2024-07-07.
Foss, S., Korshunov, D., and Zachary, S. (2013). An Introduction to Heavy-Tailed and Subexponential Distribu- tionx. Springer, 2 edition.
Gao, L.. Schulman, J., and Hilton, J. (2023). Scaling laws for reward model overoptimization. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, and Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume Proceedings of Machine Learning Research, pages 10835-10866. PMI.R.
Haize Labs (2024). Making sota adversarial attack on llms 38x faster. https://blog.haiiien haizelaba com/ posts/ack /cg// Accessed: 2024-05-22.
Hu, J.. Wu, X., Wang, W., Xianyu, Zhang. D., and Cao. (2024). Openrlhf: An casy-to-use, scalable and high-performe performance rihf framework. arXiv preprint arXiv:2405.1114 1/143.
Jaques, N., Ghandeharioun, A., Shen, H., Ferguson, C., Lapedriza, A., Jones, N., Gu, S., and Picard, (2019). Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv: arXiv:/907.0000 1907.00470
"Thre are other explanations possible. Perhaps better optimization methods would find heavy- reward in open reward models: or OpenAl's reward models have heavy-tailed error (and their results are straightforwardly explained by our Theorem I. while open reward models have light-tailed error.

---

Figure A.1: As f oc, the mean of X (blue bar) grows without bound while KL divergence DKr, Q) (orange bar) goes to O. The base distribution Q is a Student t-distribution with df 3. In this case, high values of X upweighted 1/10.8, upweighting them 1/L would cause [[ to converge to 1 while KI. divergene goes to zero faster.
Laidlaw, C.. Singhal, S., and Dragan, A. (2024). Preventing reward hacking with occupancy measure regulariza- tion. arXiv. Accessed: 2024-07-07.
Lambert, N. and Calandra, R. (2024). The alignment ceiling: Objective mismatch in reinforcement learning from human feedback.
Lambert, N., Pyatkin, V., Morrison, J.. Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick. T., Choi, Y., Smith, N. A., Hajishirzi, (2023). Rewardbench: Evaluating reward models for language modeling. 40 pages, 19 figures, 12 tables.
Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., Bowling, M. (2018). Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-560
Schulman, J., Levine, S., Moritz, P., Jordan, M. I,, and Abbeel, P. (2015). Trust region policy optimization. CoRR.
Schulman, J., Wolski, F,, Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. CoRR.
Strathern, M. (1997). 'improving ratings': audit in the british university system. European Review, 5(3):305-321.
Wierman, A. (2013). tions
Catastrophes, conspiracies, and subexponential distrihu-
(part ii).
wordereswdddd.....e. com/ 2013/12/17/
catastrophes cinserrrppees-oospprcciieee seeeees-ndd-uuuueuue subexponential witp:aaaaladdttiiiiitiiooooo paateiiiiinnnitt---eee 2024-06-26.
Accessed:
7hu, B., Frick, E., Wu, T,, Zhu, H., and Jiao, J. (2023). Starling-7B: Improving Ilm helpfulness & harmlessness with rlaif.
Zhuang, S. and Maddield-Menelle D. (2021), Consequences of misaligned AI. Advances in Neural Information Processing Systems, 33:15762 15773, arXiv:2102.03896vl [cs.Al].
Ziegler, D. M.. Stiennon, N., Wu, J.. Brown, T. B., Radford. A., Amodei, D., Christiano, P., and Irving, G. (2020). Fine-tuning language models from human preferences. arXiv: 0999088533 [es.CL].
Zou, A., Wang, Z., Kolter, J. Z.. and Fredrikson, M. (2023). Universal and transferable adversarial attacks aligned language models.
A Proofs
A.1 Theorem
Restatement of Theorem Given any heavy-tailed reference distribution over R with mean μiq and any M, > 0, there a distribution P with mean pe > M and (xi(P))) <
Intuitively, a heavy-tailed distribution, events with extremely high x are not very rare, so you don't t pay much of a KL penalty to upweight them so they happen about 1/x of the time. This is visually illustrated in Figure A.I
10

---

Proef. WLOG let na 0. We construct sequence of distributions {Pr} such that limy-se Ep, [X] 2 for any constant and - D (P: [[Q) O. We define for any ( thusly. Writing Fp, (a) for the CDF Prx-p, (X I x) and FA, (x) for Fp, (2)), we
Falt) F Fa(t) Fa(x)
1-c/t Fo(x)
x xst
FL(P(())=
x x>t
Intuitively, we rescale the part of the distribution to the right of t evenly to have total probability e/t, which is less than because 1 >
We must check that lim Ep, X =
We can write
Br, [x] (t)(t) X P f|
(Fr. Fa(()) S 6) (Pr Fx(l)) +]
[o[X]] + (F) (000)))) > - =
We know that Eq [ f) > t because is integral of values strictly greater than Because Bql [X] 0i weighted average of Egg [X] X and Eg X f). and Egg [ > f] >> O, know Eg [X] X < {] 0. So Eq X > f] - Fe X S f) > also know that for sufficiently large f. (Fp, (t) Fo) >
Intuitively, starting from Q. which has mean 0, Pt moves a probability mass approaching & from mean <0 to mean >t
Now can
lim Ex L0000
lim 10000
[o[[]] + (Fp, (1) Fo(1)(( - 0)]
lim ( FQ(()) == lim c (Fo(t)
1.000
6000
Because Q has a finite mean, lime-+00 (FQ( = 0, and so Ep, [X] 2 a
Now check that lime- (xz(P:
DxI(P:I1)= J log ((dz)
Pr(dx) (ddx)
((dx) Pi(dx) + [ log
(d
Pr(dr)
log
Q(dx) Pi(d))
n>t
(t) log (t) + Fr. (t) (t)
since ratios are constant
[((())
Fo
1- c/t
(t)
FF, log
+ Fp. (t) log
Fo(t)
Po(()
Since both c/t and Fo(() to to as t the left term goes 0, and
lim (KL(Pi s + lim Fr. log (p)(0) 10000
10000
Ft
lim ? log (F(
< lim
10 7 log (t(
10000
lim
c
log (t) since D>
600
;
Q is heavy-tailed, so by definition lim, ed FQ((t) = xo for all a > This implies that for every a >
there is sufficiently large to so that for all > to Fo > -at which means that log Fo(() > -at.
Therefore for every >> 0, lim Dxt (P < lim -e/t log (t lim, -act = ac, which
since KL divergence is nonnegative means thatlim:+ (K:(P) as desired.

---

A.2 Theorem 2
Restatement of Theorem 2 V is light-tailed, [V] is zero, and d = (kz(P)Q) is bounded, then Ep [
bounded, and [V[ -> O.
Proof. Using Lagrange multipliers, we find that when KL divergence is minimized, we have P(V) log P(V)
Q(V)
+
Au X] = for some constants \1, Au, so
P(V) V A2
log Q(V) A
(1)
V A2
P(V) = Q(V) exp
= =Q(V)
(2)
A
e 1 22/21
eQ(V)
(3)
That is, the new PDF is an exponential tilting of the old PDF Now, what is Er[V]? It's just [ CVe QXX) dV. If the distrihution of V is heavy-tailed distribution, this is oc; if it light-tailed, this some finite value.
When d
P and Q are identical, and E[V] = 0. So by a continuity argument, Br[V] - 0 as d +
A.3 Theorem 3
Restatement of Theorem Let W (S,A, P, R) be determinnisiccttrasiiion MDP with Markovian
Given W, we define the function that takes policies to trajectories Tr (S - AA) -> ASS A)", and average return function g (S x A)" + R which induces a function G A(S A)" + AR. Let S + AA be some base policy. If G o (((o)) is heavy-tailed with finite mean μiq, then for any M, > 0, there is a policy with mean return [U G Gr(()) M and Te (xL (()))) <
Proof:
We will exhibit a distribution of trajectories such that Dkz (p(T((((TT) < and B[G(p)) > M, and then construct policy with Tr(m) Note that this proof applies for continuous action spaces if trajectories are replaced with measurable sets, but this would make harder read.
Tr(mo). We have a heavy-tailed distribution of return Q A G(Pro) over IR, so we can apply Theorem [ But to define we can construct P: in the proof of Theorem in particular way. For any >>> we need Pt that uniformly upweights values of mean return such that Fp, (t) e/t. We can define P: such that any trajectory r upweighted by factor depending only on its mean return:
Let
Falti Prog (T) g(r) St c/t Potty (xo g(T) >
l-ff
(x(T) =
Then we can let P; & Go and the rest the proof Theorem applies. Therefore, applying the theorem, we G. G (..))) < e. By the chain rule for KL divergence, Dx (K (p, pso)) = Dx (Gop, Gopsa) - DD (G()) = Pxo (GT)) 7)]. Since constructed term zero, and we also have DRL (p, PTp) <.
can let for sufficiently large t, and then pCsp > M and Dx o p,
that the probabilities of each conditional its return being ? are equal, the second
Finally, since KL divergence between trajectory distributions the sum divergence between policies each action in the trajectory, and each trajectory has least one action, F.ET,,TTTGe (T()) ET-Tr()) ! (T)() (s)))) = Dx (x (p))PT)) < as desired.
To define such that Tr(m) A. we let x (s, Pr(e: alr (,.,,,,ii,..) p).
Then, the probability that any trajectory (s1, a1, ... an) ) is sampled is:
12

---

(r())(r)
(8, 0s)
(4)
Pr(o: a'lT' = (,.,,, ~ p)
(5)
[=1
Pr(a; = d'|T'
Po sei den d'zs)
(6)
p(T)
In (2), returns are Markovian, so all trajectory prefixes ending in state s have the same distribution of returns under any policy. In the construction of p. all trajectories with the same mean return have equal measure. Therefore, conditioning carlier states and actions of I docs not change the measure, can write (3). So Tr(x) P desired.
A.4 Theorem 4
Restatement of Theorem 4 I = X + V with X and V both light-tailed and V unbounded, and the distribu- fion of continuous, and n (3) 2 arg max, [[U(T)] 3D (T,, no), then limg--o+ [=V(==(())) oc.
Proof. Fix some 3. Using Lagrange multipliers, we find that for any event S, (r, = Prap (S) AU((S) Let
c(ß) be the median value of U under the policy x (8); that is, Pr(U > c())U ~ G Tr(4" (8)) = = This exists because U has continuous distribution. Then:
1
F[ FVV <d+ F 2
FVV <
2
lim [[V[
lim 8-0
V n. U <
lim
[[|]=]
2
8-01
8
The left term is c, while the right term is co, so the overall limit is co.
A.5 Theorem
Restatement of theorem 5 Let and V two independent random variables with CDFs Fx Fy and tail functions Fy 2 Fv. Fx 2 Fx such that
V has a finite mean.
subexponential; is, limg-+50 Pr(X 2 X,, X2 are two independent samples P(X
from X. This is slightly stronger property than being heavy-tailed.
The tail of is sufficiently lighter than the tail of X that lim
(C Fxt))
= 0f some >
Then lime U=
- = B[V]: that is, catastrophic Goodhart occurs in the limit of optimization for
1 (V
The proof requires expressing the conditional expectation in question the interval (-o0, oc) into four regions and bounding the integrand in the numerator above by a different quantity in each region.
J (v())(>>=0)) then partitioning
In addition to the works cited in the main paper, we make reference to the textbook Foss et al. 2013 throughout the proof. Many similar results about random variables present the textbook.
13

---

Region
Why its effect on V is small
Explanation
(-x0,- h(O)
PIV E Eriec] is too low
In this region, [V] h(() and X >> L h((), both of which are unlikely.
P2 = ((t)
E E r2 & E E r2]
The tail distribution of X is too flat to change the shape of V 's distribution within this region.
r3 =3=(h(t)
P low, and V <t.
There are increasing returns each bit of optimiza- tion for X, so it's unlikely that both X and V have moderate values. 9
11 = (t o0)
P(V e ri c] is too low
X is heavier-tailed than so the condition that V > t-h((t is much less likely than X > t t-h(t)
in r2.
Table A.1: summary of the proof strategy for Theorem
Conditional cieinbuttoon acrom the four
20
Ps(t-4]
11
R
R6
R3
Ry
Figure A.2: A diagram showing the region boundaries at -h(t), A(t), and / h(t) in an example where 1 = 25 and l(t) = 4, along with a negative log plot of the relevant distribution:
A.5.1 Proof sketch and intuitions
The conditional expectation E[[]X V 1 is given by
1 (X)
1 [X and divide the integral in
the numerator regions, showing that each region's effect on conditional expectation of V similar to that corresponding region the unconditional expectation E.[]..
regions are defined in terms slow-growing function h(t) R + R20 such that the fiddly bounds on different pieces of the proof work out. Roughly, we want it to go to infinity so that [V] is likely to he less than h(t)) in the limit, but grow slowly enough that the shape of V's distribution within the interval [-h(t), (t)) doesn't change much after conditioning.
In Table A.S.I abbreviate the condition X + V > as c.
Note that up constant vertical shift of normalization, the green curve is the pointwise of the blue and orange curves.
A.5.2 Definitions
To be more precise, we're going to make the following definitions and assumptions:
'We'll generally omit dr and de terms in the interests of compactness and conciseness; the implied differen- tials should be pretty clear.
14

---

Let (v) be the PDF of V at the value v. We assume for convenience that fv exists, is integrable, etc, though suspect that this isn't necessary. and that could work through similar proof just referring the tails of Let (x(2) = Pr(X S æ) and (x) = Pr(X 2), similarly for
V. We won't make this assumption
Fv and Fy. Assume that
V has finite mean: Sa ffv (v) du converges absolutely.
subexpenential.
Formally, this means that limz-0o Xx
This oceurs roughly whenever X has tails that heavier than e e- for any c and is reasonably well-bchaved; counterexamples to the claim "long-tailed implics subexponential" exist, but they's re nontrivial to exhibit. Examples of subexponential distributions include log-
P(X>2)
normal distributions, anything that decays like power law, the Pareto distribution, and distributions with tails asymptotic to e == for any < a < 1.
We require for that its tail function substantially lighter than X's, namely that lim some p > 1. (This implics that Fv = O(Fx (f)/t).)
1 FPF1
for Fx
With these definitions and assumptions, wc can move on to the proof.
The unnormalized PDF conditioned on X + V 2 is given by (x Its expectation given by f *fy (x) (v()))x(t-)))
Meanwhile, the unconditional expectation of V is given by J_ (fv
We'd like to show that these expectations are equal in the limit for large t. To do this, we'll introduce ((+) Ex(t-e). (More pedantically, this should really be Qr(w), which we'll occasionally use where it's (e)
helpful remember that this is a function of t.)
For given value of t, Q(v) is just a scaled version of Fx (t - v). so the conditional expectation of V is given J (v)()) by 1 fv(eeQ((e) But because Q(0) 1. numerator and denominator of this fraction are (for small v close to the unconditional expectation and 1, respectively.
(v) We'll aim to show that for all >> 0, we have for sufficiently large t that | (vv "fv(() < and I fv (v) e e, e). which implics (exercise) that the two expectations have limiting difference zero. But first we need some lemmas.
A.5.3 Lemmas
Lemma 1. There is h(I depending on Fx such that:
(a) lime h(t) = 00
(b) lim h(()) = xc
(c) lime-+00 (t-1)((0) = (x)
(d) lims-4o (1 (Q(,,)) 100,))-11=1 1|
Proof. Lemma 2.19 from (Foss al. 2013) implics that long-tailed (which it is, because subexponential
implies long-tailed), then there is h(() such that condition (a) holds and Fx is h-insensitive; by Proposition 2.20 we can take ii such that l((1) < 1/2 for sufficiently large f, implying condition (b). Conditions (c) and (d) follow from being h--insensitive.
O
Lemma 2. Suppose that Fx is whole-line subexponential and is chosen as in Lemmaa Also suppose that (t) = (F)x(t)t) Then Pr(X + +V t, V > h(t), X> h(t)) = of o(Fx (t)/t).
Proof. This is slight variation on lemma 3.8 from 2.37. Lemma 2.37 states that
Foss al. 2013), and follows from the proof of Lemma
Lemma 2.37. Let any increasing function on RR+ such that h(x)) + co. Then, for any distributions Fi, F2, Gv. and G2 on IR.
P [Ei + > x. M(z), > h(x)}
(i(
G (x)
sim P [f22 + m2
< lim sup
lim sup
> x. M(x), h(z)}
Pa(x)
C2
15

---

where &1, E2, in, and 1/2 are independent random variables with respective distributions
Fí, F2,, G1 and G2-
but is actually proved that
P [E1 + vn x k(2). m > h(x)) 1
(i sup
(1 G2(2)
sup 2>)(2) P2(2)
P (C2 + > >, M(x).22 02 > M())). (8)
#>
If we let Fi = Fv, C1 C2 = Fx, then we gel
P +( > ((t) V h(1))
(2) sup
Fx(2)
$ sup *>big Fx
P 2x h(t), x'> *(1))
s>i(t) Fx
Fv(2) P
sup <>)()) Fx(:)
>11)
(9)
where X, X' ~ Px. Multiplying by f, we have
sup (=) p (X+X' X4,,,) (10) =>)(()) Fx(2)
IP (XX-V > x, (Xx-V>,x>>,,,V>>>> V20(()
and because h(t) +
t + and (t (t)/tt)
say that for some < oc.
lime-+o sups>1(t) (x
(fy(z) < Therefore for sufficiently large IP (X 1, M(). h()) I
{P (X+X* > 1, X h(t), X X(t)(t))
By Theorem 3.6, P (X + > t, x M(), x(), is (()x so the LHS is of of (1)/t) as desired.
A.5.4 Bounds on the numerator
We want to show, for arbitrary >> 0, that 1 (fv())(()) - vfv(w) < in the limit as + co. Since | (f)(v))(v) f ufv(v) S (ofv())((())) - 1)1 f lo| (v (Q(v)- 1| will suffice to show that latter quantity less than for large
We're going to show that _ v/ (v) 1Q(x) - small by showing that the integral gets arbitrarily small on each of four pieces: (-o0, -h(t)), (-A(t), (A(t), h())., and (t h(t), oo).
We'll handle these case by case (they'll Il get monotonically trickier).
Region 1: (-o0, -t))) Since 1_ ufv(u (v) is absolutely convergent, for sufficiently large we will have (-(())) [v] < since l4(1) goes to infinity by Lemma ((a).
Since (v) monotonically increasing and Q(0) = 1, we know that this interval (Q(x) - = Q(r).
So we have (+)) fv( IQ(x) 1/= f-A(t)) (v) Q(v) (()()) /ulfy (v) < as desired.
Region 2: (-h(t), By lemma 1(d), is such that for sufficiently large (Q()) 1| < T= [o] on the interval [-h(tt, [it)). (Note that the value of this upper bound depends only on V and not on f or h.)
So we have (1(() [v) - 1| < T [o]] shit [u]sv(v) < T M f (o].fv(o)) =
Region 3: [h(t), t- [()) For the third part, we'd like to show that (i-h(tt) (fv(v)(((v) - 1) < Since
- (v) - (i())
p(- (fv = गरंक S fv(v) Fx (t-r) it would suffice to show (t))())
that the latter expression becomes less than for large i, or equivalently that M fv(v) o) = Ex(0)
16

---

The LHI in this expression is the unconditional probability that X + V > L and h(() < V < L h((). but this
event implies X + V. (c)), and X h((). So write
<V <t- k(O]
h(0)
< P V V(t),, X h(t) ((F((t)
by Lemma 2.
Region 4: (t - (t) oc) For the fourth part, we'd like to show that ( eVv(v))((v) - 0 forlarge L.
(x) would suffice to show J,_ vfv(v) = (1)). But note that since lime-s00 Ex(t-h(tt) by Lemma I(e), this is equivalent f--nee) vfv(v) ox (t h(t))), which (by (()())
Since Q(v) = (t
(x(t)
Lemma 1(b)) is equivalent to 1." ufv(v) = o(Fx(1))
Note that J (e) tfee (v) + 1,
1,"0 (v so it will suffice show Fx(1)
that both terms in this sum are ofFx (l).
The first term FFv (t) is of Fx (t)) because we assumed lime-ss
= 0 for some p >>> 1.
the second term, we have for the same reason 1, Fv(u) < 1. Fx(e) (F))).).
Fx 1, v (x(()
or
A.5.5 Bounds on the denominator
the denominator, we want to show that lim S -1- fv(e), so it'll suffice to show [ (Qr((() - 1) o(1) as - zc. Again, we'll break up this integral into pieces, though they'll '1 be more straightforward than last time. We'll look at (- -h()). (())), and (h).
1 h(tt) (v)(((())) - - (v(e)( Q())) < - h(t) fv)...
- But since ((t) goes to infinity, this left tail of the integral will contain less and less of V probability mass / increases.
[ (v) - 1)1 ( h fv(v))((v)
S suPplsh()) (Qv t) ( fv( suplola()) (Q(v,t)
-By Lemma ((d) know that this goes to zero for large t.
Shhe (v( - 1)1 Jun (v) 1) Jhi (v)QQ)
But for sufficiently large we have (t)) > 1, so we obtain Ji Qv < Jhy vfv(v)Q(v) < 1 vfv(c)Q(v)== = o(1) by the results of the previous section. This completes the proof.
A.6 Theorem
Let X, V be independent random variables such that lim implies that X has tails that dominated by e - for any though it's slightly stronger claim because it requires that X have large jumps in the decay of its tails.) Then for any V with a finite mean which has no upper bound, lime B[V[X
Restatement of theorem
Fx(t+1) Fx
0. (This
co.
Theorem gennealzzsss a consequence of the "Regressional Goodhart Identity" in (Gao et al. 2023
Proof. Let Pr(V c p> 0, which exists by assumption that V is unbounded.
Let V < c) (If this is undefined because the conditional has probability 0, we'll have the desired result anyway since then V would always be at least c.)
1 2 q (assuming it is defined). because we're we're conditioning (VIV < e) on an event which more likely for larger v (since X and are independent).
Observe that for all EVIVV c,
First, let's that lime-so P +V20
0. This ratio of probabilities equal
PVVC+IX+V20
17

---

Histogram
Normal probabiityy piet
-10
60
-10
-13-
20
-40
4.0
3.5
2.5
-20
-15
-10
-2
Figure B.1: Histogram and normal probability plot of reward assigned by Pythia RM to random length-1024 token sequences. The Q-Q plot suggests the distribution is approximately normal, which is much nigher-tailed than exponential.
Rewand og-prrbah ity
Starling 78-alpha
-1000
-1:00
-1700
-1300
1100
-1500
Reward
Figure B.2: Reward and log-probabiliig for ACG-optimized inputs Starling 7B-alpha.
1 Sv(e) 1e (x S Jen (v Fx (x (v()
1 (v (v Fx((-))
fx(t-c) Fx(t-c-Ii MVSC S
MVVC)
Px(t-c)
Fxle-c-I
which, by assumption that lim
Px will get arbitrarily small as increases for any positive p. Px(0)
Now, consider B[V|X + V 2 f). We can break this up as the sum across outcomes Z of [E Pr(Z] V t) for the three disjoint outcomes V c.c. < V + and V that we can lower bound these expectations by q, e + respectively. But then once is large enough that cel
>
1. Note
this weighted sum of conditional expectations will add more than e-g'
RVV XXV20
B Additional experiments and figures
Figures B. [.2 relate to experiments mentioned the main paper. response reviewer feedback, we added two further experiments to demonstrate the catastrophic Goodhart phenemonon with artificially heavy-taile reward, one using hest-of-N on synthetic distributions and one with PPO on Pythia 1B.
B.1 Best-of-N experiment
We created synthetic experiment by letting reward U = X + V, where X and V are independent and sampled from different probability distributions, consistent with our theoretical assumptions. We vary N from to 65536, do 100 trials of taking the best-of-N sample with highest U, and note whether V goes towards 0 (overoptimization) or not.
Possible distributions for V are normal and t-distribution with df=10. Possible distributions X are normal, t with df=3, t with df=5, lognormal, and Levy. (All of these heavy-tailed except for the normal distribution.) is scaled to a standard deviation of 2 and X has s.d. of I (except for the Levy distribution. which has infinite variance), representing that in ordinary regimes most of the variance comes from utility rather than error.
18

---

Utilty
20.
Jiiltty
Utilbt
14
1042
Utility
Procy reward
17.5
Procy reward
Prozy wward
12
Prosy rwward
1614
Prosy swoand
12.5
10"
207
107
107
101
107
807
10"
(V. 10) normel
CV. 10) vi lognormall
(C. 10) [1)
CV. 10J vs CC, 5]
CV. 2C1 vs levy
Usity
Juiltty
UVit
12.5
Utiityy Erior
Utilty
14
Eror
Eroo
95
Errrr
1044
Broo
Froxy roward
30
10-
3.0
10'
2.5
0.0
20' 30"
101 10' 20 10'
10'
10
Figure B.3: When the error X is normal and thus light-tailed, increases monotonically with N, consistent with our Theorem 6 However, when both X and V are heavy-tailed, consistent with theorem 5 In of 6 cases when X is lognormal or student-t, V first increases then starts to decline around N = 10² or 10 When X is (t, df=5) and V is (t, df=10), V instead peaks around N 10" (but declines afterwards). Finally, when X is Levy-distributed, utility never goes significantly above zero (optimization completely fails) because the Levy distribution is too heavy-tailed.
results
The results are shown in Figure B.3 Briefly, the results are consistent with the asymptotic results in theorems 6 showing that overoptimization
B.2 PPO experiment
In this experiment, examined PPO with artificially heavy-tailed rewards if catastrophic Goodhart could be observed.
OpenRLHF (Hu et al. 2024) was used to train Pythia 1B with reward model derived also from Pythia IB, on the default OpenRLHE prompt dataset. We used the reward model to represent truc utility, and heavy-tailed error term based on the number of "the" tokens was added to get the proxy reward. The kl target=O.... 0.6 option was used to dynamically adjust KL penalty, clipped. (Reward clipping be useful prevent overoptimization, but always used in PPO.) Response length was limited to 256.
mention is done in Ziegler al. (2020). Rewards were not
Two example completions are figure Midway through training, model generates reasonable text with a higher number of "the" tokens than usual, but as training progresses the model starts to generate completions with a very high number of "the" tokens while proxy reward reaches values > 100, demonstrating that the basic pattern of catastrophic Goodhart occur in RLHF under conditions of heavy-tailed emror.
19

---

The preparation of polyarylethylene which is either employed for the purpose of applications in the petroleum refining or in the production the well-known automobile oil lubes is carried out by the so-called preparation by the salt bath method described... in 1980 by D. W. Perkins in the United States. This method presupposes the usage of the salt bath as the medium for the preparation of the polyarylediene- the method of preparation of the crude oil base by the process which consists in letting it be saturated up with a particular the
mirror, the essence of the understanding of the given the conclusion that the manifestation in the form of the gender and the identification the solutions related to different fields of the modern utilization of the the functionality of the the the the modification of the the the the the the state of the the the the the the the the the the the employment and the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the and the the the the the the the the the the the the the the the the the the the the by the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the and the the the the the the the the the the the the the the the the the
Figure B.4: PPO sample generations. Top: early in training, the model generates reasonable completions. Bottom: later in training, the artifically heavy-tailed reward dominates and the model generates completions with very high number of "the" tokens.
The result depends on hyperparameters (c-g. reward clipping would prevent this), so our observation should not be taken as claim that catastrophic Goodhart is inevitable in all RLHF settings with heavy-tailed crror.
C
Assets
We use three models for our experiments: Starling 7B-alpha, Llama 2 7B chat, and Pythia-1.45 Starling was developed by Berkeley. and Pythia by EleutherAI. Starling and Pythia models are licensed under Apache-2.0
Llama 2 models were developed by Meta and licensed under license published by Meta
"hhtps:/huuginngfccc htw.r//sss..c.cccoccoooo/e//nnan
https://huggingfacc hwt.gn/gww......eete////t/aaaaaa
12 http:://aiimett..oommmm///eeeee
20

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and socictal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA]-
[NA] means either that the question is Not Applicable particular paper or the relevant
information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] is generally preferable to "[No] it is perfectly acceptable to answer "[No] provided a proper justification is given (e.g.. "error bars not reported because it would be too computationally expensive" "we were unable to find the license for dataset we used"). general, answering "[No] "r "[NA] is not grounds for rejection. While the questions are phrased in a binary way. we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to claborate. All supporting evidence appear either in the main paper the supplemental material, provided in appendix, If you answer [Yes] to question, in the justification please point the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract lists the important claims: the relationship between Goodhart's 's Law and whether the error in misspecified reward is heavy-tailed. The main limitation of independence assumptions is clearly stated in the introduction.
Guidelincs:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcoretical and experimental results, and reflect how much the results can he expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 7 lists the limitations, which we have combined with the discussion section due to heavy overlap.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
21

---

The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model -ell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope the claims made, e.g., the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to- text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
authors should discuss possible limitations their approach to address problems of privacy and fairness.
If appliccae,,,
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It a acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve integrity the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Ycs]
Justification: All proofs given inline or in the appendix, except for Theorem which appears in the supplemental material.
Guidelines:
The answer NA means that the paper does not include thcoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Sampling rewards requires no hyperparameters, and hyperparameters are provided for
ACG.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, the contribution novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but
22

---

reproducibility can also be provided via detailed instructions for how to replicate the results. access to hosted model (e.g.. in the of large language model), releasing of model checkpoint, or other means that are appropriate to research performed.
While NeurlPS does require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a way to access this model for reproducing the results a way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
new model (e.g., large language model), then there should either be
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide closed-source source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
reproducibility. In the case of
5. Open access to data and code
the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described supplemental material?
Question: Does the paper provide open
Answer: [Yes]
Justification: Code will be provided in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (httpa://mips... htcppps.cc///re cc/publcc/
guides/ siseooooirrsii for more details.
While we encourage the release of code and data, we understand that this might not be possible,
"No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https:/ hhppss://nips...// ccbpuiicc/ guides/ guides/CodeSie Submm ssionpolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs data and code permitted.
6. Experimental Deetann/Details
Question: Does the paper specify training and test details (e-g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Other than hyperparameters, there are no details required to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
23

---

Justification:
The only enror bars are inter-run variability of ACG. The standard deviation was reported rather than enror bars due the small number of runs. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
show in tables figures symmetric error bars that would yield results that are of range (c.g. negative error rates).
For asymmetric distributions, the authors should be careful
error bars reported tables plots, The authors should explain in the text how they calculated and reference the corresponding ligures or tables the text.
8. Experiments Compute Resources
Question: each experiment, does paper provide sufficient information on computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: The experiments took minimal compute resources except H100 hours for ACG, and we report the number of GPU-hours used.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for cach of the individual experimental runs well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp::/neurips
Answer: [Yes]
Justification: The research conforms to all data-related concerns. No human subjects were involved, and think the risk of harmful societal impact is minimal.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from Code Ethics.
The authors should make preserve anonymity (e.g., if there is special consideration due laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: The immediate societal impacts are limited, but we discuss some potential applications to long-tenn teng-eemmae safety.
24

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example. it is legitimate to point out that an improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, it not needed point out that a generic algorithm optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible relcase of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We have created no such artifacts.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The assets we use are Starling 7B -lpha. Llama 7B- cBachat and Pythia- 1.44, Starling and Pythia models are licensed under Apache-2.0 14 Llama models are licensed under a license published by Meta 15 We are in compliance with all licenses and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper wnwerrwwthhoooe thcode. com/ datasete has curated licenses for some datasets. Their licensing guide help determine the license of a dataset.
"httss//huggiingfac.cc htt.://nnggaea.ooooooeceeeee///a 15 http:////aiieetttooo//a/aaaa////
25

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's
13. New Assets
Question: Are introduced in the paper well documented and is the documentation provided alongside assets?
Answer: [NA]
Justification: release new assels.
Guidelines:
The answer means paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: human subjects are involved.
Guidelines:
answer means that paper does not involve crowdsourcing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Beard (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: human subjects are involved.
Guidelines:
The answer means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you ohtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as institution conducting the review.
26


Reviews:
[{"review_id": "muXy9lvafj", "reviewer": "Reviewer_G3eM", "summary": "The paper first considers a theoretical stylized model of jointly distributed utility and (mis-specified) rewards and proves a novel result that for any heavy tailed reference distribution, it is possible to find another distribution that arbitrarily approximates it in terms of the KL divergence, yet has an unbounded reward. Beginning with this basic result, the authors then attempt to apply these insights to the popular technique of regularizing the policy optimization procedure in RLHF using a learned reward model with the KL divergence to the base SFT policy. Experiments on open source LLM reward models are conducted to investigate how closely the assumptions made in the theory match with current state of the art models.", "questions": "- The definition of DMRMDP in Section 4 is unclear. Is there any precedent to this particular definition? For instance, are you always required to end a trajectory in one of the sink states? If not, are rewards also defined for partial trajectories?\n- Is the converse of Theorem 1 also true? i.e. if Q is not heavy tailed, does the maximum achievable mean stay bounded as $\\epsilon \\rightarrow 0$?\n- In the experimental results section, random sampling and adversarial sampling are used to generate the reward samples which are estimated from another model. However, it is hard to know the true utility, unlike in the case of Gao et. al. (2023), who perform a synthetic study with known controlled true utilities. Since the paper's insights and results are more conceptual than practical at the moment,  wouldn't it be interesting to study a synthetic experimental setup similar to Gao et. al. where both true utilities and proxy utilities can be measured explicitly?\n- When you speak of a \"heavy/light tail under a given policy $\\pi$, one possibility is to consider the action conditioned expected utility, or the unconditional version where action is simply a latent variable that determines the final reward. Is this distinction significant for any of the results? If so, which one is being assumed, and what are the implications for the alternate version on the basic result?   \n\n\nNits: \n\n- Figure reference in L110 seems like a typo.\n- Figure 1 has fonts which are too small to read.", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper identifies a novel conceptual insight around the heavy tailedness of the utility/rewards as being an important factor in reward hacking.\n- Theoretical results are sound and novel, along with some attempts to also ground this in experimental validation.", "weaknesses": "- The clarity in some sections can be improved -- see questions below.\n- The notion of heavy tailed versus light tailed, while theoretically appealing and convenient, seems quite sensitive to parameterization and/or scale in practice. While the authors do mention this briefly in footnote 2 (Page 6), it appears that simple reward reparametrization can transform the resulting distributions in a way that is not robust to the notion of heavy/light tailedness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for raising several questions and highlighting where the paper lacks clarity.\n\n1. *On reparameterizing rewards to change whether they are heavy-tailed:*\n  \n    Reward can be reparameterized; however, in settings where the true reward is heavy-tailed, making reward artificially light-tailed or bounded can reward behavior that is unintended. \n   *  For example, a stock-trading agent should be rewarded by profit, but stock returns are known to be heavy-tailed. If we cap or otherwise transform rewards into $[-1, 1]$, it will have no incentive to take into account huge gains or losses. Since RLHF rewards as implemented in Ziegler et al are unbounded, clipping or transforming rewards could itself cause reward misspecification. We think this also applies when rewards are bounded but potentially very large.\n    * In some cases, mostly when the reward is not the true intended one, it is possible to reparameterize the reward without adverse effects. In the RL literature for Atari games, rewards are changes in score clipped to $[-1, 1]$ [1].\n\nThank you for highlighting this, we will discuss when reward can or cannot be reparameterized in Section 7.\n\n2. *On precedent for DMRMDP*\n\n    * As for precedent, deterministic MDPs are common in the literature e.g. in [2], but we have not seen the Markovian returns property in this exact form. With our definition, we are not trying to introduce a new setting, but rather to list the properties of RLHF that imply our theorems, so the relevance of our results is clear.\n    * We intend that trajectories must terminate in a sink state. Thank you for pointing out it is unclear, we will clarify the definition.\n3. *Is the converse of theorem 1 also true?*\n\n    Theorem 3 is effectively a converse to Theorem 1, and shows that if Q is light-tailed, the maximum achievable mean is no higher than the mean of Q in the limit as $\\epsilon \\to 0$. We will restate this theorem as an equivalence statement. Very good point! \n\n4. *Synthetic experimental setup*\n\n    We agree that a synthetic experimental setup would add value, and plan to have one in the camera ready. One difficulty is in the modeling choices to make for a realistic setup, especially if we want to relax independence between utility V and error X. (A big takeaway for us was that, because independence + light-tailed error precludes overoptimization in theory, but we observed RMs to have light tails, the overoptimization observed in experiments like Gao et al is likely from broken independence, e.g. a negative correlation between error and utility. This broken independence could take many forms and different models could easily result in different outcomes.)\n5. *Precise meaning of heavy/light tail under a policy*\n\n    We are taking actions as latent variables that determine the reward, but taking the expectation over randomness in the reward assigned by the environment for each possible trajectory (note in Theorem 2, $g(\\tau)$ is the average reward of trajectory $\\tau$). So if a certain policy in a certain environment has a 50/50 chance between producing trajectories $\\tau_a$ and $\\tau_b$, and the final state of $\\tau_a$ gets reward with a normal distribution with mean 10, and the final state of $\\tau_b$ gets reward with a Student-t distribution with mean 20, the relevant distribution of $U$ will be discrete with a 50/50 chance between 10 and 20.\n\nAll these questions as well as the two nitpicks will be, or have been already, addressed in the next version of the paper. We welcome any more comments or questions you might have, as these are good feedback that has improved the paper. In addition, we invite you to revise your score if your concerns have been answered to your satisfaction. Thank you for a thoughtful review!\n\n[1]: https://arxiv.org/abs/1709.06009 Machado, Marlos C., Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. \"Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents.\" Journal of Artificial Intelligence Research 61 (2018): 523-562.\n\n[2]: Ronald Ortner, Online regret bounds for Markov decision processes with deterministic transitions, in: Algorithmic Learning Theory, 19th International Conference, ALT 2008, Proceedings, 2008, pp. 123–137.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TZYeA2UzgH", "reviewer": "Reviewer_ngHC", "summary": "This paper investigates the effectiveness of using KL divergence for regularization in reinforcement learning from human feedback (RLHF), particularly when dealing with reward functions that have misspecified errors. It introduces the concept of \"catastrophic Goodhart,\" a scenario where policies can achieve extremely high rewards with heavy-tailed errors, but without actual improvement in utility.", "questions": "Please see the weakness section", "limitations": "please see weakness", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper addresses a critically important problem within the domain of RLHF --- the challenge of dealing with reward misspecification, particularly in scenarios where the error distribution is heavy-tailed. The authors defined the problem as \"catastrophic Goodharts\" and provided analysis accordingly.", "weaknesses": "[writing / presentation] The paper contributes to the field of LLM safety/alignment, yet the writing could be made more engaging and accessible. Improving clarity, enhancing the depth of discussions, and ensuring visual and structural appeal could greatly increase its impact and readability. These changes would not only cater to experts in the field but also to readers who may be new to this area of study.\n\n[Experiments] the experimental evidence could be further bolstered to validate the presented claims comprehensively. Specifically:\n\n- The experiments primarily focus on simulated environments. Including real-world applications could demonstrate the practical implications and limitations of the findings more effectively.\n\n- Exploring a wider variety of reward functions, particularly those commonly used in commercial or industrial settings, could help ascertain the generalizability of the catastrophic Goodhart phenomenon across different contexts.\n\n- Additional experiments comparing KL divergence with other regularization or penalty methods could offer a clearer understanding of its relative efficacy and limitations.\n\n- How about other optimization methods like Best-of-N where KL divergence term is not used? \n\n- Will the implementation like LoRA affect the proposed effect?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, which gives several intriguing suggestions for experiments, as well as presentation improvements.\n\nAs for readability and clarity, we have made various edits in the latest revision of the paper, including streamlining the background section for readers unfamiliar with AI alignment, clarifying the relationship of DMRMDP to RLHF, and improving presentation of graphs. Also see several changes we have already or plan to address in the rebuttal to reviewer G3eM.\n\nWe agree that further experimental evidence could be valuable, with some caveats.\n* *Wider variety of reward functions / real-world experiments:* We think work here would be valuable, but it could risk being redundant with the wealth of examples of overoptimization in the existing literature (see concluding paragraph).\n* *Best-of-N experiments:* This would complement our theorems 5 and 6 on optimization by conditioning, which describe the best-of-N distribution. We have already done the experiment to show that best-of-N succeeds in a toy light-tailed regime but fails in a toy heavy-tailed regime, and we will include this in the appendix of the final version.\n* *Additional experiments comparing KL divergence with other regularization or penalty methods:* We will add a synthetic experiment with KL divergence, in response to this review and that of reviewer G3eM. The experiment demonstrates whether a reward function with artificially heavy-tailed error causes catastrophic Goodhart in the KL divergence setting. We are not sure which other regularization schemes we should try: the RLHF literature uses KL divergence, and Best-of-N is an alternative proposed precisely to avoid overoptimization. Perhaps an integral probability metric like the Wasserstein distance, or quantilizing optimization, which only goes up to a percentile of the misspecified reward? It would be good if you could clarify which schemes would be important to try here. \n* *LoRA:* We think studying LoRA would be valuable to nail down exactly when inductive bias prevents catastrophic Goodhart, but considering that a search over LoRA hyperparameters and environments (see paragraph 4 of rebuttal to G3eM for why a single environment is as yet insufficient) would increase the compute requirements by orders of magnitude, this would be valuable follow-up work.\n\nAs for why we do not plan to implement all experiments suggested, we see this work as primarily providing a theoretical framework for experimental work already done by others, especially Gao et al [1] and specification gaming [2, 3]. The latter gives 60 examples of real-world utility loss due to specification gaming, several of which we think are examples of \"catastrophic Goodhart\". The link to previous specification gaming work will be clarified in the latest version.\n\nWe thank reviewer ngHC for their feedback. We hope that the theorems, existing experiments, and additional experiments listed above (demonstration of catastrophic Goodhart, and best-of-N study) address the given concerns. Please let us know what weaknesses remain and what we can do to address them, and consider revising your review if we have addressed your concerns to your satisfaction. We would welcome any further feedback and comments. Thank you!\n\n[1] Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward\nmodel overoptimization. In Krause, A., Brunskill, E., Cho, K.,\nEngelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings\nof the 40th International Conference on Machine Learning,\nvolume 202 of Proceedings of Machine Learning Research,\npp. 10835–10866. PMLR, 23–29 Jul 2023. URL https://\nproceedings.mlr.press/v202/gao23h.html\n\n[2]: https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/\n\n[3]: https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EfU77hyVEm", "reviewer": "Reviewer_SFxG", "summary": "This paper analyzes a phenomenon called \"catastrophic Goodhart\": in RL training, suppose the learned reward function is the sum of the true utility function and some noise, then if (1) the true utility and the noise are independent and light-tailed, maximizing reward with KL regularization can also give high utility, while if (2) the noise is heavy-tailed, then there exists a policy with arbitrarily high reward and arbitrarily small KL penalty, but its true utility is close to the initial policy. Empirical results suggest that open-source reward models could be light-tailed.", "questions": "I have one question regarding Theorem 4: the final conclusion is that the expected utility can be arbitrarily large, but if the utility function is bounded, I don't understand why its expectation can be arbitrarily large.\n\nI also have a more open-ended question: if we make the proxy reward function bounded (e.g., by clipping), do we have similar results?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper considers whether KL regularization is enough to handle  reward misspecification, which is an important problem. This paper also gives clean answers to this question: if utility and noise are independent and light-tailed, then KL regularization is enough, while if noise is heavy-tailed, we may have very little improvement in utility even with high proxy reward and small KL penalty. I think these results are nice and enhance our understanding of this problem.", "weaknesses": "One missing piece is the implicit regularization effects of the optimization algorithms: although there exists a policy with low utility even under a KL penalty, probably the training algorithm can avoid those bad policies and still find a good one. It will be nice if we can understand whether certain optimization techniques can solve this issue.\n\nSome additional comments:\n1. The notation $U$ and $V$ are used inconsistently in the paper: on line 34, $U$ denotes the true utility and $V$ denotes the proxy reward, while on line 71 their meanings are reversed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the positive feedback. To address the weaknesses and questions:\n* *Implicit regularization:* we acknowledge this as a weakness, and have some ideas for how implicit regularization can prevent Goodharting. But there is not really any evidence yet, and we would be excited to see follow-up work like this.\n* *Typo on line 34:* we have fixed this in the latest version. Thank you for pointing it out!\n* *Question about theorem 4:* Thank you for pointing out that we need to assume unbounded light-tailed $V$, we inadvertently dropped it and will explicitly state it for the final version. For the result to hold, the proxy reward should be unbounded so that $\\lim _{\\beta \\rightarrow 0^{+}} \\frac{1}{2} E[V \\mid \\pi] = \\infty$ in the proof (for reference $\\beta$ is the regularization coefficient of the KL term).\n* *When proxy reward is bounded:* The optimal policy is Boltzmann rational. So how much utility we get will be determined by whether, for state-actions which are close to optimal, the reward is from utility or error. We roughly think that if you clip at large enough values, the asymptotic results in this paper will basically transfer.\nWith more aggressive clipping, you tend to get both error and utility components when the reward is optimized, so neither of the scenarios in our theorems holds (neither wholly error in heavy-tailed nor arbitrarily high utility in light-tailed). It is possible we can characterize how much utility or error the solution will have based on their ratios for the largest $V$, but we’ll leave that to future work.\n\nClipping can also distort the reward (see response to reviewer G3eM), so while it prevents overoptimization somewhat it also rewards incorrect behavior.\n\nWe would welcome any more questions or feedback you have. Thank you for your review!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1SXL5CXfT3", "reviewer": "Reviewer_znAw", "summary": "The authors presents a study on the RM in RLHF. Theoretical results of the paper assume that the reward from the RM consists of a true reward plus a noise term which is modeled as a random variable. Under this model the authors show that\n\n- If the noise terms are heavy-tailed there exists policies with vanishing KL-divergence but infinite proxy reward.\n- If the noise terms are IID and light-tailed, then KL divergence suffices to guarantee good outcomes\n\nThe authors also investigate the actual rewards given by real RMs. They find that the rewards on random sequences are roughly", "questions": "- The theoretical results are related to asymptotically heavy tails. What would happen in I take my RM and clip the output to be in [-10000, +10000]. I’m guessing in practice nothing would happen, but would the theoretical results would be invalid since the tails disappear?\n\n- In practice the output of the RM isn’t a random variable, it’s deterministic. Can you comment on the validity of the assumption that the rewards are random?", "limitations": "na", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and easy to read.\n- The theoretical results are relevant.", "weaknesses": "- there is probably some discrepancy between the theory and practice. The results are asymptotic, it would be better if the statements could hold in non asymptotic regimes. \n- The experimental results seems to show that the rewards are not heavy tailed, and thus the theory seems to suggest that KL-divergence should work well. We already know that it does work well from practice, so there is no novel take-home message.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful criticism. We considered writing more about how asymptotic results relate to real life, and will do so in the next version of the paper. We additionally hope that this addresses your concerns:\n* *Asymptotic vs bounded/clipped:* With a bounded RM the asymptotic results would indeed not apply. In practice, the same pattern of rare failures producing most of the reward and resulting in near zero utility would likely apply, with sufficiently regularized policies having, say, a 0.1% chance of 10000 reward and otherwise matching the base policy. This could change if error and utility are nonindependent and the base policy is close to optimal so that it actually encounters lots of clipped rewards. (See also the next paragraph, and the last bullet point of our rebuttal to SFxG.) Though it would be possible to prove things about the bounded case, we presented the asymptotic results because we thought they were much cleaner.\n* *Take-home message:* We think there is an interesting take-home message. It is true that KL divergence penalties work sufficiently well to control overoptimization in current language models, though overoptimization has been demonstrated in practice by the results of Gao et al. But our theoretical results show that light-tailed + independence *prevents* overoptimization. Therefore in such settings we conclude that overoptimization is most likely a result of independence being violated, perhaps as textual features that increase error also decreasing utility. We have improved section 7.4, which discusses this, in the latest version.\n* Deterministic RM: Our theorems still apply when the reward model is deterministic. The randomness in our theorems comes from the sentences generated by the LLM (which are random) being passed through a the deterministic reward model, which results in a joint distribution over $V$ and $X$. One of the contributions of our paper is the framing that, to study the final achieved reward, it is sufficient to examine this 2d distribution. Did we understand your question correctly?\n\nWe welcome any more comments you might have. Thank you for the review!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you to all the reviewers for their thoughtful feedback, constructive criticism, and recognition of our paper's contributions. We appreciate the time and effort invested in evaluating our work and suggesting improvements.

The reviewers unanimously agreed the paper is technically sound and that it is relevant to an important problem in RLHF: reward misspecification. The reviewers' main concerns centered around clarity, experimental depth, and the relationship between theory and practice. We have addressed these as follows:

1. Clarity and presentation:
   - We have improved the overall clarity and readability of the paper, particularly in the background section and graph presentations (see resp. to ngHC).
   - We clarified the relationship between DMRMDPs and RLHF (see resp. to G3eM).
   - We've fixed notation inconsistencies and typos (e.g., the missing unboundedness assumption in Theorem 4, thank you SFxG for pointing it out).

2. Synthetic experiments: we've conducted new experiments demonstrating our results with Best-of-N in a synthetic reward model. We also plan to do experiments optimizing the synthetic reward for KL divergence regularization, and possibly other kinds of regularization (see responses to ngHC and G3eM).

3. Implicit regularization/inductive bias in optimization: reviewers SFxG and ngHC asked about studies of how the RLHF setup regularizes the solutions found, beyond what is just optimal. As part of the synthetic experiments in point 2, we will study the implicit regularization of optimization of LLMs. However, we believe studying LoRA should be part of future work. 

4. Bounded rewards: in responses to znAw and SFxG, we have provided more intuition on what happens with bounded or clipped rewards, and will add that to the paper. Concrete theoretical results on this are difficult and should be part of future work.

We believe these changes and clarifications significantly improve the paper while maintaining its core contributions. The theoretical framework we provide offers a new lens for understanding existing experimental work on specification gaming and overoptimization (see response to ngHC).

We would be excited to continue discussing, and remain committed to addressing any remaining concerns. We kindly ask the reviewers to consider revising their ratings if they feel we've adequately addressed their initial concerns.

Thank you again for your valuable input, which has undoubtedly strengthened our paper.

Meta-review: This paper studies whether KL regularization is sufficient to mitigate the (potentially-catastrophic) errors due to reward-misspecification in RL.
They prove that KL regularization works under a fairly natural assumption on the type of reward mis-specification (independent noise with light-tails). Conversely, they prove that even KL-regularization can be catastrophic if this assumption is broken (ie if noise is heavy-tailed).

Reviewers agreed that the work presents a clean and novel theoretical insight to the important problem of reward misspecification.
There were several concerns about clarity and experimental support, which the authors addressed in their rebuttal. After the rebuttal, all reviewers recommended acceptance. I thus recommend acceptance.
Decision: Accept (poster)