Title: Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification
Venue: NeurIPS 2024
Accepted: yes
Keywords: RLHF

extreme value theory

reward misspecification
Abstract: When applying reinforcement learning from human feedback (RLHF), the reward is learned from data and, therefore, always has some error. It is common to mitigate this by regularizing the policy with KL divergence from a base model, with the hope that balancing reward with regularization will achieve desirable outcomes despite this reward misspecification. We show that when the reward function has light-tailed error, optimal policies under less restrictive KL penalties achieve arbitrarily high utility. However, if error is heavy-tailed, some policies obtain arbitrarily high reward despite achieving no more utility than the base model--a phenomenon we call catastrophic Goodhart. We adapt a discrete optimization method to measure the tails of reward models, finding that they are consistent with light-tailed error. However, the pervasiveness of heavy-tailed distributions in many real-world applications indicates that future sources of RL reward could have heavy-tailed error, increasing the likelihood of reward hacking even with KL regularization.

Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification

Abstract

When applying reinforcement learning from human feedback (RLHF), the reward is learned from data and, therefore, always has some error. It is common to mitigate this by regularizing the policy with KL divergence from a base model, with the hope that balancing reward with regularization will achieve desirable outcomes despite this reward misspecification. We show that when the reward function has light-tailed error, optimal policies under less restrictive KL penalties achieve arbitrarily high utility. However, if error is heavy-tailed, some policies obtain arbitrarily high reward despite achieving no more utility than the base model-a phenomenon we call catastrophic Goodhart. We adapt a discrete optimization method to measure the tails of reward models, finding that they are consistent with light-tailed error. However, the pervasiveness of heavy-tailed distributions in many real-world applications indicates that future sources of RL reward could have heavy-tailed error, increasing the likelihood of reward hacking even with KL regularization.

1 Introduction

Kullback-Leibler (KL) divergence constraints in reinforcement learning (RL) are employed to stay in regimes where the objective is accurate enough. Some on-policy (Schulman et al., 2015, 2017) and many off-policy (Abdolmaleki et al., 2018; Jaques et al., 2019) policy gradient algorithms employ KL constraints or penalties during optimization to prevent the policy from deviating too much from the data collection distribution. This ensures that estimates of each action's advantage are reliable enough to update the policy in a helpful way.

Reinforcement learning from human feedback (Christiano et al., 2017; Ziegler et al., 2020, RLHF) is a very popular method to induce desirable behavior in language models. RLHF starts with a base pre-trained model, then learns a reward function from human annotator data. Next, it trains an RL policy to maximize this reward, while penalizing high KL divergence from the policy to the base model. RLHF uses an on-policy algorithm and has accurate advantages, but the reward function is always somewhat misspecified compared to desired behavior, due to insufficient data, human biases, and other factors.

The main purpose of the KL penalty in RLHF is to limit the consequence of reward modeling errors by keeping the policy within a distribution similar to that on which it was trained. Ideally, in the low-KL regime the reward model's errors are small enough that it provides correct updates to the base model. Gao et al. (2023) empirically supports this view: if the KL divergence in RLHF is allowed to grow too much, with a misspecified reward, the model's performance on the true utility starts to decrease.

FAR AI

❛❞/a114✐❛❅❢❛/a114✳❛✐

We ask: can we obtain good outcomes from misspecified reward in RLHF by controlling the KL divergence? That is, if there is some error between the true reward V and the proxy reward U , can the KL help us to still optimize V ? Using mathematical proof, we answer the question in the negative for heavy-tailed errors: there exist policies which have infinite proxy reward U , but whose KL with the base model vanishes (these have undetermined V ). We term this phenomenon 'catastrophic Goodhart', after Goodhart's law.

If the misspecification errors are independent and light-tailed, the KL divergence does suffice to guarantee good outcomes. There may also be guarantees under weaker assumptions, but assumptions that intuitively seem sufficient are often not (see Section 5).

Possibly, other regularization schemes would guarantee good outcomes for heavy-tailed errors, but this is not just a problem of KL. We show that optimizing by conditioning on large reward U has similar outcomes in light- and heavy-tailed regimes.

Empirically, open-source language reward models seem to be light-tailed, which does not imply light-tailed errors but suggests it (Section 4.1). However, the errors are likely not independent and, given the prevalence of heavy-tailed distributions in the real world, error in future reward models may also be heavy-tailed. In any case, the present success of RLHF with misspecified rewards cannot be explained solely by the KL regularization in its objective.

2 Background

2.1 KL divergence and KL regularization

Recall that KL divergence between two distributions P and Q is defined as

If we have two policies π, π 0 , we define D KL ( π ∥ π 0 ) as the KL divergence between the distributions of actions taken on the states in trajectories reached by π . That is, if Tr ( π ) is the distribution of trajectories taken by π , we penalize D KL ( π ∥ π 0 ) ≜ E s ∈ T,T ∼ Tr ( π ) [ D KL ( π ( s ) ∥ π 0 ( s ))] .

In RLHF, it is common to use the regularization term βD KL ( π ∥ π 0 ) to prevent the learned policy from deviating too much from the base policy, which can prevent unstable behavior or overfitting to the reward model. If our reward model gives reward U , then the optimal policy for RLHF with a KL penalty is

Often the regularization parameter β is dynamically adjusted to keep the D KL near some target value (Ziegler et al., 2020).

2.2 Heavy-tailed distributions

A distribution P over R with cumulative distribution function (CDF) F P is heavy-tailed if its tail function ¯ F P ( x ) ≜ 1 -F P ( x ) satisfies

Heavy-tailed distributions are well-known in statistics to have a higher probability of producing a single extreme value. For example, if the sum of two independent variables from heavy-tailed distributions is large, it is most likely due to one extreme sample rather than two equally large samples. (Wierman, 2013)

2.3 Reward misspecification and Goodhart's Law

Reward misspecification has caused low-utility outcomes in practice; for example, in (Clark and Amodei, 2016), an RL agent trained to play a racing videogame according to a misspecified reward function achieves a high score while failing to complete the course.

Gao et al. (2023) introduce the concept of 'overoptimization': optimizing for a proxy objective decreases performance according to the true objective. This raises the question: in general, when RLHF reward is misspecified, when does the optimal policy produce high utility?

By applying the proxy reward and true reward functions to a distribution over text (generated by an LLM), we get two scalar random variables, which we call U for proxy reward and V for true reward / utility. Then we can define the error in the proxy reward as X ≜ U -V , so that U = X + V . Framed this way, optimization for a proxy reward U is a mix of desirable optimization for V and undesirable optimization for X . The joint distribution of V and X determines the limiting value of V as we apply more optimization. When we say that reward misspecification can have negative effects, we mean that too much variance in X can "redirect" the optimization pressure from V to X , and prevent utility gain from optimization.

Reward misspecification is also studied by (Lambert and Calandra, 2024), (Laidlaw et al., 2024), and others. Laidlaw et al show that a KL penalty between action distributions can be ineffective, and propose instead regularizing state occupancy measure. Our results show an inherent weakness of KL divergence, including when applied to state occupancy measure.

We prove that in many cases, V → 0 in the limit of optimization for some proxy U . We call this phenomenon 'catastrophic Goodhart', after Goodhart's law: 'when a measure becomes a target, it ceases to be a good measure' (Strathern, 1997). In these cases, the end result of optimizing for a proxy of V is no better than not optimizing at all. However, in other cases, V →∞ despite some reward misspecification; in these cases the reward misspecification is not severe enough to prevent good outcomes.

3 Theoretical results

When applying KL regularization, the trained model is regularized towards some base policy π 0 . One would hope that a KL penalty can produce good outcomes even in the case of reward misspecification; that is, if the reward U is the sum of true utility V and an error term X , we would hope that optimal policies under a KL penalty achieve high V even if the magnitude of X is large. We show that this is not always the case: Corollary 1 of Theorems 1, 3, and 2 establishes that when X ( π 0 ) is heavy-tailed, there are arbitrarily well-performing policies π with E π [ V ] ≈ E π 0 [ V ] . However, Theorem 4 shows that when error is light-tailed and independent of V , the optimal policy under a KL penalty results in V > 0 , and V can be made arbitrarily large. Thus, the tails of the error distribution are crucial in determining how much utility will result from optimization towards an imperfect proxy.

Theorems 5 and 6 (Section 3.4) show that the relationship of catastrophic Goodhart to heavy-tailed error is not just a quirk of KL divergence by using a different model of optimization based on conditioning on high reward values. Under this model (and given additional regularity conditions), it is also true that heavy-tailed error results in catastrophic Goodhart and light-tailed error plus independence results in arbitrarily large utility. All proofs are in the appendix.

3.1 KL divergence on heavy- and light-tailed distributions

Theorem 1. Given any heavy-tailed reference distribution Q over R with mean µ Q , and any M,ϵ > 0 , there is a distribution P with mean µ P > M and D KL ( P ∥ Q ) < ϵ .

Outline of proof (see appendix for full proof): WLOG take µ Q = 0 . If we set P t to upweight the probability mass of Pr P t ( X > t ) to c/t for some c, t , then the mean of P t will be approximately at least c . As t →∞ , the KL divergence D KL ( P t ∥ Q ) will shrink to zero.

Intuitively, in a heavy-tailed distribution, events with extremely high x are not very rare, so you don't pay much of a KL penalty to upweight them so they happen about 1 /x of the time.

Theorem 2. However, if the distribution Q is light-tailed and d = D KL ( P ∥ Q ) is bounded, then µ P is bounded, and µ P -µ Q → 0 as d → 0 .

3.2 RLHF with KL penalty under heavy-tailed return distribution

We now adapt our result to the case where the policy is a language model and we are training it using RLHF. We are now applying KL divergence over the policies rather than the return distributions.

We first formally define the properties of RLHF on language models that cause the result to hold: namely, when when considered as a Markov decision process (MDP), environmental transitions are deterministic and return depends only on the final state reached.

Definition: A deterministic-transition MDP with Markovian returns (DMRMDP) is an MDP ( S , A , P, R ) such that:

The transition function P : S ×A → S is deterministic, i.e., for each state s ∈ S and action a ∈ A , there exists a unique state s ′ ∈ S such that P ( s ′ | s, a ) = 1 . In RLHF: the transition is appending the generated token a to the context s .

There is a set of sink states E ⊆ S that terminate every trajectory, which is disjoint from the set of start states.

In RLHF: The sink states are sequences ending in ❁❊❖❙❃ or above a certain length.

Returns are Markovian; that is, for any two trajectories τ = ( s 1 , a 1 , . . . , s n ) , τ ′ = ( s ′ 1 , a ′ 1 , . . . , s ′ n ) , if s n = s ′ n , then τ and τ ′ have identical return distributions. Equivalently, for the trajectory random variable T = ( S 1 , A 1 , . . . ) distributed according to any policy, with return G , G ⊥ ⊥ ( S <i , A <i ) | S i for any i ≥ 1 .

In RLHF: the return only depends on the full generated string, which is the final state.

The language model stochastically outputs the next token a given s , and corresponds to the policy. A DMRMDP is therefore a good model of RLHF.

Theorem 3. Let W = ( S , A , P, R ) be a deterministic-transition MDP with Markovian returns. Given W we define the function that takes policies to trajectories Tr : ( S → ∆ A ) → ∆( S × A ) ∗ , and the average return function g : ( S × A ) ∗ → R , which induces a function G : ∆( S × A ) ∗ → ∆ R . Let π 0 : S → ∆ A be some base policy. If G ◦ Tr ( π 0 ) is heavy-tailed with finite mean µ Q , then for any M,ϵ > 0 , there is a policy π with mean return E [ U | U ∼ G ◦ Tr ( π )] > M and E s ∈ T,T ∼ Tr ( π ) [ D KL ( π ( s ) ∥ π 0 ( s ))] < ϵ .

Corollary 1. Theorems 2 and 3 imply that when utility is light-tailed, reward modeling errors make the proxy reward heavy-tailed, and a policy π is regularized severely enough to have KL divergence values approaching zero, the reward E [ U ( π )] can go to infinity while utility E [ V ( π )] approaches a value no higher than the base policy.

3.3 Light-tailed + independence imply E [ V ] →∞

Theorem 4. If U = X + V with X and V both light-tailed and V unbounded, and the distribution of U is continuous, and π ∗ ( β ) ≜ arg max π E [ U ( π )] -βD KL ( π, π 0 ) , then lim β → 0 + E [ V ( π ∗ ( β ))] = ∞ .

3.4 Conditioning as alternate model of optimization

Although we think a KL divergence penalty or cap is the most realistic setting for RLHF, it is not the only model of optimization where heavy-tailedness of the error determines whether catastrophic Goodhart occurs. Consider another model of optimization where U = X + V as before, but we simply condition on U being higher than some threshold t . 2 Then we are interested in the quantity lim t →∞ E [ V | X + V ≥ t ] . If we slightly strengthen the heavy-tailedness and light-tailedness assumptions, heavy-tailed error results in catastrophic Goodhart, while light-tailed error results in arbitrarily high expected utility.

3.4.1 Conditioning with heavy-tailed error produces catastrophic Goodhart

Theorem 5. Let X and V be two independent random variables with CDFs F X and F V and tail functions ¯ F V ≜ 1 -F V , ¯ F X ≜ 1 -F X such that

V has a finite mean.

X is subexponential; that is, lim x →∞ Pr ( X 1 + X 2 >x ) Pr ( X>x ) = 2 if X 1 , X 2 are two independent samples from X . This is a slightly stronger property than being heavy-tailed.

The tail of V is sufficiently lighter than the tail of X that lim t →∞ t p ¯ F V ( t ) ¯ F X ( t ) = 0 for some p > 1 .

Then lim t →∞ E [ V | X + V ≥ t ] = E [ V ] ; that is, catastrophic Goodhart occurs in the limit of optimization for U = X + V .

The proof is included in the appendix. It requires expressing the conditional expectation in question as ∫ ∞ -∞ vf V ( v ) Pr ( X>t -v ) ∫ ∞ -∞ f V ( v ) Pr ( X>t -v ) , then partitioning the interval ( -∞ , ∞ ) into four regions and bounding the integrand in the numerator above by a different quantity in each region.

3.4.2 Conditioning with light-tailed error produces arbitrarily high utility

Theorem 6. Let X,V be independent random variables such that lim t →∞ ¯ F X ( t +1) ¯ F X ( t ) = 0 . (This implies that X has tails that are dominated by e -cx for any c, though it's a slightly stronger claim because it requires that X not have large jumps in the decay of its tails.) Then for any V with a finite mean which has no upper bound, lim t →∞ E [ V | X + V > t ] = ∞ .

Theorem 6 generalizes a consequence of the "Regressional Goodhart Identity" in (Gao et al., 2023).

4 Experiments

Our theoretical results now raise the question of whether the error in reward models is heavy-tailed or light-tailed in practice. 3 If we observe the reward distribution to be light-tailed, this is a strong indication that error is light-tailed. 4

To empirically test whether the reward is heavy-tailed, we consider two lines of evidence: examining the distributions directly through random sampling and temperature-1 sampling, and finding adversarial token sequences that get high rewards. We examine one small and one medium reward model that performed reasonably well on RewardBench (Lambert et al., 2023). The small model is an OpenAssistant model based on Pythia 1.4B, and the medium model is Starling 7B-alpha (Zhu et al., 2023) 5 .

For random sampling, we sample 30000 length-1024 sequences of uniformly random tokens and observe the distribution of rewards assigned by both Pythia 1.4B and Llama 7B-chat. We also use Llama 7B-chat to generate 16000 length-133 sequences at temperature 1 and observe the distribution of rewards assigned by Starling 7B-alpha.

Because sampling is inefficient at probing the extreme tail, we also find token sequences that optimize Starling 7B-alpha for reward. We considered Greedy Coordinate Gradient (GCG) from (Zou et al., 2023), a method used to find adversarial suffixes that circumvent jailbreaking, but decided on a faster version of GCG called Accelerated Coordinate Gradient (ACG) from (Haize Labs, 2024). See Table 4 for ACG hyperparameters.

Generating plots took about 5 GPU-hours on 1x Nvidia H100, and running ACG took a further 8 hours.

4.1 Results

When sampling token sequences, both the Pythia model on random inputs (Figure B.1) and Starling 7B-alpha on Llama-generated inputs (Figure 2) appear approximately normal and, therefore, lighttailed. Starling on random inputs (Figure 1 is ambiguous, with the exponential Q-Q plot having an outlier that could indicate a heavy-tailed distribution, but the Hill estimator is consistent with a

Plots of reward from 30000 random token sequences to Starling 7B alpha reward model

Histogram

Normalprobabilityplot

2000

0

1750

1500

-1

Frequency

1250

reward

-2

1000-

750

-3

500

250

-0

5

-5

-4

-3

-2

-1

0

4

-2

0

2

4

Reward

Normal quantile

Exponentialprobabilityplot

HillEstimatorforHeavy-tailedDistribution

0.40

0.5

0.35

0°0

0.30

0.5

Hill estimator

0.25

reward

-1.0

0.20

1.5

0.15

2.0

0.10

2.5

0.05

0

2

4

6

8

10

0

25

50

75

100

125

150

175

Exponentialquantile

Number of upper order statistics (k)

light-tailed distribution. Because Llama-7B-chat is a more reasonable base model than a completely random policy, we believe that Starling 7B-alpha is more likely to be light-tailed for the purposes of our theoretical results.

The ACG results need some interpretation. The KL divergence between two distributions P and Q if P is the same as Q a fraction 1 -α of the time, but is some value x a fraction α of the time is given by D KL ( P ∥ Q ) = [(1 -α ) q ( x ) + α ] log ( (1 -α ) q ( x )+ α q ( x ) ) +(1 -α ) log(1 -α )(1 -q ( x )) .

When α is small but much larger than q ( x ) , we approximate this to first order as D KL ( P ∥ Q ) ≈ α log ( α q ( x ) ) . In Theorems 1 and 3, we prove that when the error is sufficiently heavy-tailed, a policy

Plots of reward from 16000 Llama-7B-chat generated inputs to Starling 7B alpha reward model

Histogram

Normal probability plot

500

2

400

Frequency

00

reward

0

-2

200

100

4

0

-6

-6

-4

-2

0

2

-4

-2

2

Reward

Normal quantile

Exponentialprobabilityplot

4

0.16

0.14

3

0.12

Hill estimator

0.10

reward

2

0.08

0.06

1

0.04

0.02

0

0.00

0

2

4

6

8

0

50

100

150

200250

300

Exponential quantile

Numberofupperorderstatistics(k)

that gets extremely large reward a small fraction of the time will achieve high expected reward with low KL divergence. This is not the case here because the rewards achieved through ACG were small and the log-probabilities extremely negative. For example, a policy that matches Llama 2-chat's base reward 99% of the time and uses the highest-reward input generated by ACG α = 1% of the time will have KL divergence from Llama 2-chat of α (log( α ) -1339 . 70) = 13 . 35 nats, but reward only about α ∗ (2 . 2377 -0 . 3329) = 0 . 02571 greater than the base model, far less than can be obtained with the same KL divergence by conditioning.

5 Discussion and Limitations

5.1 How likely is catastrophic Goodhart?

The low-KL policies that result in catastrophic Goodhart are not a unique optimal policy, just one family of high-performing policies. When optimizing E [ U ( π )] -βD KL ( π, π 0 ) , the outcome depends on RL training dynamics; it could be that D KL → 0 causing catastrophic Goodhart, but more likely both terms will go to infinity, potentially allowing V →∞ . Catastrophic Goodhart can be prevented by using a light-tailed or bounded reward function.

Even so, catastrophic Goodhart is likely to occur in many scenarios where KL regularization is naively employed in an attempt to avoid Goodhart's Law:

If we maximize σ ( E [ U ]) + D KL ( Tr ( π ) ∥ Tr ( π 0 )) , where σ is a bounded function (e.g. sigmoid), all near-optimal policies will have V ≈ 0 . Since we can only obtain so much reward from σ ( E [ U ]) , it pays to make the KL (and thus V) go to zero.

If we cap KL to a finite value (or dynamically adjust the KL penalty to target a finite KL, as done in Ziegler et al. (2020), then E [ V ] is also upper bounded by a finite value (see Theorem 2), and we think it is likely that E [ V ] ≈ 0 . Consider a toy model where an AI can adjust three parameters: true quality V of responses, frequency of reward hacking (producing actions with extremely high X), and severity of hacking (value of X on those actions). Adjusting the policy to increase E [ U ] without increasing KL increase the severity of hacking while decreasing either frequency of hacking or quality of responses. When E [ U ] is already large, decreasing quality has much better returns than decreasing frequency. This is similar to Theorems 5, 6 about hard-threshold optimization.

Any way we maximize E [ U ( π )] -βD KL ( π, π 0 ) results in very large values of E [ U ( π )] , and there are a number of arguments that extreme optimization for an imperfect proxy can result in decreased utility due to tradeoffs between X and V ; e.g., the constrained resource scenario in (Zhuang and Hadfield-Menell, 2021).

5.2 Independence assumptions

Theorems 1-3 do not require any independence assumption, but Theorems 4, 5, and 6 require that error X and utility V are independent, which seems to be violated in practice. Future work could weaken this assumption, although intuitively obvious ways to weaken it result in the statement being false. 6

5.3 Stronger optimization methods

We did not search the entire space of token sequences, so we cannot rule out that the reward is heavy-tailed enough to cause catastrophic Goodhart in some situations. While it is intractable to search the more than 10 2000 possible token sequences, future work could get more evidence through more powerful optimization methods.

5.4 Reparameterizing reward

In some cases, a heavy-tailed reward can be reparameterized to make it light-tailed and avoid catastrophic Goodhart; however, in settings where the true reward is heavy-tailed, making reward artificially light-tailed or bounded can result in unintended behavior.

For example, a stock-trading agent should be rewarded by profit, but financial returns are known to be heavy-tailed. If we clip or otherwise transform rewards into a bounded interval, it will have no incentive to take into account huge gains or losses. Since RLHF rewards as implemented in Ziegler et al are unbounded, clipping or transforming rewards could itself cause reward misspecification.

In some cases, e.g. when the reward is not the true intended one, it is possible to reparameterize the reward without adverse effects. In the RL literature for Atari games, rewards are changes in score clipped to [ -1 , 1] (Machado et al., 2018).

5.5 Relation to previous overoptimization work

Gao et al. (2023) found that optimizing the reward of small reward models causes overoptimization: a decrease in utility with increasing optimization. However, we observed that reward models are lighttailed, and (Theorem 4) that independence combined with light-tailed error prevents overoptimization. We think this discrepancy is explained by dependence between error and utility. Policies optimized

for high error may activate features in the proxy reward models that are undesirable according to the true utility function. 7 More research is needed to understand why high-error completions have low utility and to design reward models that do not suffer from this problem; perhaps it is possible to construct reward models whose errors are in directions orthogonal to human preferences, so that the large-reward completions do not have lower utility.

6 Conclusion

We have argued that the purpose of the KL divergence regularization in RLHF is to mitigate reward misspecification. However, we have also proven that when errors in the reward function are heavytailed, it cannot serve this purpose: even with zero KL divergence, there are policies that achieve very high misspecified reward and no actual reward.

When errors are light-tailed and independent, the KL divergence can mitigate misspecification, but when they are dependent, this may not be possible. Thus, we must look to places other than the KL objective to explain the current success of RLHF and ensure its continued success in the future.

Acknowledgments and Disclosure of Funding

This work was supported by the Long-Term Future Fund (LTFF). We also thank the anonymous reviewers for their valuable feedback and constructive suggestions.

Impact Statement

As this work aims to improve the safety of future ML systems by characterizing a possible failure mode of reward misspecification in RLHF, we hope the social impact is positive. We see no particular ethical issues to discuss.

References

Abdolmaleki, A., Springenberg, J. T., Tassa, Y., Munos, R., Heess, N., and Riedmiller, M. (2018). Maximum a posteriori policy optimisation. CoRR .

Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc.

Clark, J. and Amodei, D. (2016). Faulty reward functions in the wild. Accessed: 2024-07-07.

Foss, S., Korshunov, D., and Zachary, S. (2013). An Introduction to Heavy-Tailed and Subexponential Distributions . Springer, 2 edition.

Gao, L., Schulman, J., and Hilton, J. (2023). Scaling laws for reward model overoptimization. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J., editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 10835-10866. PMLR.

Haize Labs (2024). Making a sota adversarial attack on llms 38x faster. ❤/a116/a116♣/a115✿✴✴❜❧♦❣✳❤❛✐③❡❧❛❜/a115✳❝♦♠✴ ♣♦/a115/a116/a115✴❛❝❣✴ . Accessed: 2024-05-22.

Hu, J., Wu, X., Wang, W., Xianyu, Zhang, D., and Cao, Y. (2024). Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11143 .

Jaques, N., Ghandeharioun, A., Shen, J. H., Ferguson, C., Lapedriza, A., Jones, N., Gu, S., and Picard, R. (2019). Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456 .

0.35

0.40

Threshold: t=1.50

Q(x)

0.35

0.40

Threshold: t=10.77

Qx)

0.35

0.40

Threshold: t=100.00

0.30

0.30

0.30

0.25

0.25

0.25

0.20

0.20

0.20

0.15

0.15

0.15

0.10

0.10

0.10

0.05

0.05

0.05

0.00

1

20

25

0.00

20

25

0.00

20

25

10

P_tDX)

Q(x)

10

P_t[X)

QDx)

10#

P,t0X)

Q0x)

10-1

10-1

10-1

10~

10-4

18-

10-

18

18-I -

16°f

8[x]

0_G

10-1#

13

29

[X]

18-1#.

[X]

0_KL

20-1#

Laidlaw, C., Singhal, S., and Dragan, A. (2024). Preventing reward hacking with occupancy measure regularization. arXiv . Accessed: 2024-07-07.

Lambert, N. and Calandra, R. (2024). The alignment ceiling: Objective mismatch in reinforcement learning from human feedback.

Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., and Hajishirzi, H. (2023). Rewardbench: Evaluating reward models for language modeling. 40 pages, 19 figures, 12 tables.

Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., and Bowling, M. (2018). Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research , 61:523-562.

Schulman, J., Levine, S., Moritz, P., Jordan, M. I., and Abbeel, P. (2015). Trust region policy optimization. CoRR .

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. CoRR .

Strathern, M. (1997). 'improving ratings': audit in the british university system. European Review , 5(3):305-321.

Wierman, A. (2013). Catastrophes, conspiracies, and subexponential distributions (part ii). ❤/a116/a116♣/a115✿✴✴/a114✐❣♦/a114❛♥❞/a114❡❧❡✈❛♥❝❡✳✇♦/a114❞♣/a114❡/a115/a115✳❝♦♠✴✷✵✶✸✴✶✷✴✶✼✴ ❝❛/a116❛/a115/a116/a114♦♣❤❡/a115✲❝♦♥/a115♣✐/a114❛❝✐❡/a115✲❛♥❞✲/a115✉❜❡①♣♦♥❡♥/a116✐❛❧✲❞✐/a115/a116/a114✐❜✉/a116✐♦♥/a115✲♣❛/a114/a116✲✐✐✴ . Accessed: 2024-06-26.

Zhu, B., Frick, E., Wu, T., Zhu, H., and Jiao, J. (2023). Starling-7B: Improving llm helpfulness & harmlessness with rlaif.

Zhuang, S. and Hadfield-Menell, D. (2021). Consequences of misaligned AI. Advances in Neural Information Processing Systems , 33:15762-15773. arXiv:2102.03896v1 [cs.AI].

Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. (2020). Fine-tuning language models from human preferences. arXiv:1909.08593v2 [cs.CL].

Zou, A., Wang, Z., Kolter, J. Z., and Fredrikson, M. (2023). Universal and transferable adversarial attacks on aligned language models.

A Proofs

A.1 Theorem 1

Restatement of Theorem 1. Given any heavy-tailed reference distribution Q over R with mean µ Q , and any M,ϵ > 0 , there is a distribution P with mean µ P > M and D KL ( P ∥ Q ) < ϵ .

Intuitively, in a heavy-tailed distribution, events with extremely high x are not very rare, so you don't pay much of a KL penalty to upweight them so they happen about 1 /x of the time. This is visually illustrated in Figure A.1.

Proof. WLOG let µ Q = 0 . We construct a sequence of distributions { P t } such that lim t →∞ E P t [ X ] ≥ c for any constant c , and lim t →∞ D KL ( P t ∥ Q ) = 0 . We define P t for any t > c thusly. Writing F P t ( x ) for the CDF Pr X ∼ P t ( X ≤ x ) and ¯ F P t ( x ) for 1 -F P t ( x ) , we let

Intuitively, we rescale the part of the distribution to the right of t evenly to have total probability c/t , which is less than 1 because t > c .

We must check that lim t →∞ E P t [ X ] = c . We can write

We know that E Q [ X | X > t ] > t because it is an integral of values strictly greater than t. Because E Q [ X ] = 0 is a weighted average of E Q [ X | X > t ] and E Q [ X | X ≤ t ] , and E Q [ X | X > t ] > 0 , we know E Q [ X | X ≤ t ] < 0 . So E Q [ X | X > t ] -E Q [ X | X ≤ t ] > t. We also know that for sufficiently large t , ( F P t ( t ) -F Q ( t )) > 0 . Intuitively, starting from Q , which has mean 0, P t moves a probability mass approaching c t from mean <0 to mean >t.

Now we can say

Because Q has a finite mean, lim t →∞ t ¯ F Q ( t ) = 0 , and so lim t →∞ E P t [ X ] ≥ c .

Now we check that lim t →∞ D KL ( P t ∥ Q ) = 0 :

Since both 1 -c/t and F Q ( t ) go to 1 as t →∞ , the left term goes to 0 , and so

Q is heavy-tailed, so by definition lim t →∞ e at ¯ F Q ( t ) = ∞ for all a > 0 . This implies that for every a > 0 there is a sufficiently large t c so that for all t > t c , ¯ F Q ( x ) > e -at , which means that log ¯ F Q ( t ) > -at .

Therefore for every a > 0 , lim t →∞ D KL ( P t ∥ Q ) ≤ lim t →∞ -c/t log ¯ F Q ( t ) < lim t →∞ --act t = ac , which since KL divergence is nonnegative means that lim t →∞ D KL ( P t ∥ Q ) = 0 as desired. ■

A.2 Theorem 2

Restatement of Theorem 2. If V is light-tailed, E Q [ V ] is zero, and d = D KL ( P ∥ Q ) is bounded, then E P [ V ] is bounded, and E P [ V ] → 0 as d → 0 .

Proof. Using Lagrange multipliers, we find that when KL divergence is minimized, we have P ( V )[ λ 1 log P ( V ) Q ( V ) + λ 2 -X ] = 0 for some constants λ 1 , λ 2 , so

That is, the new PDF is an exponential tilting of the old PDF. Now, what is E P [ V ] ? It's just ∫ ∞ -∞ CVe V/λ 1 Q ( X ) dV . If the distribution of V is heavy-tailed distribution, this is ∞ ; if it is light-tailed, this is some finite value.

When d = 0 , P and Q are identical, and E [ V ] = 0 . So by a continuity argument, E P [ V ] → 0 as d → 0 . ■

A.3 Theorem 3

Restatement of Theorem 3. Let W = ( S , A , P, R ) be a deterministic-transition MDP with Markovian returns. Given W , we define the function that takes policies to trajectories Tr : ( S → ∆ A ) → ∆( S × A ) ∗ , and the average return function g : ( S × A ) ∗ → R which induces a function G : ∆( S × A ) ∗ → ∆ R . Let π 0 : S → ∆ A be some base policy. If G ◦ Tr ( π 0 ) is heavy-tailed with finite mean µ Q , then for any M,ϵ > 0 , there is a policy π with mean return E [ U | U ∼ G ◦ Tr ( π )] > M and E s ∈ T,T ∼ Tr ( π ) [ D KL ( π ( s ) ∥ π 0 ( s ))] < ϵ .

Proof: We will exhibit a distribution of trajectories ρ such that D KL ( ρ ∥ Tr ( π 0 )) < ϵ and E [ G ( ρ )] > M , and then construct a policy π with Tr ( π ) = ρ . Note that this proof applies for continuous action spaces if trajectories are replaced with measurable sets, but this would make it harder to read.

Let ρ π 0 = Tr ( π 0 ) . We have a heavy-tailed distribution of return Q ≜ G ( ρ π 0 ) over R , so we can apply Theorem 1. But to define ρ , we can construct P t in the proof of Theorem 1 in a particular way. For any t > c , we need a P t that uniformly upweights values of mean return such that ¯ F P t ( t ) = c/t . We can define ρ t such that any trajectory τ is upweighted by a factor depending only on its mean return:

Then we can let P t ≜ G ◦ ρ t and the rest of the proof of Theorem 1 applies. Therefore, applying the theorem, we can let ρ = ρ t for sufficiently large t , and then µ G ◦ ρ > M and D KL ( G ◦ ρ, G ◦ ρ π 0 ) < ϵ . By the chain rule for KLdivergence, D KL ( ρ, ρ π 0 ) = D KL ( G ◦ ρ, G ◦ ρ π 0 )+ E γ ∼ G ◦ ρ [ D KL ( ρ ( T ) | G ( T ) = γ ∥ ρ π 0 ( T ) | G ( T ) = γ )] . Since we constructed ρ so that the probabilities of each τ conditional on its return being γ are equal, the second term is zero, and we also have D KL ( ρ, ρ π 0 ) < ϵ .

Finally, since the KL divergence between trajectory distributions is the sum of KL divergence between policies at each action in the trajectory, and each trajectory has at least one action, E s ∈ T,T ∼ Tr ( π ) [ D KL ( π ( s ) ∥ π 0 ( s ))] ≤ E T ∼ Tr ( π ) ∑ s ∈ T [ D KL ( π ( s ) ∥ π 0 ( s ))] = D KL ( ρ ∥ ρ π 0 ) < ϵ as desired.

To define π such that Tr ( π ) = ρ , we let π ( s, a ) = Pr ( a i = a | τ = ( ..., s, a i , ... ) ∼ ρ ) .

Then, the probability that any trajectory τ = ( s 1 , a 1 , . . . , a n ) is sampled is:

In (2), returns are Markovian, so all trajectory prefixes ending in state s have the same distribution of returns under any policy. In the construction of ρ , all trajectories with the same mean return have equal measure. Therefore, conditioning on earlier states and actions of τ does not change the measure, so we can write (3). So Tr ( π ) = ρ as desired. ■

A.4 Theorem 4

Restatement of Theorem 4. If U = X + V with X and V both light-tailed and V unbounded, and the distribution of U is continuous, and π ∗ ( β ) ≜ arg max π E [ U ( π )] -βD KL ( π, π 0 ) , then lim β → 0 + E [ V ( π ∗ ( β ))] = ∞ .

Proof. Fix some β . Using Lagrange multipliers, we find that for any event S , Pr π ( S ) = Pr π 0 ( S ) e λU ( S ) . Let c ( β ) be the median value of U under the policy π ∗ ( β ) ; that is, Pr ( U > c ( β ) | U ∼ G ◦ Tr ( π ∗ ( β ))) = 1 2 . This exists because U has a continuous distribution. Then:

The left term is c , while the right term is ∞ , so the overall limit is ∞ .

A.5 Theorem 5

Restatement of theorem 5. Let X and V be two independent random variables with CDFs F X and F V and tail functions ¯ F V ≜ 1 -F V , ¯ F X ≜ 1 -F X such that

V has a finite mean.

X is subexponential; that is, lim x →∞ Pr ( X 1 + X 2 >x ) Pr ( X>x ) = 2 if X 1 , X 2 are two independent samples from X . This is a slightly stronger property than being heavy-tailed.

The tail of V is sufficiently lighter than the tail of X that lim t →∞ t p ¯ F V ( t ) ¯ F X ( t ) = 0 for some p > 1 .

Then lim t →∞ E [ V | X + V ≥ t ] = E [ V ] ; that is, catastrophic Goodhart occurs in the limit of optimization for U = X + V .

The proof requires expressing the conditional expectation in question as ∫ ∞ -∞ vf V ( v ) Pr ( X>t -v ) ∫ ∞ -∞ f V ( v ) Pr ( X>t -v ) , then partitioning the interval ( -∞ , ∞ ) into four regions and bounding the integrand in the numerator above by a different quantity in each region.

In addition to the works cited in the main paper, we make reference to the textbook (Foss et al., 2013) throughout the proof. Many similar results about random variables are present in the textbook.

Conditional distribution ofVacross thefourregions

20

Fx(t-α)

V

V|X+V≥t

15

-log(fv)

10

5

R1

R2

R3

R4

0

10

0

10

20

30

V

A.5.1 Proof sketch and intuitions

The conditional expectation E [ V | X + V > t ] is given by ∫ ∞ -∞ vf V ( v ) Pr ( X>t -v ) ∫ ∞ -∞ f V ( v ) Pr ( X>t -v ) , 8 and we divide the integral in the numerator into 4 regions, showing that each region's effect on the conditional expectation of V is similar to that of the corresponding region in the unconditional expectation E [ V ] .

The regions are defined in terms of a slow-growing function h ( t ) : R → R ≥ 0 such that the fiddly bounds on different pieces of the proof work out. Roughly, we want it to go to infinity so that | V | is likely to be less than h ( t ) in the limit, but grow slowly enough that the shape of V 's distribution within the interval [ -h ( t ) , h ( t )] doesn't change much after conditioning.

In Table A.5.1, we abbreviate the condition X + V > t as c .

Note that up to a constant vertical shift of normalization, the green curve is the pointwise sum of the blue and orange curves.

A.5.2 Definitions

To be more precise, we're going to make the following definitions and assumptions:

Let f V ( v ) be the PDF of V at the value v . We assume for convenience that f V exists, is integrable, etc, though we suspect that this isn't necessary, and that one could work through a similar proof just referring to the tails of V . We won't make this assumption for X . Let F X ( x ) = Pr ( X ≤ x ) and ¯ F X ( x ) = Pr ( X > x ) , similarly for F V and ¯ F V . Assume that

V has a finite mean: ∫ ∞ -∞ vf V ( v ) dv converges absolutely.

X is subexponential.

Formally, this means that lim x →∞ Pr ( X 1 + X 2 >x ) Pr ( X>x ) = 2 . This occurs roughly whenever X has tails that are heavier than e -cx for any c and is reasonably well-behaved; counterexamples to the claim "long-tailed implies subexponential" exist, but they're nontrivial to exhibit. Examples of subexponential distributions include lognormal distributions, anything that decays like a power law, the Pareto distribution,and distributions with tails asymptotic to e -x a for any 0 < a < 1 .

We require for V that its tail function is substantially lighter than X's, namely that lim t →∞ t p ¯ F V ( t ) ¯ F X ( t ) = 0 for some p > 1 . (This implies that ¯ F V ( t ) = O ( ¯ F X ( t ) /t ) .)

With these definitions and assumptions, we can move on to the proof.

The unnormalized PDF of V conditioned on X + V ≥ t is given by f V ( v ) ¯ F X ( t -v ) . Its expectation is given by ∫ ∞ -∞ vf V ( v ) ¯ F X ( t -v ) ∫ ∞ -∞ f V ( v ) ¯ F X ( t -v ) .

Meanwhile, the unconditional expectation of V is given by ∫ ∞ -∞ vf V ( v ) .

We'd like to show that these two expectations are equal in the limit for large t . To do this, we'll introduce Q ( v ) = ¯ F X ( t -v ) ¯ F X ( t ) . (More pedantically, this should really be Q t ( v ) , which we'll occasionally use where it's helpful to remember that this is a function of t .)

For a given value of t , Q ( v ) is just a scaled version of ¯ F X ( t -v ) , so the conditional expectation of V is given by ∫ ∞ -∞ vf V ( v ) Q ( v ) ∫ ∞ -∞ f V ( v ) Q ( v ) . But because Q (0) = 1 , the numerator and denominator of this fraction are (for small v ) close to the unconditional expectation and 1 , respectively.

We'll aim to show that for all ϵ > 0 , we have for sufficiently large t that ∣ ∣ ∣ ∫ ∞ -∞ vf V ( v ) Q t ( v ) -∫ ∞ -∞ vf V ( v ) ∣ ∣ ∣ < ϵ and ∫ ∞ -∞ f V ( v ) Q t ( v ) ∈ [1 -ϵ, 1 + ϵ ] , which implies (exercise) that the two expectations have limiting difference zero. But first we need some lemmas.

A.5.3 Lemmas

Lemma 1. There is h ( t ) depending on F X such that:

(a) lim x →∞ h ( t ) = ∞

(b) lim t →∞ t -h ( t ) = ∞

(c) lim t →∞ ¯ F X ( t -h ( t )) ¯ F X ( t ) = 1

(d) lim t →∞ sup | v |≤ h ( t ) | Q ( v, t ) -1 | = 0 .

Proof. Lemma 2.19 from (Foss et al., 2013) implies that if X is long-tailed (which it is, because subexponential implies long-tailed), then there is h ( t ) such that condition (a) holds and ¯ F X is h -insensitive; by Proposition 2.20 we can take h such that h ( t ) ≤ t/ 2 for sufficiently large t , implying condition (b). Conditions (c) and (d) follow from being h -insensitive.

Lemma 2. Suppose that F X is whole-line subexponential and h is chosen as in Lemma 1. Also suppose that ¯ F V ( t ) = O ( ¯ F X ( t ) /t ) . Then Pr [ X + V > t, V > h ( t ) , X > h ( t )] = o ( ¯ F X ( t ) /t ) .

Proof. This is a slight variation on lemma 3.8 from (Foss et al., 2013), and follows from the proof of Lemma 2.37. Lemma 2.37 states that

Lemma 2.37. Let h be any increasing function on R + such that h ( x ) →∞ . Then, for any distributions F 1 , F 2 , G 1 , and G 2 on R ,

where ξ 1 , ξ 2 , η 1 , and η 2 are independent random variables with respective distributions F 1 , F 2 , G 1 and G 2 .

but it is actually proved that

If we let F 1 = F V , F 2 = G 1 = G 2 = F X , then we get

where X,X ′ ∼ F X . Multiplying by t , we have

and because h ( t ) → ∞ as t → ∞ and ¯ F V ( t ) = O ( ¯ F X ( t ) /t ) , we can say that for some c < ∞ , lim t →∞ sup z>h ( t ) t ¯ F V ( z ) ¯ F X ( z ) < c . Therefore for sufficiently large t P { X + V > t, X > h ( t ) , V > h ( t ) } ≤ c t P { X + X ′ > t, X >h ( t ) , X ′ >h ( t ) } .

By Theorem 3.6, P { X + X ′ > t, X >h ( t ) , X ′ >h ( t ) } is o ( ¯ F X ( t )) , so the LHS is o ( ¯ F X ( t ) /t ) as desired.

A.5.4 Bounds on the numerator

We want to show, for arbitrary ϵ > 0 , that ∣ ∣ ∣ ∫ ∞ -∞ vf V ( v ) Q ( v ) -∫ ∞ -∞ vf V ( v ) ∣ ∣ ∣ < ϵ in the limit as t → ∞ . Since ∣ ∣ ∣ ∫ ∞ -∞ vf V ( v ) Q ( v ) -∫ ∞ -∞ vf V ( v ) ∣ ∣ ∣ ≤ ∫ ∞ -∞ | vf V ( v )( Q ( v ) -1) | = ∫ ∞ -∞ | v | · f V ( v ) · | Q ( v ) -1 | it will suffice to show that the latter quantity is less than ϵ for large t .

We're going to show that ∫ ∞ -∞ | v | · f V ( v ) · | Q ( v ) -1 | is small by showing that the integral gets arbitrarily small on each of four pieces: ( -∞ , -h ( t )] , ( -h ( t ) , h ( t )) , [ h ( t ) , t -h ( t )] , and ( t -h ( t ) , ∞ ) . We'll handle these case by case (they'll get monotonically trickier).

Region 1: ( -∞ , -h ( t )] Since ∫ ∞ -∞ vf V ( v ) is absolutely convergent, for sufficiently large t we will have ∫ -h ( t ) -∞ | v | f V ( v ) < ϵ , since h ( t ) goes to infinity by Lemma 1(a).

Since Q ( v ) is monotonically increasing and Q (0) = 1 , we know that in this interval | Q ( v ) -1 | = 1 -Q ( v ) .

Region 2: ( -h ( t ) , h ( t )) By lemma 1(d), h is such that for sufficiently large t , | Q ( v ) -1 | < ϵ ∫ ∞ -∞ | v | f V ( v ) on the interval [ -h ( t ) , h ( t )] . (Note that the value of this upper bound depends only on V and ϵ , not on t or h .) So we have ∫ h ( t ) -h ( t ) | v | f V ( v ) | Q ( v ) -1 | < ϵ ∫ ∞ -∞ | v | f V ( v ) ∫ h ( t ) -h ( t ) | v | f V ( v ) < ϵ ∫ ∞ -∞ | v | f V ( v ) ∫ ∞ -∞ | v | f V ( v ) = ϵ .

Region 3: [ h ( t ) , t -h ( t )] For the third part, we'd like to show that ∫ t -h ( t ) h ( t ) vf V ( v )( Q ( v ) -1) < ϵ . Since ∫ t -h ( t ) h ( t ) vf V ( v )( Q ( v ) -1) < ∫ t -h ( t ) h ( t ) tf V ( v ) Q ( v ) = t ¯ F X ( t ) ∫ t -h ( t ) h ( t ) f V ( v ) ¯ F X ( t -v ) it would suffice to show that the latter expression becomes less than ϵ for large t , or equivalently that ∫ t -h ( t ) h ( t ) f V ( v ) ¯ F X ( t -v ) = o ( ¯ F X ( t ) t ) .

The LHS in this expression is the unconditional probability that X + V > t and h ( t ) < V < t -h ( t ) , but this event implies X + V > t, V > h ( t ) , and X > h ( t ) . So we can write

Region 4: ( t -h ( t ) , ∞ ) For the fourth part, we'd like to show that ∫ ∞ t -h ( t ) vf V ( v ) Q ( v ) → 0 forlarge t .

Since Q ( v ) = ¯ F X ( t -v ) ¯ F X ( t ) < 1 ¯ F X ( t ) , it would suffice to show ∫ ∞ t -h ( t ) vf V ( v ) = o ( ¯ F X ( t )) . But note that since lim t →∞ ¯ F X ( t -h ( t )) ¯ F X ( t ) = 1 by Lemma 1(c), this is equivalent to ∫ ∞ t -h ( t ) vf V ( v ) = o ( ¯ F X ( t -h ( t ))) , which (by Lemma 1(b)) is equivalent to ∫ ∞ t vf V ( v ) = o ( ¯ F X ( t )) .

Note that ∫ ∞ t vf V ( v ) = t ∫ ∞ t f V ( v ) + ∫ ∞ t ( v -t ) f V ( v ) = t ¯ F V ( t ) + ∫ ∞ t ¯ F V ( v ) , so it will suffice to show that both terms in this sum are o ( ¯ F X ( t )) .

The first term t ¯ F V ( t ) is o ( ¯ F X ( t )) because we assumed lim t →∞ t p ¯ F V ( t ) ¯ F X ( t ) = 0 for some p > 1 .

For the second term, we have for the same reason ∫ ∞ t ¯ F V ( v ) < ∫ ∞ t ¯ F X ( v ) v p = ¯ F X ( t ) ∫ ∞ t v -p = t 1 -p p -1 ¯ F X ( t ) = o ( ¯ F X ( t )) .

A.5.5 Bounds on the denominator

For the denominator, we want to show that lim t →∞ ∫ ∞ -∞ f V ( v ) Q t ( v ) = 1 = ∫ ∞ -∞ f V ( v ) , so it'll suffice to show | ∫ ∞ -∞ f V ( v )( Q t ( v ) -1) | = o (1) as t → ∞ . Again, we'll break up this integral into pieces, though they'll be more straightforward than last time. We'll look at ( -∞ , -h ( t )) , [ -h ( t ) , h ( t )] , and ( h ( t ) , ∞ ) .

-But since h ( t ) goes to infinity, this left tail of the integral will contain less and less of V 's probability mass as t increases.

| ∫ h ( t ) -h ( t ) f V ( v )( Q ( v ) -1) | ≤ ∫ h ( t ) -h ( t ) f V ( v ) | Q ( v ) -1 |

-By Lemma 1(d) we know that this goes to zero for large t .

But for sufficiently large t we have h ( t ) > 1 , so we obtain ∫ ∞ h ( t ) f V ( v ) Q ( v ) < ∫ ∞ h ( t ) vf V ( v ) Q ( v ) < ∫ ∞ -∞ vf V ( v ) Q ( v ) = o (1) by the results of the previous section. This completes the proof.

A.6 Theorem 6

Restatement of theorem 6. Let X,V be independent random variables such that lim t →∞ ¯ F X ( t +1) ¯ F X ( t ) = 0 . (This implies that X has tails that are dominated by e -cx for any c, though it's a slightly stronger claim because it requires that X not have large jumps in the decay of its tails.) Then for any V with a finite mean which has no upper bound, lim t →∞ E [ V | X + V > t ] = ∞ .

Theorem 6 generalizes a consequence of the "Regressional Goodhart Identity" in (Gao et al., 2023).

Proof. Let Pr ( V > c +1) = p > 0 , which exists by our assumption that V is unbounded.

Let E [ V | V < c ] = q . (If this is undefined because the conditional has probability 0 , we'll have the desired result anyway since then V would always be at least c .)

Observe that for all t , E [ V | V < c, X + V > t ] ≥ q (assuming it is defined), because we're conditioning ( V | V < c ) on an event which is more likely for larger v (since X and V are independent).

First, let's see that lim t →∞ P ( V <c | X + V ≥ t ) P ( V >c +1 | X + V ≥ t ) = 0 . This ratio of probabilities is equal to

Histogram ofrewardfromrandom inputs

Normalprobability plot of reward from random inputs

1.0 -

140

120

1.5

100

2.0

Frequency

80

reward

2.5

60

3.0

40

3.5

20

4.0

0

4.0

3.5

3.0

2.5

2.0

1.5

1.0

-3

2

-1

0

1

2

3

Reward

Normal quantile

RewardvsLog-probabilityofACG-optimizedsequencesonStarling7B-alpha

-1000

1100

Log-probability

-1200

-1300

-1400

-1500

2

3

4

5

6

7

8

Reward

which, by our assumption that lim t →∞ ¯ F X ( t +1) ¯ F X ( t ) = 0 , will get arbitrarily small as t increases for any positive p .

Now, consider E [ V | X + V ≥ t ] . We can break this up as the sum across outcomes Z of E [ V | Z, X + V ≥ t ] · Pr ( Z | X + V ≥ t ) for the three disjoint outcomes V < c , c ≤ V ≤ c + 1 , and V > c + 1 . Note that we can lower bound these expectations by q, c, c +1 respectively. But then once t is large enough that Pr ( V <c | X + V ≥ t ) Pr ( V >c +1 | X + V ≥ t ) < 1 c -q , this weighted sum of conditional expectations will add to more than c .

B Additional experiments and figures

Figures B.1, B.2 relate to experiments mentioned in the main paper. In response to reviewer feedback, we added two further experiments to demonstrate the catastrophic Goodhart phenemonon with artificially heavy-tailed reward, one using best-of-N on synthetic distributions and one with PPO on Pythia 1B.

B.1 Best-of-N experiment

We created a synthetic experiment by letting reward U = X + V , where X and V are independent and sampled from different probability distributions, consistent with our theoretical assumptions. We vary N from 1 to 65536, do 100 trials of taking the best-ofN sample with highest U , and note whether V goes towards 0 (overoptimization) or not.

Possible distributions for V are normal and t-distribution with df=10. Possible distributions for X are normal, t with df=3, t with df=5, lognormal, and Levy. (All of these heavy-tailed except for the normal distribution.) V is scaled to a standard deviation of 2 and X has s.d. of 1 (except for the Levy distribution, which has infinite variance), representing that in ordinary regimes most of the variance comes from utility rather than error.

normal vs normal

normal vs lognormal

normal vs ('t', 3)

normal vs ('t', 5)

normal vs levy

10

Utility

20.0

Error

Utility

40

Utility

14

Utility

Error

1013

Error

Utility

8

Proxy reward

17.5

Proxy reward

Proxy reward

12 -

Proxy reward

1011

Proxy reward

15.0

OE

10

109

6

12.5

8

107

?

10.0

×

20

6

105

7.5

4

103

2

5.0

10

101

2.5

0

F0°0

-0

0

10-1

101

103

105

101

103

105

101

103

105

101

103

105

101

103

105

n

n

n

n

n

(t', 10) vs normal

('t',10) vs lognormal

('t', 10) vs ('t', 3)

('t', 10) vs ('t', 5)

('t', 10) vs levy

Utility

Utility

Utility

17.5

Utility

Utility

14

20

35

1011

Error

Proxy reward

Proxy reward

Proxy reward

15.0 -

Proxy reward

Proxy reward

12

OE

109

10

15

25

12.5 -

107

10.0

8-

20

×

10

7.5

105 -

6

15

103

5.0

10

5-

2.5

101

5

0

-0

F0

00

10-1

101

103

105

101

103

105

101

103

105

101

103

105

101

103

105

n

n

n

n

n

The results are shown in Figure B.3. Briefly, the results are consistent with the asymptotic results in theorems 5 and 6, showing that overoptimization

B.2 PPO experiment

In this experiment, we examined PPO with artificially heavy-tailed rewards to see if catastrophic Goodhart could be observed.

OpenRLHF (Hu et al., 2024) was used to train Pythia 1B with a reward model derived also from Pythia 1B, on the default OpenRLHF prompt dataset. We used the reward model to represent true utility, and a heavy-tailed error term based on the number of "the" tokens was added to get the proxy reward. The ❦❧❴/a116❛/a114❣❡/a116❂✵✳✺ option was used to dynamically adjust KL penalty, as we mention is done in Ziegler et al. (2020). Rewards were not clipped. (Reward clipping can be useful to prevent overoptimization, but is not always used in PPO.) Response length was limited to 256.

Two example completions are in figure B.4. Midway through training, the model generates reasonable text with a higher number of "the" tokens than usual, but as training progresses the model starts to generate completions with a very high number of "the" tokens while proxy reward reaches values > 10 9 , demonstrating that the basic pattern of catastrophic Goodhart can occur in RLHF under conditions of heavy-tailed error.

The preparation of polyarylethylene which is either employed for the purpose of applications in the petroleum refining or in the production of the well-known automobile oil lubes is carried out by the so-called preparation by the salt bath method described... in 1980 by D. W. Perkins in the United States. This method presupposes the usage of the salt bath as the medium for the preparation of the polyarylediene-the method of preparation of the crude oil base by the process which consists in letting it be saturated up with a particular the

A mirror, the essence of the understanding of the given the conclusion that the manifestation in the form of the gender and the identification the solutions related to different fields of the modern utilization of the the functionality of the the the the modification of the the the the the the state of the the the the the the the the the the the employment and the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the and the the the the the the the the the the the the the the the the the the the the by the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the and the the the the the the the the the the the the the the the the the

Figure B.4: PPO sample generations. Top: early in training, the model generates reasonable completions. Bottom: later in training, the artifically heavy-tailed reward dominates and the model generates completions with a very high number of "the" tokens.

The result depends on hyperparameters (e.g. reward clipping would prevent this), so our observation should not be taken as a claim that catastrophic Goodhart is inevitable in all RLHF settings with heavy-tailed error.

C Assets

We use three models for our experiments: Starling 7B-alpha, Llama 2 7B-chat, and Pythia-1.4B. Starling was developed by Berkeley, and Pythia by EleutherAI. Starling and Pythia models are licensed under Apache-2.0. 10 11 Llama 2 models were developed by Meta and licensed under a license published by Meta. 12

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract lists the important claims: the relationship between Goodhart's Law and whether the error in a misspecified reward is heavy-tailed. The main limitation of independence assumptions is clearly stated in the introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 7 lists the limitations, which we have combined with the discussion section due to heavy overlap.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All proofs are given inline or in the appendix, except for Theorem 5 which appears in the supplemental material.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Sampling rewards requires no hyperparameters, and hyperparameters are provided for ACG.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but

reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code will be provided in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( ❤/a116/a116♣/a115✿✴✴♥✐♣/a115✳❝❝✴♣✉❜❧✐❝✴ ❣✉✐❞❡/a115✴❈♦❞❡❙✉❜♠✐/a115/a115✐♦♥/a80♦❧✐❝② ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( ❤/a116/a116♣/a115✿✴✴♥✐♣/a115✳❝❝✴♣✉❜❧✐❝✴ ❣✉✐❞❡/a115✴❈♦❞❡❙✉❜♠✐/a115/a115✐♦♥/a80♦❧✐❝② ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Other than hyperparameters, there are no details required to understand the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The only error bars are inter-run variability of ACG. The standard deviation was reported rather than error bars due to the small number of runs.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The experiments took minimal compute resources except H100 hours for ACG, and we report the number of GPU-hours used.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ❤/a116/a116♣/a115✿✴✴♥❡✉/a114✐♣/a115✳❝❝✴♣✉❜❧✐❝✴❊/a116❤✐❝/a115●✉✐❞❡❧✐♥❡/a115 ?

Answer: [Yes]

Justification: The research conforms to all data-related concerns. No human subjects were involved, and we think the risk of harmful societal impact is minimal.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The immediate societal impacts are limited, but we discuss some potential applications to long-term safety.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We have created no such artifacts.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The assets we use are Starling 7B-alpha, Llama 2 7B-chat, and Pythia-1.4B. Starling and Pythia models are licensed under Apache-2.0. 13 14 Llama 2 models are licensed under a license published by Meta. 15 We are in compliance with all licenses and terms of use.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ♣❛♣❡/a114/a115✇✐/a116❤❝♦❞❡✳❝♦♠✴❞❛/a116❛/a115❡/a116/a115 has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No human subjects are involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No human subjects are involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The authors presents a study on the RM in RLHF. Theoretical results of the paper assume that the reward from the RM consists of a true reward plus a noise term which is modeled as a random variable. Under this model the authors show that

- If the noise terms are heavy-tailed there exists policies with vanishing KL-divergence but infinite proxy reward.
- If the noise terms are IID and light-tailed, then KL divergence suffices to guarantee good outcomes

The authors also investigate the actual rewards given by real RMs. They find that the rewards on random sequences are roughly

STRENGTHS:
- The paper is well written and easy to read.
- The theoretical results are relevant.

WEAKNESSES:
- there is probably some discrepancy between the theory and practice. The results are asymptotic, it would be better if the statements could hold in non asymptotic regimes. 
- The experimental results seems to show that the rewards are not heavy tailed, and thus the theory seems to suggest that KL-divergence should work well. We already know that it does work well from practice, so there is no novel take-home message.

QUESTIONS:
- The theoretical results are related to asymptotically heavy tails. What would happen in I take my RM and clip the output to be in [-10000, +10000]. I’m guessing in practice nothing would happen, but would the theoretical results would be invalid since the tails disappear?

- In practice the output of the RM isn’t a random variable, it’s deterministic. Can you comment on the validity of the assumption that the rewards are random?

LIMITATIONS:
na
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes a phenomenon called \"catastrophic Goodhart\": in RL training, suppose the learned reward function is the sum of the true utility function and some noise, then if (1) the true utility and the noise are independent and light-tailed, maximizing reward with KL regularization can also give high utility, while if (2) the noise is heavy-tailed, then there exists a policy with arbitrarily high reward and arbitrarily small KL penalty, but its true utility is close to the initial policy. Empirical results suggest that open-source reward models could be light-tailed.\n\nSTRENGTHS:\nThis paper considers whether KL regularization is enough to handle  reward misspecification, which is an important problem. This paper also gives clean answers to this question: if utility and noise are independent and light-tailed, then KL regularization is enough, while if noise is heavy-tailed, we may have very little improvement in utility even with high proxy reward and small KL penalty. I think these results are nice and enhance our understanding of this problem.\n\nWEAKNESSES:\nOne missing piece is the implicit regularization effects of the optimization algorithms: although there exists a policy with low utility even under a KL penalty, probably the training algorithm can avoid those bad policies and still find a good one. It will be nice if we can understand whether certain optimization techniques can solve this issue.\n\nSome additional comments:\n1. The notation $U$ and $V$ are used inconsistently in the paper: on line 34, $U$ denotes the true utility and $V$ denotes the proxy reward, while on line 71 their meanings are reversed.\n\nQUESTIONS:\nI have one question regarding Theorem 4: the final conclusion is that the expected utility can be arbitrarily large, but if the utility function is bounded, I don't understand why its expectation can be arbitrarily large.\n\nI also have a more open-ended question: if we make the proxy reward function bounded (e.g., by clipping), do we have similar results?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the effectiveness of using KL divergence for regularization in reinforcement learning from human feedback (RLHF), particularly when dealing with reward functions that have misspecified errors. It introduces the concept of \"catastrophic Goodhart,\" a scenario where policies can achieve extremely high rewards with heavy-tailed errors, but without actual improvement in utility.\n\nSTRENGTHS:\nThe paper addresses a critically important problem within the domain of RLHF --- the challenge of dealing with reward misspecification, particularly in scenarios where the error distribution is heavy-tailed. The authors defined the problem as \"catastrophic Goodharts\" and provided analysis accordingly.\n\nWEAKNESSES:\n[writing / presentation] The paper contributes to the field of LLM safety/alignment, yet the writing could be made more engaging and accessible. Improving clarity, enhancing the depth of discussions, and ensuring visual and structural appeal could greatly increase its impact and readability. These changes would not only cater to experts in the field but also to readers who may be new to this area of study.\n\n[Experiments] the experimental evidence could be further bolstered to validate the presented claims comprehensively. Specifically:\n\n- The experiments primarily focus on simulated environments. Including real-world applications could demonstrate the practical implications and limitations of the findings more effectively.\n\n- Exploring a wider variety of reward functions, particularly those commonly used in commercial or industrial settings, could help ascertain the generalizability of the catastrophic Goodhart phenomenon across different contexts.\n\n- Additional experiments comparing KL divergence with other regularization or penalty methods could offer a clearer understanding of its relative efficacy and limitations.\n\n- How about other optimization methods like Best-of-N where KL divergence term is not used? \n\n- Will the implementation like LoRA affect the proposed effect?\n\nQUESTIONS:\nPlease see the weakness section\n\nLIMITATIONS:\nplease see weakness", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper first considers a theoretical stylized model of jointly distributed utility and (mis-specified) rewards and proves a novel result that for any heavy tailed reference distribution, it is possible to find another distribution that arbitrarily approximates it in terms of the KL divergence, yet has an unbounded reward. Beginning with this basic result, the authors then attempt to apply these insights to the popular technique of regularizing the policy optimization procedure in RLHF using a learned reward model with the KL divergence to the base SFT policy. Experiments on open source LLM reward models are conducted to investigate how closely the assumptions made in the theory match with current state of the art models.\n\nSTRENGTHS:\n- The paper identifies a novel conceptual insight around the heavy tailedness of the utility/rewards as being an important factor in reward hacking.\n- Theoretical results are sound and novel, along with some attempts to also ground this in experimental validation.\n\nWEAKNESSES:\n- The clarity in some sections can be improved -- see questions below.\n- The notion of heavy tailed versus light tailed, while theoretically appealing and convenient, seems quite sensitive to parameterization and/or scale in practice. While the authors do mention this briefly in footnote 2 (Page 6), it appears that simple reward reparametrization can transform the resulting distributions in a way that is not robust to the notion of heavy/light tailedness.\n\nQUESTIONS:\n- The definition of DMRMDP in Section 4 is unclear. Is there any precedent to this particular definition? For instance, are you always required to end a trajectory in one of the sink states? If not, are rewards also defined for partial trajectories?\n- Is the converse of Theorem 1 also true? i.e. if Q is not heavy tailed, does the maximum achievable mean stay bounded as $\\epsilon \\rightarrow 0$?\n- In the experimental results section, random sampling and adversarial sampling are used to generate the reward samples which are estimated from another model. However, it is hard to know the true utility, unlike in the case of Gao et. al. (2023), who perform a synthetic study with known controlled true utilities. Since the paper's insights and results are more conceptual than practical at the moment,  wouldn't it be interesting to study a synthetic experimental setup similar to Gao et. al. where both true utilities and proxy utilities can be measured explicitly?\n- When you speak of a \"heavy/light tail under a given policy $\\pi$, one possibility is to consider the action conditioned expected utility, or the unconditional version where action is simply a latent variable that determines the final reward. Is this distinction significant for any of the results? If so, which one is being assumed, and what are the implications for the alternate version on the basic result?   \n\n\nNits: \n\n- Figure reference in L110 seems like a typo.\n- Figure 1 has fonts which are too small to read.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}