Reward Gaming in Conditional Text Generation
Year: 2023

Abstract:
To align conditional text generation model outputs with desired behaviors, there has been an increasing focus on training the model using reinforcement learning (RL) with reward functions learned from human annotations. Under this framework, we identify three common cases where high rewards are incorrectly assigned to undesirable patterns: noise-induced spurious correlation, naturally occurring spurious correlation, and covariate shift. We show that even though learned metrics achieve high performance on the distribution of the data used to train the reward function, the undesirable patterns may be amplified during RL training of the text generation model. While there has been discussion about reward gaming in the RL or safety community, in this discussion piece, we would like to highlight reward gaming in the natural language generation (NLG) community using concrete conditional text generation examples and discuss potential fixes and areas for future work.

heading: 1 Introduction
text: Natural language generation aims to automatically produce text that is fluent, relevant, and factual. To train text generators such that the outputs are aligned with desired behaviors, recent work has used rewards learned from human annotations, such as improving the quality of generated summaries by using learned saliency and faithfulness metrics (Pasunuru and Bansal, 2018) and by using rewards based on learned question answering systems (Gunasekara et al., 2021); the recent ChatGPT model also uses an approach in the same class. In general, this class of methods (1) collects a human annotation dataset Dreward consisting of, e.g., direct ratings of generations (Sellam et al., 2020; Nakatani et al., 2022; Ramamurthy et al., 2023), labels of error spans in the generations (Freitag et al., 2020; Amrhein and Sennrich, 2022), or pairwise comparison of generations given the same source
sequence (Stiennon et al., 2020; Wu et al., 2021; Bai et al., 2022a); (2) learns a proxy reward function that scores generations (as opposed to a true reward function which is often given by human judgment) on Dreward; and then (3) learns the text generator on a dataset Dtask, using RL with the learned reward function.
What could go wrong when we obtain the reward signal from humans? The rewards would rarely be robust. When training the text generator, the distribution induced by the policy (i.e., the generator) changes because we frequently update it, which opens up opportunities for exploiting errors in the reward. Thus, even if the reward function performs well on the dev/test split of Dreward as an evaluator, the reward can still be gamed during RL training of the generator. Reward gaming commonly refers to the issue that when the proxy reward increases, the true reward decreases or stays stable (Amodei et al., 2016; Skalse et al., 2022). In this discussion and in the context of NLP, we use “reward gaming” to broadly refer to the phenomenon that as training progresses, models produce low-quality generations that exhibit undesirable patterns while converging to high rewards.
Reward gaming can happen when an undesirable pattern is associated with a high reward in the learned metric. We identify three ways this phenomenon can happen. (1) A group of examples is misannotated systematically. For instance, suppose we train a model to do effective negotiation and annotators carelessly label all long paragraphs as effective, then the reward model would assign high scores on long generations even if they are nonsensical, and the generator would subsequently exploit this pattern. (2) Dreward contains some bias due to the data we select to annotate, or due to the people we select to be annotators. An example in the former case is that suppose every translation that contains “united nations” happens to have high quality/reward, possibly due to the way we
4746
collect Dreward; then the neural machine translation model may end up almost always generating the phrase surrounded by some gibberish. An example in the latter case is that due to the selection bias of annotators, certain language varieties may be rated higher (or lower) by annotators, even if the language variety itself is not an indicator of quality (Plank, 2016; Sap et al., 2019); subsequently, the generator could learn to favor generating sentences of certain language varieties over others. (3) Dreward does not cover certain groups of sentences. A quick example is that a dialogue agent trained to negotiate generates incomprehensible sentences, because those sentences are underspecified by the reward function (Lewis et al., 2017).
In short, among these three cases, the first two cases induce spurious correlations between the undesirable pattern and the reward, and the third case induces underspecified behavior on uncovered examples.
We use synthetic and real-world examples to illustrate the above three cases: even if the learned reward achieves a good performance on Dreward, high rewards can still be assigned to undesirable patterns. Notably, we show that these patterns get amplified during RL training of the generators. For instance, a synthetic experiment discussed later (§4.1) shows it is possible that even a reward function that gives the correct reward on 99.3% of the test split of Dreward can lead to generation failure after RL.
We also review potential fixes (§5), including restricting the policy – e.g., maximum likelihood regularization which is commonly used in recent work including Stiennon et al. (2020) and Ramamurthy et al. (2023) – and fixing the reward itself like iteratively collecting human annotations. In light of these observations, we would like to bring more attention to reward gaming in the context of conditional text generation. Leveraging learned metrics during RL is a promising approach to training aligned text generation systems. But given that the rewards can only reliably improve generators if the sampled texts are within the distribution of Dreward, extra caution is needed when interpreting the results when training text generators using learned rewards – quality control or manual inspection is required to ensure good generation quality.

heading: 2 Related Work
text: Reward gaming or similar ideas have been discussed since Goodhart (1975). More recently, it is extensively discussed in Amodei et al. (2016). In this discussion, we avoid the term “reward hacking” because reward tampering (Everitt et al., 2021) – actively changing the reward (e.g., by execution of reward-modifying code under certain circumstances in a video game) – is also reward hacking, but it is not the topic of our discussion.
Many examples have demonstrated the reward gaming behavior, usually in gameplay or autonomous driving. For example, in a boat racing game in Amodei et al., the boat would hit objects in circles mid-way in the race instead of completing the race (the latter being the intended goal), because the reward increases faster by hitting a certain set of objects than completing the race; Baker et al. (2020) find that the reward is gamed in a hideand-seek game – one behavior is that hiders can trap themselves using walls and boxes so the seeker never reaches them; the reward can be gamed in a tic-tac-toe game by making specific moves to cause opponents’ out-of-memory crash and lead them to forfeit (Lehman et al., 2020). Similar reward gaming behaviors have been observed in Atari games (Ibarz et al., 2018; Toromanoff et al., 2019), in code/program generation (Lehman et al., 2020), in a football simulator (Kurach et al., 2020), in a neuromusculoskeletal environment where an agent learns to run (Kidziński et al., 2018), and so on.
Reward gaming is rarely concretely discussed in conditional text generation. A quick example by Lewis et al. (2017) and Kenton et al. (2021) is that a dialogue agent trained to do successful negotiation ends up generating nonsensical sentences, because those generations are underspecified by the reward function that is used to train the dialogue model.
Recently, there have been two findings that indicate the seriousness of reward gaming, albeit not in the context of NLP. First, more capable models may exacerbate reward gaming: Pan et al. (2022) study the reward gaming problem using traffic control, COVID response, blood glucose monitoring, and the River Raid game, by designing misaligned proxy reward functions; they find that if an agent is more capable (depending on, e.g., model size, the number of training steps), then it is better at exploiting loopholes in the reward function, and therefore ends up with a lower true reward compared to a less capable model.
More recently, Skalse et al. (2022) has suggested a strict definition of the hackability of a pair of reward functions, where “a pair” can be understood as an original reward and a proxy reward.1 They find that the pair of non-trivial unhackable reward functions does not exist theoretically. The question then becomes whether it is safe to use a proxy reward function empirically.
In this discussion, we aim to demonstrate the effect of reward gaming in text generation using concrete examples. Here are the two main differences of our discussion from the aforementioned examples. First, we focus on conditional text generation; in particular, the experiments in this discussion do not rely on state-of-the-art large language models – we aim to use smaller specialized conditional generation models instead. Second, we aim to investigate the reward gaming categories when the reward signal is learned from human annotations.

heading: 3 Background
text: Conditional text generation systems usually model p(y | x) where x = (x1, . . . , xTs) is a source sequence and y = (y1, . . . , yT ) is a target sequence. Most models use an autoregressive factorization: log p(y | x) = ∑Tt=1 log pθ(yt | y<t,x), where y<t = (y1, . . . , yt−1), and pθ is parameterized with a neural network. Maximum likelihood estimation (MLE) leads to mismatched train/test history and objectives during sequence generation (Bengio et al., 2015; Huszár, 2015; Ranzato et al., 2016; Schmidt, 2019; Pang and He, 2021; Arora et al., 2022). In addition, recent work aims to better align training objectives with human-annotated quality of generated texts (e.g., translation quality judgments, summarization faithfulness, human preference of generations).
The generation process can be considered a sequential decision making process suitable for RL. Given state st = (x,y<t), the policy πθ (i.e., pθ) takes action at (a token in the vocabulary), transits to the next state st+1, and receives a reward rt ∈ R learned from human annotations. Assume discount factor γ = 1. To maximize the objective J(θ) = Eτ∼πθR(x,y), where R(x,y) =∑T
t=1 rt, one way is to use policy gradient (RE1In short, reward functions r1, r2 are hackable w.r.t. a
policy set and an environment, if there exist policies π, π′ such that J1(π) < J1(π′) but J2(π) > J2(π′) where Ji denotes the expected return corresponding to reward function ri. See Definition 1 in Skalse et al. (2022) for details.
INFORCE; Williams, 1992; Sutton et al., 1999): ∇θJ(θ) = Eτ∼πθ ∑ t∇θ log πθ(at | st)Q̂(st, at),
where Q̂(st, at) = ∑T
t′=t rt′ is the estimated return. Our work uses REINFORCE with tricks of advantage estimation and value function fitting, described in the appendix. Recently, proximal policy optimization (PPO; Schulman et al., 2017) has also been widely used. It aims to avoid reward performance collapse, but we argue that the choice of algorithm that makes generations achieve high rewards is orthogonal to the issue that high rewards can correspond to undesirable generations.
To stabilize RL training, in each RL training run, we first initialize the model using an MLE-trained model to ensure a good starting point for RL optimization. In addition, we also use KL regularization which helps RL optimization (Jaques et al., 2019; Stiennon et al., 2020; Ramamurthy et al., 2023), so J(θ) = Eτ∼πθ [R(x,y) − β[log πθ(y | x) − log pMLE(y | x)]] where pMLE is the model trained using standard MLE. To demonstrate reward gaming behaviors, we tune β to achieve the highest validation reward in the synthetic Sudoku experiments, unless explicitly mentioned. Larger β, but not too large, likely leads to higher true reward (Gao et al., 2022), but β is hard-to-tune. But in some examples (e.g., §4.3), even large β does not eliminate undesirable behaviors. We will discuss using KL regularization as a remedy in §5.

heading: 4 Examples of Reward Gaming in Conditional Text Generation
text: As a reminder, we consider the class of conditional text generation learning algorithms where we:
(1) have a human annotation dataset Dreward;
(2) use this dataset to train a reward function fϕ that scores generations;
(3) learn the text generator on a dataset Dtask, using RL with the learned reward function.
Reward gaming happens when some undesirable pattern is associated with a high reward. We identify three such scenarios:
(1) spurious correlation due to annotation errors;
(2) naturally occurring spurious correlation;
(3) underspecified behavior in the reward function due to covariate shift.
We use both synthetic and real-world tasks to demonstrate the reward gaming behavior. The full experimental details can be found in the appendix. The experiments in this discussion do not rely on large language models; instead, we aim to build smaller specialized conditional generation models.
For synthetic tasks, we simulate all three settings using the following framework. We adapt Sudoku as a conditional text generation task.2 A valid Sudoku is a 9x9 grid with each cell containing a number from 1 to 9, such that no rows/columns and none of each of the nine non-overlapping 3x3 regions contains duplicates. For this task, let the input be the first k (k randomly chosen from 36 to 80) cells in a valid Sudoku after flattening it row by row. Let the reference output be the rest of the cells (i.e., the last 81− k cells). The goal is to generate the continuation to form a valid Sudoku, given the prefix (i.e., first k cells). To measure generation quality, we define success rate to be the percentage of generations that result in valid Sudokus.
While the sequence generator can be rule-based without using neural nets in this synthetic setting, to illustrate reward gaming, we consider learning the generator from a learned reward function.

heading: 4.1 Noise-Induced Spurious Patterns
text: We want to study settings where there is noise in human annotations. If we inject a small amount of high-reward but low-quality examples in Dreward, the reward function could put a high reward incorrectly on these examples.
Synthetic example: modified Sudoku. Dreward is a balanced dataset containing 500k positive and 500k negative examples. Out of the 500k positive examples, 0.5k (0.05% of all examples) are false positives, i.e., invalid Sudokus. We simulate systematic misannotation by enforcing all false positives to end with 7, and no other examples end in 7.3 This design is intended to simulate system-
2Controlling spurious correlations in the reward is difficult on experiments using real-world generation tasks. Therefore, we rely on the Sudoku framework, which has all the key elements we need for such experiments: (1) it is a conditional generation task (where the model needs to learn the relation between the input and the output); (2) it has clearly defined ground-truth rewards which enable easy evaluation; (3) it allows for easy manipulation of spurious correlations in the reward function. Therefore, we use the Sudoku experiments to show that reward gaming exists in conditional generation, and the reward gaming effect can be severe.
3For positive examples, we first create a set of 2M valid Sudokus, and then sample from the set. Many negative examples are small modifications of positive examples (§B.1) to
atic errors in human annotation; e.g., a group of sentences on rare topics getting mislabeled.
The reward is the probability of the Sudoku being valid, estimated by a classifier fϕ. fϕ, based on a tiny RoBERTa (§B.1), achieves 99.3% accuracy on the i.i.d. test split of Dreward. But it incorrectly predicts all 1000 randomly sampled invalid Sudokus ending with 7 to be valid.4
As a sanity check, a baseline generator trained by MLE on the 500k positive examples achieves a 74.7% success rate in spite of the noise. However, the RL-trained generator produces a large fraction of invalid generations that end in 7 despite achieving a high reward. Figure 1 shows that the reward increases to above 0.8 (a large reward given the range [0, 1]), and the amount of Sudokus ending with 7 oscillates around 85%; however, only 0.1% of the actual correct reference generations end with 7. Additionally, given a reward of 0.85 in the figure, we would expect around 85% of generations to be valid; however, the success rate (i.e., the proportion of valid generations) turns out to be always smaller than 15% throughout training.
In short, in this specific example, even 0.05% of noise in Dreward could lead to generation failure (>80% of generations are invalid), as the RL training of the generation model amplifies the failure mode.
Experimental details for the above example. The RoBERTa-tiny-based (Liu et al., 2019) reward function has 4 encoder layers and 2 atten-
ensure a high-quality fϕ. 4The reward makes the wrong prediction on those examples, but they represent a small portion of the dataset used to train the reward.
tion heads; the encoder embedding dimension is 64, and the dimension for FFN for 256. For the sequence generator, we use a smaller version of the transformer_iwslt_de_en architecture in fairseq (Ott et al., 2019). The encoder embedding dimension and the decoder embedding dimension are both 32. We use 2 attention heads in both the encoder and the decoder. The dimension for FFN in both the encoder and the decoder is 64. There are 2 encoder layers and 2 decoder layers. Please refer to the appendix for more details.
Takeaway. Even a small amount of noise in Dreward can enable the reward function to assign high reward on sequences containing certain undesirable patterns. After RL training, a large proportion of generations could incorrectly contain those undesirable patterns.

heading: 4.2 Naturally Occurring Spurious Patterns
text: The spurious correlation is not necessarily noiseinduced but can be naturally occurring. Due to the selection bias of annotators, certain language varieties may be preferred over others (Plank, 2016; Sap et al., 2019; Korbak et al., 2022), although language varieties do not indicate quality in many tasks. In addition, due to the selection bias of examples that are annotated, some attributes that are irrelevant to the quality get correlated with the reward (Wiegreffe and Marasovic, 2021; Pezeshkpour et al., 2022). If high rewards are assigned to these spurious patterns (e.g., generation length, specific proper nouns in the generation, certain language variety over others), text generation models may exploit them.
Synthetic example: Sudoku revisited. Dreward is dataset with 200k randomly sampled valid Sudokus as positive examples and 200k randomly sampled invalid Sudokus as negative examples. Using this dataset, we simulate the setting where a
simple feature (the feature that “the last nine numbers of the output do not repeat”) is predictive of the reward (validity) on a biased Dreward. Repetitions co-occur with 99.9% of negative examples, and therefore the repetition is a highly predictive feature of the reward.
The reward function, fϕ, achieves 99.9% accuracy on the test split of Dreward. We then train the conditional text generation model using RL where fϕ is the reward.
Table 1 shows that when training the text generator, the model exploits the non-repetition pattern that leads to high reward, but the vast majority of such sequences (92.8%) are in fact incorrect.
Real-world example: machine translation (MT) using dense reward. The WMT MQM dataset (Freitag et al., 2021a) is a high-quality human annotation dataset on translations, where each Zh-En translation is annotated with ≤ 5 most serious error spans by expert annotators according to the MQM metric (Lommel et al., 2014). Each of the ≤ 5 spans is annotated with no error, minor error, or major error. In Dreward, an example annotation of a generated translation is as follows:
state-owned enterprises and <major> advantageous </major> private enterprises entered the <major> revolutionary base area </major> <major> of </major> <minor> south ji@@ ang@@ xi </minor> .
Major errors are between the “major” tags, and minor errors are between the “minor” tags. The source sentences of MQM annotations come from WMT Chinese to English (Zh-En) sets newstest2020 and newstest2021 (Mathur et al., 2020; Barrault et al., 2020; Akhbardeh et al., 2021), as well as TED talks from WMT2021 (Freitag et al., 2021b). Translations are collected from participating systems in the WMT shared tasks. Human-written
references are also integrated into the annotation dataset.
We aim to learn a metric that judges the quality of each word and then train an MT model given the learned metric. fϕ is a scorer that predicts whether each token in a given translation is in a no-error span. Let the reward rt be the score that fϕ outputs at time-step t. Our key observation is that certain tokens are spuriously correlated with no-error annotations in the dataset. The ellipses punctuation (“...”) is one of them: experts annotated 98.3% of the occurrences as no-error.
Figure 2 shows that during RL training of the MT model on WMT17 Zh-En, as training goes on, the percentage of translations with ellipses increases and the ellipses achieve high rewards. The frequent occurrence of ellipses, however, are undesirable. Moreover, there is no improvement in BLEU: the MLE-trained translator achieves a dev BLEU of 23.1, but in different runs of our algorithm, the BLEU never exceeds 21.0. All experiments are run using a KL coefficient 0.1.5
In other training runs of fϕ and MT model, we found other tokens that are spuriously correlated with the reward. The token “conduct” is one example. Only 0.01% of all tokens are “conduct” but about 95% of them are labeled as “no error.” Here are some example model generations where “conduct” is integrated but its use is incorrect and nonsensical.
Reference: 66 countries and regions have reported the evidence of the mosquito-borne zika virus transmission since 2015. Generation: the 66 countries and regions have been able to conduct the evidence in the dissemination of the virus in 2015.
Reference: the ankara newspaper daily hurryiet, citing military sources, added turkey’s military attaché in jordan was also missing. Generation: the newspaper in ankara has been able to conduct the military information and the military work in jordan and the disappearance of military work.
Reference: i found out after the sports day some of the parents went on facebook and made a fuss about it and that got a lot of people interested. Generation: the some parents have been able to conduct the campaign day and the some comments on this matter and the many persons have been able to conduct attention.
In another run, we also find that “in accordance with” starts occurring often in generations. We hypothesize that even if we suppress particular tokens
5In comparison, Bai et al. (2022a) use a coefficient of 0.001.
from occurring, the model could frequently generate other tokens or discover more spurious features that are more obscure and hard-to-interpret.
Experimental details on the above examples. For the Sudoku experiment, the hyperparameters are selected from the same sets as in §4.1. For the MT experiment, to train the classifier fϕ, the model is initialized by a WMT17 Zh-En MLEtrained model. Then, the source sentence is fed into the encoder, and the target sentence is fed into the decoder. However, we remove the attention mask in the decoder that prevents hidden states at token t from seeing future hidden states. The reward rt is the probability that the t-th token is erroneous, according to fϕ. For Dtask, our translation task uses the WMT17 Zh-En dataset, and fϕ is fine-tuned from an MLE-trained MT checkpoint using the WMT17 Zh-En dataset. We use a transformer model with 6 encoder layers and 6 decoder layers. The number of attention heads is 8 in both the encoder and the decoder. The FFN embedding dimension is 2048 in both the encoder and the decoder.
Takeaway. Even a small amount of examples with spurious patterns in Dreward can enable the reward function to assign high reward on sequences containing those patterns. After RL training, a large proportion of generations could incorrectly contain those patterns.

heading: 4.3 Covariate Shift
text: During RL training, the policy (i.e., the generator) may sample examples out of the support of the reward model. Therefore, in these examples, the reward model’s behavior is underspecified – it may or may not assign high rewards to these low-quality examples.
Synthetic example: another Sudoku variant. Dreward contains 200k positive and 200k negative examples.6 We design Dreward in such a way that the model behavior would be undefined for certain inputs. All examples end with 1; continuations that end with 2–9 are not in the support on the data used to train the reward function fϕ. fϕ achieves 96.5% accuracy on the test split of Dreward. We sample 1000 in-support (i.e., ending with 1) and 1000 out-of-support (i.e., ending with 2–9) invalid Sudokus. The model only misclassifies
6Negative examples are obtained by swapping two different tokens of a positive example 1–20 times.
1 out of 1000 example as valid on the in-support set; in contrast, 659 out of 1000 examples are misclassified as valid on the out-of-support set.
During RL training of the conditional text generation model, the reward for sampled generations increases above 0.8. We expect the reward to imply that more than 80% continuations are estimated to be valid by the reward; however, only <10% of the continuations are actually valid.
Real-world example 1: AgreeSum. One simple example reproduces the multi-doc AgreeSum summarization (Pang et al., 2021). The input of the task is a cluster of articles, and the expected output is a summary that is faithful to every article in the cluster. We consider Dreward that consists of faithfulness annotations on article-summary pairs provided by the AgreeSum paper. The reward function fϕ is a summary-article faithfulness classifier. fϕ achieves 79% dev accuracy, which we use as the reward. However, the shortest summary in Dreward is 7-token-long, so the behavior of the reward for shorter summaries is underspecified. Training a summarizer using the faithfulness classifier as the reward leads to short summaries – most of which (>90%) are ≤ 2 tokens. Even though these nearempty summaries can be technically considered as being entailed in the article, we have not specified in Dreward that these summaries are acceptable. Real-world example 2: MT using BLEURT. BLEURT (Sellam et al., 2020) is a metric trained on expert annotations provided by WMT metric tasks.
We train a text generator by RL using BLEURT-20D3, a distilled version of BLEURT-20. BLEURT is trained on very few repetitive generations and very few long generations as discussed in the next paragraph. WMT15–19 human rating data (Stanojević et al., 2015; Bojar et al., 2016, 2017; Ma et al., 2018, 2019) are used to train BLEURT. We use BLEURT to train a MT model on the IWSLT14 De-En task (Cettolo et al., 2014). MLE-trained model achieves 63.9 in BLEURT on test set and RL-trained model achieves 65.5, so RL is successful judging by the increase in BLEURT.
Repetitive translations are out-of-support in our case, where repetition (rep) is measured the percentage of repeated 3-grams. In fact, only 0.02% (58/247,157) translations have rep >0.4 and 0.05% translations have rep >0.3 in Dreward. In addition, long translations are also out-of-support: only 0.01% of translations in Dreward has length longer than 180 BPE tokens.7
In the below analysis, we only examine the set of examples whose source length is larger than 180 tokens.8 We find that BLEURT does not punish for excessive repetition in the samples during RL: average BLEURT for translations with rep >0.4 (>40% of 3-grams are repetitions – an example is shown in the footnote to demonstrate that 40% is an undesirably large proportion)9 in the first 45,000 steps of training10 is 42.7, and average BLEURT for translations with rep <0.2 is 42.3.11 So the reward does not discourage the MT model from generating repetitions.
Next, we show in Figure 3 that as training goes on, translations get more and more repetitive as BLEURT increases. To summarize, given that long repetitive translations are rare in Dreward, the reward is underspecified on them. This repetition pattern
7BPE rules are learned using the IWSLT14 De-En dataset. 8Longer source sequences likely imply longer translations. We threshold based on source length so that the BLEURT comparison later is fair.
9As an example, the following sentence has rep = 0.397: pip was adopted from "great expectations; superman was a foster child; and the azbeth salander," the girl with the dragon tattoo, "was a foster child and a pure man; lyra belacqua from philip pullman," and a foster child, jane eyre, adopted, and roald’s james, and the great, and he was a parent, and a parent, and then, "and then, you know," and then, "and then, you know," and then, "and, you know," the "– and, you know," the "– and, you know," the "the" – and "you know," the "
10Using β = 0.05 which leads to the best dev BLEURT. 110.2 is an acceptable threshold, given that 93% of transla-
tions whose source sentence length >180 have rep <0.2.
is not discouraged by the reward, and thus it is subsequently exploited by the MT model.
Experimental details for the above examples. For AgreeSum, given URLs in the original dataset, to find the corresponding articles, we use the newspaper3k library. The reward function (classifier) is based on RoBERTa-large. The summarizer is based on BART-large (Lewis et al., 2020). For the MT experiment, the MT model has an embedding dimension of 512 for both the encoder and the decoder. The FFN embedding dimension is 1024 for both the encoder and the decoder. Both the encoder and the decoder have 4 attention heads and 6 layers. More details can be found in the appendix.
Takeaway. Suppose examples with certain undesirable patterns occur only rarely (or never occur) in Dreward. Then, they could be out of support of the reward model, and the reward model could assign high reward to these examples. After RL training, many generations could contain those patterns.

heading: 5 Possible Remedies
text: As discussed in §2, Skalse et al. (2022) has suggested that a pair of unhackable nontrivial originalproxy reward functions do not exist in theory. Then, when is it safe to use the proxy reward function? While this is still an open question, it is possible to reduce the extent of generating undesirable sentences through the following approaches.
The fundamental problem is that errors in the reward functions, specifically the over-confident errors where low-quality outputs have high rewards, can be exploited during RL training of text generators. Thus, one solution is to avoid OOD states that incur such errors by restricting the policy.
Restricting the policy by regularizing toward the ML solution. A common strategy is to regularize toward the ML solution. In practice, we can interpolate RL and ML losses (Wu et al., 2016), interleave RL and ML updates (Lewis et al., 2017; Guo et al., 2018), or use KL-regularized RL (Jaques et al., 2019; Stiennon et al., 2020; Ramamurthy et al., 2023). Here are a few potential issues. First, RL exploration could be important in case the reference dataset is small and consequently the ML solution is sub-optimal. In these tasks, it is often easier to verify or rate a generation than to provide a reference generation (unless we have access to a large language model). For example, in AgreeSum, there are not enough reference summaries due to
data collection costs, but given a decent articlesummary faithfulness classifier, we can discover new summaries that have high rewards. Similarly, in creative generation tasks like story generation and textual style transfer, or in code generation, there may not be a large enough high-quality reference dataset, but a reward function is often available. Second, ML solution may not be optimal even with an adequately large reference dataset; e.g., degeneracies like unreasonably short translations (Stahlberg and Byrne, 2019; Kulikov et al., 2022) and repetitive generations (Welleck et al., 2020b,a; Chiang and Chen, 2021) may often have high probabilities.
By relying on ML, we are essentially optimizing toward a different objective from the reward (Korbak et al., 2022); thus, we may need to find another automatic evaluator (instead of the proxy reward) to do hyperparameter tuning and model selection, which is difficult empirically and may require numerous tricks (Khalifa et al., 2021).12 In addition, KL-regularized RL cannot enforce distributional conditions of the set of all generations (Khalifa et al., 2021; Santurkar et al., 2023).
Restricting the policy by leveraging a discriminator. Following Goodfellow et al. (2014), Pang et al. (2021), and Vuong et al. (2022), another idea similar to ML-regularization is to leverage a discriminator that distinguishes between sampled generations and the set of dataset-provided generations.13 During RL training, we force the model to produce generations that are indistinguishable from references according to the discriminator. Discriminator and RL updates are interleaved. It is difficult to use GAN to train a high-quality text generator, but we hypothesize that the discriminator can reduce easy-to-identify low-quality examples during RL training.
Fixing the reward itself. Another thread of remedies is to fix the reward itself. An effective approach is to iteratively collect human annotations (Stiennon et al., 2020; Bai et al., 2022a; Fan et al.,
12Gao et al. (2022) has recently discovered that larger coefficients for the KL penalty (for ML regularization) does not improve the frontier of the curve of the gold reward-model score vs. the KL divergence (between the RL-optimized model and the ML model), so the coefficients only impact gold reward’s convergence speed. See Figure 9 of Gao et al. (2022). Thus, tuning and empirical tricks would be crucial to the success of KL-regularized RL.
13The discriminator predicts whether the generation is machine-generated or comes from the set of references. This technique is useful when there are only few parallel datapoints.
2022): the reward is iteratively updated with human annotations on latest model generations; thus, the generations with low human preferences but high rewards will be corrected through this iterative process. One concern is the cost (e.g., may require bilingual speakers or professional translators for MT annotations), which may prohibit an adequate amount of iterations or adequately frequent iterations. Krakovna et al. (2020) has discussed the possibility that a machine can learn to fool human evaluators in robotics, but it is unclear what the equivalence in conditional text generation is. So far, this approach has been successful, with the critical assumption that there is little budget/resource constraint to obtain enough high-quality annotations and enough iterations of annotations.
Another caveat is that as Perez et al. (2022) has recently discovered, RL-with-human-feedback may amplify one-sided views (e.g., on political issues); they claim that this phenomenon can be explained by the selection bias of annotators, leading to unrepresentative reward. Similarly, if selection bias is unavoidable in the context of conditional text generation (therefore the unrepresentative reward is unavoidable), we may need another way of fixing the reward and preventing the generation model from amplifying the bias – e.g., by hardcoding a set of principles as in Constitutional AI (Bai et al., 2022b).
More discussion. An additional method in the RL literature is conservative Q learning (Kumar et al., 2020); it aims to push down all high rewards to ensure that the out-of-distribution states do not achieve high Q values, but the approach requires extensive hyperparameter tuning (Zheng et al., 2022). Another possibility to avoid the reward gaming issue is to simply avoid interaction with the environment using methods like Pang and He (2021) to learn from demonstrations, so the errors in the reward function will be less exploited; additionally, non-RL objectives that can learn from both positive and negative examples (Adolphs et al., 2022) are also potential solutions.

heading: 6 Conclusion
text: We use synthetic and real-world tasks to demonstrate that even if a learned reward achieves high performance on Dreward, a high reward may still get assigned to undesirable patterns which get amplified during RL training of the conditional text generation model. A critical future direction is to
detect obscure or hard-to-interpret gaming behaviors especially in long generations. Then, we can investigate when or how easily a spurious feature could be exploited, by exploring the relationship among the minimum description length of a spurious feature (Voita and Titov, 2020) or similar statistics, the proportion of datapoints that contains the spurious feature, the choice of RL algorithm, and the degree of the reward gaming behavior. Additionally, research on new approaches of reward or preference learning is needed.
Limitations
First, off-policy algorithms like Q learning are not explored in this discussion. Second, the reward gaming issue is not a novel topic in the RL community for tasks like gameplay or autonomous driving (Amodei et al., 2016; Koch et al., 2022). However, we hope to highlight issues in the NLG community (specifically on conditional text generation tasks without use of large language models) especially given the recent endeavors on learning from learned metrics.
In addition, the paper aims to demonstrate the existence of reward gaming in conditional text generation, not the certainty regardless of experimental settings (hyperparameters, architectures, etc.). Given that our experiments use reasonable settings which lead to degenerate texts, we argue that reward gaming could be a common issue when learning a text generation model using RL based on learned rewards, and the issue deserves attention from researchers and practitioners. We leave it to future work to investigate the easiness of reward gaming in practice, which is missing in this work.

heading: Acknowledgement
text: We thank Nitish Joshi, Nicholas Lourie, Angelica Chen, Chen Zhao, Sebastian Gehrmann, and anonymous reviewers for valuable discussion and feedback. This work is supported by the Google Research Collabs program. In addition, VP and HH are supported by Samsung Advanced Institute of Technology (under the project Next Generation Deep Learning: From Pattern Recognition to AI) and AWS AI.

heading: A More Background
text: For our policy gradient algorithms, we use the standard REINFORCE algorithm with tricks that are introduced in the following paragraphs.
Specifically, in all RL experiments, we first initialize the model using an MLE-trained model to ensure a good starting point for RL optimization. During training, we collect a set of trajectories through sampling from the current policy (i.e., generator). Then, we compute the estimated return Q̂t at each time-step t.
Next, the estimated return Q̂t is subtracted by a baseline. Therefore, the actual gradient update is as follows: ∇θJ(θ) = Eτ∼πθ ∑ t∇θ log πθ(at |
st)[Q̂(st, at)−b(st)], where Q̂(st, at) = ∑T t′=t rt′ assuming discount factor γ = 1, and b is possibly state-dependent. In particular, for Sudoku experiments as well as the experiment where we train an MT model using BLEURT as the reward, we attempt two variants of baseline: (1) using the average reward for the past 50 updates, which is an effective strategy in training models using sequencelevel rewards (Kiegeland and Kreutzer, 2021), and (2) using the value function fitted by mean-squared error (so the estimated return subtracted by the value ends up being the advantage), introduced in full detail here.14 For case (1), the results are shown in the blue lines in the plots; for case (2), the results are shown in the purple dotted lines in the plots. We use the Adam optimizer (Kingma and Ba, 2014) for all our experiments.
In particular, we use KL-regularized RL, as discussed in §3. Regularization toward ML may stabilize RL optimization, but it may still lead to higher rewards that correspond to undesirable behaviors, as discussed in §5. The coefficient for the KL term is tuned in {0.01, 0.05, 0.1} for Sudoku experiments and {0.01, 0.03, 0.05, 0.1, 0.25} for other
14https://spinningup.openai.com/en/latest/ algorithms/vpg.html#pseudocode
experiments. For the purpose of this discussion, to illustrate the effect of reward gaming, the coefficient is tuned to achieve the highest validation reward; due to optimization issues in practice, a lower coefficient does not necessarily correspond to a higher reward. Larger coefficients may lead to lower proxy rewards but higher true rewards. While it may address the reward gaming problem in some experiments, we have shown in §4.3 that even large coefficients may lead to reward gaming.
Proximal policy optimization (PPO; Schulman et al., 2017) is a widely used algorithm that aims to avoid reward collapse. Our conclusion, however, does not depend on the RL algorithm. Using PPO prevents the optimization from converging to a very low reward, but it does not eliminate the possibility that high reward generations have undesirable patterns. In addition, Q learning, an off-policy RL algorithm that can leverage existing trajectories, is recently applied to also be applied in text generation (Kohita et al., 2020; Pang et al., 2022) but is not explored in this discussion.

heading: B More Experimental Details
text: B.1 Details for the Experiments on Noise-Induced Spurious Correlation
Examples that are used to train the reward function. As explained in §4.1, there are 1M examples in total, 500k of which are positive examples and 500k are negative examples. The negative examples consist of the following parts: (i) 100k invalid Sudokus that are randomly sampled. None of the above examples end with 7. (ii) 100k invalid Sudokus obtained by removing l cells randomly from a random positive Sudoku, where l is an integer randomly sampled from 1 to 80. (iii) 300k invalid Sudokus that are obtained by swapping cell i and cell j of a random positive Sudoku; after swapping, we verify that the Sudoku is in fact invalid. The train/dev/test split of Dreward is 900k/50k/50k. Reward. The RoBERTa-tiny-based (Liu et al., 2019) reward function has 4 encoder layers and 2 attention heads; the encoder embedding dimension is 64, and the dimension for FFN for 256. All the Sudoku-related experiments are done on either a single NVIDIA V100 GPU with 32G of memory or a single NVIDIA RTX 8000 GPU with 48G of memory. The reward training typically takes 1 hour. The batch size is tuned in {128, 256, 512}. The dropout rate is tuned in {0.01, 0.1}, and we find
that 0.01 always works better. The max number of epochs is set to 60. The learning rate is tuned in {1e-4, 5e-4, 1e-3}. For the best configuration, we use batch size 512 and learning rate 5e-4. It achieves a 99.3% accuracy on the dev set (5% split), and a 99.3% accuracy on the test set (5% split).
Out of 1000 samples of invalid Sudokus that end with 7 and contain 81 tokens, the trained classifier predicts (incorrectly) that 1000 are valid. Out of 1000 samples of invalid Sudokus that end with 7 and contain fewer than 81 tokens, the trained classifier predicts that 0 is valid. The performance of Sudokus longer than 81 tokens is irrelevant, given that during RL sampling as well as during generation test time, the sequences are constrained such that they can at most generate 81− k tokens where k is the length in the given source sequence.
Sequence generator. Suppose the input to the generator contains k numbers. During RL sampling and during test-time of the generator, the sequence generator is constrained to generate at most 81− k numbers. However, it can generate fewer than 81− k numbers. To avoid sequence generators from generating overly short continuations, part (ii) of the negative examples, described above, contains examples that are too short.
For the sequence generator, we use a smaller version of the transformer_iwslt_de_en architecture in fairseq (Ott et al., 2019). The encoder embedding dimension and the decoder embedding dimension are both 32. We use 2 attention heads in both the encoder and the decoder. The dimension for FFN in both the encoder and the decoder is 64. There are 2 encoder layers and 2 decoder layers. All the text generation models in the Sudoku experiments have 43k parameters.
The batch length (i.e., number of tokens in a batch) is tuned in {8192, 16,384, 32,768, 65,536}. The learning rate is tuned in {1e-4, 1.5e-4, 2e-4}. The dropout rate is tuned in {0.01, 0.1, 0.3}. For optimal reward, we choose a batch length of 32,768, a learning rate of 1.5e-4, and a dropout rate of 0.01. The training algorithm is detailed in §A.
B.2 Details for the Experiments on Naturally Occurring Spurious Correlations
Sudoku revisited. For the second Sudoku example, the hyperparameters are selected from the same sets as in §B.1. For the best-performing classifier, the learning rate is 5e-4 and the dropout rate is 0.01. For the sequence generator, we use the same
hyperparameters as before. The lack of repetition in the last nine numbers (of the output) is spuriously correlated with a high reward, given that nonrepetition is a necessary but not sufficient condition for a valid Sudoku. fϕ achieves 99.9% accuracy on test set of Dreward. The text generator learns to exploit the non-repetition pattern which leads to high rewards, but the generations are mostly wrong.
Training an MT model using the WMT MQM dataset. To train the reward function, the learning rate is selected from {1e-4, 2e-4, 5e-4}, and dropout is selected from {0.01, 0.1, 0.3}. For optimal performance, we use a learning rate of 2e-4 and a dropout rate of 0.3. Training the reward function takes around 3 hours.
For Dtask, our translation task uses the WMT17 Zh-En dataset, and fϕ is fine-tuned from an MLEtrained MT checkpoint using the WMT17 Zh-En dataset. We use a transformer model with 6 encoder layers and 6 decoder layers. The number of attention heads is 8 in both the encoder and the decoder. The FFN embedding dimension is 2048 in both the encoder and the decoder. There are 82.6M parameters in the model.
The algorithm is detailed in §A. We use a KL coefficient of 0.1. We use a dropout rate of 0.3, a learning rate of 1e-4, and a batch length of 4096. All the MT experiments are done on a single NVIDIA RTX 8000 GPU with 48G of memory. Training time is only 24 hours, given that we do not need to train the model till convergence to see the undesirable patterns in generations.
B.3 Details for the Experiments on Covariate Shift
An example of computing rep. The sentence ‘a b c e d c e d c d’ has rep = 2/5 = 40%, given that among ‘e d c,’ ‘d c e,’ ‘c e d,’ ‘e d c,’ ‘d c d,’ two 3-grams are the same with the existing ones.
Experimental details for AgreeSum. Given URLs in the original dataset, to find the corresponding articles, we use the newspaper3k library. We use slightly different architectures from the AgreeSum paper. The reward function (classifier) is based on RoBERTa-large (Liu et al., 2019) with 355M parameters. We use a learning rate of 5e-4, a dropout rate of 0.1. The submitted job for the classifier is 24-hour-long. The summarizer is based on BART-large (Lewis et al., 2020) with 406M parameters. We use a learning rate of 3e-5, a batch length of 2048, and a dropout rate of 0.1. We use a single NVIDIA RTX 8000 GPU for AgreeSum experiments.
Experiment Details for MT with BLEURT as Reward. The BLEURT-20-D3 evaluator has around 30M parameters. For the MT model that is trained on the IWSLT14 De-En dataset (train/dev/test size: 160,239/7,283/6,750), the embedding dimension is 512 for both the encoder and the decoder. The FFN embedding dimension is 1024 for both the encoder and the decoder. Both the encoder and the decoder have 4 attention heads and 6 layers. There are 39.5M parameters in the model. The learning rate is selected from {1e-4, 3e-4}. The batch length (i.e., number of tokens in a batch) is set to be 4,096 and the dropout rate is set to be 0.3 – these are the optimal choices for IWSLT14 De-En experiments trained using MLE. KL coefficient is selected from in {0.01, 0.03, 0.05, 0.1}. We choose the hyperparameter settings that lead to the highest validation BLEURT. Training time is around 20 hours on a single NVIDIA RTX 8000 GPU.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: "Limitations" section.
A2. Did you discuss any potential risks of your work? Not applicable. This paper aims to identify risks.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and introduction.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4.2 (MQM, WMT), and Section 4.3 (AgreeSum, BLEURT, IWSLT) – some datasets are composed of multiple years’ WMT datasets, and we have cited each year’s creators.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. The machine translation datasets are all in the public domain, widely used by industry and academia. The newly created Sudoku datasets are purely synthetic.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. The machine translation datasets are all in the public domain, widely used by industry and academia. The newly created Sudoku datasets are purely synthetic.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. The collected dataset (Sudoku) is purely synthetic.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? The MT datasets’ languages are specified in Section 4.2 and Section 4.3.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix, as well as Section 4.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 4.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix. All three items are reported in the paper.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix.
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Given that we aim to demonstrate the existence of reward gaming, most experiments reported in the paper are from a single run (for a given set of hyperparameters, after careful hyperparameter search).

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? The package fairseq is discussed in the Appendix. BLEURT is discussed in Section 4.3.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: A critical future direction is to
detect obscure or hard-to-interpret gaming behaviors especially in long generations. Then, we can investigate when or how easily a spurious feature could be exploited, by exploring the relationship among the minimum description length of a spurious feature (Voita and Titov, 2020) or similar statistics, the proportion of datapoints that contains the spurious feature, the choice of RL algorithm, and the degree of the reward gaming behavior. Additionally, research on new approaches of reward or preference learning is needed