MIXCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies
Year: 2023

Abstract:
Autoregressive language models are trained by minimizing the cross-entropy of the model distribution Qθ relative to the data distribution P – that is, minimizing the forward cross-entropy, which is equivalent to maximum likelihood estimation (MLE). We have observed that models trained in this way may “over-generalize”, in the sense that they produce non-human-like text. Moreover, we believe that reverse crossentropy, i.e., the cross-entropy of P relative to Qθ, is a better reflection of how a human would evaluate text generated by a model. Hence, we propose learning with MIXCE, an objective that mixes the forward and reverse crossentropies. We evaluate models trained with this objective on synthetic data settings (where P is known) and real data, and show that the resulting models yield better generated text without complex decoding strategies. https://github.com/bloomberg/ mixce-acl2023

heading: 1 Introduction
text: Rapid advances in pre-trained large-scale autoregressive language models (LMs) have dramatically improved the performance of a variety of tasks (Radford et al., 2019; Brown et al., 2020; Zhang et al., 2022; Chowdhery et al., 2022). However, these systems still struggle in many openended generation settings, where they are asked to produce a long text following a short prompt. In these cases, we seek systems that generate sensical, coherent, fluent, and engaging, or in short, human-like text (Pillutla et al., 2022).
Different decoding strategies to generate such text from pretrained LMs suffer from different degeneration problems. Unbiased sampling1 usually
∗ Work done during an internship at Bloomberg. 1Unbiased sampling is vanilla random sampling, i.e., sampling with temperature=1.0. It is also called ancestral sampling (Eikema and Aziz, 2020) or pure sampling (Holtzman
results in incoherent and nonsensical text, while greedy and beam searches often get stuck in repetition loops (Holtzman et al., 2020). These observations suggest that the learned LM distribution Qθ still differs substantially from the human LM distribution P . A possible reason is that the autoregressive modeling of Qθ gives a non-zero probability to every possible sequence of tokens, while many sequences are impossible under P . Nevertheless, we still hope that Qθ(x) is as small as possible when P (x) = 0. To this end, maximum likelihood estimation (MLE), i.e., minimizing the cross-entropy (CE) −Ex∼P [logQθ(x)], is the most widely used objective to train Qθ(x) using sequences sampled from P . In an idealized setting, with unlimited training data and model capacity, as well as a perfect optimizer, fitting Qθ with MLE will learn a distribution as close to P as we like. However, in practice, we only have finite and noisy data.
We argue that the MLE objective only weakly penalizes generations x from Qθ that are “bad”,
et al., 2020). We call it unbiased sampling because it allows unbiased exploration of the model distribution.
2Note that logP (x) is infinite when P (x) = 0. But in practice, we use logP (x) = ∑ t log(P (xt|x<t)+ϵ) to avoid log 0 and ϵ = 1e− 30.
9027
in the sense that P (x) = 0. When Qθ puts a small amount of probability mass onto P (x) = 0 space, MLE cannot sufficiently discourage this behavior (see Figure 3 in Appendix C). Moreover, minimizing forward CE, −Ex∼P [logQθ(x)], is equivalent to minimizing the forward KL divergence between P and Qθ, i.e., KL(P ||Qθ) = Ex∼P [logP (x)/Qθ(x)]. Forward KL has a zeroavoiding property – avoiding Qθ(x) = 0 when P (x) ̸= 0 (Murphy, 2012). Therefore, if there is noise in the data, Qθ will try to cover the noise as well, which leads the model to over generalize, in the sense of putting non-trivial probability mass over P (x) = 0 generations (Huszár, 2015; Theis et al., 2016; Ott et al., 2018; Kang and Hashimoto, 2020). As a result, we observe samples from the model deviating from human-like text. A common strategy is to modify the decoding method, e.g., top-k, top-p, typical, contrastive (Fan et al., 2018; Holtzman et al., 2020; Meister et al., 2022; Li et al., 2022) samplings, to tailor the model distribution Qθ in a post-hoc manner to avoid unwanted generations. In contrast, our approach differs: how can we obtain a better Qθ to obviate the need for these sampling strategies?
We propose a novel training objective for autoregressive LMs – MIXCE that Mixes the forward and reverse Cross-Entropies: −η · Ex∼P [logQθ(x)]− (1− η) · Ex∼Qθ [logP (x)]. MIXCE can be understood in two ways. First, we want model generations to be high-quality as well as diverse. Reverse cross-entropy reflects how we conduct human evaluations, sampling from the model Qθ and evaluating it by the human P , where the focus is text quality. Forward cross-entropy emphasizes the diversity of model generations (Hashimoto et al., 2019). Second, MIXCE works similarly to a mixture of the forward and reverse KL divergences. The reverse KL divergence (KL(Qθ||P )) is zero-forcing – forcing Qθ(x) = 0 when P (x) = 0 – and thus more strongly penalizes generating non-human-like samples compared to MLE. Overall, MIXCE combines two complementary driving forces to better fit Qθ to P (Figure 1). We elaborate on these interpretations in § 3.1.
Unfortunately, optimizing reverse cross-entropy is intractable because we do not know P . Hence, we propose an approximation of the reverse crossentropy (see § 3.2), which ends up being a selfreinforced loss function that encourages the model to produce generations in which it is already con-
fident. This loss function has the same computational complexity as forward cross-entropy, making MIXCE easy to implement and as fast as MLE.
We demonstrate the effectiveness of MIXCE in both a synthetic setting, where the “human” distribution P is known, as well as a real setting. For the synthetic case, we evaluate six learning objectives: MIXCE, MIXCE∗ (MIXCE without approximation), forward KL (=MLE), reverse KL, the mixture of two KL divergences, and Jensen–Shannon (JS) divergence. We show that MIXCE∗ works slightly worse than the mixture of KLs while outperforming other objectives, and MIXCE works worse than MIXCE∗ but generally outperforms MLE. In real settings, we finetune GPT-2 (Radford et al., 2019) of different sizes on three English text domains using MIXCE or MLE. Our results show that, compared to MLE, unbiased sampling from MIXCE-finetuned models produces text that has diversity (Meister et al., 2022) closer to that of human text, has higher Coherence (Su et al., 2022), has higher Mauve (Pillutla et al., 2021), and is preferred by humans. When using top-p sampling (Holtzman et al., 2020) and carefully tuning p, generations from MLE-finetuned models are similar to those generated from MIXCE-finetuned models. Nonetheless, MIXCE models have tuned p values closer to 1, implying a less noisy model distribution. In addition, we modify the original Mauve to make it more robust to spurious features (e.g., text length), under which MIXCE still improves over MLE when using unbiased sampling.

heading: 2 Background and Related Work

heading: 2.1 Autoregressive Language Modeling
text: Language generation is mostly based on the autoregressive language modeling methodology. The generation of one word is conditioned on previously generated words, Qθ(xt|x<t), and the final probability of the sequence x is the product of probabilities of each step, Qθ(x) = ∏ tQθ(xt|x<t). Early works build n-gram neural LMs (Bengio et al., 2000) and then RNN-based LMs (Mikolov et al., 2010), and now Transformers (Vaswani et al., 2017) have become the dominant architecture. Language generation models have either a decoderonly (Mikolov et al., 2010) or an encoder-decoder architecture (Sutskever et al., 2014; Bahdanau et al., 2015). In this work, we focus on decoder-only LMs. In recent years, many large-scale pre-trained decoder-only LMs have been introduced (Radford
et al., 2019; Brown et al., 2020; Zhang et al., 2022; Chowdhery et al., 2022). They can be finetuned for downstream tasks and even perform surprisingly well in a zero-shot or few-shot manner. Despite the impressive performance, language degeneration is one of the key issues that remain to be solved.
2.2 Language Degeneration According to Holtzman et al. (2020), language degeneration refers to output text that is bland, incoherent, or gets stuck in repetitive loops. It is widely observed in open-ended generations from pretrained LMs. Two commonly observed patterns of degeneration are the incoherent text from unbiased sampling and the repetitive text from greedy or beam search. Degeneration also appears in sequence-to-sequence generation tasks but in a slightly different form (Stahlberg and Byrne, 2019).
There is no agreement on what causes degeneration. Ott et al. (2018) attribute it to data noise and the smooth class of model functions. It is inherent in the model’s structure to have support everywhere, in particular, because all probabilities are produced by softmax, which is strictly positive. Therefore, Hewitt et al. (2022) assume that an LM distribution is the true data distribution plus a uniform-like smoothing distribution. Based on the observation that human-like text has a large but not too large likelihood under the learned LM distribution (Zhang et al., 2021), a lot of works propose empirically useful decoding methods beyond unbiased sampling and greedy/beam search (Fan et al., 2018; Holtzman et al., 2020; Eikema and Aziz, 2020; Basu et al., 2021; Meister et al., 2022; Li et al., 2022; Hewitt et al., 2022; Su et al., 2022; Krishna et al., 2022). One of these approaches is the canonical top-p (or nucleus) sampling method (Holtzman et al., 2020), which samples from top tokens that take up p proportion (e.g., 95%) of the probability mass at each decoding step. Even though these decoding methods work impressively well, they are post-hoc fixes rather than learning the LM accurately in the first place. Therefore, some other works criticize the MLE training objective and propose alternative loss functions.

heading: 2.3 Objectives Beyond MLE
text: Unlikelihood training (Welleck et al., 2020; Li et al., 2020) was proposed to penalize repetition (or any undesirable phenomenon) explicitly during training. The idea is to minimize the likelihood of a set of negative tokens at each generation step
during training. The selection of negative tokens is pre-defined, e.g., tokens that appear often in the previous context. MIXCE shares the same goal with unlikelihood training – matching the human LM distribution, but provides a more general approach without targeting any specific problem.
Similar to our motivation, Kang and Hashimoto (2020) think that the zero-avoiding property of MLE makes the model sensitive to dataset noise. To cover these noisy examples, the model has to put non-trivial probability mass on the P (x) = 0 area. To combat this problem, they propose a loss truncation method that drops high-loss (low-likelihood) examples during training time.
Pang and He (2021) want to address the mismatch of learning objective and human evaluation (likelihood vs. quality) and introduce the GOLD algorithm to approximate reverse cross-entropy. Our approximation is similar to theirs but has a different derivation process (see § 3.2). Moreover, GOLD is evaluated on controlled generation tasks (e.g., summarization and translation) in which the goal is to generate one high-quality text for each input, and diversity is not so important. In contrast, if we train the LM only with reverse CE till convergence, the model will deterministically produce the most likely text for each prompt, which is undesirable for an LM. Therefore, mixing forward and reverse CEs is necessary.
The idea of MIXCE is also relevant to GANs (Goodfellow et al., 2014). GANs optimize the Jensen–Shannon (JS) divergence between model and data distributions. Essentially, JS divergence is also for balancing the two driving forces of forward and reverse KL divergences (Huszár, 2015), and it has been successfully used for evaluating LM-generated text (Pillutla et al., 2021). However, probably due to the discrete nature of text, GANs have not been well applied to LM training. Caccia et al. (2020) show that previous language GANs often give up diversity for quality.
Another related work is Popov and Kudinov (2018), which finetunes LMs with the sum of the forward cross-entropy loss and reverse KL divergence. They train a discriminator to estimate reverse KL, similar to a GAN. On the other hand, we directly approximate reverse cross-entropy in our objective function, without training an additional discriminator.
Concurrently, with the same motivation as ours, Ji et al. (2023) propose to replace MLE with min-
imization of the total variation distance (TVD) (Van Handel, 2014) between data and model distributions. Notably, their final approximation of TVD, which they call TaiLr, is equivalent to forward cross-entropy when the hyperparameter γ = 0 and equals our approximated reverse cross-entropy when γ = 1.

heading: 3 Methodology

heading: 3.1 MIXCE
text: Our MIXCE learning objective for training LMs is the combination of forward and reverse crossentropies, written as
−η ·Ex∼P [logQθ(x)]− (1−η) ·Ex∼Qθ [logP (x)] (1) where η is the mixing ratio. When η = 1, it becomes the normal MLE objective; and when η = 0, it is the reverse cross-entropy only.
The MIXCE loss can be understood in two ways. First, reverse and forward cross-entropy (CE) emphasize quality and diversity respectively. The reverse CE, −Ex∼Qθ [logP (x)], focuses on quality because it resembles how we conduct human evaluations – sampling from the model Qθ and evaluating it by the human P . In human evaluations, the focus is more on the quality of the model-generated text. So, it is possible that a model always generates the same few high-quality texts, but still gets high human evaluation scores. This is similar to the mode collapse problem of GANs. The forward CE, −Ex∼P [logQθ(x)], instead focuses more on diversity because it needs any sample from P to have a non-trivial probability under Qθ (Hashimoto et al., 2019). Note that it does not mean forward CE has zero effect on quality, rather, the model likelihood Qθ(x) only loosely correlates with the human-perceived quality of x (Zhang et al., 2021).
Second, we hypothesize that MIXCE works similarly to a mixture of forward and reverse KL divergences, which we will show empirically in our synthetic experiments (§ 4.1). On the one hand, minimizing forward KL is equivalent to optimizing forward CE. On the other hand, reverse KL divergence, Ex∼Qθ [log Qθ(x) P (x) ], has two parts: reverse CE and negative entropy of Qθ, Ex∼Qθ [logQθ(x)]. Reverse CE is minimized when the model deterministically outputs the most likely example, i.e., Qθ(x) = δ(the most likely x under P ). Instead, minimizing the negative entropy (maximizing the entropy) of the model encourages it to be as un-
certain as possible, i.e., having a large support and uniform distribution. This entropy term counteracts the narrowing-down effect of reverse CE. As discussed above, forward CE pushes the Q distribution to fully cover the support of P . In this case, forward CE can also help counteract the narrowingdown effect of reverse CE, i.e., the maximizing entropy term becomes less important when forward CE is present. Hence, we think it is reasonable to drop it from reverse KL.
Overall, MIXCE combines two complementary training signals, as shown in Figure 1. Reverse CE prevents the model distribution from being broader than the data distribution, while forward CE is more helpful for preventing the model distribution from being narrower than the data distribution. Although forward CE also has non-zero loss when the model distribution is too wide, its loss magnitude is much smaller than what reverse CE provides (see Appendix C for more discussion). When data is clean, two CEs work jointly to help learn the data distribution better. When data is noisy, the mixing ratio η allows us to trade-off between emphasizing a good coverage of the data and putting more weight on the actually high-quality sequences.

heading: 3.2 Optimization of Reverse CE
text: Optimizing MIXCE is non-trivial. The obstacle is to minimize the reverse CE, −Ex∼Qθ [logP (x)] with respect to θ. To this end, we need to know P and to have a differentiable sampling operation from Qθ. In our synthetic experiments (§ 4.1), we use a distribution P of our own construction and use Gumbel-Softmax (Jang et al., 2017; Maddison et al., 2017) to make the sampling operation differentiable.
However, in a real setting, we do not know P . To deal with this, we take the following steps to derive an approximated reverse cross-entropy (we omit the negative sign for simplicity):
∇θEx∼Qθ [logP (x)] (2) ≈∇θEx∼Qθ [P (x)] (3) = ∑
x
∇θQθ(x)P (x) (4)
= ∑
x
Qθ(x)∇θ logQθ(x)P (x) (5)
= ∑
x
P (x)Qθ(x)∇θ logQθ(x) (6)
=Ex∼P [Qθ(x)∇θ logQθ(x)] (7)
=Ex∼P [ T∏
t=1
Qθ(xt|x<t) T∑
t=1
∇θ logQθ(xt|x<t)]
(8)
≈Ex∼P [ T∑
t=1
Qθ(xt|x<t)∇θ logQθ(xt|x<t)] (9)
First, from (2) to (3), we substitute expected log-likelihood by expected accuracy. Irsoy (2019) shows that expected accuracy is a comparable or better alternative loss function to cross-entropy for classification tasks. Then, following the Policy Gradient theorem (Williams, 1992; Sutton et al., 1999), we get (4) and (5), where we view model Qθ as the policy and P (x) as the reward we want to optimize for the whole sequence. Next, we switch from the expectation of Qθ to the expectation of P (from (5) to (6) and (7)), so that we can use the offline samples from P (data samples in the training set) instead of online sampling from Qθ. We unfold Qθ(x), which results in (8). Until this point, theoretically, we are already able to optimize the model using Equation (8) without knowing P . However, the product of Qθ(xt|x<t) has a very high variance, and in practice, it underflows when T is large. Therefore, we apply a final rough approximation that leads to (9).
Equations (8) and (9) are apparently not equivalent to each other. Nonetheless, they have similar effects. Intuitively, in (8), we weigh the gradients of each sequence differently based on their sequencelevel probabilities, Qθ(x); in other words, it promotes high-likelihood sequences. Similarly, (9) weighs gradients at each step by Qθ(xt|x<t), i.e., promoting high-likelihood tokens at each step. So essentially, they both encourage the model to produce generations in which it is already confident. We call it a self-reinforced objective. To further illustrate why self-reinforcement makes sense, we conduct an analysis using GPT-2 (Radford et al., 2019). Please refer to Appendix B for a detailed discussion. In short, we show that MLE-pretrained GPT-2 on average assigns a higher probability to human text than to text sampled from the model. Therefore, when we promote high-probability sequences or tokens, it is like “pushing” the model distribution toward the human distribution. But, we need to avoid overly “pushing” it to the extremely high-probability region where repetitive greedy search outputs locate.
Note that our approximation of reverse crossentropy is relevant to the method proposed by Pang
and He (2021), though we have a different derivation process from theirs. Please see Appendix A for a detailed comparison.
Finally, combining forward CE and Equation (9), our approximated MIXCE objective is to maximize
Ex∼P [ T∑
t=1
(η+(1−η) ·Qθ(xt|·))∇θ logQθ(xt|·)],
(10) where Qθ(xt|·) is short for Qθ(xt|x<t). This loss function has the same computational complexity as forward CE (MLE). Since Qθ(xt|x<t) is strictly lower than 1 (it is around 0.017 to 0.13 when using GPT-2), the gradient from approximated reverse CE is smaller than that from forward CE. Therefore, it is important to tune η to balance the effects of two CEs.

heading: 4 Experiments

heading: 4.1 Synthetic Experiments
text: We first conduct experiments in a synthetic ideal setting, where we know P , to show the effectiveness of mixing two cross-entropies with or without approximation. Moreover, during evaluation, we can directly compare the learned model parameters against the ground truth parameters of P .
Define the “human” LM P . We start by defining P as a bi-gram LM. Bi-gram means that the prediction of the next token only depends on the immediately previous token, i.e., P (xt|xt−1). Therefore, P is determined by a transition matrix among words M ∈ RV×V (V =vocabulary size) and a start token probability distribution π ∈ RV , i.e., stochastic finite-state automata. The last token in the vocabulary is the end-of-sequence (EOS) token. For simplicity, we initialize π as a uniform distribution. To initialize M, we use two methods. The first is random initialization. We sample categorical distributions from a Dirichlet (α=0.5) prior to initialize each row of M. However, one remaining problem is that P has support everywhere. To have P = 0 areas, we randomly assign 0s to a certain percent of values in each row of M and then re-normalize to sum to 1.3 We test 3 percentages: 10%, 50%, and 90%. The second is initialization using real data. We sample 5000 pieces of text from WebText (Radford et al., 2019), count the occurrence of bigrams, and then use the occurrence to
3When we assign 0s, we make sure every token has nonzero transition probability to EOS.
initialize M. In this case, there are naturally 0s in M, and the larger the vocabulary size is, the sparser M is. No matter which initialization is used, we reserve the last row of M for EOS and it has all 0s, i.e., will not transit to any token. We set the vocabulary size V =20, 50, 100, 500, or 1000.4
Learn an LM Qθ. We implement model Qθ as a simple neural bigram LM. Given the word embedding ei−1 of the previous token xi−1, the next token is predicted via a simple neural network f :
hi−1 = Dropout(ReLU(W1ei−1 + b1)),
Q(xi|xi−1) = Softmax(W2hi−1 + b2), where W1 ∈ Rd×d (d is the hidden dimension size), b1 ∈ Rd, W2 ∈ Rd×V , and b2 ∈ RV are model parameters. After training this model, the learned transition matrix can be obtained by M′ = f(E), E is the word embedding matrix.
Synthetic data. We sample sequences from P . We set the max sequence length as 500. We sample 50K and 5K sequences as the training and validation set, respectively. There is no test set because we directly compare the learned transition matrix M′ to the gold M during evaluation.
Metrics. (1) avg. js: we compute the JS divergence between each row (except the last row) of M′ and the corresponding row in M, and then average across rows. This metric evaluates the overall divergence of M′ from M, and equals 0 iff M′ = M; (2) avg. 0s: we get the probabilities from M′ from positions where the corresponding gold probabilities are 0 in M, and take their average. If M′ = M, avg. 0s = 0, but vice versa is not true.
Objectives. (1) Forward KL, KL(P ||Qθ) = Ex∼P [logP (x)/Qθ(x)], which is equivalent to MLE; (2) Reverse KL, KL(Qθ||P ) = Ex∼Qθ(x)[logQθ(x)/P (x)]; (3) Mixture of two KLs, η · KL(P ||Qθ) + (1 - η) · KL(Qθ||P ); (4) JS, we use a general definition of JS divergence (Huszár, 2015), η · KL(P ||M ) + (1 - η) · KL(Qθ||M), where M=η · P + (1 - η) · Qθ;5 (5) Oracle mixture of cross-entropies (MIXCE∗), where we use the known P . (6) Approximated
4Our defined bi-gram LMs are always tight, i.e., do not “leak” probability mass onto infinite sequences because we make sure that all accessible tokens also have non-zero paths to other tokens. Please refer to Du et al. (2022) for the proof.
5When η = 0.5, it is the same as the objective of GAN (Goodfellow et al., 2014). But instead of using GAN’s min-max loss, we directly optimize JS because we know P .
mixture of cross-entropies (MIXCE), where we assume P is unknown. Except for Forward KL and MIXCE, the other four objectives all need to sample from Qθ and require gradients to pass through this sampling operation. To this end, we use Gumbel-Softmax (Jang et al., 2017; Maddison et al., 2017) to make sampling differentiable.
Model selection. During training, we check the validation loss (the value of the objective function) after every epoch and only save the best checkpoint that has the lowest validation loss. For objectives with η, we choose the best η based on the avg. js result on the validation set. We report a 5-seed average for each experiment. The search space of η is [0.99, 0.9, 0.5, 0.1, 0.01]. Selected best ηs are reported in Table 11 in the Appendix.
Results. Table 1 (and Table 6 in the Appendix) shows the results of our synthetic experiments. Across 4 kinds of initialization of M and 5 vocabulary sizes, we observe some common patterns. First, the mixture of two KLs often gets the best avg. js compared to other objectives, and MIXCE∗ usually comes second. This supports our expectation that the mixture of two cross-entropies approximates the mixture of two KLs (§ 3.1), as well as demonstrates that combining two KLs or CEs can help learn the data distribution more accurately compared to MLE. Second, the approximated MIXCE usually under-performs MIXCE∗ but outperforms forward KL (MLE). Third, reverse KL generally works best for the avg. 0s metric, due to its property of zero-forcing – forcing Qθ(x) = 0 when P (x) = 0. Lastly, JS divergence oftentimes works similarly to reverse KL, which is consistent with the observation made by Caccia et al. (2020) – language GANs trade off diversity for quality.

heading: 4.2 GPT-2 Experiments
text: Next, we test MIXCE in a real setting where we do not know P , but we have finite samples from P . We use GPT-2 (Radford et al., 2019) as the LM Qθ. Though GPT-2 models are already pretrained by MLE, for simplicity, we use different objectives to finetune it. We test GPT-2 in 3 sizes:
small (24M), medium (355M), and large (774M). See more implementation details in Appendix G.
Real data. We use English text data from 3 domains: (1) WikiText (Merity et al., 2017): text from Wikipedia; (2) WebText (Radford et al., 2019): text from the Web. It was used for pretraining GPT-2; and (3) WritingPrompts (Fan et al., 2018): text from the writing prompts forum of Reddit. We sample from each of these 3 datasets to form our training, development, and test sets. By default, our training/development/test set contains 50K/5K/5K examples. Please find more details about these datasets in Appendix G.
Metrics. (1) Perplexity (ppl) is defined as e− 1 N∗T ∑ N ∑ T logeQθ(xt|x<t), where N is the number of examples and T is the sequence length. Perplexity is not necessarily correlated with human perceived quality (Zhang et al., 2021). (2) Diversity (div): following Meister et al. (2022), we define n-gram diversity as the average fraction of unique vs. total n-grams for n ∈ {1, 2, 3, 4} in each piece of text. (3) Mauve (Pillutla et al., 2021) compares model-generated text against human text via a KL divergence curve and is the stateof-the-art metric for open-ended text generation. We use Mauve as our primary metric. (4) Coher-
ence (coh) (Su et al., 2022) computes the cosine similarity between the embedding of prompt and the embedding of continuation, and embeddings are from SimCSE (Gao et al., 2021). All metrics are the closer to human scores the better.
Objectives. Since we have no access to P , we can only implement two out of the six objectives we test in the synthetic setting: (1) MLE, which is equal to forward CE or forward KL; (2) MIXCE, the approximated mixture of cross-entropies.
Decoding. We use unbiased sampling (see footnote 1) as our primary decoding method as it allows us to explore the learned distribution in an unbiased way (Eikema and Aziz, 2020). Additionally, we test top-p sampling (Holtzman et al., 2020) to check if MIXCE is complementary to advanced decoding methods, and we carefully tune p on the development set. For each text, we take the first 50 tokens (by GPT-2 tokenizer) as the prompt and set the max generation length as 512.
Model selection. We finetune the model for 5 epochs on the training set and save the best checkpoint with the lowest dev loss. We select the best mixing ratio η and the best p based on the Mauve score on the dev set. The search space of η is [0.99, 0.9, 0.7, 0.5, 0.3, 0.1, 0.01, 0.0] and that of p is [0.85, 0.87, 0.89, 0.91, 0.93, 0.95, 0.97, 0.99]. Selected best ηs are reported in Table 12 in the Appendix. Best ps are reported in Table 3. Metric scores are reported on the test set and are 3-run averages because sampling is stochastic.
Results. Table 2 shows unbiased sampling results of models in different sizes and finetuned with different objectives on three datasets. As you can see, MIXCE-finetuned models usually get worse perplexity but consistently better diversity, mauve, and coherence, compared to MLE-finetuned models. Table 3 shows top-p sampling results from the same models as Table 2. Since perplexity will not change as the decoding method changes, we instead report the selected best p in this table. It can be seen that after carefully applying top-p sampling, MIXCE-finetuned models work on par with MLE-finetuned models for diversity, mauve, and coherence. Nonetheless, the best p for MIXCE models is always 0.99, while MLE models have smaller and more diverse ps. This indicates that MIXCE leads to a less noisy model distribution.
Human evaluation. Besides automatic metrics, we also conduct a human evaluation. Following Krishna et al. (2022), we conduct blind A/B testing. We randomly sample 105 examples from each dataset. For each example, we ask humans to read two generations from MLE and MIXCE-finetuned GPT-2 large models, respectively, and the order of showing these two generations is random. We use unbiased sampling to get the generations. Then, we ask them to judge which one is better (or they are the same) and justify their preference, based on fluency, coherence, informativeness, and whether it is sensical. We conduct this evaluation on Amazon Mechanical Turk and collect 3 responses for each example. Please refer to Appendix F for more details and examples. The final results are shown in Table 4. As you can observe, MIXCE-finetuned models significantly outperform MLE-finetuned models on both WikiText and WebText domains, while the two methods perform similarly on WritingPrompts. It is also worth noting that, compared to the results shown in Table 2, none of the 4 automatic metrics share the same trend with human evaluation.

heading: 4.3 Robustness & Analysis
text: Varying training data sizes. We test 3 other training data sizes: 10K, 25K, and 100K using GPT-2 small. Table 5 in the Appendix contains the results, and it shares the same story trend as Table 2: MIXCE-finetuned models get worse perplexity but in general work better than MLE-finetuned models for diversity, mauve, and coherence.
Varying η and max generation length. To examine how the mixing ratio η and the max generation length affect the performance, we show the mauve score curves on the dev set in Figure 4. The x-axis is the mixing ratio η from 0 to 1 (MIXCE=MLE when η = 1), and the y-axis is the mauve score with different max generation lengths (128, 320,
6The significance test is conducted following the bootstrap test setup (Efron and Tibshirani, 1994).
and 512). First, reasonable performances are usually observed when η ≥ 0.1, and only training the models with approximated reverse CE (i.e., η = 0) leads to degeneration. Second, the advantage of MIXCE is more prominent when the max generation length is longer.
Controlled Mauve. The max generation length is not the actual text length because when sampling from the model, EOS can be generated at any step. We find that the actual text length can affect the mauve computation. Even if we truncate all texts to the same length, the incompleteness caused by truncation can be another confounding factor. Both text length and text completeness are irrelevant to text quality but can be used by mauve to distinguish model generations from human texts. Therefore, to eliminate the influence of these confounding factors, we propose a controlled mauve (or c-mauve) computation approach. Concretely, for human texts and model generations, we randomly sample 10K L-length text fragments from each of these two sets. L is the number of tokens. Then, we compute the mauve between these two sets of fragments. Table 8 shows the results. As you can see, c-mauve scores are in general very high (≥ 0.90), which may indicate that, after controlling the confounding factors, the ability of mauve to distinguish model text from human text has been weakened. MIXCE still gets better performance than MLE in most cases. Besides, we also compute controlled coherence in the same fashion, and MIXCE retains its advantage. Please refer to Appendix D.4 for more details about controlled Mauve and Coherence.

heading: 5 Conclusion
text: We propose a novel training objective, MIXCE, for autoregressive language modeling. MIXCE combines forward and reverse cross-entropies, which can be viewed as combining two complementary driving forces for better fitting the model distribution to the data distribution. We demonstrate the superiority of MIXCE over MLE in both synthetic and real settings via both automatic and human evaluations. In the future, MIXCE can be potentially used for pretraining language models.

heading: Acknowledgments
text: We thank anonymous reviewers for their valuable comments. We thank Xiang Zhou for the helpful discussions. This work was supported by a Bloomberg Data Science Ph.D. Fellowship.
Limitations
One apparent disadvantage of MIXCE is the mixing ratio η. As shown in Table 12 and Figure 4, the best η changes as the experimental setting changes. It may be because we use mauve as the model selection criteria or because different datasets have different noise levels. In general, we do not have a good answer to which η should be used. The ideal solution is to select η based on the performance of the development set like what we did. However, in pretraining settings, it is too expensive to search over multiple ηs. Therefore, how to find a universal η or how to determine η automatically is an important problem to resolve before MIXCE can be reliably used for pretraining.
As we mentioned in § 1, language degeneration of open-ended generation shows two distinct patterns: the non-sensical text from unbiased sampling and the repetition loops from greedy search. Though MIXCE helps improve the performance of sampling, we still see repetition loops when using greedy search.
Ethical Considerations
As the OpenAI team pointed out, GPT-2 does not distinguish fact from fiction, so it can not support use cases that require the generated text to be true. Additionally, GPT-2 reflect the biases inherent to the systems they were trained on, so it can not be deployed into systems that interact with humans unless the deployers first carry out a study of biases relevant to the intended use case. Though our MIXCE-finetuned GPT-2 gets improved performance with respect to the metrics we used, the above statement still holds. At this point, we are not sure whether MIXCE can help improve factuality or lead to less biased generations, but we are sure that the generations still have non-factual content and biases.

heading: A Connection to Pang and He (2021)
text: In Section 3.2, we introduce an approximation of the reverse cross-entropy (CE) objective. Similarly,
Pang and He (2021) also propose to approximate reverse CE, and the resulting GOLD algorithm is similar to our Equation 9. Here, we would like to clarify the difference and connection.
The following equation is the start policy gradient equation used by Pang and He (2021).
Eτ∼πθ [ ∑
t
∇θ log πθ(at|st)Q̂(st, at)]
They used different notations from ours. πθ is the same as our Qθ, i.e., πθ(at|st) is the same as our Qθ(xt|x<t). Q̂ is the accumulated future reward from timestamp t, ∑T t′=t γ
t′−trt′ , γ is the decay factor and rt′ is the reward for each step. We will discuss Q̂ in detail later.
Then, they apply importance sampling to sample from a different behavioral policy πb. Since they also use examples from the training set, their πb is the same as our human (or data) distribution P .
Eτ∼πb [ ∑
t
wt∇θ log πθ(at|st)Q̂(st, at)]
wt is the importance weight. They use a per-action approximation: wt ≈ πθ(at|st)πb(at|st) , which is similar to how we get Equation 9 from Equation 8.
Since πb is unknown, they assume a uniform distribution: πb ≈ 1/N (N is the number of training examples). Hence, their final approximated gradient is:
Eτ∼πb [ ∑
t
πθ(at|st)∇θ log πθ(at|st)Q̂(st, at)]
They define rt′ and Q̂ in three ways. The first is called δ-reward, i.e., Q̂ = 1. In this case, their final gradient is exactly the same as our Equation 9. However, as you can see, we take a different path of derivation. Instead of using this δ-reward, our Q̂ is the sequence-level reward P (x). The reward P (x) nicely helps us to switch from the expectation of Qθ to the expectation of P (from Equation 5 to Equation 7). Therefore, without assuming a uniform distribution of πb, our πb is just P .
When using the other two rewards, they also need to know P . To address this, they use an MLEpretrained model as a proxy of P .
Overall, we introduce a different derivation approach for approximating reverse CE. Moreover, as we mentioned in § 2.3, Pang and He (2021) focused on improving controlled generation tasks where the focus is on the quality of the text, while we focus
on open-ended generations where quality and diversity are both important. Therefore, we mix reverse CE with forward CE to form our MIXCE learning objective.
B Intuition behind the Self-reinforced Objective
To further illustrate why this self-reinforced objective (Equation (8) or (9)) makes sense and their shortcomings, we conduct an analysis using GPT-2 large (Radford et al., 2019). We first sample 5000 pieces of text from WikiText, WebText, and WritingPrompts, respectively, and we call them human texts. Then, using the first 50 tokens of each human text as a prompt, we get 5000 sampling and greedy search generations from pretrained GPT-2 large (max generation length = 512). Next, we use the same model to score human texts and model generations and get the sequence-level and tokenlevel negative log-likelihoods. Figure 2 shows the histograms of these negative log-likelihoods.
In Figure 2, we take the human text histogram (in blue) as a proxy of human distribution and the sampling text histogram (in red) as a proxy of model distribution. As you can see, the support of model distribution usually contains the support of human distribution. It supports our previous claim that MLE-trained models tend to over-generalize. Meanwhile, at both the sequence and the token levels, the model on average assigns a higher probability to human text than to text sampled from the model. Therefore, when we promote highprobability sequences or tokens, it is equivalent to pushing the model distribution toward the human distribution. However, we need to avoid overly pushing it to the extremely high-probability region where greedy search outputs locate (in yellow) because they are known to be poor-quality and repeti-
byryuer
December 2022
1 Introduction
1
tmp
byryuer
December 2022
1
tive. Also, as shown in the figure, when promoting high-probability sequences, even if we overdo it, it will still be within the support of human distribution. In contrast, when promoting high-probability tokens, it can go outside the support of the human distribution, which is the drawback of Equation (9) compared to Equation (8).
Lastly, if we train the model only with the selfreinforced objective till convergence, it is inevitable to end up with a model that can only output greedy search generations. Hence, we need to combine it with the forward cross-entropy.

heading: C Loss Magnitude
text: As shown in Figure 1, we use reverse cross-entropy (CE) to provide a driving force for narrowing the model distribution down when it is broader than the data distribution. And forward CE is to broaden the model distribution up. However, it does not mean forward CE does not have the opposite drive force because forward CE is minimized if and only if Qθ(x) = P (x). However, as shown in Figure 3, the loss magnitude is greatly smaller than the loss magnitude we get from reverse CE.

heading: D Additional Results
text: D.1 Additional synthethic experiments
Table 6 shows the results of additional synthetic experiments besides Table 1 in the main paper. Here, the goal transition matrix M is randomly initialized with 10% and 90% zero probabilities.
As the magnitudes of both avg. js and avg. 0s are fairly small, we examine the 95% confidence intervals under one synthetic experimental setting – initializing the transition matrix M by the bigram occurrence in the WebText data and setting vocabulary size as 1000. Table 7 contains the results. We can see that 95% confidence intervals are small enough to maintain the trend of the results.
D.2 Varying training data sizes Table 5 shows the results of using different training data sizes in the real-data setting.
D.3 Varying η and max generation length Figure 4 illustrates the curves of mauve scores on the development sets.
D.4 Controlled Mauve and Coherence We find that the actual length of the text is a confounding factor of mauve computation. For example, when we compute mauve between a set of texts and the same set with an extra new line token after each text (or the same set with the last k tokens being truncated), the score will be lower
than 0.01. Though you may think truncating all texts to the same length can resolve this problem, we find that the incompleteness caused by truncation can also be a confounding factor. For instance, keeping human texts intact, we truncate texts generated by two systems by their shorter lengths (i.e., for each example, we truncate text1 and text2 by min_length(text1, text2)). Then, the system whose texts get truncated less will get a greatly larger mauve score than the other system. Therefore, to eliminate the influence of these two confounding factors, we propose a controlled mauve computation approach. Concretely, for the set of human texts Th and the set of model-generated texts Tm, we randomly sample 10K L-length text fragments from each of these two sets. L is the number of tokens in each text fragment. After that, we compute the mauve between these two sets of 10K text fragments. We denote this controlled mauve as
c-mauveL.
Fh,L = {f ih,L}10Ki=1 , f ih,L ∼ Th
Fm,L = {f im,L}10Ki=1 , f im,L ∼ Tm c-mauveL = mauve(Fh,L,Fm,L)
To sample each fragment, we first randomly sample a text ti from the set, and then randomly select a start token s (as long as there are more than L tokens from s to the end of ti), then the fragment is ti[s : s+L]. Finally, Table 8 shows the results. We set L = 100, 200, and 300, except that we could not get 10K 200-token fragments from WikiText because its texts are shorter.
The Coherence score (Su et al., 2022) computes the cosine similarity between the prompt and the continuation. We suspect that the length of the continuation may affect the score. Therefore, following the same idea of controlled mauve, we also sample 10K fragments of the same length from the set of texts for evaluation and compute coherence on the fragments. And for each fragment, we take the first 50 tokens as the prompt and the rest as the continuation. Table 9 shows the results. As you can observe, under this controlled setting, MIXCE-finetuned models generally achieve better coherence over MLE-finetuned models.
D.5 Text length of model generations
Though by default we set the max generation length as 512, the actual text length can vary as the EOS token can be sampled at any time step. Therefore, we list the average text length of the human text and GPT2-large generations in Table 10. We observe that model generations are always shorter than human text. Compared to MLE, our MIXCEfinetuend model produces shorter text on WikiText while producing longer text on the other two datasets. We suspect that the shorter length of MIXCE on WikiText is due to the small mixing ratio (0.1) chosen based on mauve (see Table 12). However, we do not think shorter text length leaves
to better mauve, as shown by the other two datasets and discussed in D.4.

heading: E Best ηs
text: Table 11 has the best ηs for synthetic experiments. Table 12 contains the best ηs selected for GPT-2 experiments.

heading: F Human Evaluation Details
text: We conduct A/B testing (or pairwise comparison) to compare generations from two models. As shown in Figure 5, in each job, we give the evaluator two text paragraphs (in random order) that share the same beginning part (the prompt) but have different continuations. Then, they need to choose which one they think is better (or nondistinguishable). To avoid random selections, they are also asked to provide a justification for their
choice. We find this justification not only gives us additional explanations of their choices but also helps us easily identify bad workers, because bad workers tend to use one single justification or several repeated justifications.
We instruct them by defining a good text paragraph as being:
• Fluent: Should have no obviously ungrammatical sentences, missing components, etc. that make the text difficult to read.
• Coherent: Should stay on topic with the prompt and build from sentence to sentence to a coherent body of information.
• Informative: Should have diverse and interesting content.
• Sensical: Should generally make sense.
Since short text has little information and long text is difficult to read, we only use paragraphs with 5 to 8 sentences for evaluation. If a paragraph has more than 8 sentences, we truncate it to 8 sentences. And we remove paragraphs with less than 400 or more than 2000 characters. Besides, to eliminate the influence of length difference, we do not select examples whose length difference between two
paragraphs is more than 1 sentence or more than 200 characters.
We conduct this evaluation on Amazon Mechanical Turk. We only allow workers, who are located in the US, have a Masters Qualification,7 have an approval rate larger than 97%, and have more than 10000 HITs approved, to do our tasks. In addition, we first ran a testing batch, then manually checked the results, and selected 44 qualified workers to continue doing the rest of our tasks.
For each of the 3 datasets, we sampled 105 examples and collected 3 responses per example. In total, we received 945 human evaluations. We pay workers $1 per response, and it takes around 5 minutes to finish one response, i.e., the hourly rate is around $12.
7https://www.mturk.com/worker/help
Table 13 shows that inter-annotator agreements.
Figure 6-11 are 6 randomly sampled examples from human evaluation results, 2 examples per dataset.

heading: G Reproducibility
text: In our GPT-2 experiments, we use English text data from 3 domains: (1) WikiText (Merity et al., 2017): text from Wikipedia, and we use wikitext-103-rawv1 from Hugging Face.8 Its license is Creative Commons Attribution-ShareAlike License (CC BYSA 4.0). (2) WebText (Radford et al., 2019): text from the Web. It was used for pretraining GPT-2. The full WebText is not available but they released a subset on Github9. The GitHub repository con-
8https://huggingface.co/datasets/ wikitext
9https://github.com/openai/ gpt-2-output-dataset
tains an MIT license, and they did not specify the license of the data. But they indicated in the readme: “We look forward to the research produced using this data!” (3) WritingPrompts (Fan et al., 2018)10: text from the writing prompts forum of Reddit. Its GitHub repository also contains an MIT license without specification of the data license. However, WritingPrompts has been used by many other research works, e.g., Pillutla et al. (2021). We use their official dev and test sets as much as possible. If they have fewer than 5K examples, we sample from their official training set to make up the rest.
All of our experiments were conducted on NVIDIA Tesla V100 32G GPUs. We use a single GPU to run each experiment and change the batch size to fit models of different sizes. When fine-tuning GPT-2 small using a single GPU with MLE or MIXCE, it took less than 1 hour to finish 5 epochs on 50K WikiText training data and took less than 2 hours to finish 5 epochs on 50K WebText or WringPrompts training data.
We implemented our GPT-2 based models based on the GPT-2 modeling code from Hugging Face Transformers11. For training and evaluation, we modified the example script of causal language model training12. We used the default optimizer, learning rate, scheduler, etc. in that script. But we set the maximum training epochs as 5 and changed the batch size and gradient accumulation steps based on the model size to fit it in one 32Gmemory GPU.
10https://github.com/facebookresearch/ fairseq/tree/main/examples/stories
11https://github.com/huggingface/ transformers/blob/main/src/transformers/ models/gpt2/modeling_gpt2.py
12https://github.com/huggingface/ transformers/blob/main/examples/pytorch/ language-modeling/run_clm_no_trainer.py
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
The limitation section on page 9
3 A2. Did you discuss any potential risks of your work? The ethical consideration section on page 9
3 A3. Do the abstract and introduction summarize the paper’s main claims? abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4.2, Appendix F, and Appendix G
3 B1. Did you cite the creators of artifacts you used? Section 4.2, Appendix F, and Appendix G
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Appendix G
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix G
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We used public datasets and our data collection does not introduce identifications or offensive content.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 4.2, Appendix F, and Appendix G
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.2, Appendix F, and Appendix G
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4 and Appendix G
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix G
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4 and Appendix D
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix G
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 4.2 and Appendix F
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Section 4.2 and Appendix F
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 4.2 and Appendix F
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Appendix F
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Appendix F, by a Bloomberg legal team
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Appendix F

heading: Future Work
text: Q̂ is the accumulated future reward from timestamp t, ∑T t′=t γ
t′−trt′ , γ is the decay factor and rt′ is the reward for each step. We will discuss Q̂ in detail later. Then, they apply importance sampling to sample from a different behavioral policy πb. Since they also use examples from the training set, their πb is the same as our human (or data) distribution P. Eτ∼πb [ ∑
t
wt∇θ log πθ(at|st)Q̂(st, at)]
wt is the importance weight. They use a per-action approximation: wt ≈ πθ(at|st)πb(at|st) , which is similar to how we get Equation 9 from Equation 8. Since πb is unknown, they assume a uniform distribution: πb ≈ 1/N (N is the number of training examples). Hence, their final approximated gradient is:
Eτ∼πb [ ∑
t
πθ(at|st)∇θ log πθ(at|st)Q̂(st, at)]
They define rt′ and Q̂ in three ways. The first is called δ-reward, i. e. , Q̂ = 1. In this case, their final gradient is exactly the same as our Equation 9. However, as you can see, we take a different path of derivation. Instead of using this δ-reward, our Q̂ is the sequence-level reward P (x). The reward P (x) nicely helps us to switch from the expectation of Qθ to the expectation of P (from Equation 5 to Equation 7). Therefore, without assuming a uniform distribution of πb, our πb is just P. When using the other two rewards, they also need to know P. To address this, they use an MLEpretrained model as a proxy of P. Overall, we introduce a different derivation approach for approximating reverse CE. Moreover, as we mentioned in § 2. 3, Pang and He (2021) focused on improving controlled generation tasks where the focus is on the quality of the text, while we focus
on open-ended generations where quality and diversity are both important. Therefore, we mix reverse CE with forward CE to form our MIXCE learning objective. B Intuition behind the Self-reinforced Objective
To further illustrate why this self-reinforced objective (Equation (8) or (9)) makes sense and their shortcomings, we conduct an analysis using GPT-2 large (Radford et al. , 2019). We first sample 5000 pieces of text from WikiText, WebText, and WritingPrompts, respectively, and we call them human texts. Then, using the first 50 tokens of each human text as a prompt, we get 5000 sampling and greedy search generations from pretrained GPT-2 large (max generation length = 512). Next, we use the same model to score human texts and model generations and get the sequence-level and tokenlevel negative log-likelihoods. Figure 2 shows the histograms of these negative log-likelihoods. In Figure 2, we take the human text histogram (in blue) as a proxy of human distribution and the sampling text histogram (in red) as a proxy of model distribution. As you can see, the support of model distribution usually contains the support of human distribution. It supports our previous claim that MLE-trained models tend to over-generalize. Meanwhile, at both the sequence and the token levels, the model on average assigns a higher probability to human text than to text sampled from the model. Therefore, when we promote highprobability sequences or tokens, it is equivalent to pushing the model distribution toward the human distribution. However, we need to avoid overly pushing it to the extremely high-probability region where greedy search outputs locate (in yellow) because they are known to be poor-quality and repeti-
byryuer
December 2022
1 Introduction
1
tmp
byryuer
December 2022
1
tive. Also, as shown in the figure, when promoting high-probability sequences, even if we overdo it, it will still be within the support of human distribution. In contrast, when promoting high-probability tokens, it can go outside the support of the human distribution, which is the drawback of Equation (9) compared to Equation (8). Lastly, if we train the model only with the selfreinforced objective till convergence, it is inevitable to end up with a model that can only output greedy search generations. Hence, we need to combine it with the forward cross-entropy