Title: QUANTIFYING EXPOSURE BIAS FOR NEURAL LANGUAGE GENERATION Anonymous authors Paper under double-blind review

Abstract: The exposure bias problem refers to the training-testing discrepancy caused by teacher forcing in maximum likelihood estimation (MLE) training, for autoregressive neural network language models (LM). It has been regarded as a central problem for natural language generation (NLG) model training. Although a lot of algorithms have been proposed to avoid teacher forcing and therefore to alleviate exposure bias, there is little work showing how serious the exposure bias problem actually is. In this work, we first identify and analyze the self-recovery ability of MLE-trained LM, which casts doubt on the seriousness of exposure bias. We then develop a precise, quantifiable definition for exposure bias. Based on that definition, we design experiments to measure the seriousness of exposure bias. Surprisingly, we find that removing the training-testing discrepancy only brings very little performance gain, in both real and synthetic settings. With these results, we conclude that on the contrary to popular belief, the exposure bias problem is only a minor problem for MLE-based LM training.

1. INTRODUCTION:

Language model (LM) is a central module for natural language generation (NLG) tasks (Young et al., 2017) such as machine translation (Wu et al., 2017) , dialogue response generation (Li et al., 2017) , image captioning (Lin et al., 2014) , etc. For decades, maximum likelihood estimation (MLE) has been the the most widely used objective for LM training. However, there is a popular belief in the natural language processing (NLP) community that standard MLE training will cause "exposure bias" and lead to a performance degradation during the test-time language generation.

The exposure bias problem (Bengio et al., 2015; Ranzato et al., 2016) refers to the following discrepancy between MLE training and test-time generation for language models: During training, the language model predicts the next word conditioned on history words sampled from the groundtruth data distribution. And during generation, the model generates words conditioned on history sequences generated by the model itself. However, due to the exposure to real data during training, the language model is biased to only perform well on the ground-truth history distribution. As a result, during generation the errors will accumulate along the generated sequence, and the distribution generated by the model will be distorted. The forced exposure to ground-truth data during training is also referred to as "teacher forcing".

Given its defintion, the exposure bias problem could rise in the general cases when the model needs to make a sequence of decisions or generations (e.g. music/pixel/speech generation (Lamb et al., 2016) ). In this work, we focus on the task of language generation, because the exposure bias problem is originally proposed in this field (Bengio et al., 2015) , and has since attracted huge research attention.

In order to avoid teacher forcing, many training algorithms (Bengio et al., 2015; Lamb et al., 2016; Ranzato et al., 2016; Yu et al., 2016; Zhu et al., 2018; Lu et al., 2018; Lin et al., 2017; Guo et al., 2017; Rajeswar et al., 2017; Wiseman & Rush, 2016; Nie et al., 2019; Shi et al., 2018) have been proposed as alternatives to MLE training. Most of these works utilize techniques from generative adversarial network (GAN) (Goodfellow et al., 2014) or reinforcement learning (RL) (Sutton & Barto, 1998) . In this paper, we refer to these algorithms as non-MLE methods or text GANs.

Despite the huge research efforts devoted to alleviate exposure bias, surprisingly, its existence or significance is much less studied. In particular, to the best of our knowledge, no existing work Model Samples as Hisotry → Model Samples When asked about how she thinks about the games, Flocke → dislikes most of those about it, citing instances of paranoia in her heart and trembling temper, which infuriated him. Approximately 500 Finns became sick since early October when sleeping → in their sleeping bags. On 3 October, the "Red Guard", which had been organized two months previously by Marius Kuusinen ... The entire key results of the arc be obtained through → unifying methods to construct the prologue, three pieces could be combined instead of need to provide a final chapter. Data Samples as History → Model Samples Most of what is known of Du Fu <unk> s → life is clear and graphic descriptions, memoirs, commentaries on storyboards, and descriptions of Canadian settlers. More than 60 biographies and ... In the early 730s, he travelled in the Jiangsu province → of Asia after Ashras ibn Abdallah al-Sulami visited Quanzhou in Bukhara, the capital of Turkmenistan and a native of the ... Since the Song dynasty, critics have called Du Fu the → "master poet", a product of his use of Du Fu scenes to establish the empress's nature and to emphasize his ... Shuffled Data Samples as History → Model Samples is Du of <unk> s known Fu of life what → Most claimed was his tragic adaptation of John Ching's The Janus of Hades, translated by disgraced performer just months before ... in the he travelled, the early Jiangsu In 730s, he → attended a mission on the peninsula. He soon moved to Monkwearmouth, on the northern shores of Baffin Bay in The ... Since, the called have Song Fu critics dynasty Du Fu, → who Zhang historians have included, have not rivaled HABS's Web site held for 253 years. In 2015, HABS-based producers Oronoco ... Random Sequences as History → Model Samples ...RANDOM... surface leader Game → after a failed attempt to test her effectively in three fleets falling to I-30. This went unnoticed by most ichthyologists; none understood either strict rules ... ...RANDOM... faster elephant emperor → decorations with Rocky Mountain state exploit by linking all black geese to 1970s planning regulations that prohibit slaughter of snake species. ...RANDOM... hitting remained prominently → from the system as she witnessed no mention of criteria in the text. Douglas Turner noted then that Gottesfeld may have assumed ... Table 1 : Samples of a MLE-trained STOA transformer LM when fed with different types of length-10 history prefix. To save space, we omitted the first 7 words of the random history. attempts to directly show the seriousness of exposure bias in an empirical or theoretical way. This work is motivated by the belief that a good solution should be built upon a testable and quantifiable problem definition. In this rest of this paper, we first identify the "self-recovery" ability of popular LM models, which casts doubt on the original claim of exposure bias. We then develop a precise and quantifiable definition of exposure bias, and validate its seriousness in controlled experiments.

2. MOTIVATION: THE SELF-RECOVERY ABILITY:

To study the seriousness of exposure bias in standard MLE LM training, we first stress that the following methodology, although tempting, is wrong: If we can rigorously show that the non-MLE methods proposed to avoid teacher forcing do indeed bring solid generation performance gain, then we can conclude exposure bias is a meaningful problem for the original MLE training. The reason is that we typically do not know the exact underlying reason for the performance gain. For example, despite the huge success of the batch normalization technique in deep learning, whether "internal covariate shift" (which is the motivation of batch norm) exists in deep neural network training remains a question (Santurkar et al., 2018) . Therefore, in this work we seek a direct way to validate the seriousness of exposure bias.

We focus on the following informal claim that immediately follows from the original definition of exposure bias: During generation, if we set the history distribution to be the ground-truth data distribution instead of the model's own distribution (now that there is no discrepancy between training and testing), then the model's language generation quality should be much better (we will formalize this notion in Section 4 and 5).

We start with the following qualitative analysis. We feed a MLE-trained transformer LM on wiki-103 data-set (Baevski & Auli, 2018) with four kinds of prefixes: model's own samples, data samples, shuffled (word-level) data samples or samples from a uniform random distribution. Then we let the model complete the sentence given these prefixes as history. We list some samples in Table 1 and more in Appendix A (this experiment is also repeated for a LSTM LM).

Assuming the seriousness of exposure bias, we expect the quality of generated sentence-completion samples with real-data prefixes to be significantly better than the ones from prefixes of model samples. However, by manual inspection, we do not observe noticeable differences in sample quality. More surprisingly, the model is still able to generate relevant and fairly high-quality samples from shuffled prefixes. Even in the extreme case where random sequences are fed, the model is able to generate reasonable sentences. Due to the recent increasing interest of solving exposure bias in the field of neural machine translation (NMT) (Zhang et al., 2019) , we repeat the above experiment in a standard NMT setting in Appendix A, and get very similar observations. These experiments clearly show that the MLE-trained auto-regressive LMs have the self-recovery ability, i.e. the model is able to recover from artificially distorted history input, and generate reasonably high-quality samples. This phenomenon is clearly in contradiction with the popular claim of exposure bias, that the error induced by the mismatch between history and data distribution should accumulate during the generation process.

Motivated by these experiments, in the following sections, we turn to more rigorous methods to quantify the significance of exposure bias. Note that our quantification approaches will be independent of the training procedure and only require inference from the trained model.

3. NOTATIONS:

The task of auto-regressive language modelling is to learn the probability distribution of the (l + 1) th word W l+1 in a sentence conditioned on the word history W 1:l := (W 1 , . . . , W l ). Here, we use the uppercase W i ∈ V to denote a discrete random variable distributed across the vocabulary V . The lower-case w is used to denote some particular word in V . Given a training data-set D consisting of sentences of length L, the standard MLE training minimizes the negative log-likelihood below:

EQUATION

Note that in this work we assume all sentences are of length L for simplicity.

We denote the generation distribution of the trained LM as P M , and the ground-truth data distribution as P D . Readers can assume P M refers to the generation distribution of a LSTM LM (Hochreiter & Schmidhuber, 1997; Sundermeyer et al., 2012) or a transformer LM (Baevski & Auli, 2018; Dai et al., 2019) trained with MLE objective, which is the major subject of this study. We will mainly present results on LSTM based models to facilitate comparison with text-GAN works (listed in Section 1), which are mostly implemented on LSTM models. We will also provide results with the transformer model, with very similar observations or measurements.

Our quantification mainly relies on the measurements of the distance from the model's generation distribution to the data distribution. Hence we define the following notations to simplify expressions. Let P denote the set of probability distributions on the vocabulary V . Let d denote a distance measure between distributions (e.g. total variation distance), d : P × P → R ≥0 .

4. A (WRONG) QUANTIFICATION USING MARGINAL DISTRIBUTION:

In this section, we propose an intuitive and seemingly correct quantification approach using marginal distributions. The approach can be applied to real-world text data experiments, but it has some lethal weak point. The discussion will lead us to our final precise definition of exposure bias in Section 5.

4.1. METHOD:

Assuming a given history length l, we consider the marginal distribution of W l+1 from the following three random process:

• Draw word sequences of length L from the data distribution P D . Denote the marginal distribution of the random variable at position l + 1 (W l+1 ) as P l+1 D|D , where

EQUATION

• Draw word sequences of length L from the model distribution P M . Denote the marginal distribution of the random variable at position l + 1 as P l+1 M |M , where

P l+1 M |M (w) = E W 1:l ∼P M [P M (w | W 1:l )].

(3)

• First draw W 1..l from P D , then draw W l+1 from P M (•|W 1..l ). Denote the marginal distribution of the random variable at position l + 1 as P l+1 M |D , where

EQUATION

By the definition of exposure bias, P l+1 M |M suffers from the training-testing discrepancy, while P l+1 M |D should be closer to the true distribution P l+1 D|D . To measure this discrepancy, define the marginal generation deviation (MGD) at history length l of history distribution P H with metric d as

EQUATION

where P H ∈ {P M , P D } denotes the history distribution. MGD measures the deviation of the marginal distribution of W l+1 from ground-truth data distribution.

Finally, we define the rate of exposure bias (EB-M) at history length l of model P M as the ratio (discrepancy) between the MGD measurements when two different history distributions are fed:

EQUATION

For MLE-trained models, EB-M1 is expected to be larger than 1, and larger EB-M indicates a more serious exposure bias problem for the trained model. For the metric d, we consider two popular probability metrics: total variation distance (denoted as d T V ), and Jensen-Shannon divergence (denoted as d JS ).

4.2. EXPERIMENTS AND DISCUSSION:

In this section, we focus on answering the following question: "Does the EB-M measurement correctly reflect the significance of exposure bias?" In short, our answer is not really. The problem is that the distortion of the marginal P l+1 M |M is not only affected by the presumably existing exposure bias problem alone, but also by the mismatch between the history distribution P M from P D for W 1:l , which grows with the length of the history. Therefore, even if the measured EB-M is significantly larger than one, we can not conclude that exposure bias causes serious deterioration. We provide an example to illustrate this argument: Example 1. Suppose L = 2, and V = {A, B}. P D and P M are crafted as follows: P D (AA) = P D (BB) = 0.5, P D (AB) = P D (BA) = 0; And P M (W

1 = A) = 1, P M (W 2 = A|W 1 = A) = 1, P M (W 2 = B|W 1 = B) = 1. In Example 1, MGD(P M |M , 1, d T V ) = 0.5 and MGD(P M |D , 1, d T V ) = 0, which gives EB-M(P M , 1, d T V ) = ∞.

However, the only problem P M has is the mismatch between the history distributions (P M and P D ) for W 1 .

The next set of experiments also suggest that EB-M does not precisely reflect exposure bias. On the EMNLP-news data-set (specified in Appendix B), we compare EB-M measurements for several non-MLE training methods with the baseline MLE model. We include results for Scheduled Sampling (SS) (Bengio et al., 2015) , Cooperative Training (CoT) (Lu et al., 2018) , and Adversarial Ranking (RankGAN) (Lin et al., 2017) . We provide implementation details for non-MLE methods in Appendix C. Intuitively, these methods will cause the model to be biased to behave well with model samples as history, instead of data samples. Therefore, we expect EB-M measurement for non-MLE trained models to be smaller than MLE trained models. However, Figure 1 shows that the measurements for different training frameworks are almost the same. We believe the reason is that the EB-M measurements are only reflecting the trivial mismatch between the history distributions.

Is it possible that the original definition of exposure bias (Bengio et al., 2015; Ranzato et al., 2016) exactly refers to this mismatch between the model and data history distributions? However, note that this mismatch is inevitable for any imperfect model, and non-MLE training algorithms can not solve it. We believe a better, more precise definition is needed to discriminate exposure bias from this trivial mismatch. Motivated by this view, we propose a second approach in the section below. Following the discussion in the last section, we wish our measurement to be independent of the quality of the history distribution. In light of that, we design a quantity to measure the model's conditional generation quality. Let P H ∈ {P M , P D } denote the history distribution as in the MGD definition (5). With history length l fixed, we define the conditional generation deviation (CGD) with history distribution P H for P M using metric d as:

EQUATION

where we assume that P D (• | W 1:l )) is computable, and use it to measure the quality of the model's conditional distribution. For the choice of the distribution distance d, in addition to d T V and d JS , we introduce greedy decoding divergence (d GD ) defined as:

d GD (P, Q) = 1(arg max i P i = arg max i Q i ) (8)

where 1 is the indicator function, and P, Q ∈ P. The distance d GD2 reflects the model's accuracy during greedy decoding.

Similar to MGD, exposure bias should imply a significant gap between CGD(P M |M , l, d) and CGD(P M |D , l, d). We again define rate of exposure bias at history length l with metric d to be:

EQUATION

For our definition of EB-C, a natural question is why we only focus on the generation distribution of the very next word. The reason is we want to precisely measure how the error caused by the history part affect the generation part, by keeping them separate. If we measure the deviation of, for example, two sampled tokens, the definition will be confusing: Because the second sampled token will be affected not only by the accumulated error induced by the history (sampled from the model), but also by the first generated token as history. To get a better understanding of the intuition behind the definition of EB-C, we recommend readers to read Appendix A about our NMT experiment.

5.2. SYNTHETIC EXPERIMENTS AND DISCUSSION:

Since CGD requires inference for ground-truth data distribution P D , we first consider experiments in a synthetic setting. In text-GAN literature (Yu et al., 2016; Lin et al., 2017) , a randomly-initialized one-layer LSTM model with hidden dimension of 32 is usually used as P D in synthetic experiments (we denote this setting as M random

32:

). However, the model is small-scale and does not reflect any structure existing in real-world text. To improve upon this approach, we take the MLE baseline model trained on EMNLP-news data (described in Appendix B) as P D in this synthetic setting. We denote the data model (P D ) as M news 512 . We then train two LSTM LM (P M ) with different capacities using samples from the data model, with the standard MLE objective. One is a one-layer LSTM with hidden width of 512 (denoted as LSTM-512), the other one is with hidden width of 32 (denoted as LSTM-32).

We train P M for 100 epochs using the Adam optimizer with learning rate 0.001. In each epoch, 250k sentences (same to the size of the original EMNLP-news data) of length L = 50 are sampled from M news-512 as training data to avoid over-fitting. We show perplexity (PPL) results of the trained models in Appendix F. Finally, EB-C is calculated using 100k3 samples from P M and P D . In Figure 2 , we show EB-C measurements with different metrics d m , and the two models give similar results. It is shown that EB-C has a steady but slow increasing trend as history length increases. This is expected as a consequence of exposure bias, because P M deviates farther from P D as history length increases. However, the average value of EB-C is less than 1.03 (the largest average value is from d JS for the LSTM-512 experiment), meaning that the gap between CGD(P M |M , l, d) and CGD(P M |D , l, d) is not large. Also, note that in most NLG applications (such as machine translation or image captioning), the generated sequence typically has short length (less than 20). In that range of history length, the EB-C measurements that exposure bias only has minimal influence.

In Appendix E, we repeat the experiment for a transformer LM (Dai et al., 2019) , and get very similar EB-C measurements. These measurements imply a striking conclusion : (Informal) Even if all the bad effects from exposure bias for MLE LM training are removed, the relative performance gain is at most 3%. If the sequence length is not very long, the gain is less than 1%..

To dive deeper into the cause of the gap in CGD, we experiment with corrupted versions of P M as history distribution. We first specify a corrupt rate c ∈ [0, 1], and randomly substitute words in a history sample from P M to a "noise" word drawn uniformly from the vocabulary with probability c. Consequently, larger c will cause the history distribution to deviate farther from the groundtruth P D . In Figure 3 , we show CGD measurement versus the corrupted history P corrupt M . Large gaps are observed between CGD(P M |M corrupt ) and CGD(P M |D ). Therefore, the small gap between CGD(P M |M ) and CGD(P M |D ) in Figure 2 results from the small deviation between the history distribution P M and P D . In other word, P M has learned a "good enough" distribution that is able to keep it in the well-behaving region during sampling.

With these observations, we conclude that, in the synthetic setting considered, exposure bias does exist, but is much less serious than it is presumed to be. Although there exists mismatch between the history distribution P M and P D , the mismatch is still in the model's "comfortable zone". In other words, the LSTM LM is more robust than exposure bias claims it to be. To concretize the this argument, we provide an example LM and show that MLE training is unlikely to generate models with a large EB-C value. Example 2. Again suppose L = 2, and V = {A, B}, the ground-truth data distribution is uniform on {AA, AB, BB, BA}. P M is crafted as follows: P M (W 1 = A) = 0.9, P M (W 2 = A|W 1 = A) = 0.9, P M (W 2 = A|W 1 = B) = 0.5. Note that the model behaves bad when W 1 = A, which is of high probability during sampling. setting 5 , as we find it hard to do a fast implementation of RankGAN for the LSTM-512 setting. We find that RankGAN and CoT gives lower EB-C measurements than MLE, which is expected, as these methods avoid teacher forcing. For CoT, at short History: in august 2009 , vh1 classic aired music videos from the tv special around the beatles ( 1964 ) , Choices: 0: the 1: including 2: which 3: featuring 4: and 5:[None of the above is plausible] History: contingent units were accused of being armed with antiquated weapons , while some local peasants took claims of having been Choices: 0: conscripted 1: armed 2: involved 3: tricked 4: intimidated 5:[None of the above is plausible] hisotry length, EB-C is even less than 1. We believe the reason is that CoT trys to make the model be biased to behave better when fed with model samples. However, SS gives worse EB-C measurements comparing to MLE, which we currently do not have a good explanation. We refer readers to Huszr (2015) for a discussion about the SS objective.

In Example 2, CGD(P:

M |D , 1, d T V ) = 0.2 and CGD(P M |M , 1, d T V ) = 0.36, so EB-C(P M , 1, d T V ) = 1.8.

To the best of our knowledge, this is the first direct empirical evidence that text GAN does indeed alleviate the exposure bias problem. It also indicates that EB-C correctly reflect the significance of exposure bias. We believe the reason for why EB-C is still not less than 1 is that, text GANs still rely on MLE pre-training a lot.

5.3. EB-C MEASUREMENTS WITH REAL HUMAN AS P D:

In this section, we design experiments to efficiently estimate EB-C for a SOTA transformer LM with real human as P D , by utilizing the Amazon Mechanical Turk (AMT) platform. Given a MLE-trained LM as P M , by examining the definition of EB-C in Equation 9 and 7, it is clear the only obstacle is that we don't have access to P D (• | W 1:l ) with a given history W 1:l . So, in this section, we focus on the greedy decoding divergence (d GD ) metric (Equation 8), which only requires the turkers to give the most probable next word prediction, instead of the full distribution (which is clearly intractable).

In our preliminary trials, we find it is still very hard for a person to guess the next word, even with real data history samples. The reason is that the vocabulary is very big, and the turkers may be not familiar with the context (e.g. wikipedia). To alleviate that problem, we design the following simplification: For a given history, we let the model output its top-5 next word prediction, then we only ask the turkers to choose among the 5 choices (the turker can also express that he/she thinks none of them is likely). Finally, we examine whether the turker's choice is indeed the model's top-1 prediction. We illustrate this process in Table 2 .

We use the code of Transformer-XL (Dai et al., 2019) to train a SOTA transformer LM on the wiki-103 data-set. We favour the wiki-103 data-set because it is large-scale and has long (over 30 words) paragraphs, which is useful for the measurements of exposure bias. The model is a 16layer transformer-xl model with hidden dimension of 410. Since the estimation of CGD(P M |D , l, d) requires large amounts of unseen real data samples, we use half of the wiki-103 training data (around 900k sentences and 50m words) to train the model P M , and save the other half as samples from P D .

Other training configurations ( learning rate, batch size, etc.) are not changed6 . The resulting model P M has a test-set PPL of 27.81 (if trained on full training data, the PPL will be 24.02).

We collect data to estimate EB-C at history length 10, 20, and 30. For each length and history model (P M or P D ) pair, we collect 10k d GD samples (via next-word prediction) from turkers on the AMT platform. More details about the AMT setup are provided in Appendix D. The results are shown in Caccia et al. (2018) tunes a "temperature" parameter in the softmax output, and evaluate models over the whole quality-diversity spectrum. Semeniuta et al. (2018) proposes to use "Reverse Language Model score" or "Frechet InferSent Distance" to evaluate the model's generation performance. Tevet et al. (2018) proposes a method for approximating a distribution over tokens from a GAN, and then evaluate the model with standard LM metrics.

These works arrive at a similar conclusion: The general performance of Text GANs is not convincingly better, or even worse, than standard MLE training. Hence to some extent, they imply that exposure bias may be not a serious problem in MLE training. However, as we argued in Section 2, one can not draw direct conclusions about exposure bias with these results. For example, it is also possible that exposure bias is indeed serious for MLE training, but text GAN does not solve the problem well enough.

7. CONCLUSION AND DISCUSSION:

In this work, we first identify the self-recovery ability of MLE-trained LM, which casts doubt on the seriousness of exposure bias, which has been regarded as a central problem for MLE training by the LM community. We then explore two intuitive approaches to quantify the significance of exposure bias for LM training. The first quantification EB-M relies on the marginal generation distribution and reveals some vagueness in the original definition of exposure bias. We argue that we should focus on the model's generation performance in terms of its conditional distribution and propose a second quantification EB-C, which we regard as the precise definition for exposure bias.

We design a evaluation of EB-C at different history length with real human (turkers from AMT) as the data model, for a SOTA transformer LM. It is shown that removing the training-testing discrepancy only gives around 2% of performance gain. Our synthetic experiments also gives very similar measurements. By analyzing EB-C measurements with perturbed history samples, we hypothesise that although the mismatch between the data and model distribution for history prefix exists, it is still in the model's "comfortable zone". With these results, we claim that on the contrary to the popular belief, exposure bias is only a minor problem in MLE-based LM training.

To wrap up, we discuss the fundamental question "Is MLE training really biased?", from the perspective of objective functions. Note that the MLE objective (1) can be re-written as:

EQUATION

where D KL denotes the Kullback-Leibler divergence, and θ denotes the trainable parameters in P M . Therefore, MLE training is minizing the divergence from P M , which is exactly the model's sampling distribution, from P D . While it's true that the training is "exposed" to data samples, we can not simply deduce the objective is "biased".

We want to end our discussion with two remarks. First, the proposed quantification approaches should not be used as the only metric for NLG. For example, a position-aware uni-gram LM, which generates words independent of previous context, has no exposure bias problem and can pass our test easily. Second, the intention of this work is not to discourage researchers from exploring non-MLE training algorithms for LM. It is completely possible that an training objective different from D KL (P D ||P M ), such as JSD(P D ||P M ), can lead to better generation performance (Lu et al., 2018; Huszr, 2015) . However, though non-MLE algorithms avoid teacher forcing, these algorithms (using GAN or RL for example) are usually less stable and more difficult to tune. Given that the quantified measurement of exposure bias is insignificant, we think it should be questioned whether adopting these techniques to avoid exposure bias is a wise trade-off.

A THE AUTO-RECOVERY ABIITY IN GENERAL LM AND NMT:

In Table 6 , we provide more samples of a MLE-trained transformer LM model (discussed in Section 2) when fed with different kinds of history. And in Table 7 we repeat the experiment for a LSTM-LM trained on the EMNLP-News data.

In Table 4 we repeat the preliminary experiment in Section 2 for a standard NMT setting.We train a 6-layer transformer model with hidden dimension 1024 on the IWSLT'14 German to English data set. We feed the trained model with types of prefix during decoding which represents different level of training-decoding discrepancy. Note that the source input is kept intact.

The result is very similar (or more striking) to our language model experiment, the data prefix does not seem to help, and in the extreme case of random prefix, the model still generates fairly good translation. In Section 2 we summarize this observation as the auto-recovery ability.

To interpret the UNREL3 results, we should not directly compare the translation generated from unrelated prefix to the target translation. In fact, we cannot even compare part of it (e.g. the part after the length-3 prefix). Instead, we highlight the surprising fact that although the model is forced to begin (conditioned) with a wrong prefix, it still comes up with a reasonable translation. This is not an 

F PERPLEXITY OF THE TRAINED MODELS:

We show PPL results for model trained on EMNLP-news data-set in Model Samples as Hisotry → Model Samples it was only a pieces that had gone up to → the forest and forces the shoppers about their chronic young i mean we didn ' t know what i haven → ' t considered through , " she told bbc radio if he were the president -elect , he was → known that he would run a force in business at but these are not as tired of " the same → message that the harry actor does have been hours in first opinion the agent have taken four seconds , or → if they don ' t only know anything , were " the economy of the uk is low enough of → people of defending where americans think that " brexit , the economy grew on 1 . 6 % since the → us voted , and when it turned around 200 streets i was able to produce on my own , which → is good ; now that the theatre i ' ve " i ' ve not buying boys i addressed many → nervous times before , as a teenager made me is we think about one -third of the struggles we → actually want to see those very well that even more the story of a album -which made public -→ was still fantastic , and for the second time in " the test comes up before tuesday and when we → ' re feeling ahead again soon , " she posted a year on when he was last seen in his → home and he did not see him , his suffering brady has forced the 9 -known targets to get → all -of -12 gun migration and performing communication i asked if he himself did , i managed to → show all my charges at all , it used to Data Samples as Hisotry → Model Samples what this group does is to take down various different → players in the future and we play in paris we over 1 , 600 a day have reached greece this → gone in 2013 and it planned to allow civilians on " we ' re working through a legacy period , → and i am proud of the experience of the worker ' the first time anyone says you need help , → you don ' t have put accurate press into the out of those who came last year , 69 per → cent of women can really take the drive to avoid he has not played for tottenham ' s first team → this season then and sits down 15 -0 with so you have this man who seems to represent this → bad story , which he plays minutes -because he cnn : you made that promise , but it wasn → ' t necessarily at all the features he had in this is a part of the population that is unk → lucky to have no fault today , and it would they picked him off three times and kept him out → of the game and was in the field , the the treatment was going to cost $ 12 , 000 → as a result of the request of anyone who was but if black political power is so important , why → doesn ' t we becomes the case that either stands local media reported the group were not looking to hurt → the animals , but would never be seen to say 

:

Note that one can also directly measure d(P l+1 M |M , P l+1 M |D ), but in that way, we can not tell which distribution is better.

dGD qualifies as a pseudometric in mathematics.

We show that we can get stable measurements using 100k samples in Appendix E.

If we change to PM (W1 = A) = 0.1, then EB-C(PM , 1, dT V ) will be 0.2, meaning that the model has better conditional generation performance during sampling

The MLE model is used as the pre-trained model for the RankGAN generator. The MLE model has an oracle NLL of 8.67, and RankGAN's oracle NLL is 8.55.

https://github.com/kimiyoung/transformer-xl/blob/master/pytorch/run_ wt103_base.sh

The wikitext-103 data is available at https://blog.einstein.ai/the-wikitext-long-\ term-dependency-language-modeling-dataset/.

Figures and Tables:

Figure 1: EB-M (with metric d JS ) comparison for MLE and non-MLE training on EMNLP-News data. For each training method, we show corpus-BLEU (Yu et al., 2016) measurement using the test-set as reference set in the legend.

Figure 2: EB-C measurement for the LSTM-32 and LSTM-512 model with different metrics. Also average value of EB-C along all history length is shown in the legend.

Figure 3: CGD measurement for corrupted P M (with d T V ) for the LSTM-512 synthetic experiment.

Figure 4: (a): EB-C measurements (with d JS ) for comparing non-MLE methods in the LSTM-512 synthetic experiment. (b): EB-C measurements for comparing RankGAN and MLE for the M random 32

Figure 7: A histogram of the number of HITs done by a unique turker.

Figure 9: measurements (with metric d T V ) using different number of samples on wiki103 data.

An illustration for the next word collection process. The choices are shuffled. The first history sample is from real data, and the second history sample is from the trained model.

Table content: <table><tr><td colspan="4">History Length (l) CGD(P M |D , l, d GD ) CGD(P M |M , l, d GD ) EB-C</td></tr><tr><td>10</td><td>0.723</td><td>0.738</td><td>1.019</td></tr><tr><td>20</td><td>0.736</td><td>0.752</td><td>1.021</td></tr><tr><td>30</td><td>0.732</td><td>0.754</td><td>1.029</td></tr></table>

EB-C measurements with human as P D .

Table content: <table/>

The EB-C measurements are strikingly similar to the results in our synthetic experiments in that, removing the training-testing discrepancy only gives around 2% of relative performance gain. This result further strengthens our claim that exposure bias is only a minor problem for MLE-based LM training.

Table content: <table><tr><td>6 RELATED WORKS</td></tr><tr><td>Several recent works attempt to carefully evaluate whether the non-MLE training methods (e.g.</td></tr><tr><td>adversarial training) can give superior NLG performance than standard MLE training for RNN LM.</td></tr></table>

The MLE model for wiki-103 data-set discussed in Section 4.2 has PPL 84.58. Note that due to our special setting 8 , our PPL result is not directly comparable to state-of-art LM results on these data-sets. We only keep sentences of length longer than L, and for wiki-103, only half of training data is used.

Table content: <table><tr><td>Model</td><td>PPL</td></tr><tr><td>MLE Baseline M news-512</td><td>55.85</td></tr><tr><td>LSTM-512 (MLE, synthetic)</td><td>115.3</td></tr><tr><td>LSTM-32 (MLE, synthetic)</td><td>156.3</td></tr><tr><td colspan="2">Transformer-512 (MLE, synthetic) 103.6</td></tr><tr><td>CoT-512 (synthetic)</td><td>115.6</td></tr><tr><td>SS-512 (synthetic)</td><td>113.7</td></tr><tr><td>CoT</td><td>56.83</td></tr><tr><td>RankGAN</td><td>53.43</td></tr><tr><td>SS</td><td>56.43</td></tr></table>

PPL results for model trained on EMNLP-news data-set.

Table content: <table/>

Random Sequences as History → Model Samples ...RANDOM... big winter deserve → , but they just say it your things goes wrong ...RANDOM... playoff north realise → at its lowest level , improving their understanding in danger ...RANDOM... vital childhood registration → , not previously planned for ¡unk¿ to each and reduced ...RANDOM... treated ship find → one as an actual three points contained at a time ...RANDOM... faith five crazy → schools and could give them a " sleep " necessary ...RANDOM... domestic jason follows → a 12 -year cruise line over the christmas track ...RANDOM... ownership generous tourist → accounts for more than 1 per cent every month -...RANDOM... spending raped since → the file returns in january , joining groups of foreign ...RANDOM... netflix worker four → centre -and said facebook text ¡unk¿ to see how ...RANDOM... race labor witnessed → is great , with more to an active the ¡unk¿ ...RANDOM... treatments airlines hidden → real -time out to sell on benefits to our ...RANDOM... intention short reflects → showing the nature of flying in his space rather than ...RANDOM... conversation pace motion → them further , but as late as they ' ve ...RANDOM... export feb president → obama agreements with president obama and her being on trump ...RANDOM... entering pocket hill → and made it later in the united states and make

Table content: <table/>

Samples of a MLE-trained LSTM LM (on the EMNLP-news data-set) when fed with different kinds of history. To save space, we omitted the first 7 words of the random history.

Table content: <table/>

Peer review (iclr2020):
This paper studies the seriousness of the exposure bias problem in language modeling. Many NLP researchers believe that the fact that an auto-regressive language model (LM) is conditioned on its own prediction at test time whereas it was conditioned only on training data at training time induces exposure biases. In this work, based on the observation that conditioning on even random inputs or model*s past history will still generate valid continuations, questions whether exposure bias is a serious problem. The first experiment in this paper is based on measuring the distance between the marginal distribution of the model*s predicted word at a certain position to the ground truth marginals, where the model is either fed with ground truth prefix or its own predictions. While this metric shows around 10% difference between ground truth prefix and prediction prefix as the length of the generation goes to around 20, the authors argued that there might be multiple causes since the model*s prediction prefix might have a different distribution to the ground truth prefix, leading to different marginals. To separate out the effect, this work considers measuring the distance between conditionals instead of marginals at a certain position, where conditionals are either conditioned on ground truth prefix or language model prefix. However, since in real data we do not get access to the true conditionals (especially when prefix is generated from the model), this work did a synthetic experiment, using an LSTM language model to generate data as the true data distribution, and use other LSTM models to fit. On this synthetic task, 1) the difference in distances either conditioned on ground truth prefix or predicted prefix is within 3%, and 2) scheduled sampling does not help, but seqGAN helps a bit. Pros: 1. This paper raises a very interesting question of quantifying exposure bias and proposes a metric to quantify the phenomenon of exposure bias. 2. The experiment on the synthetic task is convincing, showing that exposure bias is not a huge issue there. Cons: 1. In terms of motivation, the observation here is that auto-regressive LMs are able to self-correct, and authors found similar results for transformers. However, as shown by Holtzmann et al 2019, when directly beam search from the LM, it will produce repetitions that would be further reinforced by the model itself. Those two observations seem contradictory. 2. In the synthetic experiment, the data generator is also of an LSTM architecture, so this setting is similar to knowledge distillation where the student and teacher networks have similar structures. It is well established that the data generated from an LSTM LM is biased (smoother and simpler) compared to the data distribution it*s trained on. It is thus not very surprising that the distribution of student*s samples is similar to the *true* data distribution (teacher*s samples), whereas this is not the case in the real world. One way to evaluate this is to repeat the first (*wrong*) experiment on this synthetic dataset, and I would expect the measured differences to be also smaller here. Another way would be using a much more powerful generator and a less powerful LM (what if we use transformer LM to generate but use RNN w/o attention to learn?) 3. What does 3% difference in TV/JS mean? It is a token level metric anyway. Would it mean a less than 3% difference in sequence-level metrics such as BLEU? Does it mean 3% wouldn*t compound over time steps? I don*t think we can draw any conclusions by looking at this number. Overall, I think this paper asks a really interesting question, but this paper doesn*t provide a convincing answer, and I am still not sure if exposure bias is a serious problem or not. I am inclined to reject this work unless I can get a convincing assessment of the exposure bias problem. References: 1. Holtzmann et al 2019. The Curious Case of Neural Text Degeneration. https://arxiv.org/abs/1904.09751