Quantifying Exposure Bias for Open-ended Language Generation
Authors: Tianxing He
Jingzhao Zhang
Zhiming Zhou
James R. Glass
Venue:  (2021)
Keywords: exposure bias
natural language generation
autoregressive

Abstract: The exposure bias problem refers to the incrementally distorted generation induced by the training-generation discrepancy, in teacher-forcing training for auto-regressive neural network language models (LM). It has been regarded as a central problem for LMs trained for open-ended language generation. Although a lot of algorithms have been proposed to avoid teacher forcing and therefore alleviate exposure bias, there is little work showing how serious the exposure bias problem actually is. In this work, we propose novel metrics to quantify the impact of exposure bias in the generation of MLE-trained LMs. Our key intuition is that if we feed ground-truth data prefixes (instead of prefixes generated by the model itself) into the model and ask it to continue the generation, the performance should become much better because the training-generation discrepancy in the prefix is removed. We conduct both automatic and human evaluation in our experiments, and our observations are two-fold: (1) We confirm that the prefix discrepancy indeed induces some level of performance loss. (2) However, the induced distortion seems to be limited, and is not incremental during the generation, which contradicts the claim of exposure bias.

Under review as a conference paper at ICLR 2021
QUANTIFYING EXPOSURE BIAS FOR OPEN-ENDED LANGUAGE GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
The exposure bias problem refers to the incrementally distorted generation in- duced by the training-genne seneringgenertiinnn discrepancy, in teacher-forcing training for auto- regressive neural network language models (LM). It has been regarded as central problem for LMs trained for open-ended language generation. Although a lot of algorithms have been proposed to avoid teacher forcing and therefore alleviate exposure bias, there is little work showing how serious the exposure bias problem actually is. In this work, we propose novel metrics to quantify the impact of exposure bias in the generation of MLE-trained LMs. Our key intuition is that if we feed ground-truth data prefixes (instead of prefixes generated by the model itself) into the model and ask it to continue the generation, the performance should become much better because the -eniinnggnnrttionn discrepancy in the prefix is removed. We conduct both automatic and human evaluation in experiments, and our observations are two-fold: indeed induces some level of performance loss. (2) However, the induced distortion seems to be limited, and is not incremental during the generation, which contradicts the claim of exposure bias.
(1) We confirm that the prefix discrepancy
INTRODUCTION
Language model (LM) is central module for natural language generation (NLG) tasks (Young et al., 2017) such as open-ended language generation (Radford al., 2018; Nadeem et al., 2020), machine translation (Wu et al., 2017), dialogue response generation (Li et al., 2017), image captioning (Lin et al., 2014), etc. For decades, maximum likelihood estimation (MLE) has been the most widely used objective for LM training. However, there is a popular belief in the natural language processing (NLP) community that standard MLE training suffers from the exposure bias problem which leads to an incremental performance degradation during test-time generation.
The claim of the exposure bias problem (Bengio et al., 2015; Ranzato et al., 2016) is originated from the following discrepancy between MLE training and test-time generation for nugo-regressive language models: During training, the model is trained to predict the next word conditioned on prefix (or history) words sampled from the ground-truth data distribution; While during generation, the model generates words conditioned on prefix sequences generated by the model itself. Hence, due to the exposure to real data during training, the language model could potentially be biased only perform well with data prefixes. Therefore, it is claimed (and widely believed among researchers) that during generation the errors should accumulate along the generated sequence, and the distribution generated by the model would be incrementally distorted. The forced exposure to ground-truth data during training also referred to as teacher forcing.
In order to avoid teacher forcing, many training algorithms (Bengio et al., 2015; Lamb et al., 2016;
Ranzato et al., 2016; Yu al., 2016; Zhu al., 2018; Lu et al., 2018; Lin al., 2017: Guo et al., 2017: Rajeswar et al., 2017; Wiseman & Rush, 2016; Nic et al., 2019; Shi et al., 2018; de Masson d' Autume et al., 2019; Rennie et al., 2016) have been proposed as alternatives to MLE training for open-ended language generation. Most of these works utilize techniques from generative adversarial networks (GANs) (Goodfellow et al., 2014) or reinforcement learning (RL) (Sutton & Barto, 1998). In this paper, we refer to these algorithms as non-MLE methods.
With the huge research efforts devoted to alleviate exposure bias, interestingly, the existence or significance of exposure bias much less studied. On the other hand, despite the criticism, MILE

---

Under review as a conference paper at ICLR 2021
(teacher forcing) has remained to be the dominant objective for LM training (Radford et al., 2018; Keskar et al., 2019). To make the situation more curious, multiple recent works show that the proposed non-MLE methods actually have inferior generation performance (Caccia et al., 2018; de Masson d' Autume et al., 2019) than the MLE baseline. These negative results lead us to question: Is exposure bias truly serious problem for MLE training?
In this work we seek a direct answer to the above question. Here we briefly summarize our contribu- tions: We conduct controlled experiments in which we remove the the prefix, and design various metrics to quantify the performance improvement of the generation as the prefix length grows. On the contrary to our expectation, our measurements consistently show that the performance gain is limited, and the incremental distortion as claimed by exposure bias is not observed (the performance gap does not become larger with longer prefixes). In the next section, we begin by introducing notations and background.
discrepancy in
2 PRELIMINARIES
The task of auto vettorregreesvve language modelling is to learn the probability distribution of the (1 + Du word (or token) Wi+1 in a sentence W conditioned on the prefix W1: := (W1.... W.). We use W; V to denote discrete random variable distributed across the vocabulary V. For simplicity, assume all sentences are of length L in the formulations. Denoting the ground-truth data distribution as PD. the standard MLE training aims to minimize the negative log-likelihood (NLL) loss below:
(1) W1:) denotes the conditional distribution of W1+1 of Po given a prefix W1.1, and A stands for the set of parameters to be trained. Note that the concept of "sentence" (W) can be naturally generalized paragraphs or even articles, depending on the target task.
LMLE =
E W~PP
- log (W
where (o
We denote the distribution of a ML.E-rained LM as PMr which is the major subject of this study. We will experiment with two popular model architectures: LSTM LM (Hochreiter & Schmidhuber, 1997; Sundermeyer et al., 2012) and transformer LM (Baevski & Auli, 2018; Dai et al., 2019). For generation, we do classical ancestral sampling without invoking sampling algorithms such as top-k sampling (Fan et al., 2018) for the following reasons: (1) The sampling algorithms are known to trade quality out of diversity (Nadeem et al., 2020; Caccia et al., 2018). So. invoking them could "hide" the exposure bias problem because the prefixes from the model will be of higher quality. (2) The sampling algorithms requires tunigg of syper-parameters, which will complicate the comparison.
In addition to popular measures in natural language generation (NL.) such as BLEU (Papineni et al., 2002) or METEOR (Denkowski & Lavie, 2014), our quantification approaches also rely on the measurements of the divergence between two distributions. Let P denote the set of probability distributions on the vocabulary V. and let faiv : P x P P R20 be a divergence function between two distributions (e.g., total variation distance). We will adopt two popular probability divergence functions: total variation distance (denoted as drv) and Saasen-Shannon divergence (denoted as dys). We provide definitions of dry and dıs in Appendix A.
Our experiments will be focused on the task of open-ended language generation, which is arguably good test bed for exposure bias because of the following reasons: (1) The generation length is long. (2) Different from typical seq2seq tasks such as machine translation, the generation space is only weakly constrained and the topics can be very diverse, which means the training-generatinn discrepancy could be large.
3 A QUALITATIVE ATTEMPT
We begin with a qualitative attempt to verify the significance of exposure bias. We design a prefix- switching experiment as follows: We feed a MI.E-trained transformer L.M on the wiki-103 dataset with four types of prefixes of the same length: (1) test-data samples, (2) model's own samples, (3) test-data samples shuffled on word-level, or (4) samples from a uniformly random distribution on V. Then we let the model continue the generation given these prefixes and compare the quality of the samples in a qualitative manner. We defer details of the model and dataset to Section
The intuition behind the prefix-switching experiment follows immediately from the original claim of exposure bias: During generation, if we set the prefix distribution to be the ground truth data distribution instead of the model's own distribution, then the discrepancy between training and
2

---

Under review as a conference paper at ICLR 2021
<LEN- 20 PROMPT>... (<LEN-100 DATA PREFIX>) in February 1942, Headlam & Generation: was captain of No. 4 Squadron which had recently been withdrawn from service in France. He was promoted to squadron leader in November that year, , and appointed
<LEN-20 PROMPT>... ( (<LEN-100 MODEL PREFIX>) ... when he was
Generation: appointed chief flying instructor at Moth City He flew both jet aircraft and Mitsubishi A6M Zero fighters in large numbers during the weeks following the war when he
(<LEN- 120 SHUFFLED DATA PREFIX>) ... (ir Marshal against West April July & Generation: ). Wing Commander WS Marais, No. 3 Wing ( (Wing Commander replaced Headlam in October 1942. The Wing scored 9 victories in the three months between ...
... I<LEN-120 RANDOM PREFIX>) ... Canterbury Oxford flagship looks person + Generation: in New Canterbury City thousands more Yap State Party vice president Datuk Bambang Ishii had died that same day of mild cancer of the left foot in
Table 1: Samples of a MLE-trained transformer LM when fed with different types of prefixes. The prompt is "= Frank Headlam Air Vice Marshal Frank Headlam, CB, CBE ( 15 July 1914 ", which is the beginning of an article in the wiki-103 test set. The generation length is fixed to 30. To save space, we omit the long prefix and only show the last few words. The examples are not cherry-picked.
generation in the prefix is removed, and hence the model's generation quality should be much better. In the extreme case of shuffled or random prefixes, due to the claim from exposure bias that the errors should accumulate, we expect the model generate also badly distorted sequences.
The samples with different types of prefixes are shown in Table 1. To make the generation from data and model prefix more comparable, we force the same prompt the beginning of the context to constrain the topic. Moreover, we intentionally use long prefixes of length 100, in the hope that the incremental distortion of generation (as claimed by exposure bias) would become observable. We include another set of examples with the same transformer LM, and examples with a LSTM LM in Appendix B, which gives similar observations.
On the contrary to our expectation, we do not observe a noticeable difference in sample quality comparing samples from model and data prefixes. More surprisingly, the model is still able to generate fairly good samples from shuffled prefixes. Even in the extreme case of random prefixes, we still observe basic language structures in the sample.
This experiment suggests that the MLE-trained ruto-regressive LMs have the self-recoven ability, i... the model is able to recover from artificially distorted history input, and generate samples with reasonable quality. This phenomenon not be explained by exposure bias which claims that the errors along the generation process should, on the contrary, accumulate.
We conclude that this qualitative attempt fails to show the significance of exposure bias, indicating its impact could be more subtle than expected. In the following sections, we turn to more rigorous quantification methods to measure the impact of exposure bias.
4 QUANTIFICATION METHODS
Following the intuition of the prefix switching experiment, we design our quantification metrics to be a simple ratio of the relative performance gain when length-/ data prefixes in fed to the model as opposed to the original model prefixes. And we compute the measurements for different prefix lengths.
For a systematic assessment of exposure bias, we decompose the claim of exposure bias into two factors: (1) The discrepancy or mismatch in the prefix distribution would indeed hurt the generation performance, in general. (2) Moreover, the distortion should be incremental along the generation. The first factor can be reflected by the average magnitude of the measurements (the values are expected to be larger than by a meaningful margin), and the second factor can be reflected by whether the measurements
increasing along the prefix length.
We now go into the detailed definitions of our metrics. We attempt to quantify the impact of exposure bias on three key aspects of open-endee ended language generation: quality, diversity, and consistency. We first introduce EB-M, which covers the quality and diversity aspects, and then EB-C, which covers the consistency aspect.
3

---

Under review as a conference paper at ICLR 2021
4.1 DEFINITION OF EB-M
In this section, we propose the EB-M metric. Since the key idea is to compare the generation quality with different types of prefixes, denoting the prefix distribution as PH E (PM, Pb} (model or data prefixes), we first formalize the following generation process:
Given prefix length and a prefix distribution Ph. we sample WIt from PH.
Conditioned on the prefix WI... we sample W1+1:++mm from Par. where Wi+j is sampled from (M Woi+i-1) with 0, and Ige is the length of generation.
Wig If
We denote the marginal distribution of W1+1:4+bm of the above random process as M exposure bias is indeed serious, we expect the quality or diversity of W1+1:++-gn to better when PD is used as Pn than PM. In our experiments we fix ggen to be 30, and vary the prefix length I.
With these ingredients in hand, we now propose the EB-M quantification for exposure bias. reflects the relative performance gain when the length-/ prefix is from PD instead of from PM, and is formulated as below:
MB, (sonr)) fscore (P (MDDD
WiLL Wel1:11-g
,PD
Sccoro (P
P ,P We+sittamn
Wet
where /score is a pre-defined scoring function' of the generation samples, and we assume higher value of fecore indicates that the generation is of higher quality or diversity. In our experiments, will use popular NLG metrics including BLEU (Yu et al., 2016; Caccia et al., 2018) / Nist (Doddington, 2002) METTOR (Denkowski & Lavie, 2014), which mainly capture the quality aspect, and backward- BLEU (Shi et al., 2018) / n-gram entropy (Zhang et al., 2018), which capture the diversity aspect. We will show that the observations from different metrics are consistent.
EB-M has a potential weakness that it doesn't 't reflect how the generation is consistent with the given prefix Wi.,, because only focuses on the marginal distribution of Wi+1++ To cover this shortcoming, in the next section we propose another quantification method named EB-C, which focuses on the model's conditional generation distribution of Wi-1 given a prefix W14-
4.2 DEFINITION OF EB-C
Again, let PH (PM,PD) denote the prefix distribution. With a given prefix length first define the conditional generation deviation (CGD) the expected distance between Par and Pd conditioned on the prefix samples from PH. measured by divergence faw:
CGD(M)H,1,
E [Jav (Par( (Wia), Po( (olW))))))
smaller CGD value suggests a better-modeled conditional word distribution, which captures the consistency aspect of language generation. For the choice of faiv, we will use the standard dry and dss metrics introduced in Section 2.
Exposure bias should induce meaningful gap between CGD(MJM, fav) and CGD DD,,, fav). We now define the EB-C quantification metric for exposure bias at prefix length with metric d to be:
FB-C(M,4, =
CGD/ CGD MI M.L.fa) CGD D..Ja.)
(4)
which describes the relative gain in CGD value when the prefix distribution is replaced by Pp from Pas- Since the computation CGD requires access to the data distribution, in our experiments we will first consider synthetic setting, where existing model is used as PD-
5 EXPERIMENT RESULTS
In this section we use the proposed approaches to quantify the impact of exposure bias. Most of our experiments are conducted on the wiki-103 dataset?. It has around 1 1.8m sentences / 101m words for training, and 4k sentences 241k words for testing. We favour the wiki-102 103 dataset because is large-scale and has long and complex paragraphs, which is useful for the measurements of exposure bias. is also among the most popular datasets for LM bench-marking.
We include Pn into the input of faom, because some of the metrics require data samples as reference.
link to the w0ki-103 dataset.
4

---

Under review as a conference paper at ICLR 2021
prefix length ())
20
30
40
50
60
(Ls
0.350 - .011
0.348 - .001
0.346 - 001
0.342 = .001
0.336
001
BLEU(MS MLs)
0.343 + .002
0.342 + .001
0.340
.001
0.336 + .001
0.332
002
EB-M (Mis)
1.020 .008
1.017 + .004
1.015 + .004
1.016 + .006
1.013
005
(M DD)
0,437 .001
0.438 = .001
0,438
= .001
0,437 = .001
0,432 = 001
BLEU(MTT Mr)
0.430 .001
0.431 - .001
0,433 :00
0.432 = .001
0.428
0011
ER-M (MTr)
1.016 .002
1.015 + .002
1.012 + .004
1.010
.004
1.011
.004
(LEU(Mr+ Daaut)
0.408 + .001
0.406 - .001
0.403 - 0001
0,398 = .001
0.393 = 001
EB-M (Mm Datut)
1.071 + .003
1.080 - .002
1.088 = .006
1.096 - .004
1.100 - .003
Table 2: EB-M measurements with BLEU on the wiki-103 dataset. MLs refers to the LSTM model, We+ S
.PD .P (++1)t)))) The 3-gram BLEU score is adopted. The ratios (EB-M) in bold show that switching from model to data prefix does not give large performance gain.
and MfF refers to the transformer model. (LEU(M (H) refers BLEU(P M1
To prepare MLE-trained PM. we use the code of Transformer-XL. (Dai et al., 2019) to train transformer LM on the wiki- 103 dataset. The model is 16-layer Transformer-Xi XL model with hidden dimension of 410 and an inner dimension of 2100. Since the computation of BLEU / METEOR Nist scores for EB-M requires large amounts of unseen real-data samples as references, we use half of the wiki-103 training data (around 900k sentences and 50m words) to train the model Pas, and save the other half as samples from PD (used as reference for BLEU). More training details provided in Appendix C. The resulting model Par has test-set perplexity (PPL) of 27.81 (if trained on full training data, the PPL will be 24.02). In addition, we also train a 3-layer LSTM LM (Sundermeyer al., 2012) with hidden layer dimension of 600, which has test-set PPL of 34.80.
5.1 RESULTS WITH EB-M
We show the EB-M measurements with different prefix length / in the upper (LSTM) and middle (transformer) part of Table 2. Due to space constraint we first show results with BLEU as /score it has been widely adopted in recent NLG works (Yu et al., 2016; Caccia et al., 2018). . In Appendix B, we repeat this experiment with other popular metries, and the observations are highly consistent and similar. We show the mean and standard deviation as error bar from 10 runs with different random seeds, and for each run 10k samples from the model are used to calculate BLEU (or other metrics) with 10k data samples as references. The observations with longer prefix length (e.g., 80 or 100) are very similar and we omit them to save space.
We observe that the EB-M measurements are around 1.01 or 1.02, for both the LSTM or transformer model. While this confirms that the prefix discrepancy indeed hurts the performance, its impact seems to be limited. Moreover, neither the ratio the absolute gap becomes larger the prefix length grows, which contradicts the incremental distortion claim of exposure bias.
We then check whether "worse" prefix would induce larger performance loss. Similar to the prefix- switching experiment (Table 1), we feed the transformer model with word-level shuffled data prefix, and then compute BLEU score for the generations, denoted as BLEU(Mrr Dshut). Likewise, compute the ratio between BLEU(MT) D) and BLEU( Datur), denoted as EB-M((Tel Damt), and report them in the lower part of Table 2. We find that the measured EB-M(MTr Datar) is much larger than EB-M( MTF), which follows our intuition that a large-enough mismatch would indeed induce significant performance loss in the model's generation.
From this analysis, we put forward the following hypothesis to explain the weak impact of exposure bias, indicated by the EB-M measurements:
Hypothesis 1. While the mismatch between Pı and PD prefix distributions exists and indeed leads to some level of distortion, the distortion seems to be limited and is not large enough to induce an incremental performance loss in the model's 's generation.
We will further validate this hypothesis in the EB-C experiments.
5.2 RESULTS WITH EB-C
Recall that the estimation of EB-C (Equation 4) requires knowledge of the data distribution, therefore we consider a synthetic setting where we treat the 16-layer transformer-X XL model trained on wiki- 103 full training data Pp. And we construct pseudo training set which is roughly the same size of the original training set by sampling from it. We then randomly initialize a 4-layer transformer-J model as PM and train it on the pseudo training set with the same hyper-parameters. The resulting
5

---

Under review as a conference paper at ICLR 2021
prefix length(!)
40
50
60
CG D)
0.102 + .001
0.104 1: .001
0.108
1 .001
0.109 1 .001
0.109 1 .01
CGD(M)MT (M)
0.104 +: .001
0.107 1: .001
0.109
1 .011
0.110.1.00
0.111 1 001
EB-C(M)
1.019 + 0011
1.027 1 .013
1.013
.010
1.013
.012
1.011 1 .011
CGD(M
0.139 1 .001
0.173
.001
0,180
001
0,183
.001
0.184 1 000
EB-C(M aauor)
1.553 + .016
1.652 + .027
1.662
.019
1.675 + .019
1.681 + .019
Table EB-C measurements with dıs for the transformer synthetic setting. We have also tried longer prefix length (e.g.. 80 or 100), and get very similar observations.
1.00
1.015
E8-C
1.005
MLE avg:1.008
avg:8.996
ScratchGAN avg:0. 945
10
20
30
20
25
Length of history
Longth nistor
(a) RankGAN
(b) ScratchGAN
Figure 1: EB-C measurements (with dis) for comparing non-MLE methods in the synthetic experi- ment. The ScratchGAN model behaves better with model prefix than data prefix.
model has perplexity of 84 on wiki 103 test set, indicating that the pseudo data model is not fully recovered by the training process. Finally, EB-C estimated using 10k samples from Par and Pp-
We show EB-C measurements with dys in Table 3. We also show the standard deviation as error bar from 10 runs with different random seeds. The observations with metric div are very similar, and are deferred to Table 7 (Appendix B). We observe that most EB-C measurements is around 1.01 1.02, and do not increase with longer prefix length. Similar to the EB-M measurements, this suggests a weak impact from exposure bias.
is also shown that if we feed shuffled prefix² to the model, the EB-C measurements become much larger. Similar the EB-M case, this suggests that when the mismatch between the prefix distribution and Pp is large enough, the generation will indeed be distorted. On the other hand, the mismatch between Pas and PD seems still in the model's "comfortable zone" (Hypothesis 1).
What measurements do EB-C give for text GANs? We further compare MLE baseline against ScratchGAN (de Masson d' Autume al., 2019) and RankGAN (Lin et al., 2017) the synthetic setting. The results are shown in Figure and implementation details are given in Appendix C. We find that RankGAN and ScratchGAN give lower EB-C measurements than MLE, which as expected, these methods avoid teacher forcing. Most EB-C values in the ScratchGAN case are less than 1, which matches our intuition that GAN models should behave better when fed with model prefixes than data prefixes. On the other hand, EB-C in the RankGAN case is still slightly larger than We believe the reason that RankGAN still relies on MLE pre- pre-training.
To the best of our knowledge, this is the first direct empirical evidence showing that MLE training could indeed avoid the exposure bias problem in that the model behaves better with model prefix than data prefix. It also suggests that EB-C correctly captures how the graining-testing discrepancy affects generation. Note that lower EB-C value does not mean the generation performance better (e.g.. the authors of ScratchGAN acknowledge their performance is still inferior the MLE baseline).
5.3 HUMAN EVALUATION
To confirm observations from the EB-M and EB-C experiments, utilize the Amazon Mechani- cal Turk (AMT) platform to conduct a human evaluation. The subject model (PM) is the 16-layer Transformer-Xi XL. LM trained the full wiki-103 dataset.
Our goal is to compare the scores of generations from the model with prefixes from PD or PM. We follow the standard evaluation protocol for open-ended language generation: The turkers shown
We do shuffle the last word the prefix, otherwise the distortion will be trivial.

---

Under review as a conference paper at ICLR 2021
quality
consistency
len
data prefix model prefix abs. gap
rel. ratio
data prefix
model prefix abs. gap
rel. ratio
20
4.27
.07
4.21 = .16
0.06 = 10 1.016 - .025
4,40
- .05
4.34
= 007
0.06 = 04 1.015 .011
30
4.37
08
4.26
= .05
0.11 = 09 1.027
- .02
4.25
.10
4.14
= .10
0.10 00 1.025 .06
40
4.39
066
4.28 1: .08
0.11 = 07 1.026
1 0017
4.28
4 .12
4.28
0.00 00 1.001 018
50
4.24
+ .07
4.14
+ .12
0.09
= 10 1.023
+ 0044
4.26
+ 110
4.19
- .15
0.07 : .9 1.018
- .022
60
4.51
+ .11
4.42
4: .05
0.09 = 111 1.020
+ .26
4.55
+: .07
4.53
+ .04
0.02 + .08 1.004 .017
Table 4: Human ratings of length-30 generations with prefixes of different length from PD or PM- The absolute gap and relative ratio between the performance with data prefix and model prefix are also shown. The leading zeroes are omitted to save space.
quality
consistency
modal profix
60%
medal profix
Context: = Rakie Ayola = Rakie <unk> Ayola bor May 1968 Welsh actress. best for her role Kyln lysau the BBC medical drama Holby City She first rose to prominence Generation: lon ber prolessional uctiny. Belore this, she went on to play the character of Kyln 's oldar sister, Rakas Ayola. She was later oflered the lead
data prefix
data preftx
40%
40%
20%
10%
15%
2
SCOTM
ScOre
(a) An example of context-generation pair.
(b) The distribution of human ratings.
Figure 2: Left: An example of context-generatte prompt part is underlined, and the remaining part of the context is either from P'D or PMs. A screenshot of our web interface included in Figure 3 (Appendix B). Right: The distribution of scores for generation from model or data prefix of length 60.
pair used for human evaluation, the fixed-length
with a context (prefix) and a corresponding generation, then they are asked to rate the quality (how grammatical/ informative / logical the generation is) and the consistency (how related the generation is to the context) of the generation. We explicitly ask turkers to judge the quality aspect disregarding the context. They can use score from (invalid) to 5 (completely meet the expectation of natural language), and scores like 0.5 or 4.5 are also allowed.
The context consists of a length-20 prompt (from data) and a prefix, which is either from PD or PM. of different length". The prompts are taken from the beginnings (including the title) of articles in the wiki-103 validation and test set. The length of the generation is fixed to 30. We show an example in Figure 2(a). In each assignment, the turker is asked to rate 10 context-generation pairs in shuffled order, 5 of them are with data prefixes and the other are with model prefixes. For every configuration, we collect scores of 250 context-generantt variance, for each context- seneext-gnneatiion pair we collect five replicas of scores from independent turkers, and compute the average, with inter-annotator agreement of around 70%. We report mean and standard deviation as error bar of the average scores from five independent runs (cach run consists of 50 pairs).
pairs from pool of 130 turkers. To reduce
The results are shown in Table 4. We observe that in all configurations, both the absolute gap (around 0.1) and the relative improvement (around 2%) between generation from the two types of prefixes small. More importantly, the gap does not become larger as the prefix length grows. Additionally in Figure 2(b), we observe that not only the average of scores are close, but also the distributions of scores are similar.
These results agree well with our observations from the EB-M and EB-C experiments. We conclude that in the setting we consider, the performance loss induced by the in the prefix is limited. Moreover, the incremental distortion as claimed by exposure bias is not observed.
discrepancy
Foor example, if the prefix is of length 60, then the whole context is of length 80.
7

---

Under review as a conference paper at ICLR 2021
6 DISCUSSION AND LIMITATIONS
What kind of model has a large EB-C measurement? We now discuss a concrete toy example LM with a large EB-C value. However, we argue that this model is unlikely to be product of MLE training.
Example 1. Suppose L 2, and V = {A, B}, the ground-truth data distribution is uniform on {AA, AA,, AB, BA). PM is crafted as follows: PM (W1 A) 0.9, Pas (W2 A = A) 0.9, (W AW1 = B) = 0.5. Note that the model behaves worse when W1 = A, which is of high probability during sampling.
For Example 1, we can easily get CGD(M)D, 1, dTv) 0.2 and CGD(M 1, drv) 0.36, which gives us EB-C(M, 1, dTv) = 1.8. However, this crafted model is unlikely to be an outcome MLE training. The fact that (- W1 B) is better modeled suggests that in the training data, there are more sentences beginning with W1 = than W1 = A. So MLE training should assign more probability (W B), not the other way around. From this perspective, the claim of exposure bias seems to be conflicting with the MLE principle. Due to space constraint, we defer further discussion in the aspect of teacher forcing an objective function to Appendix D.
We devote this rest of the section to discuss the limitations of this work. Firstly, the proposed quantification approaches should not be used as the only performance metric for NLG. For example, a position-aware uni-gram LM, which generates words independent of previous context, has no exposure bias problem and can pass our test easily. More importantly, since the original claim of exposure bias is not well defined, our approaches can only act as reasonable proxies to measure its seriousness, and we humbly acknowledge they have limitations.
Finally, the results from this work should not discourage researchers from exploring non-MLE training algorithms for LM. As shown by Holtzman al. (2020), there exists important problems other than exposure bias for the current NLG models (e.g.. the likelihood trap). Therefore, it completely possible that a training objective different from DRI((Po]|PM can lead to better generation performance (Lu al., 2018; Huszár, 2015).
7 RELATED WORKS
We first clarify that "Whether exposure bias is serious for MLE training?" and "Whether new algorithms improve generation performance?" are two related but different questions, and our work has clear focus on the first question. Despite the large amount of works (listed in Section 1) devoted to alleviate exposure bias, to the best of our knowledge, its actual impact has rarely been systematically studied or validated in the literature. Zhang et al. (2019) attempts to measure the gain from alleviating exposure bias by counting the ground truth words whose probabilities in the predicted distributions produced by their proposed model are greater than those produced by the baseline model. However, it is unclear why this quantification can be linked to exposure bias. Schmidt (2019) provides valuable discussions in the definition of exposure bias, but they did not propose quantifiable definition or metric.
In relevant direction to answer the second question, several recent works attempt carefully evaluate whether the non-MLE training methods can really give superior NLG performance than standard MLE training. Caccia al. (2018) tunes "temperature" parameter in the softmax output, and evaluates models over the whole quality-diversity spectrum. Semeniuta et al. (2018) proposes to use "Reverse Language Model score" or "Frechet InferSent Distance" to evaluate the model's generation performance. Tevet et al. (2018) proposes method for approximating a distribution over tokens from GANs, and then evaluates models with standard LM metrics.
These works arrive at a similar conclusion: The generation performance of text GANs is not convincingly better, or even worse, than standard MLE training. Our work provides an explanation for these observations, that the exposure bias problem, which serves as the major motivation of text GANs, could have been overestimated.
CONCLUSION
In this work, aim to check whether exposure bias is indeed serious problem for MLE-based auto-regressive LM training. We first identify the relf-recovery ability of MLE-trained LM, which casts doubt on the original incremental-distortion claim from exposure bias. We then design and
8

---

Under review as a conference paper at ICLR 2021
experiment with two approaches proxies to quantify the significance of exposure bias in the aspects of quality, diversity, and consistency. We also conduct human evaluation, with consistent observations.
discrepancy indeed induces some level of performance loss. (2) However, the performance gap seems to be limited, and not enlarged as prefix length grows, which contradicts the incremental distortion claim of exposure bias. To explain our observations, we hypothesize that although the mismatch between the data and model prefix distribution exists and indeed leads to some level of distortion, it is still in the model's "comfortable zone", and is not large enough induce incrementally drastic performance loss during generation.
Our observations are two-fold: (1) We confirm that the
REFERENCES
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. CoRR, abs/1809.10853, 2018. URL http: //tp:i////aii/////aa/eaeeaaaiaia wwwpaawwwww...aaa..t./oo//e....e org/abs//80 10853.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS' 15, pp. 1171-1179, Cambridge, MA, USA, 2015. MIT Press. URL http::ttt//// http:/dl.acct arg. wtag/iiiain cttg:/iitiioonno cfm2i.==2969239 2969370.
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. Language gans falling short. CoRR, abs/1811 .02549, 2018.
Zihang Dai, Zhilin Yang, Yiming Yang. Jaime G. Carbonell, Quoe V. Le, and Ruslan Salakhutdinov. Transformer-xi Attentive language models beyond a fixed-length context. CoRR, htt//990.02860) 2019. URL http: aataaaiivvvv......a.aaaa bzgg/bbs. / 1901. C2860.
Cyprien de Masson d' Autume, Mihaela Rosca. Jack W. Rae, and Shakir Mohamed. Training language gans from scratch. CoRR, htss/1955.0922., 2019. URL http:://rxiv/ org/abs//////oooo//////////////s abs/1905. 1905. 09922.
Michael Denkowski and Alon Lavie. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the FACL 2014 Workshop on Statistical Machine Translation, 2014.
George Doddington. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the Second International Conference on Human Language Technology Research, HLT '02, pp. 138-145 145, San Francisco, CA. USA, 2002. Morgan Kaufmann Publishers Inc.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the Association for Computational Linguistics, pp. 889-898. Melbourne, Australia, July 2018. Association for Computational Linguisties. doi: 10.18653/v1/PP 1/P13/ 1082. URL https: www. ac balweb. /rgg antho /
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS' 14, pp. 2672-2680, Cambridge, MA, USA, 2014. MIT Press.
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang Long text generation via adversarial training with leaked information. CoRR, abs/1709.00661 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Ari Holtzman, Jan Buys Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https: //opereviiew net. itteea/mwreeeeeeeeoooooomoooooro fore/ffforrrrrrryyorrr.....

---

Under review as a conference paper at ICLR 2021
Ferenc Huszár. How (not) to train your generative model: Scheduled sampling, likelihood, adversary? CoRR, abs/1511.00 1.05101, 2015.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. Ctrl: conditional transformer language model for controllable generation, 2019.
Alex Lamb, Anirudh Goyal, Ying Zhang, Saizheng Zhang, Aaron C. Courville, and Yoshua Bengio. Professor forcing: A algorithm for training recurrent networks. In NIPS, 2016.
Jiwei Li, Will Monroe, Tianlin Shi, Alan Ritter, and Dan Jurafsky. Adversarial learning for neural dialogue generation. CoRR, abs/ 0oss/7001.00 701.06547, 2017.
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-ting Sun. Adversarial ranking for language generation. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp.
3155-3165. Curran Associates, Inc., 2017.
Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zimick. Microsoft COCO: common objects in context. CoRR, abs/1405.00 03.50031222220 2014.
Sidi Lu, Lantao Yu, Weinan Zhang, and Yong Yu. Cot: Cooperative training for generative modeling. CoRR, abs/1804.03782, 2018.
Moin Nadcem, Tianxing He, Kyunghyun Cho, and James Glass. A systematic characterization of sampling algorithms for open-ended language generation, 2020.
Weili Nic, Nina Narodytska, and Ankit Patel. ReIGAN: Relational generative adversarial networks
for text generation. In International Conference on Learning Representations, 2019. URL net. htt.:/ewwweee/ooooiiieeeeeeeeee 1
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguisties. doi: 10.3115/1073383.077333 URL https: www. aclweb. htgg/anhology/// /PO2 P00-P00200100011 1040.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- guage models are unsupervised multitask learners. 2018. URL https: //tttt:::///meeeee d4mmucppsywy sewndffoont.et//betteee language w
models poodelsllangaage models s .df.
Sai Rajeswar, Sandeep Subramanian, Francis Dutil, Christopher Joseph Pal, and Aaron C, Courville. Adversarial generation of natural language, 2017.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In ICLR, 2016.
Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self- critical sequence training for image captioning. CoRR, abs/1612.00563, 2016. URL http: // //aaxxiv. wwtp:ii.org/ab///101..
00563.
Florian Schmidt. Generalization in generation: A closer look at exposure bias, 2019.
Stanislau Semeniuta, Aliaksei Severyn, and Sylvain Gelly. On accurate evaluation of gans for language generation. CoRR, abs/1 806.049366 2018.
Zhan Shi, Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. Towards diverse text generation with inverse reinforcement learning. CoRR, abs/1 004.1804.11 1258, 2018.
Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. LSTM neural networks for language modeling. In INTERSPEECH 2012, 13th Annual Conference of the International Speech Communication Association, Portland, Oregon, USA, September 9-13, 2012, pp. 197, 2012. URL http: //www. i isca wareecch.rg//rcce htt.rrhwwweeeeeeeeeeeee.eeeeeee 201eehhh_2001111
94, html.
10

---

Under review as a conference paper at ICLR 2021
Richard S. Sutton and Andrew G. Barto. Introduction Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981.
Guy Tevet, Gavriel Habib, Vered Shwartz, and Jonathan Berant. Evaluating text gans as language 2018.
models. CoRR,
Sam Wiseman and Alexander M. Rush. saneenceto-sequene learning as beam-search optimization. CoRR, (es/1606.0296)) 2016.
Lijun Wu, Yingee Xia, Li Zhao, Fei Tian, Tao Qin, Jianhuang Lai, and Tic- Yan Liu. Adversarial neural machine translation. CoRR, abs/1704.00633, 2017.
Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. Recent trends in deep learning based natural language processing. CoRR, 7000002799, 2017. URL http: wrx.v.orrr abs/ 7088.170.00 02 279..
Lantao Yu, Weinan Zhang. Jun Wang, and Yong Yu. Seggan: Sequence generative adversarial nets with policy gradient. CoRR, haw/1609905777 2016. URL http: /tap:://bblll /dblp.llppp..pm www.:piiintintncoocmmcemoommmeee de/ db/ atbjjounalss cttp:/orrclccceccooooo 609. html #YuzWY1 6.
Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. Bridging the gap between training and inference for neural machine translation. CoRR, abs/1906.00244. 2019. URL httt:://axiiii org/abs/ 1906. 9966 02448.
Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. Gen- erating informative and diverse conversational responses via adversarial information maximization. CoRR, abs/ 009905572, 2018. URL http: /ttpii:////xvvv.//e/ org/ abs 18099..... 05972.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. SIGIR, 2018.

---

Under review as a conference paper at ICLR 2021
<LEN-20 PROMPT>.. (<LEN- 100 DATA PREFIX>) the gambia &
Generation: international victory on 6 august 2016 howell wife italia was on win, core shortened to move 2 3 murray was contracted to take the wwe
<LEN-20 PROMPT> ( (<LEN-100 MODEL PREFIX>1
the field L
Generation: and public appearances of the university also made way into law interception
and <unk> to improve measurable cricket. days played a part in the football association 's my
(<LEN-10) 100 SHUFFLED DATA PREFIX>) ... ngo for by addition in early & Generation: ) 4 years ) to <unk> the global borders relationship with the witch, because of the huge crowds of such adults, is that she has gone through
(<LEN-100 RANDOM PREFIX>)
fire. mercury 's
Generation: player protects the ire of playing to remove players from working, while the other in gold. similar purpose is to track and parade the rules, yet
Table 5: Samples of a MIEE-rriined LSTM LM when fed with different types of prefixes. The prompt "the development of women 's football in africa faces several challenges, including limited access to education, poverty amongst". The generation length is fixed to 30. To save space, we omit the long prefix and only show the last few words.
A DEFINITIONS OF drv AND djs
In this section, we formally define different probability divergences used the paper. We first give the definition of the total variation distance drv between two distributions P and Q on vocabulary V:
drv div(P,Q)
I Q(w)))-
P(w))
(5)
For dys, we first define the Kullback-Leible divergence dKL:
P(w)
dxa (PIIQ)
Σ WEV
P(w) log Q(w)
(6)
Finally, we define Jensen-Shannee Shannon divergence dis:
(ss(P.Q) } dkk (PIIM) J dxl (QIIM)
1
where M - +P)
B AUXILIARY RESULTS AND PLOTS
We show more samples for the prefix switching experiment for models trained the wiki-103 dataset in Table (LSTM), and Table (transformer).
We show EB-M measurements with different metries for the transformer LM in Table 8. Note that all the metrics we use are the higher the better (in quality or diversity), so the denominator and numerator in the EB-M formulation do need be switched.
In Figure 3, we show an example of the interface used for human evaluation discussed in Section 5.3.
Table 7 contains EB-C measurements with drv as metric for the transformer synthetic setting.
C IMPLEMENTATION DETAILS OF TRANSFORMER- XLL, RANKGAN, AND SCRATCHGAN
In our preliminary tries, we find that the generation behavior of the transformer-Xt XL model is not good when the prefix length is short (e.g. 5). We believe the reason is that in its training the model
httpps: / //g thub. com/kimi young//raanffoomee-- /b1 httpa////////bboo/////tttt////// httla///ttteer/pttrcc//m/ 03 base. sh
12

---

Under review as a conference paper at ICLR 2021
SLEN-20 PROMPT> (<LEN-100 DATA PREFIX>) on the American thrash metal band Slayer 's 1986 album Reign in Blood. The lyries and music were written by Slayer guitarist, Jeff Hanneman and are based on Nazi physician Josef Mengele, who conducted human experiments at the Auschwitz concentration camp during World War II Angel of Death " led to the band facing accusations of Nazi sympathizing and racism throughout their career Despite the controversy surrounding the song and its contribution to the delay in the release of Reign in Blood, " Angel of Death is featured on all of 4
Generation: Slayer 's compilation albums, Reign in Blood Part One ( 1992). The song has received critical acclaim from music critics Notable recordings and music videos
<LEN-20 PROMPT> ICLEN-100 MODEL PREF IX>) on Slayer 's fifth studio album Blood Rush (004)) ) Terry Richardson composed the track with contributions from other band members This is the first Slayer song not to be featured on Megadeth album due to positive reviews from reviewers Although a commercial success. Blood Rush entered the Translations album chart number 11 on 18 April 2005, in the US and there was band project also called War Lovegood to re e- release Ghost on the Knees The Lion King Angel of Death " was 4
Generation: released as single in Europe 1 June 2005. It appeared on the UK Singles Chart at number 16 on 8 March 2005 and reached a peak of
(<LEN-120 SHUFFLED DATA PREFIX>) ) mph after passed of 2011 of Beatriz 20 the winds... following (Hurricane ( on km mi June disturbed with gradually that an coast km of morning was interaction hundred hours the June of intensified Originating of increasingly people status several on 90 later south and brushing Mcxico jeos; four Mexico of as western its Gaining weather became Beatriz 22 miles roughly 20 weakened the June evening / latitude in land Hurricane June, Beatriz Due 19 Category 2011 1 15 area hurricane it abruptly attained, Beatriz reached Mexico.. 150 organized system Early hurricane hurricane from ) to the The killed
Generation: or injured doctor and his loved ones. Due to the crashed unit, Juan marked the formation of the character of Razzi, the first modern hurricane record
(<LEN-120 RANDOM PREFTX>> Parker holding appearing feeling Special wider destroy none metres downtown forward physical Rolling gathered volume dead kingdom Cooper aged 1953 voted Vincent future Delaware alcohol CR improvements Soviet considered field Impact animation 400 Philippines promote leaving Who Beatles Ocean commonly hoped Series comes permitted venue % nature would temporarily 66 91 younger Creek European sing battalion tip phase setting panel cruiser soul E. Children museum shared Big lower Department Mountain J. victims Rangers difficulty limit although Hospital Mountains news pilot anime tropical sisters determined tropical black Paul north lane interior 1955 Nicholas bottom ago heading Born bear Carl dogs blue 1920s improved border Medical course Boys Story Welsh Building listed Iron sea father view link gone faced candidate +
Generation: according Doctor Richard Geng FEU of Hong Kong. girls who were infected with <unk> syndrome became a positive factor rooting in the film: PPeppe working
Table 6: Samples of a MLE-trained transformer LM when fed with different types of prefixes. The prompt is "= Angel of Death Slayer song = Angel of Death is the opening track". The generation length is fixed to 30.
prefix length CGD(M D)
20
30
40
50
60
0.293 = .001
0.300 + .002
0.308
+ .001
0.312 + .001
0.312 + .001
MG
0.294 .002
0.303 + .001
0.309
+ .001
0.312
.001
0.313 .002
EB-C(M)
1.005 = .008
1.010 + .009
1.001 + .006
1.002 + .008
1.002 + .007
Table 7: EB-C measurements with dry as the metric for the transformer synthetic setting.
is almost always fed with very long history context. So it's behavior with short prefixes becomes undefined.
To alleviate that problem, we slightly modify the implementation by randomly emptying the history context of each mini-batch with very small probability (0.03). In this way, the model is made aware of possible short context. We find that this modification is very effective and doesn't 't have noticeable performance degradation.
13

---

Under review as a conference paper at ICLR 2021
Context: Hurricane Beatriz (201)) ) Hurricane Beatriz was a Category hurricane that killed four people after brushing the western coast of Mexico in June 2011 Originating from an area of disturbed weather June
the central Atlantic Ocean, , the storm initially moved generally weslward, intensified slowly until becoming a 40 mph (65 km h tropical storm
Generation:
How good is the generation by itself, disregarding the context? Ploase enter quality score (0 for "invalid", 5 for "completaly meet your expectation of natural language"):
Please type the quality score number the (10.055111552222533344444..333
How consistent (related) is the generation to the given context? Please enter consistency score (0 for "not related at all", for "naturally and consistently continues the context"):
Please type consistency score (a number set (00511115222533344445553
Figure 3: An example of the interface used for human evaluation.
prefix length backward BLEU
20
30
40
50
60
1.013 001
1.015 E 000
1.014 - .001
1.014 = .002
1.013 = .002
Nist
1.007 + .011
1.008 1 .001
1.008 1: .001
1.007 4: .001
1.006 1 .001
METEOR
1.012 0001
1.012
.002
1.011
1 .001
L007
= .003
1.005 1 .002
3-gram entropy
0.999 001
0.999 E 0001
1.000 = .001
1.000 = .001
1.006 + .001
Table 8: EB-M measurements with different metrics for the transformer LM on the wiki- 103 dataset.
Other training configurations of transformer-X XL (learning rate, batch size, etc.) are not changed.
For ScratchGAN, we implement the synthetic experiment based on https: / For RankGAN, we use
/ hggptuut..b.....gggggg..gg..b... com/
deepmind, deepminds nd- weeeeacch/rrre//aatttts//eteeeee TensorFlow implementation https: /gtthtb.bb..gg wttp:/ttww...oooe..oettttotm cetood/deiire http://esrre2200/eee/ 2200200002//RKA.....
Since both code from ScratchGAN and RankGAN are based LSTM, focus LSTM LMs for this set experiments. We train a standard MLE model on the EMNLP-news data, and use it as PD for synthetic setting. It refers to the EMNLP 2017 WMT News Section, which has around 268k sentences 7.5m words for training and 10k sentences 277k words for testing. It has been widely used text GAN literature (Yu et al., 2016; Lu al., 2018). The PD model is a one-layer LSTM LM with hidden dimension of 512. We randomly initialize another one-layer LSTM LM with hidden dimension of 32 PM. We then use samples from P'D train it either with MLE or with the GAN objective.
D MORE DISCUSSION
Here discuss the fundamental question "Is teacher forcing really biased?", from the perspective of objective functions. Note that the teacher forcing (MLE) objective (1) can be re-written as:
arg min E
- log Po
W~PD
arg min E - log Pa (W)
WPPp
(8)
arg min log Po(W)
W~ID Po(W]
arg min (PoIIP))
where DRi. denotes the Kullback-Leible divergence, and 0 denotes the trainable parameters in Par- Therefore, teacher forcing (MLE) training is minimizing model's sampling distribution, from Pp. While it's true that the training is "exposed" to data samples as prefixes, we should not simply deduce the objective is "biased".
divergence Pe. which exactly
We also note that is straightforward to prove that when Po has enough capacity and the MLE objective is minimized, there will be no discrepancy and thus no exposure bias problem, because P converges Pp. Therefore, believe exposure bias, assuming it exists, is more of empirical issue than a theoretical one.
14


Reviews:
[{"review_id": "rPBiy9jmR1", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an empirical study of exposure bias, showing that it does not appear to be an especially significant issue.  Both automated metrics and human evaluations are presented, and I found the experiments to be fairly convincing.  While the paper could have more material and depth, I think it makes an important point that people will appreciate seeing in the conference.\n\nDesigning metrics for exposure bias is a novel task, and this paper invented appropriate approaches.  The experiments cover the two most important model classes (LSTMs and transformers) and use representative model settings and corpora.\n\nConcerns:\nThe experiments only look at pure sampling from the model---i.e., they don't use temperature, nucleus sampling, greedy or beam search.  These other generation settings are more important for applications, compared to pure generation.  Also, they have been reported to exhibit more deviation from the corpus distribution (especially with respect to particular pathologies like repetition).  Evaluating generation in these other settings would increase the impact of the paper's conclusions.\n\nThe human evaluation is helpful, but needs measures of inter-annotator agreement.  \n\nLess significant: The qualitative experiment that starts the paper is very anecdotal, and I think the paper would be better off without it.  The concrete examples in Table 1 are helpful for illustration, but I would not dedicate a section to them or call it an ‘experiment,’ as it is too limited in scope.  It detracts from the stronger experiments later in the paper.  Likewise, the model in Example 1 seems too inaccurate to be illustrative, and the associated discussion in Appendix D is already well known I believe (that the “Teacher forcing” MLE objective just aims to choose parameters that maximize the likelihood of the corpus, using the chain rule).", "extra_scores": {}, "extra_text": {}}, {"review_id": "nQ173DvHOfk", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper makes a key observation: “exposure bias” is blamed for many of the issues with Neural Language Generation but it lacks both a concrete definition or any obvious evidence that it truly exists. The authors begin by defining exposure bias as the decrease in quality and relevancy (to the conditioning text) in generations as the model conditions on its own output. A limited qualitative study fails to find evidence of exposure bias, and the authors propose two metrics, EB-M and EB-C, to measure the quality and relevancy degradation, respectively. Quantitative results find that degradation on these two axes as the model conditions on itself are either minor or non-existent. To calibrate our understanding of these metrics, the authors do a human study as well as comparing two GAN frameworks. In the discussion the authors discuss limitations of the work, mostly that the given metrics are not a complete notion of evaluation and connect their work to related literature.\n\nStrong Points:\n- The authors correctly call out the lack of scientific work on actually defining and understanding exposure bias.\n- Their intuition that exposure bias should mean quality and relevancy are highly dependent on the prefix is good idea.\n\nWeak Points:\n- The quantification of exposure bias is largely unjustified.\n- The qualitative evaluation is ad-hoc.\n- The discussion section is rushed and does not make a strong argument that the author’s interpretation is supported.\n- The choice of wiki-103 as a testbed is strange.\n\n\nI recommend rejecting this paper, as the way exposure bias is quantified is dubious and the authors do not make a strong argument that they would have detected exposure bias if it is present.\n\nI want to start by saying I think that the authors correctly point out a deep flaw in the literature around text generation: exposure bias is a casually defined issue that is nowhere well-defined or proven to exist. Even more importantly, I think the intuition that this paper gives around sampling different kinds of prefixes and looking at the resulting generations is exactly right. The main issue I have is that the formalization of exposure bias is not well-founded and I do not believe a convincing argument is made the claims of relatively little (or even no) exposure bias are actually supported by the experiments.\n\n While the ways the authors try to quantify exposure bias are intuitive, there is really no evidence that these metrics measure what they claim to. Corpus-BLEU has been used in many papers, though I don’t know of any showing actual correlation with a human metric, but we can at least say that it has precedent being used cumulatively. However, using the corpus-BLEU of _individual_ samples in a fraction creates the potential for completely inaccurate estimates. Even assuming corpus-BLEU works, there is no guarantee that it acts as an estimate of individual samples, in the exact same way that BLEU is known not to be accurate for individual sentences (Burch, 2006). Furthermore the EB-C metric is basically proposed from thin air. It makes sense to compare the distributions, but how big of a difference should we expect? GANs do badly on EB-C, but they are known to be significantly worse even on normal metrics, so it’s not clear what this proves.\n\nThe qualitative evaluation is just the narrative of what the authors think. No intuition is given, simply “we didn’t find exposure bias when we looked at the data”. It especially shouldn’t be described as failing “to show the significance of exposure bias”. The authors refer back to these thoughts as experiments, e.g. on page 5 with “This agrees with our observations in the qualitative experiments.” This seems inappropriate.\n\nThe discussion is extremely rushed and it is not very clear what we are supposed to conclude from it. The authors show that their metric is not complete via a toy example, but talk about how MLE does not produce these kinds of solutions. That’s fine and I believe that part of their argument, but I still feel that there is not much evidence that the metric would actually show significant differences if the quality of the writing went down.  Quality is a tricky thing to measure. To the authors’ credit they conduct a human study.  However, it is unclear to me whether humans that are exposed to a generated prefix (which may already deviate somewhat from human language) would feel that the mistakes made due to exposure bias “match” the mistakes in the prefix.  If the generations were truly horrible, I agree we would see more deviation in this score, but the paper attempts to disprove the very existence of exposure bias. A more appropriate reframing would be to talk about the kinds of effects exposure bias couldn’t possibly be having, e.g. that exposure bias may only result in strange word choice but not total degeneration. This over-claim, and the lack of smaller claims to build-up to it that might have been more appropriate, make it difficult for me accept the described conclusions.\n\nFinally, previous works on open-ended text generation have used the portion of the corpus released by OpenAI. It’s hard to know how comparable these results are, and since both GPT-2 and a significant number of validation text examples from WebText are freely available this seems like a flaw.  Why was wiki103 chosen instead of WebText used alongside the pretrained GPT-2?\n\nCallison-Burch, Chris, Miles Osborne, and Philipp Koehn. \"Re-evaluation the role of bleu in machine translation research.\" 11th Conference of the European Chapter of the Association for Computational Linguistics. 2006.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Tp9oPDjdmiw", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe so-called \"exposure bias problem\" (EBP) is often cited as a serious issue when training sequential model with MLE and teacher forcing. This paper attempts to quantify whether the problem is real on open-ended generation experiments, and concludes that it is actually minor.\n\nPositive aspects of the paper:\n\n- EBP is often accepted as an obvious problem in the generation community, despite the lack of experimental evidence. Thus, a paper such as this one that tackles such evidence head-on can be very useful.\n\n- Close to the end, the paper argues, I think correctly, that \"the original claim of exposure bias is not well defined, [and] our approaches can only act as a reasonable proxy to measure its seriousness\". The paper however attempts to design (1) experiments and (2) quantitative measures that correspond with our general intuition of the EBP.\n\n- According to these experiments and measures, and also to some human evaluations, the paper compares two situations: (D) the trained generator is provided with a prefix from (an unseen portion of) the dataset and asked to generate a continuation; (M) it is provided with a prefix that it generated itself and asked to generate a continuation. The conclusion from these experiments is that the continuations in the (M) case are only very slightly worse than in the (D) case. This contradicts the usual expectation about the EBP, namely that in the (M) case, the model would be \"lost\" in unknown territory and start to produce very poor text. The main conclusion of the authors is that: \"although the mismatch between the data and model prefix distribution exists, it is still in the model’s “comfortable zone”, and is not large enough induce drastic performance loss during generation\".\n\n- The authors are careful to avoid a possible misunderstanding of their results. They do *not* make the claim that MLE + teacher forcing is the best way to train a generation model, but only the *different* claim that exposure bias is not such a serious problem as is often assumed.\n\nSome issues and questions:\n\n- The fact that the generation model does not move away from its \"comfort zone\" when generating prefixes could be related to two different dimensions: First, the prefixes that you generate are only moderately long, thus alleviating the EBP issue. Second, you do not describe exactly how the generation is done. I gather that Transformer-XL uses a form of \"top-k\" sampling, that is, a generation mode more restricted than a standard (from the pure probabilistic viewpoint) \"ancestral sampling\". Such restricted sampling (as also beam-search) is known to improve the quality but limit the diversity of the generations. It would be interesting to see if you would obtain the same EBP conclusions with ancestral sampling.\n\n- The corpus-BLEU measure that you use is most of your experiments was designed for Machine Translation and appears to me to be a very weak measure of quality for open-ended generation. It would actually be informative to compute a baseline for that measure in terms of continuations not from the model, but from the training data itself, giving a measure of the average \"quality\" of a gold-standard continuation relative to all the other gold-standard continutations from the training data. How good would this \"gold corpus-BLEU\" be? Probably not so good, and it would be interesting to compare the (D) and (M) continuations to these gold continuations.\n \n\nMinor points:\n\n- The JS divergence is not a distance.\n\n- Please clarify exactly when \"prompts\" are used in your experiments. While they are used in the human evaluations, it was not clear to me whether they were used in the EB-M experiments, for example.\n\n- The paper https://arxiv.org/abs/1906.05664 \"Calibration, Entropy Rates, and Memory in Language Models\" might be relevant as related work: in that paper the authors argue that neural language models tend to suffer from \"entropy drift\", namely the tendency to entropy of the next token prediction to be higher when conditioned on an (M) type prefix than on a (D) prefix. However, if I am correct, they assume standard hierarchical sampling, and their empirical evaluations are not extensive.\n\n-----\n\nNov. 30th: On a second reading of the authors’ exchanges with the reviewers as well as of the updated paper, I am lowering my overall score. While I still believe that the *questions* that the paper raise are very worthwhile to the community, I agree with several reviewers that the *answers* provided in the paper are insufficiently supported by a convincing formalization and by experiments.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "-_lLzJYM_Df", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the exposure bias in auto-regressive neural language models. This problem is known to cause incremental performance degradation, and attempts to mitigate this problem have received significant attention in the community (using, e.g., RL and GANs). The paper claims that prior work has mostly focused on addressing the problem rather than measuring how severe the exposure bias problem actually is. Despite extensive previous work on mitigating exposure bias, the paper suggests that the exposure bias is not “large enough [to] induce drastic performance loss during generation” (e.g., a human evaluation controlling for exposure bias show relative differences of only < 3%).\n\nOn the positive side, I agree with the paper that it is worthwhile to study the extent of the exposure bias problem, and the approach of the paper (comparing generation with model-based prefixes vs. data prefixes) is quite intriguing, as it attempts to compare generation with exposure bias again without such a bias. That said, I have several concerns that make me question some of the claims of the paper:\n\n1) The human evaluation shows very little performance difference between generation with data prefix (D-prefix) and model prefix (M-prefix), suggesting exposure bias is not a problem. But I would argue this is mostly an artifact of the experimental setup, as prefixing generation with D-string (of length L) only eliminates exposure bias up to position L, and then both evaluated systems (according to the generation process defined Section 4.1) use a standard auto-regressive LM generation process that makes them both subject to the exposure bias problem. Since these generated strings are of length 30, there is plenty of room for exposure bias to crop in. (Indeed, Zhang et al., (2019) and Holtzman et al. (2019) have shown concrete examples of exposure bias artifacts appearing in much shorter sequences). So, the small performance difference between a preference M-prefixed and D-prefixed generation may very well be due to both setups being almost identical (same model, same auto-regressive inference algorithm, and the only difference is the prefix – which human raters are not even asked to judge directly). If exposure bias is indeed a problem, then it would affect both systems almost the same way, so this human evaluation can’t be used to either affirm or deny exposure bias is a significant problem. \n\n2) I also have concerns regarding the automatic evaluation based on BLEU. What the authors call “corpus-BLEU” is actually not the standard version of BLEU (see “Other Comments” below), as the version of the paper looks at n-gram (n=1 to 3) matches between the generated sentences and a large set of references that is *not* specific to a particular context. As the paper’s automatic evaluation is done in a completely context-agnostic way and relative to a large pool of references, it essentially only measures whether the model (with or w/o exposure bias) is able to generate plausible trigrams, but that sets the bar very low as we already have plenty of evidence showing neural language models are quite capable of generating reasonable trigrams (whether there is exposure bias or not), and in fact often much longer n-grams. The authors claim that auto-regressive models appear to have “self-recovery ability” that mitigate any exposure bias, but I would say that it is hard to claim anything about self-recovery (at least in terms of automatic evaluation) when the evaluation metric operates over such a small window (<=3 words). \n\n3) Abstract: “Although a lot of algorithms have been proposed to avoid teacher forcing and therefore alleviate exposure bias, there is little work showing how serious the exposure bias problem actually is.” \nI find this claim a bit misguided, as the numerous papers addressing exposure bias are *empirical* ones. They have shown improvements in various tasks such as machine translation, image captioning, and other generation tasks thanks to techniques aimed at reducing exposure bias. Now, one could claim that significant improvements shown in these papers are due to reasons other than exposure bias, but if that is the case then the submission doesn’t do a much better job at isolating exposure bias from other factors (given my concerns in (1) and (2)). Note that Zhang et al. (2019) actually does include a short section attempting to quantify the effect of exposure bias. \n\n4) For a paper that attempts to challenge the current understanding of a problem that has received very significant attention (i.e., exposure bias), it is quite thin in terms of related work (2 paragraphs). The paper omitted important related work (e.g., Schmidt, 2019; Tan et al., 2019; Rennie et al., 2016), including an ACL best paper on the same topic and whose findings appear to be at odds with the current submission.\n\nIn sum, the paper tries to improve our understanding of exposure bias and its impact on open-ended language generation, and this is totally a worthy goal. I also recognize that isolating (i.e., ablating) the effect of exposure bias is difficult. That said, the paper makes rather strong claims (e.g., that “performance gain is minimal” if exposure bias is supposedly eliminated) that I find unfounded given my points in (1) and (2). In both evaluations, the setups evaluate a given model sequence W prefixed by a string sampled either from the data or the model, but that does not eliminate the fact that exposure bias is bound to appear within the generated sequence (length 30), which is the sequence that is evaluated in the end.\n\nOther comments:\n\n“Corpus BLEU”: Note that BLEU, as originally defined in (Papineni et al., 2002), is in fact a corpus-level metric, as it aggregates n-gram statistics over an entire (test) corpus. So “corpus” in “corpus BLEU” is redundant and possibly confusing. The term “corpus BLEU” or “corpus-level BLEU” is generally used to contrast with various versions of “sentence-level BLEU.” Now it appears that “corpus BLEU” in this submission refers to the version of BLEU used in SeqGAN (Yu et al.), which is therein not called “corpus BLEU.” That distinction should be made clearer, considering that the use of “a large number of sentences from ground-truth data as references” is a significant departure from how BLEU was originally designed to work. Indeed, (corpus-level) BLEU (Papineni et al., 2002) doesn’t allow matching hypotheses and references across test instances, as opposed to the submission. The authors justify their use of BLEU as it is a “well-established [metric] in the NLG literature,” but this is rather misleading as their specific version of BLEU is not well established and not what is commonly used in MT and NLG.\n\nTable 1 doesn’t actually illustrate what the experimental setup of the paper does (i.e., Section 4.1), as the latter doesn’t include a shared prompt of 20 words that is supposed to make the two generated strings “more comparable”. Since this prompt supposed to make the two strings more comparable is absent from the actual evaluation setup, I gather from the authors’ own words that they implicitly admit that string comparisons in their evaluation setup are not so comparable. \n\n* Zhang et al., 2019: https://arxiv.org/pdf/1906.02448\n* Schmidt, 2019: https://arxiv.org/abs/1910.00292\n* Rennie et al., 2016: https://arxiv.org/abs/1612.00563\n* Tan et al., 2019: https://arxiv.org/abs/1811.09740\n* Holtzman et al., 2019: https://arxiv.org/abs/1904.09751\n* Chen and Cherry, 2014: https://www.aclweb.org/anthology/W14-3346/\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Sequence generation models trained via maximum likelihood estimation (or variants of so called 'teacher-forcing') condition on *data* samples during training and on *model* samples for predictions. The susceptibility to this potential "mismatch" in input distribution is often referred to as exposure bias (EB). 

This paper stresses that most research around EB is focused on addressing it, rather than defining and/or quantifying it. Thus the submission questions the severity of EB and attempts to operationalise a testable definition for it. Myself and all the reviewers strongly support the observations and the agenda, we find the question this paper asks an important one. 

Despite our appreciation for this paper's relevance, we have identified a number of problems that prevent me from recommending this paper. I will comment on the two most important points:

1. The 'operational definition' of EB in this paper is not sufficiently precise to be testable. It builds on the somewhat commonly accepted view that the effects of EB accumulate as the conditioning context grows longer, and that this causes a model to generate badly distorted sentences. This definition still leaves quite some room for interpretation (without specifying reasonable expectation about how these effects 'accumulate' and what/how bad they are, it seems difficult to design tests). We acknowledge that the submission attempts to shed light onto some of these aspects by having some 'control groups' using gold data and shuffled strings, but we did not find those sufficient (mostly in light of the next point).

2. MT evaluation metrics (essentially, string similarity metrics), most notably (but not exclusively) BLEU, are used in this work in a setting where we cannot easily grant that they have the discriminating power that the authors expect of them. See this is not a criticism about the imperfections of BLEU (or any other automatic metric), but about the lack of evidence supporting its use against unrelated sentences. We do not find it sufficient that some recent NLG papers have made similar use of it (I, for example, would have criticised those papers on similar grounds).

Overall, we believe this submission asks a relevant question, the insight about dependence on prefix is nice and might lead to a first operational definition of EB (which might be only a few refinements away from the version proposed here). The current evaluation is unconvincing and I believe the authors should be able to find more credible strategies, especially, strategies that have already gone through some scrutiny (for example, in literature around OOD detection and tests for distribution shift). 

Though I do not recommend this paper for acceptance, I hope the authors will find valuable feedback in the expert reviews attached.
Decision: Reject