Quantifying Exposure Bias for Open-ended Language Generation

1 INTRODUCTION . Language model ( LM ) is a central module for natural language generation ( NLG ) tasks ( Young et al. , 2017 ) such as open-ended language generation ( Radford et al. , 2018 ; Nadeem et al. , 2020 ) , machine translation ( Wu et al. , 2017 ) , dialogue response generation ( Li et al. , 2017 ) , image captioning ( Lin et al. , 2014 ) , etc . For decades , maximum likelihood estimation ( MLE ) has been the most widely used objective for LM training . However , there is a popular belief in the natural language processing ( NLP ) community that standard MLE training suffers from the exposure bias problem which leads to an incremental performance degradation during test-time generation . The claim of the exposure bias problem ( Bengio et al. , 2015 ; Ranzato et al. , 2016 ) is originated from the following discrepancy between MLE training and test-time generation for auto-regressive language models : During training , the model is trained to predict the next word conditioned on prefix ( or history ) words sampled from the ground-truth data distribution ; While during generation , the model generates words conditioned on prefix sequences generated by the model itself . Hence , due to the exposure to real data during training , the language model could potentially be biased to only perform well with data prefixes . Therefore , it is claimed ( and widely believed among researchers ) that during generation the errors should accumulate along the generated sequence , and the distribution generated by the model would be incrementally distorted . The forced exposure to ground-truth data during training is also referred to as teacher forcing . In order to avoid teacher forcing , many training algorithms ( Bengio et al. , 2015 ; Lamb et al. , 2016 ; Ranzato et al. , 2016 ; Yu et al. , 2016 ; Zhu et al. , 2018 ; Lu et al. , 2018 ; Lin et al. , 2017 ; Guo et al. , 2017 ; Rajeswar et al. , 2017 ; Wiseman & Rush , 2016 ; Nie et al. , 2019 ; Shi et al. , 2018 ; de Masson d ’ Autume et al. , 2019 ; Rennie et al. , 2016 ) have been proposed as alternatives to MLE training for open-ended language generation . Most of these works utilize techniques from generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) or reinforcement learning ( RL ) ( Sutton & Barto , 1998 ) . In this paper , we refer to these algorithms as non-MLE methods . With the huge research efforts devoted to alleviate exposure bias , interestingly , the existence or significance of exposure bias is much less studied . On the other hand , despite the criticism , MLE ( teacher forcing ) has remained to be the dominant objective for LM training ( Radford et al. , 2018 ; Keskar et al. , 2019 ) . To make the situation more curious , multiple recent works show that the proposed non-MLE methods actually have inferior generation performance ( Caccia et al. , 2018 ; de Masson d ’ Autume et al. , 2019 ) than the MLE baseline . These negative results lead us to question : Is exposure bias truly a serious problem for MLE training ? In this work we seek a direct answer to the above question . Here we briefly summarize our contributions : We conduct controlled experiments in which we remove the training-generation discrepancy in the prefix , and design various metrics to quantify the performance improvement of the generation as the prefix length grows . On the contrary to our expectation , our measurements consistently show that the performance gain is limited , and the incremental distortion as claimed by exposure bias is not observed ( the performance gap does not become larger with longer prefixes ) . In the next section , we begin by introducing notations and background . 2 PRELIMINARIES . The task of auto-regressive language modelling is to learn the probability distribution of the ( l + 1 ) th word ( or token ) Wl+1 in a sentence W conditioned on the prefix W1 : l : = ( W1 , . . . , Wl ) . We use Wi 2 V to denote a discrete random variable distributed across the vocabulary V . For simplicity , we assume all sentences are of length L in the formulations . Denoting the ground-truth data distribution as PD , the standard MLE training aims to minimize the negative log-likelihood ( NLL ) loss below : LMLE = E W⇠PD ⌃L 1 l=0 logP✓ ( Wl+1|W1 : l ) , ( 1 ) where P✓ ( · | W1 : l ) denotes the conditional distribution of Wl+1 of P✓ given a prefix W1 : l , and ✓ stands for the set of parameters to be trained . Note that the concept of “ sentence ” ( W ) can be naturally generalized to paragraphs or even articles , depending on the target task . We denote the distribution of a MLE-trained LM as PM , which is the major subject of this study . We will experiment with two popular model architectures : LSTM LM ( Hochreiter & Schmidhuber , 1997 ; Sundermeyer et al. , 2012 ) and transformer LM ( Baevski & Auli , 2018 ; Dai et al. , 2019 ) . For generation , we do classical ancestral sampling without invoking sampling algorithms such as top-k sampling ( Fan et al. , 2018 ) for the following reasons : ( 1 ) The sampling algorithms are known to trade quality out of diversity ( Nadeem et al. , 2020 ; Caccia et al. , 2018 ) . So , invoking them could “ hide ” the exposure bias problem because the prefixes from the model will be of higher quality . ( 2 ) The sampling algorithms requires tuning of hyper-parameters , which will complicate the comparison . In addition to popular measures in natural language generation ( NLG ) such as BLEU ( Papineni et al. , 2002 ) or METEOR ( Denkowski & Lavie , 2014 ) , our quantification approaches also rely on the measurements of the divergence between two distributions . Let P denote the set of probability distributions on the vocabulary V , and let fdiv : P ⇥ P ! R 0 be a divergence function between two distributions ( e.g. , total variation distance ) . We will adopt two popular probability divergence functions : total variation distance ( denoted as dTV ) and Jensen-Shannon divergence ( denoted as dJS ) . We provide definitions of dTV and dJS in Appendix A . Our experiments will be focused on the task of open-ended language generation , which is arguably a good test bed for exposure bias because of the following reasons : ( 1 ) The generation length is long . ( 2 ) Different from typical seq2seq tasks such as machine translation , the generation space is only weakly constrained and the topics can be very diverse , which means the training-generation discrepancy could be large . 3 A QUALITATIVE ATTEMPT . We begin with a qualitative attempt to verify the significance of exposure bias . We design a prefixswitching experiment as follows : We feed a MLE-trained transformer LM on the wiki-103 dataset with four types of prefixes of the same length : ( 1 ) test-data samples , ( 2 ) model ’ s own samples , ( 3 ) test-data samples shuffled on word-level , or ( 4 ) samples from a uniformly random distribution on V . Then we let the model continue the generation given these prefixes and compare the quality of the samples in a qualitative manner . We defer details of the model and dataset to Section 5 . The intuition behind the prefix-switching experiment follows immediately from the original claim of exposure bias : During generation , if we set the prefix distribution to be the ground-truth data distribution instead of the model ’ s own distribution , then the discrepancy between training and generation in the prefix is removed , and hence the model ’ s generation quality should be much better . In the extreme case of shuffled or random prefixes , due to the claim from exposure bias that the errors should accumulate , we expect the model to generate also badly distorted sequences . The samples with different types of prefixes are shown in Table 1 . To make the generation from data and model prefix more comparable , we force the same prompt at the beginning of the context to constrain the topic . Moreover , we intentionally use long prefixes of length 100 , in the hope that the incremental distortion of generation ( as claimed by exposure bias ) would become observable . We include another set of examples with the same transformer LM , and examples with a LSTM LM in Appendix B , which gives similar observations . On the contrary to our expectation , we do not observe a noticeable difference in sample quality comparing samples from model and data prefixes . More surprisingly , the model is still able to generate fairly good samples from shuffled prefixes . Even in the extreme case of random prefixes , we still observe basic language structures in the sample . This experiment suggests that the MLE-trained auto-regressive LMs have the self-recovery ability , i.e. , the model is able to recover from artificially distorted history input , and generate samples with reasonable quality . This phenomenon can not be explained by exposure bias which claims that the errors along the generation process should , on the contrary , accumulate . We conclude that this qualitative attempt fails to show the significance of exposure bias , indicating its impact could be more subtle than expected . In the following sections , we turn to more rigorous quantification methods to measure the impact of exposure bias . 4 QUANTIFICATION METHODS . Following the intuition of the prefix switching experiment , we design our quantification metrics to be a simple ratio of the relative performance gain when length-l data prefixes in fed to the model as opposed to the original model prefixes . And we compute the measurements for different prefix lengths . For a systematic assessment of exposure bias , we decompose the claim of exposure bias into two factors : ( 1 ) The discrepancy or mismatch in the prefix distribution would indeed hurt the generation performance , in general . ( 2 ) Moreover , the distortion should be incremental along the generation . The first factor can be reflected by the average magnitude of the measurements ( the values are expected to be larger than 1 by a meaningful margin ) , and the second factor can be reflected by whether the measurements are increasing along the prefix length . We now go into the detailed definitions of our metrics . We attempt to quantify the impact of exposure bias on three key aspects of open-ended language generation : quality , diversity , and consistency . We first introduce EB-M , which covers the quality and diversity aspects , and then EB-C , which covers the consistency aspect . 4.1 DEFINITION OF EB-M . In this section , we propose the EB-M metric . Since the key idea is to compare the generation quality with different types of prefixes , denoting the prefix distribution as PH 2 { PM , PD } ( model or data prefixes ) , we first formalize the following generation process : • Given a prefix length l and a prefix distribution PH , we sample W1 : l from PH . • Conditioned on the prefix W1 : l , we sample Wl+1 : l+lgen from PM , where Wl+j is sampled from PM ( · | W1 : l+j 1 ) with j > 0 , and lgen is the length of generation . We denote the marginal distribution of Wl+1 : l+lgen of the above random process as P Wl+1 : l+lgen M |H . If exposure bias is indeed serious , we expect the quality or diversity of Wl+1 : l+lgen to be better when PD is used as PH than PM . In our experiments we fix lgen to be 30 , and vary the prefix length l. With these ingredients in hand , we now propose the EB-M quantification for exposure bias . It reflects the relative performance gain when the length-l prefix is from PD instead of from PM , and is formulated as below : EB-M ( M , l , fscore ) = fscore ( P Wl+1 : l+lgen M |D , P Wl+1 : l+lgen D ) fscore ( P Wl+1 : l+lgen M |M , P Wl+1 : l+lgen D ) , ( 2 ) where fscore is a pre-defined scoring function1 of the generation samples , and we assume higher value of fscore indicates that the generation is of higher quality or diversity . In our experiments , we will use popular NLG metrics including BLEU ( Yu et al. , 2016 ; Caccia et al. , 2018 ) / Nist ( Doddington , 2002 ) / METEOR ( Denkowski & Lavie , 2014 ) , which mainly capture the quality aspect , and backwardBLEU ( Shi et al. , 2018 ) / n-gram entropy ( Zhang et al. , 2018 ) , which capture the diversity aspect . We will show that the observations from different metrics are consistent . EB-M has a potential weakness that it doesn ’ t reflect how the generation is consistent with the given prefix W1 : l , because it only focuses on the marginal distribution of Wl+1 : l+lgen . To cover this shortcoming , in the next section we propose another quantification method named EB-C , which focuses on the model ’ s conditional generation distribution of Wl+1 given a prefix W1 : l .

Summary: The paper studies the exposure bias in auto-regressive neural language models. This problem is known to cause incremental performance degradation, and attempts to mitigate this problem have received significant attention in the community (using, e.g., RL and GANs). The paper claims that prior work has mostly focused on addressing the problem rather than measuring how severe the exposure bias problem actually is. Despite extensive previous work on mitigating exposure bias, the paper suggests that the exposure bias is not “large enough [to] induce drastic performance loss during generation” (e.g., a human evaluation controlling for exposure bias show relative differences of only < 3%).