The Curious Case of Neural Text Degeneration

Context : In a shocking finding , scientist discovered a herd of unicorns living in a remote , previously unexplored valley , in the Andes Mountains . Even more surprising to the researchers was the fact that the unicorns spoke perfect English . Beam Search , b=32 : `` The study , published in the Proceedings of the National Academy of Sciences of the United States of America ( PNAS ) , was conducted by researchers from the Universidad Nacional Autónoma de México ( UNAM ) and the Universidad Nacional Autónoma de México ( UNAM/Universidad Nacional Autónoma de México/Universidad Nacional Autónoma de México/Universidad Nacional Autónoma de México/Universidad Nacional Autónoma de ... ” Pure Sampling : They were cattle called Bolivian Cavalleros ; they live in a remote desert uninterrupted by town , and they speak huge , beautiful , paradisiacal Bolivian linguistic thing . They say , 'Lunch , marge . ' They do n't tell what the lunch is , '' director Professor Chuperas Omwell told Sky News . `` They 've only been talking to scientists , like we 're being interviewed by TV reporters . We do n't even stick around to be interviewed by TV reporters . Maybe that 's how they figured out that they 're cosplaying as the Bolivian Cavalleros . '' Figure 1 : Even with substantial human context and the powerful GPT-2 Large language model , Beam Search ( size 32 ) leads to degenerate repetition ( highlighted in blue ) while pure sampling leads to incoherent gibberish ( highlighted in red ) . When b ≥ 64 , both GPT-2 Large and XL ( 774M and 1542M parameters , respectively ) prefer to stop generating immediately after the given context . 1 INTRODUCTION . On February 14th 2019 , OpenAI surprised the scientific community with an impressively highquality article about Ovid ’ s Unicorn , written by GPT-2.1 Notably , the top-quality generations ob- 1https : //openai.com/blog/better-language-models/ tained from the model rely on randomness in the decoding method , in particular through top-k sampling that samples the next word from the top k most probable choices ( Fan et al. , 2018 ; Holtzman et al. , 2018 ; Radford et al. , 2019 ) , instead of aiming to decode text that maximizes likelihood . In fact , decoding strategies that optimize for output with high probability , such as beam search , lead to text that is incredibly degenerate , even when using state-of-the-art models such as GPT-2 Large , as shown in Figure 1 . This may seem counter-intuitive , as one would expect that good models would assign higher probability to more human-like , grammatical text . Indeed , language models do generally assign high scores to well-formed text , yet the highest scores for longer texts are often generic , repetitive , and awkward . Figure 2 exposes how different the distribution of probabilities assigned to beam search decoded text and naturally occurring text really are . Perhaps equally surprising is the right side of Figure 1 , which shows that pure sampling — sampling directly from the probabilities predicted by the model — results in text that is incoherent and almost unrelated to the context . Why is text produced by pure sampling so degenerate ? In this work we show that the “ unreliable tail ” is to blame . This unreliable tail is composed of tens of thousands of candidate tokens with relatively low probability that are over-represented in the aggregate . To overcome these issues we introduce Nucleus Sampling ( §3.1 ) . The key intuition of Nucleus Sampling is that the vast majority of probability mass at each time step is concentrated in the nucleus , a small subset of the vocabulary that tends to range between one and a thousand candidates . Instead of relying on a fixed top-k , or using a temperature parameter to control the shape of the distribution without sufficiently suppressing the unreliable tail , we propose sampling from the top-p portion of the probability mass , expanding and contracting the candidate pool dynamically . In order to compare current methods to Nucleus Sampling , we compare various distributional properties of generated text to the reference distribution , such as the likelihood of veering into repetition and the perplexity of generated text . The latter reveals that text generated by maximization or top-k sampling is too probable , indicating a lack of diversity and divergence in vocabulary usage from the human distribution . On the other hand , pure sampling produces text that is significantly less likely than the gold , corresponding to lower generation quality . Vocabulary usage and Self-BLEU ( Zhu et al. , 2018 ) statistics reveal that high values of k are needed to make top-k sampling match human statistics . Yet , generations based on high values of k often have high variance in likelihood , hinting at qualitatively observable incoherency issues . Nucleus Sampling can easily match reference perplexity through tuning the value of p , avoiding the incoherence caused by setting k high enough to match distributional statistics . Finally , we perform Human Unified with Statistical Evaluation ( HUSE ; Hashimoto et al. , 2019 ) to jointly assess the overall quality and diversity of the decoding strategies , which can not be captured using either human or automatic evaluation alone . The HUSE evaluation demonstrates that Nucleus Sampling is the best overall decoding strategy . We include generated examples for qualitative analysis – see Figure 3 for a representative example , and further examples in the appendix.2 2Code and all generations are available at https : //github.com/ari-holtzman/degen 2 BACKGROUND . 2.1 TEXT GENERATION DECODING STRATEGIES . A number of recent works have alluded to the disadvantages of generation by maximization , which tend to generate output with high grammaticality but low diversity ( Kulikov et al. , 2019 ; Holtzman et al. , 2018 ; Fan et al. , 2018 ) . Generative Adversarial Networks ( GANs ) have been a prominent research direction ( Yu et al. , 2017 ; Xu et al. , 2018 ) , but recent work has shown that when quality and diversity are considered jointly , GAN-generated text fails to outperform generations from language models ( Caccia et al. , 2018 ; Tevet et al. , 2019 ; Semeniuta et al. , 2018 ) . Work on neural dialog systems have proposed methods for diverse beam search , using a task-specific diversity scoring function or constraining beam hypotheses to be sufficiently different ( Li et al. , 2016a ; Vijayakumar et al. , 2018 ; Kulikov et al. , 2019 ; Pal et al. , 2006 ) . While such utility functions encourage desirable properties in generations , they do not remove the need to choose an appropriate decoding strategy , and we believe that Nucleus Sampling will have complementary advantages in such approaches . Finally , Welleck et al . ( 2020 ) begin to address the problem of neural text degeneration through an “ unlikelihood loss ” , which decreases training loss on repeated tokens and thus implicitly reduces gradients on frequent tokens as well . Our focus is on exposing neural text degeneration and providing a decoding solution that can be used with arbitrary models , but future work will likely combine training-time and inference-time solutions . 2.2 OPEN-ENDED VS DIRECTED GENERATION . Many text generation tasks are defined through ( input , output ) pairs , such that the output is a constrained transformation of the input . Example applications include machine translation ( Bahdanau et al. , 2015 ) , data-to-text generation ( Wiseman et al. , 2017 ) , and summarization ( Nallapati et al. , 2016 ) . We refer to these tasks as directed generation . Typically encoder-decoder architectures are used , often with an attention mechanism ( Bahdanau et al. , 2015 ; Luong et al. , 2015 ) or using attention-based architectures such as the Transformer ( Vaswani et al. , 2017 ) . Generation is usually performed using beam search ; since output is tightly scoped by the input , repetition and genericness are not as problematic . Still , similar issues have been reported when using large beam sizes ( Koehn & Knowles , 2017 ) and more recently with exact inference ( Stahlberg & Byrne , 2019 ) , a counter-intuitive observation since more comprehensive search helps maximize probability . Open-ended generation , which includes conditional story generation and contextual text continuation ( as in Figure 1 ) , has recently become a promising research direction due to significant advances in neural language models ( Clark et al. , 2018 ; Holtzman et al. , 2018 ; Fan et al. , 2018 ; Peng et al. , 2018 ; Radford et al. , 2019 ) . While the input context restricts the space of acceptable output generations , there is a considerable degree of freedom in what can plausibly come next , unlike in directed generation settings . Our work addresses the challenges faced by neural text generation with this increased level of freedom , but we note that some tasks , such as goal-oriented dialog , may fall somewhere in between open-ended and directed generation . 3 LANGUAGE MODEL DECODING . Given an input text passage as context , the task of open-ended generation is to generate text that forms a coherent continuation from the given context . More formally , given a sequence of m tokens x1 . . . xm as context , the task is to generate the next n continuation tokens to obtain the completed sequence x1 . . . xm+n . We assume that models compute P ( x1 : m+n ) using the common left-to-right decomposition of the text probability , P ( x1 : m+n ) = m+n∏ i=1 P ( xi|x1 . . . xi−1 ) , ( 1 ) which is used to generate the generation token-by-token using a particular decoding strategy . Maximization-based decoding The most commonly used decoding objective , in particular for directed generation , is maximization-based decoding . Assuming that the model assigns higher probability to higher quality text , these decoding strategies search for the continuation with the highest An unprecedented number of mostly young whales have become stranded on the West Australian coast since 2008 .. likelihood . Since finding the optimum argmax sequence from recurrent neural language models or Transformers is not tractable ( Chen et al. , 2018 ) , common practice is to use beam search ( Li et al. , 2016b ; Shen et al. , 2017 ; Wiseman et al. , 2017 ) . However , several recent studies on open-ended generation have reported that maximization-based decoding does not lead to high quality text ( Fan et al. , 2018 ; Holtzman et al. , 2018 ) . 3.1 NUCLEUS SAMPLING . We propose a new stochastic decoding method : Nucleus Sampling . The key idea is to use the shape of the probability distribution to determine the set of tokens to be sampled from . Given a distribution P ( x|x1 : i−1 ) , we define its top-p vocabulary V ( p ) ⊂ V as the smallest set such that ∑ x∈V ( p ) P ( x|x1 : i−1 ) ≥ p. ( 2 ) Let p′ = ∑ x∈V ( p ) P ( x|x1 : i−1 ) . The original distribution is re-scaled to a new distribution , from which the next word is sampled : P ′ ( x|x1 : i−1 ) = { P ( x|x1 : i−1 ) /p′ if x ∈ V ( p ) 0 otherwise . ( 3 ) In practice this means selecting the highest probability tokens whose cumulative probability mass exceeds the pre-chosen threshold p. The size of the sampling set will adjust dynamically based on the shape of the probability distribution at each time step . For high values of p , this is a small subset of vocabulary that takes up vast majority of the probability mass — the nucleus . 3.2 TOP-k SAMPLING Top-k sampling has recently become a popular alternative sampling procedure ( Fan et al. , 2018 ; Holtzman et al. , 2018 ; Radford et al. , 2019 ) . Nucleus Sampling and top-k both sample from truncated Neural LM distributions , differing only in the strategy of where to truncate . Choosing where to truncate can be interpreted as determining the generative model ’ s trustworthy prediction zone . At each time step , the top k possible next tokens are sampled from according to their relative probabilities . Formally , given a distribution P ( x|x1 : i−1 ) , we define its top-k vocabulary V ( k ) ⊂ V as the set of size k which maximizes ∑ x∈V ( k ) P ( x|x1 : i−1 ) . Let p′ = ∑ x∈V ( k ) P ( x|x1 : i−1 ) . The distribution is then re-scaled as in equation 3 , and sampling is performed based on that distribution . Note that the scaling factor p′ can vary wildly at each time-step , in contrast to Nucleus Sampling . Difficulty in choosing a suitable value of k While top-k sampling leads to considerably higher quality text than either beam search or sampling from the full distribution , the use of a constant k is sub-optimal across varying contexts . As illustrated on the left of Figure 5 , in some contexts the head of the next word distribution can be flat across tens or hundreds of reasonable options ( e.g . nouns or verbs in generic contexts ) , while in other contexts most of the probability mass is concentrated in one or a small number of tokens , as on the right of the figure . Therefore if k is small , in some contexts there is a risk of generating bland or generic text , while if k is large the top-k vocabulary will include inappropriate candidates which will have their probability of being sampled increased by the renormalization . Under Nucleus Sampling , the number of candidates considered rises and falls dynamically , corresponding to the changes in the model ’ s confidence region over the vocabulary which top-k sampling fails to capture for any one choice of k .

Summary: This paper studies an important problem, i.e., how to find a good decoding strategy for open-ended text generation. To this end, the authors provide a deep analysis of the most common decoding methods, and propose Nucleus Sampling, a very simple yet effective method to generate higher-quality text. Compared with top-k sampling, the key idea behind the proposed method is to sample from the dynamic nucleus of tokens containing the majority of the probability mass. Experiments demonstrate that nucleus sampling is an effective decoding strategy in practice.