The Curious Case of Neural Text Degeneration

Context : In a shocking finding , scientist discovered a herd of unicorns living in a remote , previously unexplored valley , in the Andes Mountains . Even more surprising to the researchers was the fact that the unicorns spoke perfect English . Beam Search , b=32 : `` The study , published in the Proceedings of the National Academy of Sciences of the United States of America ( PNAS ) , was conducted by researchers from the Universidad Nacional Autónoma de México ( UNAM ) and the Universidad Nacional Autónoma de México ( UNAM/Universidad Nacional Autónoma de México/Universidad Nacional Autónoma de México/Universidad Nacional Autónoma de México/Universidad Nacional Autónoma de ... ” Pure Sampling : They were cattle called Bolivian Cavalleros ; they live in a remote desert uninterrupted by town , and they speak huge , beautiful , paradisiacal Bolivian linguistic thing . They say , 'Lunch , marge . ' They do n't tell what the lunch is , '' director Professor Chuperas Omwell told Sky News . `` They 've only been talking to scientists , like we 're being interviewed by TV reporters . We do n't even stick around to be interviewed by TV reporters . Maybe that 's how they figured out that they 're cosplaying as the Bolivian Cavalleros . '' Figure 1 : Even with substantial human context and the powerful GPT-2 Large language model , Beam Search ( size 32 ) leads to degenerate repetition ( highlighted in blue ) while pure sampling leads to incoherent gibberish ( highlighted in red ) . When b ≥ 64 , both GPT-2 Large and XL ( 774M and 1542M parameters , respectively ) prefer to stop generating immediately after the given context . 1 INTRODUCTION . On February 14th 2019 , OpenAI surprised the scientific community with an impressively highquality article about Ovid ’ s Unicorn , written by GPT-2.1 Notably , the top-quality generations ob- 1https : //openai.com/blog/better-language-models/ tained from the model rely on randomness in the decoding method , in particular through top-k sampling that samples the next word from the top k most probable choices ( Fan et al. , 2018 ; Holtzman et al. , 2018 ; Radford et al. , 2019 ) , instead of aiming to decode text that maximizes likelihood . In fact , decoding strategies that optimize for output with high probability , such as beam search , lead to text that is incredibly degenerate , even when using state-of-the-art models such as GPT-2 Large , as shown in Figure 1 . This may seem counter-intuitive , as one would expect that good models would assign higher probability to more human-like , grammatical text . Indeed , language models do generally assign high scores to well-formed text , yet the highest scores for longer texts are often generic , repetitive , and awkward . Figure 2 exposes how different the distribution of probabilities assigned to beam search decoded text and naturally occurring text really are . Perhaps equally surprising is the right side of Figure 1 , which shows that pure sampling — sampling directly from the probabilities predicted by the model — results in text that is incoherent and almost unrelated to the context . Why is text produced by pure sampling so degenerate ? In this work we show that the “ unreliable tail ” is to blame . This unreliable tail is composed of tens of thousands of candidate tokens with relatively low probability that are over-represented in the aggregate . To overcome these issues we introduce Nucleus Sampling ( §3.1 ) . The key intuition of Nucleus Sampling is that the vast majority of probability mass at each time step is concentrated in the nucleus , a small subset of the vocabulary that tends to range between one and a thousand candidates . Instead of relying on a fixed top-k , or using a temperature parameter to control the shape of the distribution without sufficiently suppressing the unreliable tail , we propose sampling from the top-p portion of the probability mass , expanding and contracting the candidate pool dynamically . In order to compare current methods to Nucleus Sampling , we compare various distributional properties of generated text to the reference distribution , such as the likelihood of veering into repetition and the perplexity of generated text . The latter reveals that text generated by maximization or top-k sampling is too probable , indicating a lack of diversity and divergence in vocabulary usage from the human distribution . On the other hand , pure sampling produces text that is significantly less likely than the gold , corresponding to lower generation quality . Vocabulary usage and Self-BLEU ( Zhu et al. , 2018 ) statistics reveal that high values of k are needed to make top-k sampling match human statistics . Yet , generations based on high values of k often have high variance in likelihood , hinting at qualitatively observable incoherency issues . Nucleus Sampling can easily match reference perplexity through tuning the value of p , avoiding the incoherence caused by setting k high enough to match distributional statistics . Finally , we perform Human Unified with Statistical Evaluation ( HUSE ; Hashimoto et al. , 2019 ) to jointly assess the overall quality and diversity of the decoding strategies , which can not be captured using either human or automatic evaluation alone . The HUSE evaluation demonstrates that Nucleus Sampling is the best overall decoding strategy . We include generated examples for qualitative analysis – see Figure 3 for a representative example , and further examples in the appendix.2 2Code and all generations are available at https : //github.com/ari-holtzman/degen 2 BACKGROUND . 2.1 TEXT GENERATION DECODING STRATEGIES . A number of recent works have alluded to the disadvantages of generation by maximization , which tend to generate output with high grammaticality but low diversity ( Kulikov et al. , 2019 ; Holtzman et al. , 2018 ; Fan et al. , 2018 ) . Generative Adversarial Networks ( GANs ) have been a prominent research direction ( Yu et al. , 2017 ; Xu et al. , 2018 ) , but recent work has shown that when quality and diversity are considered jointly , GAN-generated text fails to outperform generations from language models ( Caccia et al. , 2018 ; Tevet et al. , 2019 ; Semeniuta et al. , 2018 ) . Work on neural dialog systems have proposed methods for diverse beam search , using a task-specific diversity scoring function or constraining beam hypotheses to be sufficiently different ( Li et al. , 2016a ; Vijayakumar et al. , 2018 ; Kulikov et al. , 2019 ; Pal et al. , 2006 ) . While such utility functions encourage desirable properties in generations , they do not remove the need to choose an appropriate decoding strategy , and we believe that Nucleus Sampling will have complementary advantages in such approaches . Finally , Welleck et al . ( 2020 ) begin to address the problem of neural text degeneration through an “ unlikelihood loss ” , which decreases training loss on repeated tokens and thus implicitly reduces gradients on frequent tokens as well . Our focus is on exposing neural text degeneration and providing a decoding solution that can be used with arbitrary models , but future work will likely combine training-time and inference-time solutions . 2.2 OPEN-ENDED VS DIRECTED GENERATION . Many text generation tasks are defined through ( input , output ) pairs , such that the output is a constrained transformation of the input . Example applications include machine translation ( Bahdanau et al. , 2015 ) , data-to-text generation ( Wiseman et al. , 2017 ) , and summarization ( Nallapati et al. , 2016 ) . We refer to these tasks as directed generation . Typically encoder-decoder architectures are used , often with an attention mechanism ( Bahdanau et al. , 2015 ; Luong et al. , 2015 ) or using attention-based architectures such as the Transformer ( Vaswani et al. , 2017 ) . Generation is usually performed using beam search ; since output is tightly scoped by the input , repetition and genericness are not as problematic . Still , similar issues have been reported when using large beam sizes ( Koehn & Knowles , 2017 ) and more recently with exact inference ( Stahlberg & Byrne , 2019 ) , a counter-intuitive observation since more comprehensive search helps maximize probability . Open-ended generation , which includes conditional story generation and contextual text continuation ( as in Figure 1 ) , has recently become a promising research direction due to significant advances in neural language models ( Clark et al. , 2018 ; Holtzman et al. , 2018 ; Fan et al. , 2018 ; Peng et al. , 2018 ; Radford et al. , 2019 ) . While the input context restricts the space of acceptable output generations , there is a considerable degree of freedom in what can plausibly come next , unlike in directed generation settings . Our work addresses the challenges faced by neural text generation with this increased level of freedom , but we note that some tasks , such as goal-oriented dialog , may fall somewhere in between open-ended and directed generation . 3 LANGUAGE MODEL DECODING . Given an input text passage as context , the task of open-ended generation is to generate text that forms a coherent continuation from the given context . More formally , given a sequence of m tokens x1 . . . xm as context , the task is to generate the next n continuation tokens to obtain the completed sequence x1 . . . xm+n . We assume that models compute P ( x1 : m+n ) using the common left-to-right decomposition of the text probability , P ( x1 : m+n ) = m+n∏ i=1 P ( xi|x1 . . . xi−1 ) , ( 1 ) which is used to generate the generation token-by-token using a particular decoding strategy . Maximization-based decoding The most commonly used decoding objective , in particular for directed generation , is maximization-based decoding . Assuming that the model assigns higher probability to higher quality text , these decoding strategies search for the continuation with the highest An unprecedented number of mostly young whales have become stranded on the West Australian coast since 2008 .. likelihood . Since finding the optimum argmax sequence from recurrent neural language models or Transformers is not tractable ( Chen et al. , 2018 ) , common practice is to use beam search ( Li et al. , 2016b ; Shen et al. , 2017 ; Wiseman et al. , 2017 ) . However , several recent studies on open-ended generation have reported that maximization-based decoding does not lead to high quality text ( Fan et al. , 2018 ; Holtzman et al. , 2018 ) . 3.1 NUCLEUS SAMPLING . We propose a new stochastic decoding method : Nucleus Sampling . The key idea is to use the shape of the probability distribution to determine the set of tokens to be sampled from . Given a distribution P ( x|x1 : i−1 ) , we define its top-p vocabulary V ( p ) ⊂ V as the smallest set such that ∑ x∈V ( p ) P ( x|x1 : i−1 ) ≥ p. ( 2 ) Let p′ = ∑ x∈V ( p ) P ( x|x1 : i−1 ) . The original distribution is re-scaled to a new distribution , from which the next word is sampled : P ′ ( x|x1 : i−1 ) = { P ( x|x1 : i−1 ) /p′ if x ∈ V ( p ) 0 otherwise . ( 3 ) In practice this means selecting the highest probability tokens whose cumulative probability mass exceeds the pre-chosen threshold p. The size of the sampling set will adjust dynamically based on the shape of the probability distribution at each time step . For high values of p , this is a small subset of vocabulary that takes up vast majority of the probability mass — the nucleus . 3.2 TOP-k SAMPLING Top-k sampling has recently become a popular alternative sampling procedure ( Fan et al. , 2018 ; Holtzman et al. , 2018 ; Radford et al. , 2019 ) . Nucleus Sampling and top-k both sample from truncated Neural LM distributions , differing only in the strategy of where to truncate . Choosing where to truncate can be interpreted as determining the generative model ’ s trustworthy prediction zone . At each time step , the top k possible next tokens are sampled from according to their relative probabilities . Formally , given a distribution P ( x|x1 : i−1 ) , we define its top-k vocabulary V ( k ) ⊂ V as the set of size k which maximizes ∑ x∈V ( k ) P ( x|x1 : i−1 ) . Let p′ = ∑ x∈V ( k ) P ( x|x1 : i−1 ) . The distribution is then re-scaled as in equation 3 , and sampling is performed based on that distribution . Note that the scaling factor p′ can vary wildly at each time-step , in contrast to Nucleus Sampling . Difficulty in choosing a suitable value of k While top-k sampling leads to considerably higher quality text than either beam search or sampling from the full distribution , the use of a constant k is sub-optimal across varying contexts . As illustrated on the left of Figure 5 , in some contexts the head of the next word distribution can be flat across tens or hundreds of reasonable options ( e.g . nouns or verbs in generic contexts ) , while in other contexts most of the probability mass is concentrated in one or a small number of tokens , as on the right of the figure . Therefore if k is small , in some contexts there is a risk of generating bland or generic text , while if k is large the top-k vocabulary will include inappropriate candidates which will have their probability of being sampled increased by the renormalization . Under Nucleus Sampling , the number of candidates considered rises and falls dynamically , corresponding to the changes in the model ’ s confidence region over the vocabulary which top-k sampling fails to capture for any one choice of k .

Summary: This paper is motivated by an observation that maximization-based decoding approaches such as beam search can lead to incoherent and repetitive sentences when open-ended long-form text generation based on neural language model such as GPT-2 is performed. To solve the problem, this paper proposes a sampling method called Nucleus Sampling. Similar to Top-k sampling, Nucleus Sampling truncates the probability distribution of the words in the vocabulary. Instead of re-normalizing the probabilities for the top-k words, Nucleus Sampling re-normalizes the original probabilities for the words with values above a pre-chosen threshold p. Some quantitative and qualitative results show that the proposed sampling method can generate long-form texts with some nice properties.