The Curious Case of Neural Text Degeneration
Authors: Ari Holtzman
Jan Buys
Li Du
Maxwell Forbes
Yejin Choi
Venue:  (2020)
Keywords: generation
text
NLG
NLP
natural language
natural language generation
language model
neural
neural language model

Abstract: Despite considerable advances in neural language modeling, it remains an open question what the best decoding strategy is for text generation from a language model (e.g. to generate a story). The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, maximization-based decoding methods such as beam search lead to degeneration — output text that is bland, incoherent, or gets stuck in repetitive loops.

To address this we propose Nucleus Sampling, a simple but effective method to draw considerably higher quality text out of neural language models than previous decoding strategies. Our approach avoids text degeneration by truncating the unreliable tail of the probability distribution, sampling from the dynamic nucleus of tokens containing the vast majority of the probability mass.

To properly examine current maximization-based and stochastic decoding methods, we compare generations from each of these methods to the distribution of human text along several axes such as likelihood, diversity, and repetition. Our results show that (1) maximization is an inappropriate decoding objective for open-ended text generation, (2) the probability distributions of the best current language models have an unreliable tail which needs to be truncated during generation and (3) Nucleus Sampling is currently the best available decoding strategy for generating long-form text that is both high-quality — as measured by human evaluation — and as diverse as human-written text.

Published as a conference paper at ICLR 2020
THE CURIOUS CASE OF NEURAL TEXT DeGENERATION
Ari Holtzman
Jan Buysst
Du
Maxwell Forbestt
Yejin Choi
Paul G. Allen School of Computer Science & Engineering, University of Washington
$ Allen Institute for Artificial Intelligence
$ Department of Computer Science, University of Cape Town (ahai, dul2, mbforbes, yejiing fic rashington.eddu jbuys@cs. ac..
ABSTRACT
Despite considerable advances in neural language modeling, it remains an open question what the best decoding strategy is for text generation from language model (e.g. to generate a story). The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, maximization based decoding methods such beam search lead to degeneration output text that is bland, incoherent, or gets stuck in repetitive loops.
To address this we propose Nucleus Sampling, simple but effective method to draw considerably higher quality text out of neural language models than previ- ous decoding strategies. Our approach avoids text degeneration by truncating the unreliable tail of the probability distribution, sampling from the dynamic nucleus of tokens containing the vast majority of the probability mass.
To properly examine current maximization- based and stochastic decoding meth- ods, we compare generations from each of these methods to the distribution human text along several axes such as likelihood, diversity, and repetition. Our re- sults show that (1) maximization is an inappropriate decoding objective for open- ended text generation, (2) the probability distributions of the best current language models have an unreliable tail which needs to be truncated during generation and (3) Nucleus Sampling currently the best available decoding strategy for gener- ating long-form text that is both high-quality as measured by human evaluation and as diverse as
text.
Context: In shocking finding, scientist discovered herd of unicorns living in remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicoms spoke perfect English.
Beam Search, b=32:
Pure Sampling:
"The study, published in the Proceedings of the National Acaclemy of Sciences of the United States of America (PNAS), was conducted by researchers from the Universidad Nacional Autónoma de México (UNAM) and the Universidad Nacional Autónoma de México JJNAM/Univere Nacional Autónoma de México/ Universidad Nacional Autónoma de México/Univee Universidad Nacional Auténoma de México/Universicon Nacional Autónoma de ..."
They were cattle called Balivian Cavalleras; they live in
remote desert uninterrpteed by town, and they speak huge, beautiful, paradisiacal Bolivian linguistic thing. They say, 'Lunch, marge. They don't tall what the lunch is," director Professor Chuperas Omwell told Sky News. "They vo only TV been talking scientists, like we re being interviewed reporters. We don't it even stick around to be interviewed by TV reporters. Mayoe that's how they figured out that they cosplaying as the Bolivian Cavalleros."
Even with substantial human context and the powerful GPT-2 Large language model, Beam Search (size 32) leads to degenerate repetition (highlighted in blue) while pure sampling leads to incoherent gibberish (highlighted in red). When b > 64, both GPT-2 Large and XL (774M and 1542M parameters, respectively) prefer to stop generating immediately after the given context.
Figure 1:
INTRODUCTION
On February 14th 2019, OpenAl surprised the scientific community with an impressively high- quality article about Ovid's Unicorn, written by GPT-2 Notably, the top-quality generations ob-

---

Published as a conference paper at ICLR 2020
tained from the model rely on randomuness in the decoding method, in particular through t sam- pling that samples the next word from the top k most probable choices (Fan et al. 2018 Holtzman et al. 2018 Radford et al., 2019), instead of aiming to decode text that maximizes likelihood.
In fact, decoding strategies that optimize for output with high probability, such as beam search, lead to text that is incredibly degenerate, even when using state-of-the-an art models such GPT-2 Large, as shown in Figure [ This may seem counter-intuitive, as one would expect that good models would assign higher probability
more human-like, grammatical text. Indeed, language models do generally assign high scores to well-formed text, yet the highes/ scores for longer texts are often generic, repetitive, and awkward. Figure 2 exposes how different the distribution of probabilities assigned to beam search decoded text and naturally occurring text really are.
Perhaps equally surprising is the right side of Figure which shows that pure sampling - sampling directly from the probabilities predicted by the model results in text that is incoherent and almost unrelated to the context. Why is text produced by pure sampling so degenerate? In this work we show that the "unreliable tail"" is to blame. This unreliable tail is composed of tens of thousands of candidate tokens with relatively low probability that are over-represented the aggregate.
To overcome these issues we introduce Nucleus Sampling 33.1 The key intuition of Nucleus Sampling that the vast majority of probability mass at each time step concentrated in the nucleus, small subset of the vocabulary that tends to range between one and thousand candidates. Instead of relying on a fixed top-k, or using a temperature parameter to control the shape of the distribution without sufficiently suppressing the unreliable tail, we propose sampling from the top-p portion of the probability mass, expanding and contracting the candidate pool dynamically.
In order to compare current methods to Nucleus Sampling, we compare vari- ous distributional properties of generated text to the reference distribution, such as the likelihood of veering into repetition and the perplexity of generated text. The latter reveals that text generated by maxi- mization or top-k sampling too probable, in- dicating a lack of diversity and divergence in vocabulary usage from the human distribution. On the other hand, pure sampling produces text
Beam Search Text Less Surprising
0.8
Probabiity
that is significantly less likely than the gold, corresponding to lower generation quality.
M
0.
Vocabulary usage and Self-BLEU (Zhu et al. 2018) statistics reveal that high values of l are needed to make top-k sampling match human statistics. Yet, generations based high val- ues of & often have high variance in likelihood, hinting at qualitatively observable incoherency issues. Nucleus Sampling can easily match ref- erence perplexity through tuning the value of avoiding the incoherence caused by setting k high enough match distributional statistics.
100
Tirastop
Bram Smarch
Humaan
Beam Search
Human
provide overview of the current ttate-g--hh-srt the field
...which grant incressed life
end three years warranty. The
computer vision and machine learning,
Antec HCG series cons'sts of five
to provide overview of the the field of
models with capacilies spanning
from 40DW to 902W. Here have already
state oa---oo-hheeee
should di note tested the HCG-624 -20-620 previcus quito satisfied today's review will rigorouzly its model number implies, has 520W capacity contraly Antac's strang ballefs mult-rail PSUs equipped...
computer vision and machine to provide overview of the state of
and With performance
the field of computer vision and machine to provide overview of the
the
Finally, we perform Human Unified with Sta- tistical Evaluation (HUSE: Hashimoto et al. 2019) to jointly assess the overall quality and diversity of the decoding strategies, which can- not be captured using either human or auto- matic evaluation alone. The HUSE evaluation demonstrates that Nucleus Sampling is the best overall decoding strategy. We include gener- ated examples for qualitative analysis- see Fig- ure 33 for a representative example, and further examples in the appendix
Antec HCG-520 520,
of the the fiold of computer vision and machine learning,
Figure 2: The probability assigned to tokens gen- erated by Beam Search and humans, given the same context. Note the increased variance that characterizes human text, in contrast with the end- less repetition of text decoded by Beam Search.
"Code and all generations are available attttps://gtthbbbccm com wtwooo/ttmm//e/eeeee
2

---

Published as a conference paper at ICLR 2020
2 BACKGROUND
2.1 TEXT GENERATION DECODING STRATEGIES
A number of recent works have alluded the disadvantages of generation by maximization, which lend to generate output with high grammaticality but low diversity (Kulikov et al. 2019 Holtzman al. 2018 Fan al., 2018). Generative Adversarial Networks (GANs) have been prominent research direction (Yu et al. 2017 Xu et al. 2018), but recent work has shown that when qual- ity and diversity are considered joinly, GAN generated text fails outperform generations from language models (Caccia et al. 2018 Tevet et al. 2019: Semeniuta et al. 2018). Work on neural di- alog systems have proposed methods for diverse beam search, using a task-specific diversity scoring function or constraining beam hypotheses to be sufficiently different (Li et al.) [20 Vijayakumar itt al. 2018, Kulikov et al., 2019; Pal et al. (206). While such utility functions encourage desirable properties generations, they do not remove the need to choose an appropriate decoding strategy, and we believe that Nucleus Sampling will have complementary advantages in such approaches. Finally, Welleck et al.] (2020) begin to address the problem of neural text degeneration through an "unlikelihood loss", which decreases training loss on repeated tokens and thus implicitly reduces gradients on frequent tokens as well. Our focus is on exposing neural text degeneration and provid- ing a decoding solution that can be used with arbitrary models, but future work will likely combine training- time and inference-tine solutions.
2.2 OPEN-ENDED DIRECTED GENERATION
Many text generation tasks are defined through (input, output) pairs, such that the output is a con- strained transformation of the input. Example applications include machine translation (Bahdanau et al. 2015), data-to-text generation (Wiseman et al. 2017). and summarization (Nallapati et al., 2016). We refer to these tasks as directed generation. Typically encoder-decoder architectures used, often with an attention mechanism (Bahdanau et al.) 2015: Luong et al. 2015) or using iatention-based architectures such as the Transformer (Vaswani et al. 2017). Generation is usually performed using beam search; since output is tightly scoped by the input, repetition and generic- ness are not as problematic. Still, similar issues have been reported when using large beam sizes (Koehn & Knowles 2017) and more recently with exact inference (Stahlberg & Byrne 2019), counter-intuitive observation since more comprehensive search helps maximize probability.
Open-ended generation, which includes conditional story generation and contextual text continua- tion (as in Figure|] I]. has recently become a promising research direction due to significant advances in neural language models (Clark et al. 2018 Holtzman et al.) 2018 Fan al. 2018 Peng et al. 2018 Radford et al., 2019). While the input context restricts the space of acceptable output genera- tions, there considerable degree of freedom in what can plausibly come next, unlike in directed generation settings. Our work addresses the challenges faced by neural text generation with this increased level of freedom, but we note that some tasks, such as goal-oriented dialog, may fall somewhere in between open-ended and directed generation.
3 LANGUAGE MODEL DECODING
Given an input text passage as context, the task of open-ended generation is to generate text that forms coherent continuation from the given context. More formally, given a sequence of m tokens #1 ... as context, the task is to generate the next n continuation tokens to obtain the completed sequence 31 ... Tn We assume that models compute P P(Zian+n) using the common left-to-right decomposition of the text probability,
m+t
P(Timen)
II P(a ........... 2.-1),
which is used to generate the generation token-by-token using a particular decoding strategy.
Maximization-basee decoding The most commonly used decoding objective, in particular for directed generation, is maximization-base decoding. Assuming that the model assigns higher prob- ability to higher quality text, these decoding strategies search for the continuation with the highest
3

---

Published as a conference paper at ICLR 2020
An unprecedented number of mostly young whales have become stranded on the West Australian coast since 2008
WebText
The number of stranded whales has increased by more than 50 per cent in the past year, with the number stranded whales the West Australian coast increasing by more than 50 per cent in the past year. The number of whales stranded on the West Australian coast has increased by more than 50 per cent in the past year, with the number stranded whales the West Australian coast increasing by than 50 per cont in the past year.
Beam Search, b-16
The Australian Food Safety Authority has warned Australia's beaches may be revitalised this year because healthy seabirds and seals have been on the move. More than 50,000 seabirds, sea mammals and seahorses CS118 and Adelaide Airport CS300 from 2013. A major white-bat and umidauda migration across Australia under way in Australia for the first time, with numbers reaching estimated 50,000.
been swept
the sea by the Holden
Pure Sampling
Last week's intense storms and a series of powerful cyclones have been officially blamed for the deaths of species globally. The fin whales; packed in the belly of one killer whale thrashing madly in fear as another tries to bring it to safety. When
nine large fin whales near Whitsundays the largest loss any
colossal animal breached the waters of Whitsundays, he' d been seen tagged for decade.
Sampling, t=9.9
Pumping Station #3 shut down due to construction damage Find more at: www.abc.ne htt.bo.eetta/eeviinnmmeea serviccoment/epecieswoyyy in-the-top- tette-top-10-kklleeee 10----00r-llewaallllo "In the top 10 killer whale catastrophes in history:
K
catastrophes-in-hite
1986: Up to 12 orcas struck by lightning: many drowned and many more baclly injured.
Top-k, K-640
The whale's ss fate was confirmed late last week when the animal was found by fishermen coast of Bundaberg. Experts believe the whale was struck by a fishing vessel off the coast of Bundaberg, and died after being sucked into the ocean. The whale's fate was confirmed late last week when the animal was found by fishermen off the coast of Bundaberg.
K
Top-k, k-40, t-0.7
There has been an unprecedented number of calves caught in the nets of whaling stations that operate WA. Pilot whales continue to migrate feeding grounds feed their calves. They are now vulnerable due to the decline of wild populations: they restricted with sharp decline in wild populations the size of the Petrels are shrinking and dwindling population means there will only be room for a few new fowl.
breeding site each year. Image copyright Yoon Bo Kim But,
Nucleus, p 0.95
Poor nutrition has led rise in the number of stranded humpback whales the West Australian coast, veterinary researchers have said. Carly Holyoake, from Murdoch University, the Australian Veterinary Association' 's annual conference in Perth Wednesday, said an unprecedented number of mostly young whales had become stranded on the coast since 2008
WebText
Figure 3: Example generations continuing an initial sentence. Maximization and top-k truncation methods lead to copious repetition (highlighted in blue), while sampling with and without tempera- ture tends to lead to incoherence (highlighted in red). Nucleus Sampling largely avoids both issues.
likelihood. Since finding the optimum argmax sequence from recurrent neural language models or Transformers is not tractable (Chen et al. 2018). common practice is to use beam search (Li et al. 2016b Shen et al.) 2017 Wiseman al. 2017). However, several recent studies open-ended
generation have reported that maximization-based decoding does not lead to high quality text (Fan
al. 2018 Holtzman et al. 2018)
3.1
NUCLEUS SAMPLING
We propose a new stochastic decoding method: Nucleus Sampling. The key idea is to use the shape of the probability distribution to determine the set of tokens to sampled from. Given distribution P(r( we define its top-p vocabulary (P)) C V as the smallest set such that
Σ z
P(z( p.
(2)
4

---

Published as a conference paper at ICLR 2020
Token Probabilities for "I don't know." Repeated 200 times
64
probability
0.5
66
55
60
know
Figure 4: The probability of a repeated phrase increases with each repetition, creating positive feedback loop. We found this effect hold for the vast majority of phrases we tested, regardless of phrase length or if the phrases were sampled randomly rather than taken from human text.
0.66
Hought
kren
kealng tush
58e
SA4
TEAY
1802
Wil
401
301
Flat
would
Peaked
bakig
Distribution
Distribution
mool
Figure 5: The probability mass assigned to partial human sentences. Flat distributions lead to many moderately probable tokens, while peaked distributions concentrate most probability mass into just a few tokens. The presence of flat distributions makes the use of a small k in top-k sampling problematic, while the presence of peaked distributions makes large k's problematic.
Let p' - P(x(#1a-1). The original distribution is re-scaled to new distribution, from which the next word sampled:
P'(x((1a-1) P(z( if æ E V+)
otherwise.
In practice this means selecting the highest probability tokens whose cumulative probability mass exceeds the pre-chosen threshold The size of the sampling set will adjust dynamically based on the shape the probability distribution at each time step. For high values of p. this a small subset of vocabulary that takes up vast majority of the probability mass the nucleus.
3.2 TOP-k SAMPLING
Top-k sampling has recently become a popular alternative sampling procedure (Fan et al., 2018 Holtzman et al. 2018 Radford et al. 2019). Nucleus Sampling and top-k both sample from trun- cated Neural LM distributions, differing only in the strategy where to truncate. Choosing where to truncate can be interpreted as determining the generative model's 's trustworthy prediction zone.
At each time step, the top possible next tokens are sampled from according to their relative prob- abilitics. Formally, given a distribution P(x ((x1)11-1). we define its top-k vocabulary ((k) C as the set of size & which maximizes Evev P(x((11-1). Let p' Σ (k) P(z(215-1). The distribution is then re-scaled as equation 3 and sampling is performed based on that distribution. Note that the scaling factor p' can vary wildly each time-step. in contrast to Nucleus Sampling.
Difficulty choosing a suitable value of k While top-k sampling leads to considerably higher quality text than either beam search sampling from the full distribution, the of constant is
5

---

Published as a conference paper at ICLR 2020
sub-optimat across varying contexts. As illustrated on the left of Figure in some contexts the head of the next word distribution can be flat across tens or hundreds of reasonable options (e.g. nouns or verbs in generic contexts), while in other contexts most of the probability mass is concentrated in one or a small number of tokens, as on the right of the figure. Therefore k is small, in some contexts there is risk of generating bland generic text, while if k is large the top-k vocabulary will include inappropriate candidates which will have their probability of being sampled increased by the renormalization. Under Nucleus Sampling, the number of candidates considered rises and falls dynamically, corresponding to the changes in the model's ' confidence region over the vocabulary which top-k sampling fails to capture for any one choice of k.
3.3 SAMPLING WITH TEMPERATURE
Another common approach sampling- -basin generation is to shape a probability distribution through temperature (Ackley et al. 1985). Temperature sampling has been applied widely to text generation (Ficler & Goldberg 2017 Fan al. 2018 Caccia al. 2018). Given the logits "1:[V| and temperature I, the soltmax is re- re-estimated as
exp(ur/t) p(= = Vi(a1=-1)= = Er exp(u//t)
(4)
Setting / E [0, 1) skews the distribution towards high probability events, which implicitly lowers the mass in the tail distribution. Low temperature sampling has also been used to partially alleviate the issues of top-k sampling discussed above, by shaping the distribution before top-k sampling (Radford et al. 2018 Fan al., 2018). However, recent analysis has shown that, while lowering the temperature improves generation quality, it comes at the cost of decreasing diversity (Caccia et al. Hashimoto al. 2019))
2018
4
LIKELIHOOD EVALUATION
4.1
EXPERIMENTAL SETUP
While many neural network architectures have been proposed for language modeling, including LSTMs (Sundermeyer et al.] 2012) and convolutional networks (Dauphin et al.] (0017). the Trans- former architecture (Vaswani et al. 2017) has been the most successful in the extremely large-scale training setups in recent literature (Radford et al. 2018 2019). In this study we use the Generatively Pre-trained Transformer, version 2 (GPT2; Radford et al. 2019), which was trained on WebText, a 40GB collection of text scraped from the web We perform experiments using the Large model (762M parameters). Our analysis is based on generating 5,000 text passages, which end upon reach- ing an end-of- ally, conditioned on the initial paragraph (restricted 1-40 tokens) of documents in the held-out portion of WebText, except where otherwise mentioned.
token or maximum length of 200 tokens. Texts are generated condition-
4.2 PERPLEXITY
Our first evaluation is to compute the perplexity of generated text using various decoding strategies, according to the model that is being generated from. We compare these perplexities against that of the gold text (Figure 6) Importantly, we argue that the optimal generation strategy should produce text which has a perplexity close to that of the gold text: Even though the model has the ability to generate text that has lower perplexity (higher probability), such text tends to have low diversity and get stuck in repetition loops, as shown in and illustrated in Figure
We see that perplexity of text obtained from pure sampling is worse than the perplexity of the gold. This indicates that the model confusing itself: sampling too many unlikely tokens and creating context that makes it difficult to recover the human distribution of text, as in Figure I Yet, setting the temperature lower creates diversity and repetition issues, as we shall see in 5 Even with our relatively fine-graine parameter sweep, Nucleus Sampling obtains closest perplexity human text, as shown in T
Available https: / / thub. cww//ppeai/ggaa 2otttauttttttottt---aa--aae

---

Published as a conference paper at ICLR 2020
Method
Perplexity
Self-BLEU4
Zipf Coefficient
Repetition %
HUSE
Human
12.38
0.31
0.93
0.28
Greedy
1.50
0.50
1.00
73.66
Beam, b=16
1.48
0.44
0.94
28.94
Stochastic Beam, b-16
19.20
0.28
0.91
0.32
Pure Sampling
22.73
0.28
0.93
0.22
0.67
Sampling, t=0.9
10.25
0.35
0.96
0,66
0.79
Top-k=40 Top-k=640
6.88
0.39
0.96
0.78
0.19
13.82
0.32
0.96
0.28
0.94
Top-k=40, (=0.7
3,48
0.44
1.00
8,86
0.08
Nucleus y=0.95
13.13
0.32
0.95
0.36
0.97
Table I: Main results for comparing all decoding methods with selected parameters of each method. The numbers closest to human scores are in bold except for HUSE (Hashimoto et al.] (209)), a combined human and statistical evaluation, where the highest (best) value is bolded. For Top-k and Nucleus Sampling, HUSE is computed with interpolation rather than truncation (see (6.1)
4.3 NATURAL LANGUAGE DOES NOT MAXIMIZE PROBABILITY
One might wonder if the issue with maximization is search error, i.e., there are higher quality sentences to which the model assigns higher probability than to the decoded ones, beam search has just failed to find them. Yet Figures 2 & 6 show that the per-token probability of natural text is, on average, much lower than text generated by beam search. Natural language rarely remains high probability zone for multiple consecutive time steps, instead veering into lower- swwer-probability but more informative tokens. Nor does natural language tend to fall into repetition loops, even though the model tends to assign high probability this, as seen in Figure 4
Why is wwritenwritten text not the most probable text? We conjecture that this is an intrinsic property of human language. Language models that assign probabilities word time without global model of the text will have trouble capturing this effect. Grice's Maxims of Communication (Grice, 1975 show that people optimize against stating the obvious. Thus, making every word as predictable as possible will be disfavored. This makes solving the problem simply by training larger models or improving neural architectures using standard per-word learning objectives unlikely: such models are forced to favor the lowest common denominator, rather than informative language.
DISTRIBUTIONAL STATISTICAL EVALUATION
5.1 ZIPF DISTRIBUTION ANALYSIS
In order to compare generations to the reference text, we begin by analyzing their use of vocabu- lary. Zipf's law suggests that there is an exponential relationship between the rank of a word and its frequency in text. The Zipfian coefficient s can be used to compare the distribution in a given text
Beam Search
Sampling
Top-k (t=1.0)
Top-k (t=0.7)
Nucleus
Conditional PPR
20
10'
107
10a
10
107
10
0
1.99
Temperature
Figure 6: Perplexities of generations from various decoding methods. Note that beam search has unnaturally low perplexities. A similar effect seen using temperature of 0.7 with top- as both Radford et al.] (2019) and Fan et al.] (2018). Sampling, Top-k, and Nucleus can all be calibrated to human perplexities, but the first two face coherency issues when their parameters are set this high.

---

Published as a conference paper at ICLR 2020
Vocabulary Distrubution and Zipf's Coefficient 1.0, Supt 0.926
Supt 0.967
gold, Sgpr = 0.934
40, t= 0.7, Spp = 1.000
tog(fequency)
p = 0.95, Supr 0.949
== 640, Supr 0.958
10000
20000
30000
A0000
Figure 7: A rank-frequency plot of the distributional differences between n-gram frequencies of human and machine text. Sampling and Nucleus Sampling are by far the closest to the human distribution, while Beam Search clearly follows very different distribution than natural language.
Self-BLEU of Generations Over 5000 Documents
Self-BL Self-BLEU4
0.9
Belf-BBEUS
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1-0.
1-02
1-a.
p. 994475
p. p- prpeaaae
6 1715
Figure 8: Self-BLEU calculated on the unconditional generations produced by stochastic decoding methods; lower Self-BLEU scores imply higher diversity. Horizontal blue and orange lines represent human self-BLEU scores. Note how common values of t E [0.5, 1] and [I, 100] result in high self-similarity, whereas "normal" values o [0.9, 1) closely match the human distribution of text.
shows the vocabulary distributions along with estimated Zipf coefficients for selected parameters of different decoding methods. As expected, pure sampling is the closest the human distribution, followed by Nucleus Sampling. The visualization of the distribution shows that pure sampling slightly overes- timates the use of rare words, likely one reason why pure sampling also has higher perplexity than human text. Furthermore, lower temperature sampling avoids sampling these rare words from the tail, which is why it has been used some recent work (Fan et al. 2018 Radford al. 2019).
to theoretically perfect exponential curve, where 1 (Piantadosi 2014). Figure
5.2 SELF-BLEU
We follow previous work and compute Self-BLEU (Zhu et al. 2018) as a metric of diversity. Self- BLEU is calculated by computing the BLEU score of each generated document using all other generations in the evaluation set as references. Due to the expense of computing such an operation, sample 1000 generations. each of which is compared with all 4999 other generations ences. A lower Self BLEU score implies higher diversity. Figure shows that Self BeEE results largely follow that of the Zipfian distribution analysis diversity measure. is worth noting that
refer-
8

---

Published as a conference paper at ICLR 2020
Likelihood of Degeneration into Repetition
% Generation Resulting Repeation
0.8
greedy
Sampling
0.7
Nucleus
0.-
is bewn width in Beam Search
Beam Search
Top-K
Top-K, t=0.7
64-
u.-
Topsk sampplngg
622
6:
1w:20 2-30
1-510
1-10240
60
gold
01
02 0.2 04 0.5
0.5
0.7
0.
Nuctous / in Sampling
Figure 9: We visualize how often different decoding methods get "stuck" in loops within the first 200 tokens. A phrase (minimum length 2) is considered repetition when it repeats at least three times at the end of the generation. We label points with their parameter values except for and which follow the x-axis. Values greater than 100 are rarely used in practice and values of p are usually in [0.9, 1): therefore Nucleus Sampling is far closer to the human distribution in its usual paramcter range. Sampling with temperatures lower than 0.9 severely increase repetition. Finally, although beam search becomes less repetitive according this metrie beam width increases, this is largely because average length gets shorter as increases (see Appendix
very high values of l and t are needed to get close to the reference distribution, though these result in unnaturally high perplexity (44).
5.3 REPETITION
One attribute of text quality that we can quantify repetition. Figure 9 shows that Nucleus Sam- pling and top-k sampling have the least repetition for reasonable parameter ranges. Generations from temperature sampling have more repetition unless very high temperatures are used, which we have shown negatively affects coherence methods face repetition issues when their tuning parameters are set too low, which tends to over- truncate, mimicking greedy search. Therefore we conclude that only Nucleus Sampling satisfies all the distributional criteria for desirable generations.
measured by high perplexity). Further, all stochastic
HUMAN EVALUATION
6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Statistical evaluations are unable to measure the coherence of generated text properly. While the metrics in previous sections gave vital insights into the different decoding methods compare, human evaluation is still required to get full measure of the quality of the generated text. However, pure human evaluation does not take into account the diversity of the generated text; therefore we use HUSE (Hashimoto et al. 2019) to combine human and statistical evaluation. HUSE is computed by training a discriminator to distinguish between text drawn from the human and model distributions, based on only two features: The probability assigned by the language model, and human judgements of typicality of generations. Text that is close to the human distribution in terms of quality and diversity should perform well on both likelihood evaluation and human judgements.
As explored in the previous sections, the current best- pest-performin decoding methods rely on frunca- tion of the probability distribution, which yields a probability of 0 for the vast majority of potential tokens. Initial exploration of applying HUSE directly led top-k and Nucleus Sampling receiving scores of nearly 0 due truncation, despite humans favoring these methods. As proxy, when generating the text used to compute HUSE, we interpolate (with mass 0.1) the original probability distribution with the top-k and Nucleus Sampling distribution, smoothing the truncated distribution.
For each decoding algorithm we annotate 200 generations for typicality, with each generation re- ceiving 20 annotations from 20 different annotators. This results in total of 4000 annotations per

---

Published as a conference paper at ICLR 2020
decoding scheme. We use KNN classifier to compute HUSE, as in the original paper, with k = 13 neighbors, which we found led to the higher accuracy in discrimination. The results in Table shows that Nucleus Sampling obtains the highest HUSE score, with Top-k sampling performing second best.
6.2 QUALITATIVE ANALYSIS
Figure shows representative example generations. Unsurprisingly, beam search gets stuck repetition loop it cannot escape. Of the stochastic decoding schemes, the output of full sampling is clearly the hardest to understand, even inventing new word "umidauda", apparently a species of bird. The generation produced by Nucleus Sampling isn't perfect the model appears to confuse whales with birds, and begins writing about those instead. Yet, top-k sampling immediately veers off into unrelated event. When top-k sampling is combined with temperature of 0.7, as is commonly done (Radford et al. 2019 Fan et al. 2018), the output devolves into repetition, exhibiting the classic issues of -ewteepeeature decoding. More generations available in Appendix B
7 CONCLUSION
This paper provided a deep analysis into the properties of the most common decoding methods for open-ended language generation. We have shown that likelihood maximizing decoding causes repe- tition and overly generic language usage, while sampling methods without truncation risk sampling from the low-confidence tail a model's predicted distribution. Further, we proposed Nucleus Sam- pling as a solution that captures the region of confidence of language models effectively. In future work, we wish to dynamically characterize this region of confidence and include a more semantic utility function to guide the decoding process.
ACKNOWLEDGMENT
This research was supported in part by NSF (IS--1224371) the National Science Foundation Gradu- ate Research Fellowship under Grant No. DGEI 256082, DARPA CwC through ARO (W91 INFI5- 1- 0543), DARPA MCS program through NIWC Pacific (N66001-19-2-400)) Centre for Artificial Intelligence Research, and the Allen Institute for AI.
DGEI256082,
the South African
REFERENCES
David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive science, 9():1477-69, 1985.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. Proceedings of the 2015 International Conference on Learning Representations, 2015.
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and I Laurent Char- lin. Language gans falling short. Critiquing and Correcting Trends in Machine Learning: NewrIPS 2018 Workshop, 2018. URLp:tt:///aati
/1wl../lll... C2549
Yining Chen, Sorcha Gilroy, Andreas Maletti, Jonathan May, and Kevin Knight. Recurrent neu- ral networks as weighted language recognizers. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 2261-2271, New Orleans, Louisiana, June 2018.
Elizabeth Clark, Yangfeng Ji, and Noah A. Smith. Neural text generation in stories using entity rep- resentations as context. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 2250-2260 New Orleans, Louisiana, June 2018.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks, In Proceedings of the 34th International Conference on Machine Learn ing, pp. 933-941, 2017.
10

---

Published as a conference paper at ICLR 2020
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume I. pp. 889-898, 2018.
Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects in neural language generation. In Proceedings of the Workshop on Stylistic Variation, pp. 94-104, 2017.
H Paul Grice. Logic and conversation. In P Cole and J L Morgan (eds.), Speech Acts, volume 3 of Syntax and Semantics, pp. 41-58. Academic Press, 1975.
Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation for natural language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. In Proceedings of the Association for Computational Linguistics, 2018.
Philipp Koehn and Rebecca Knowles. Six challenges for neural machine translation. In Proceedings of the First Workshop Neural Machine Translation, pp. 28 38-39, 2017.
Ilya Kulikov, Alexander H Miller, Kyunghyun Cho, and Jason Weston. Importance of search and evaluation strategies in neural dialogue modeling. International Conference on Natural Language Generation, 2019.
Jiwei Li, Will Monroe, and Dan Jurafsky. A simple, fast diverse decoding algorithm for neural generation. arXiv preprint arXiv:1611.0856. 2016a.
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep rein- forcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1192-1202, 2016b.
Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention- based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412-1421, 2015.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. Abstractive text summarization using -ootencce---eeuencee rnss and beyond. Proceedings of The 20th SIGNLL Conference Computational Natural Language Learning, pp. 280-290, 2016
Chris Pal, Charles Sutton, and Andrew McCallum. Sparse bawkard using minimum diver- gence beams for fast training of conditional random fields. In 2006 IEEE International Confer- ence on Acoustics Speech and Signal Processing Proceedings, volume 5, May 2006.
Nanyun Peng, Marjan Ghazvininejad, Jonathan May, and Kevin Knight. Towards controllable story generation. In Proceedings of the First Workshop on Storytelling, pp. 43-49, New Orleans, Louisiana, June 2018. doi: 10. 11558v11111811111
Steven T Piantadosi. Zipf's word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, (5): 1130000 2014.
Alec Radford, Karthik Narasimhan, Tim Salimans, and llya Sutskever. Improving language under-
standing by generative pre-training, 2018. URL http:s::///3--s-eeeett 2ama--eeeesttt222aaaaaaa www.oovveesslllaagggeeeee Laaguagee-unnnnn uns.ueriise.. httpaeee_undestttaddaaee pwcplddiinnppppppeeeeee Unpublished manuscript.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
Language models are unsupervised multitask learners, February 2019. URL https:
pddf Unpublished manuscript.
Stanislau Semeniuta, Aliaksei Severyn, and Sylvain Gelly. On accurate evaluation of gans for lan- guage generation. arXiv preprint arXiv: 0866044366 2018

---

Published as a conference paper at ICLR 2020
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non-parallel text by cross-aligmment. In Advances in neural information processing systems, pp. 68306683444 2017.
Felix Stahlberg and Bill Byrne. On nmt search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Prucessing (EMNLP-IJCNLP), pp- 3347-3355. 2019.
Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. Lstm neural networks for language mod- eling. In Thirteenth annual conference of the international speech communication association, 2012.
Guy Tevet, Gavriel Habib, Vered Shwartz, and Jonathan Berant. Evaluating text gans as language models. Proceedings of the 2019 Conference the Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 2241-2247, 2019.
the North American Chapter of the Association for
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search for improved description of complex scenes. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In Proceedings the International Conference on Learning Representations (ICLR), 2020.
Sam Wiseman, Stuart Shieber, and Alexander Rush. Challenges data-to-document generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp- 2253-2263, Copenhagen, Denmark, September 2017.
Jingjing Xu, Xuancheng Ren, Junyang Lin, and Xu Sun. Diversity-promoting gan: A cross-entropy based generative adversarial network for diversified text generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3940-3949, Brussels, Belgium, oct 2018
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI, 2017.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. SIGIR, 2018.
12

---

Published as a conference paper at ICLR 2020
A BEAM WIDTH EFFECT
Beam Width Distinct Trigrams
Number of Distinct Trigrams
800000
Total Number of Trigrams
Over 5000 Generation ont
600000
400000
200000
0
gold
4
b 8
- 16
Figure 10: The total number of trigrams produced by Beam Search with varying beam widths, with gold (human) data for comparison. Note how the average length of generations goes down linearly with beam width, while the number of distinct trigrams stays constant and extremely low in comparison gold data.
13

---

Published as a conference paper at ICLR 2020
B EXAMPLE GENERATIONS
We include a set of examples for further qualitative comparison.
Top Customer Questions
WebText
Q: have question about the new "S" series. have question about the \"S\"
series A: The new "S" series is a new line of high-performance, high-performance, high- performance, high-perfomance, high-perfornance, high-performance, high- performance, high-perfomance, nigh-performance, perff--rrrrrmc,e, high- performance, high-performance high-performance, high-performancee high- performance, high-performance, high-performance, high-perffrmance,
Beam Search, b=16
When will you start sending this stuff for register until someone else orders? If anyone wants to order, we would like to keep the next batch of binders as being a vaiible sample. It ASKED for the logistics and this are to set us back another beat. Are there any instructions for understanding the CDI CAL 0519-0199 9? If there are,
of shipping. wish refund if actions like
Pure Sampling
very keen find them.
What's s the best way to read this one? The blog post from the authors discusses the differences between how the marketing mix plays out in terms of visibility, trustworthiness, and engagement, among other things. In my experience, this difference isn't 't necessarily obvious; it's hard to explain to layperson, but in all honesty it's probably the most important point to consider when reviewing social media.
*
Sampling, t=0.9
questions Asked By: Bizzabo 28th February 2016 Product: Jaybo Comfort: Durability calf and
K
have small pad how do you new t shirts. Thanks. Has the compression system at all moved????? Asked By: Kiran R 28th February 2016 Product: Jaybo Comfort: Durability
chance all breaks would to know that up the circumference of the pad with something as comfortable
Top-k, k-640
How change the wallpaper for my computer screen? If you have the latest (current version) of your operating system, then you can change the background wallpapeer for your screen. How app. Tap on your user name and tap on Settings. Click on the Background. The wa wallpaper change will apply. How can change the background wallpaper for my screen?
K
change the wallpaper
my computer screen? Open the settings
Top-k, 40 1=7.
Where can purchase my prototypes and printed books the USA? Currently we have a sign for purchase available in the guide. Since
based the States will be shipping to by FedEx. Do you make books in America? We do! We love our folks in the US. We haven't had chance make documentary so love for our culture. How coess the book publisher pay for the book and art?
our love
Nuclous, p=0.55
Why your caustics superior? Purest Grades Available. All US Manufactured, No Asian Imports. Quality Control on all incoming and outgoing shipments. Most suppliers only supply tech grade product. What difference between Food and Tech Grade Lye? Food grade has lower parts per million of heavy metal impurities. 80% of our soap makers prefer our grade product.
WebText
Figure 11: More example generations from initial tag line. All generations available at https: //github. com/ ari ho zmman/deee
14

---

Published as a conference paper at ICLR 2020
So what's new in my life?
09/11/18 Just got back from vacation.
WebText
Just got back from vacation.
09/10/18 10188111 Just got back from vacation. Just got back from vacation. 09/09/18 Just got back from vacation. Just got back from vacation. 09/08/18 Just got back from vacation. Just got back from vacation,
Beam Search, b- 16
01/05/19 Visited The Subaru Canada Heritage15tt 15th Star for my vehicle research. That wonderful experience. Thanks again for your kn knowledge and support. 05/23/19 was thrilled to receive my Subaru Legacy character! It fits me perfectly and makes me feel like a kid again! 06/15/19 Just regained my licence for my different coloured 2016 WRX / STI...
Pure Sampling
Just got back from vacation,
09/07/18-
Happy 9th birthday to me! Happy 9th birthday to me!
09/01/18 Sofwidden alsol Sofwidden also!
08/25/18 Happy birthday
me! Happy birthday to me!
08/17/18 Night shift! t a Night shift! D0
Sampling, t=0.9
08/17/18 Lucky me! Lucky me!
Just got back from vacation.
K
09/03/18 After more than years, might have an apartment in Virginia Beach.
After more than 40 years, might have apartment in Virginia Beach.
08/20/18 18 Going for a hike at Mount Eerie in Northeast Virginia Spring
Going for hike Mount Eerie Northeast Virginia Spring
Top-k, 640
Just got back from vacation. 09/08/18
gotten really sick. I've gotten really sick.
09/07/18 My wife and are getting married in February. My wife are getting married February. 09/06/18 I'm excited go back college this fall.
Top-k, k- 640, 0.7
I'm so excited to go back to college this fall.
Just got back from vacation. 07/12/18 Happy birthday Swingu, who nearly years old. would like say hi to him on the road as well as when ride with him. You cannot go to work without feeling physically sick psychologically exhausted because you barely breathe. Even you ricee rollercoaster even once, it is easy recover from the physical side of it.
Nucleus, 0-0.55
juutt got back from much needed and really great nine day vacation to remote Arizona property. It was really restful and relaxing visit. got a lot accomplished while was there, but still found time to just goof off have fun too, got do some astronomy, even though the weather was pretty cloudy most of the time. Here is 50 minute exposure of M101. It turned out pretty good.
WebText
Figure 12: More example generations from an initial tag line. Note that Pure Sampling and Nu- cleus Sampling is the only algorithms that can escape the repetition loop, with Nucleus Sam- pling's generation far closer in style the ground truth text. All generations available https:: 7/qithub. com/ ari ari-holt zman / degen
15

---

Published as a conference paper at ICLR 2020
University of Wisconsin Madison
WebText
University of Wisconsin Madison University of Wisconsin Milwaukee University of Wisconsin Oshkosh
Beam Search, b-16
Mental Health
4240 New Florence Rd, Henderson, Wi 54711 (262)-335-7433 MentalHealtt@uuiioe wettth@uiiww.eeee eduu University of Wisconsin Madison (at UW Campus)
Pure Sampling
Jonathan Hedberg Department Comparative and Applied Economics University of California Berkeley Michael Amante Department of Economics
*
Sampling, t-0.9
Colorado State University Denver
Claremont McKenna College Merrimack College New England Institute of Technology UConn School of Medicine Wisconsin School of Engineering
K
Top-k, K-640
University Wisconsin Milwaukee
K
University of Wisconsin Oshkosh
University of Wisconsin Racine
University Wisconsin Red Wing
University of Wisconsin Stevens Point
Top-k, 404 t-0.7
University Wisconsin Stevens Point
Thomas Jefferson, 1777-1826 Who Is Mike Barnacle? Well, at the start, this was clearly project clesigned to help people get the gist of classic
myriad ways. Now we find ourseives with interesting of recordings of 200 BC, Who are these guys? This one, apparently, are the descendants of Greek historian Euclid. famously analyzed straight
Nucleus, P-0.9 0095
so we know those are straight.
In late 1998, UW-Madison group led by James Thomson was the first to isolate culture human embryonic stem cells, master undifferentiated cells that arise at the earliest stages of development and are capable of becoming any of the 220 types of cells and tissues in the human body.
WebText
Figure 13: More example generations from an initial tag line. All generations available at https: 7/github. com/ ari-holt zman/degen)
16


Reviews:
[{"review_id": "rJeVL4b6KS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contributions:\n\nThis paper studies an important problem, i.e., how to find a good decoding strategy for open-ended text generation. To this end, the authors provide a deep analysis of the most common decoding methods, and propose Nucleus Sampling, a very simple yet effective method to generate higher-quality text. Compared with top-k sampling, the key idea behind the proposed method is to sample from the dynamic nucleus of tokens containing the majority of the probability mass. Experiments demonstrate that nucleus sampling is an effective decoding strategy in practice. \n\nStrengths:\n\n(1) Writing & Clarity: The proposed method is well motivated, the paper is carefully written, and clearly presented. I enjoyed reading the paper.  \n\n(2) Experiments: The experiments are also carefully designed. Both quantitative and human evaluation are provided. Quality examples are also shown. \n\nWeaknesses:\n\n(1) Novelty: The biggest concern that I have is its technical novelty. The proposed method is effective, but it acts more like a useful trick. Also, no theoretical justification is provided, but only some intuitions. So, I would say the novelty is indeed limited. However, given the comprehensive evaluation, and high writing quality, I lean to accept this paper due to its empirical contribution. It seems that this nucleus sampling method can be applied in a wide range of text generation applications.  \n\n\n** Minor **\nTypo: In the line below Eqn. (2), \"x \\in V^{(k)}\" => \"x \\in V^{(p)}\", same typo in Eqn. (3).\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Skxc3_RpFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is motivated by an observation that maximization-based decoding approaches such as beam search can lead to incoherent and repetitive sentences when open-ended long-form text generation based on neural language model such as GPT-2 is performed. To solve the problem, this paper proposes a sampling method called Nucleus Sampling. Similar to Top-k sampling, Nucleus Sampling truncates the probability distribution of the words in the vocabulary. Instead of re-normalizing the probabilities for the top-k words, Nucleus Sampling re-normalizes the original probabilities for the words with values above a pre-chosen threshold p. Some quantitative and qualitative results show that the proposed sampling method can generate long-form texts with some nice properties.\n\nPros:\n\nThe problem addressed in this paper is highly interesting, and the proposed method is simple and intuitive. The paper is well motivated and the method is clearly presented.\n\nExtensive quantitative and qualitative experiments are conducted to compare different sampling methods.\n\nCons:\n\n1) Although the raised problem in this paper is interesting, the proposed Nucleus Sampling seems to be a trivial variant of Top-k sampling. With a reasonably large k suitable for different practical problems in question, it is unclear that Nucleus Sampling produces significant advantages over commonly used Top-k sampling. \n\n2) The argued difficulty in choosing k in Top-k sampling is not that different from that of choosing the threshold p in Nucleus Sampling.\n\n3) In section 4.3, the argument that natural language rarely remains in a high-probability zone is questionable. This happens only because our current neural language models are not well-specified for generating long texts and modeling long-range contexts. \n\n4) In section 6.2, the qualitative comparison between Nucleus Sampling and Top-k sampling might be caused by randomness. With a large k, there is no technical barrier that prevents Top-k sampling from generating the sentences produced by Nucleus Sampling.\n\n5) A recent stochastic beam search method based on Gumbel-max-k (Kool, Hoof, and Welling, ICML 2019) should be discussed and compared. \n\nIn summary, although the studied problem in this paper is highly interesting, the proposed Nucleus Sampling is not technically significant compared to Top-k sampling.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1lugevycS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the domain of language models, the paper introduces a new heuristic sampling method called top-p sampling, or nucleus sampling (NS). It is a variant of top-k sampling where the smallest k is selected to ensure the combined likelihood is no less than p. The paper centers on claiming and showing that the generated samples are of higher quality and more diverse than common alternatives such as beam search, pure sampling, top-k sampling, and low-temperature sampling.\n\nWhile overall I think the proposed method is sound as an alternative to other heuristics such as beam search, I have reservations on the presentation and arguments made in the paper. \n\nPros:\n1. NS is sound as a heuristic sampling method.\n2. The paper contains many interesting experimental observations and I speculate that some of them will find future uses. For example, the selection of parameter values (not just for NS, also for top-k) and the nontrivial perplexity of generated text.\n\nCons:\n1. The ultimate performance measure (open-ended generation) of “high quality” and “diversity” is very vague. It seems that the authors end up doing is to evaluate by high self-BLEU, HUSE, few repetitions, and perplexity. Furthermore, it is unclear why one _should_ train with cross-entropy (trying to match the distributions) and then rely on the sampling procedure to fulfill these desiderata (See also Min1 and Min2).\n2. The comparison with beam search (BS) is not well motivated. BS is devised to find the maximal sentence and it is not stochastic. It seems out of place in the context of generating a “diverse” set of samples.\n3. The arguments in the comparison with pure sampling is vague and sometimes misplaced. The key argument seems to hinge on the idea that the low likelihood tail is of “low confidence.” But this claim is problematic. If the estimate is wrong on the low probability tail, then so is the estimate on p(head) = 1-p(tail) by virtue of p being a probability measure. \n4. The arguments in the comparison with top-k is vague and sometimes misplaced. The main argument against top-k is the “[d]ifficulty in choosing a suitable value of k” but the same can be said for choosing p. After all, top-k and top-p (NS) can be thought of as a variant of each other (by dynamically choosing k or p respectively). Moreover, in Figure 5, a selection for k value is suggested. I agree that this value might not _appear_ as intuitive as p, and maybe other works have chosen a smaller k than they should have, but similarly, people might intuitively choose too high a value for p (Figure 5). \n\nPossible mistakes/typos:\n1. (2), “>=“ -> ≥.\n2. Figure 7, the human self-BLEU4 < human self-BLEU5 and that seems wrong, especially when all other bars show the opposite ordering.\n3. In References, “Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In ACL, 2018a” is duplicated.\n4. In References, the citation of “Unifying human and statistical evaluation for natural language generation” is from NAACL 2019, not “2018.”\n5. In References, the first names are shown as initials in “Sparse forward-backward using minimum divergence beams for fast training of conditional random fields.”\n\nQuestions:\n1. In Table 1, how is Human perplexity estimated?\n\nMinor issues:\n1. Partly due to what the authors position NS to solve, i.e. open-ended generation, the core arguments is not as precise or rigorous as it could have been in my opinion. I feel that focusing on comparing NS to other heuristics as a heuristic might make the text appeal to a wider audience and the discussion more precise.  \n2. The distinction drawn between open-ended generation and directed generation is unpersuasive to me. In the context of language modeling, the former is to approximate a distribution (over an extended alphabet) whereas the latter is to approximate a conditional distribution (given the input). However, the most common formulation to solve the former is to decompose the distribution into a product of conditional distributions (1).\n3. The caption in Figure 1 draws a misleading comparison. The “admirable” generation (presumably referring to the OpenAI blog post) was from the full GPT-2 model, not the initially released GPT-2-117M.\n\nPlease point out my misunderstanding directly. I am open to acknowledging them and revising my assessment.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents nucleus sampling, a sampling method that truncates the tail of a probability distribution and samples from a dynamic nucleus containing the majority of the probability mass. Likelihood and human evaluations show that the proposed method is a better alternative to a standard sampling method and top-k sampling.

This is a well-written paper and I think the proposed sampling method will be useful in language modeling. All reviewers agree that the paper addresses an important problem. 

Two reviewers have concerns regarding the technical contribution of the paper (i.e., nucleus sampling is a straightforward extension of top-k sampling), and whether it is enough for publications at a venue such as ICLR. R2 suggests to have a better theoretical framework for nucleus sampling. I think these are valid concerns. However, given the potential widespread application of the proposed method and the strong empirical results, I recommend to accept the paper.

Also, a minor comment, I think there is something wrong with your style file (e.g., the bottom margin appears too large compared to other submissions).
Decision: Accept (Poster)