Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In the domain of language models, the paper introduces a new heuristic sampling method called top-p sampling, or nucleus sampling (NS). It is a variant of top-k sampling where the smallest k is selected to ensure the combined likelihood is no less than p. The paper centers on claiming and showing that the generated samples are of higher quality and more diverse than common alternatives such as beam search, pure sampling, top-k sampling, and low-temperature sampling.

While overall I think the proposed method is sound as an alternative to other heuristics such as beam search, I have reservations on the presentation and arguments made in the paper. 

Pros:
1. NS is sound as a heuristic sampling method.
2. The paper contains many interesting experimental observations and I speculate that some of them will find future uses. For example, the selection of parameter values (not just for NS, also for top-k) and the nontrivial perplexity of generated text.

Cons:
1. The ultimate performance measure (open-ended generation) of “high quality” and “diversity” is very vague. It seems that the authors end up doing is to evaluate by high self-BLEU, HUSE, few repetitions, and perplexity. Furthermore, it is unclear why one _should_ train with cross-entropy (trying to match the distributions) and then rely on the sampling procedure to fulfill these desiderata (See also Min1 and Min2).
2. The comparison with beam search (BS) is not well motivated. BS is devised to find the maximal sentence and it is not stochastic. It seems out of place in the context of generating a “diverse” set of samples.
3. The arguments in the comparison with pure sampling is vague and sometimes misplaced. The key argument seems to hinge on the idea that the low likelihood tail is of “low confidence.” But this claim is problematic. If the estimate is wrong on the low probability tail, then so is the estimate on p(head) = 1-p(tail) by virtue of p being a probability measure. 
4. The arguments in the comparison with top-k is vague and sometimes misplaced. The main argument against top-k is the “[d]ifficulty in choosing a suitable value of k” but the same can be said for choosing p. After all, top-k and top-p (NS) can be thought of as a variant of each other (by dynamically choosing k or p respectively). Moreover, in Figure 5, a selection for k value is suggested. I agree that this value might not _appear_ as intuitive as p, and maybe other works have chosen a smaller k than they should have, but similarly, people might intuitively choose too high a value for p (Figure 5). 

Possible mistakes/typos:
1. (2), “>=“ -> ≥.
2. Figure 7, the human self-BLEU4 < human self-BLEU5 and that seems wrong, especially when all other bars show the opposite ordering.
3. In References, “Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In ACL, 2018a” is duplicated.
4. In References, the citation of “Unifying human and statistical evaluation for natural language generation” is from NAACL 2019, not “2018.”
5. In References, the first names are shown as initials in “Sparse forward-backward using minimum divergence beams for fast training of conditional random fields.”

Questions:
1. In Table 1, how is Human perplexity estimated?

Minor issues:
1. Partly due to what the authors position NS to solve, i.e. open-ended generation, the core arguments is not as precise or rigorous as it could have been in my opinion. I feel that focusing on comparing NS to other heuristics as a heuristic might make the text appeal to a wider audience and the discussion more precise.  
2. The distinction drawn between open-ended generation and directed generation is unpersuasive to me. In the context of language modeling, the former is to approximate a distribution (over an extended alphabet) whereas the latter is to approximate a conditional distribution (given the input). However, the most common formulation to solve the former is to decompose the distribution into a product of conditional distributions (1).
3. The caption in Figure 1 draws a misleading comparison. The “admirable” generation (presumably referring to the OpenAI blog post) was from the full GPT-2 model, not the initially released GPT-2-117M.

Please point out my misunderstanding directly. I am open to acknowledging them and revising my assessment.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper is motivated by an observation that maximization-based decoding approaches such as beam search can lead to incoherent and repetitive sentences when open-ended long-form text generation based on neural language model such as GPT-2 is performed. To solve the problem, this paper proposes a sampling method called Nucleus Sampling. Similar to Top-k sampling, Nucleus Sampling truncates the probability distribution of the words in the vocabulary. Instead of re-normalizing the probabilities for the top-k words, Nucleus Sampling re-normalizes the original probabilities for the words with values above a pre-chosen threshold p. Some quantitative and qualitative results show that the proposed sampling method can generate long-form texts with some nice properties.

Pros:

The problem addressed in this paper is highly interesting, and the proposed method is simple and intuitive. The paper is well motivated and the method is clearly presented.

Extensive quantitative and qualitative experiments are conducted to compare different sampling methods.

Cons:

1) Although the raised problem in this paper is interesting, the proposed Nucleus Sampling seems to be a trivial variant of Top-k sampling. With a reasonably large k suitable for different practical problems in question, it is unclear that Nucleus Sampling produces significant advantages over commonly used Top-k sampling. 

2) The argued difficulty in choosing k in Top-k sampling is not that different from that of choosing the threshold p in Nucleus Sampling.

3) In section 4.3, the argument that natural language rarely remains in a high-probability zone is questionable. This happens only because our current neural language models are not well-specified for generating long texts and modeling long-range contexts. 

4) In section 6.2, the qualitative comparison between Nucleus Sampling and Top-k sampling might be caused by randomness. With a large k, there is no technical barrier that prevents Top-k sampling from generating the sentences produced by Nucleus Sampling.

5) A recent stochastic beam search method based on Gumbel-max-k (Kool, Hoof, and Welling, ICML 2019) should be discussed and compared. 

In summary, although the studied problem in this paper is highly interesting, the proposed Nucleus Sampling is not technically significant compared to Top-k sampling.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Contributions:

This paper studies an important problem, i.e., how to find a good decoding strategy for open-ended text generation. To this end, the authors provide a deep analysis of the most common decoding methods, and propose Nucleus Sampling, a very simple yet effective method to generate higher-quality text. Compared with top-k sampling, the key idea behind the proposed method is to sample from the dynamic nucleus of tokens containing the majority of the probability mass. Experiments demonstrate that nucleus sampling is an effective decoding strategy in practice. 

Strengths:

(1) Writing & Clarity: The proposed method is well motivated, the paper is carefully written, and clearly presented. I enjoyed reading the paper.  

(2) Experiments: The experiments are also carefully designed. Both quantitative and human evaluation are provided. Quality examples are also shown. 

Weaknesses:

(1) Novelty: The biggest concern that I have is its technical novelty. The proposed method is effective, but it acts more like a useful trick. Also, no theoretical justification is provided, but only some intuitions. So, I would say the novelty is indeed limited. However, given the comprehensive evaluation, and high writing quality, I lean to accept this paper due to its empirical contribution. It seems that this nucleus sampling method can be applied in a wide range of text generation applications.  


** Minor **
Typo: In the line below Eqn. (2), "x \in V^{(k)}" => "x \in V^{(p)}", same typo in Eqn. (3).

review_title: Official Blind Review #2

Meta-review: This paper presents nucleus sampling, a sampling method that truncates the tail of a probability distribution and samples from a dynamic nucleus containing the majority of the probability mass. Likelihood and human evaluations show that the proposed method is a better alternative to a standard sampling method and top-k sampling.

This is a well-written paper and I think the proposed sampling method will be useful in language modeling. All reviewers agree that the paper addresses an important problem. 

Two reviewers have concerns regarding the technical contribution of the paper (i.e., nucleus sampling is a straightforward extension of top-k sampling), and whether it is enough for publications at a venue such as ICLR. R2 suggests to have a better theoretical framework for nucleus sampling. I think these are valid concerns. However, given the potential widespread application of the proposed method and the strong empirical results, I recommend to accept the paper.

Also, a minor comment, I think there is something wrong with your style file (e.g., the bottom margin appears too large compared to other submissions).
Decision: Accept (Poster)