Neural Text Generation With Unlikelihood Training
Authors: Sean Welleck
Ilia Kulikov
Stephen Roller
Emily Dinan
Kyunghyun Cho
Jason Weston
Venue:  (2020)
Keywords: language modeling
machine learning

Abstract: Neural text generation is a key tool in natural language applications, but it is well known there are major problems at its core. In particular, standard likelihood training and decoding leads to dull and repetitive outputs. While some post-hoc fixes have been proposed, in particular top-k and nucleus sampling, they do not address the fact that the token-level probabilities predicted by the model are poor. In this paper we show that the likelihood objective itself is at fault, resulting in a model that assigns too much probability to sequences containing repeats and frequent words, unlike those from the human training distribution. We propose a new objective, unlikelihood training, which forces unlikely generations to be assigned lower probability by the model. We show that both token and sequence level unlikelihood training give less repetitive, less dull text while maintaining perplexity, giving superior generations using standard greedy or beam search. According to human evaluations, our approach with standard beam search also outperforms the currently popular decoding methods of nucleus sampling or beam blocking, thus providing a strong alternative to existing techniques.

NEURAL TEXT DEGERRRAIION WITH UNLIKELIHOOD TRAINING
Sean Welleck 1.2+
Ilia Kulikoy
Stephen Roller?
Emily Dinan?
Kyunghyun 1..3 & Jason Weston 1,2
NNww York University, Faaebook AI Research, CCIAR Azrieli Global Scholar
ABSTRACT
Neural text generation is key tool in natural language applications, but it is well known there are major problems at its core. In particular, standard likelihood training and decoding leads to dull and repetitive outputs (Holtzman et al. (219)) While some post-hoc fixes have been proposed, in particular top-k and nucleus sampling, they do not address the fact that the token-level probabilities predicted by the model are poor. In this paper we show that the likelihood objective itself is at fault, resulting in model that assigns too much probability to sequences con- taining repeats and frequent words, unlike those from the human training distri- bution. We propose a new objective, unlikelihood training, which forces unlikely generations to be assigned lower probability by the model. We show that both token and sequence level unlikelihood training give less repetitive, less dull text while maintaining perplexity, giving superior generations using standard greedy or beam search. According to human evaluations, our approach with standard beam search also outperforms the currently popular decoding methods of nucleus sam- pling or beam blocking, thus providing strong alternative to existing techniques.
INTRODUCTION
Neural text generation is a vital tool in wide range of natural language applications. However, the standard approach- training sequence sequence model, e.g. Transformer (Vaswani al., 2017), to maximize log- ligelikel and approximately decoding the most likely sequence is known to be flawed. Generated text in open-ended applications such as language modeling or dialogue has been observed to be dull, with high frequency tokens used too often and interesting content words used too rarely (Holtzman et al. 2019; Dinan et al. 2019). Moreover, the models repeat themselves at the token, phrase, and sentence levels, and statistics comparing set of human-generated utterances and model-generated responses indicate discrepancy between the human and model word distributions. This does not appear to be rectified by training on more data (Radford al.) 2019). Recent fixes involve modifying the decoding strategy using sampling or more sophisticated beam search variants. However, these decoding strategies do not address the core issue: the model's underlying sequence probabilities are clearly not correct.
Several reasons for exactly why neural text is degenerate have been posited, with the cause currently unknown. Possible candidates include the problem being (i) a by-product of the model architecture, e.g. the Transformer architecture preferring repeats (Holtzman al. 2019 Vig 2018), (ii) an intrin- sic property of human language (Holtzman et al.) 2019) rather than a modeling deficiency, or that (ii) a training objective relying on fixed corpora cannot take into account the real goal of using the language (Choi, 2018). Our work shows that, while the above may be factors, a primary factor is the use of the likelihood objective itself, as we demonstrate that degeneration alleviated if we replace the likelihood objective with our proposal.
While low perplexity in the limit should lead to predicting the correct next target word, there are two major flaws of the likelihood objective: (i) it pays relatively little attention to the argmax the top of the ranked list next token probabilities, instead optimizing the likelihood of the entire distribution;
Equal contribution; the ordering was decided by a coin flip.

---

(ii) it is not focused on optimizing sequence generation, only on producing the next token. The first issue means that greedy or beam search decoding, which rely on the top of the list to generate, are not optimized there is discrepancy between maximizing the log-probability of ground-truth token and ensuring the rank of the ground-truth token to be one. The second issue means that during sequence generation, any imperfection in next token prediction leads error accumulation that is not addressed by likelihood training.
In this work, we introduce unlikelihood training, an approach that addresses the two aforementioned issues. It combines two types of updates: a likelihood update on the true target tokens so that they are assigned high probability, and an unlikelihood update on tokens that are otherwise assigned too high probability. We can collect these unlikely token candidates either during next-token prediction or from generated sequences, allowing us to train at both the token and sequence levels. Both token and sequence level unlikelihood training are shown to improve metrics that measure dullness and repetition of the model, while maintaining performance in other metries such as perplexity or token accuracy compared to the maximum likelihood baseline. Finally, we assess our models using human evaluations. We find that our generations have vastly improved quality compared to likelihood trained models when both models use beam search decoding. Moreover, our approach when using beam search also significantly improves over likelihood trained models using either beam blocking or nucleus sampling, thus outperforming the current oate--of-theart.
RELATED WORK
Neural Text Degeneration Recently, several papers have observed various forms of neural text degeneration, especially in open-ended generation tasks. In dialogue, it has been shown that there is mismatch between model and human word distributions, where generative models are more likely to output frequent words, but less likely to produce rare words compared to humans. For example, this was observed across all generative models submitted to the A12 NeurIPS 2018 competition (Dinan et 2019). In language modeling, the work Holtzman et al. (2019) highlighted problems with the word frequency distribution and level of repetition in model generations compared to human text. These issues are not remedied by simply increasing the amount of the training data; e.g. large- scale GPT-2 language models (Radford et al. 2019) display the same issues.
Improved Decoding Algorithms Several methods have been proposed to rectify these issues. The primary ones involve changing the decoding method to a sophisticated beam search variant or stochastic decoding, e.g. sampling. Different variants of beam search have been explored (Li et al., 2016 Vijayakumar et al., 2018 Kulikov et al.) 2018 Holtzman et al. 2018) which can decrease a model's 's level of repetition by selecting candidates that are unlike previously chosen ones. Separately, hard or soft beam blocking has been investigated (Paulus et al. 2017 Klein et al.) (017)), whereby previously generated n-grams are blocked from subsequent generation. This approach is often used in dialogue generation, fixing some token or phrase level repetitions but removing repetitions that would naturally occur in human text.
The second major approach is that of sampling from the model at generation time. Top k-sampling (Fan et al.] 2018) and nucleus sampling (Holtzman et al. 2019) are two methods that sample se- quences based on function of the predicted next token probability distribution given by the model. Both approaches vastly improve the repetition issue, as the randomization often reduces the number of duplicate tokens in decoded sequence, even if highly scored paths under the model (represented by beam search candidates) contain repetitions. However, as the underlying model is unchanged, often prefers semantically similar phrasing, depending on the temperature parameter of the sampling (Holtzman et al. 2019). Furthermore, this solution is less relevant in less open-ended tasks such as machine translation, where beam search variants are the preferred method. Ideally we would like model that can work with both beam and sampling decoding methods
Improved Learning Algorithms The proposed learning criteria are closely related to structured output prediction methods in which the goal is to increase the scores assigned by a model to true examples while decreasing those assigned to negative examples often generated by the model it- self. Some representative algorithms include structured perceptron (Collins) 2002). energy- models (LeCun et al., 2006) and more recently reflective likelihood (Dieng et al. (018). A par- ticular variant in this family of algorithms, called negative training, was recently used by He and
2

---

Glass 2019 to prevent generic and malicious responses in dialogue models. Similarly, these struc- tured prediction algorithms with neural language models have been applied to machine translation in recent years by Shen al. (2013) and Edunov et al. (2017).
NEURAL TEXT GENERATION
Language Modeling In language modeling, goal is to model a probability distribution p- (x) over variable-length text sequences x = (x1) composed of tokens from a vocabulary. 2t E
Po are similar to samples from p., and po (x) R p. (x) for all x. When (x) is parameterized by neural network, we call pe a neural language model. We assume that pa takes the form px(x) = I1 (
V. We wish to find model (x) which resembles p. (x), meaning that samples
The de facto approach to training such a model is to find parameters 0 that maximize the log- likelihood of finite set of samples 'D from p. by minimizing:
1 x
Lpo D) =- Σ
E
log (1 (0)))
Sequence Completion A closely related problem consists of sampling a sub-sequence, or prefix, Xi:k ~ P.r then using Po conditionally decode a continuation, N po(-(x1:a). We now want the resulting completion (fi....,Ik;$ 06-1,. , to resemble sample from p.-
Tk;
We use sequence completion its generality. For instance, sequence completion encompasses story generation (Fan et al. 2018). contextual text completion (Radford al. 2019), language modeling (for k 0), and dialogue modeling (Zaagg et al., 2018) where 11:k is dialogue history and continuation is a next utterance.
setting study the behavior of neural language models due to
Given pe and prefix X1.k. finding the optimal continuation not tractable, so in practice approxi- mate deterministic or stochastic decoding strategies are used to generate continuations.
Deterministic Decoding Two widely used deterministic decoding approaches are greedy search and beam search. The former can be seen as a special case of the latter. Greedy search selects the highest probability token at each time step: 2; = arg max pe (ze x<t)). Beam search maintains fixed-size set of partially-decoded sequences, called hypotheses. At each time step. beam search forms new hypotheses by appending each token in the vocabulary to each existing hypothesis, scor- ing the resulting sequences then selecting the highest scoring sequences. As describe in Section these deterministic decoding strategies, which depend highly on underlying model probabilities, expose issues with conventionally trained neural language models.
Stochastic Decoding An alternative is to sample from a model-dependent distribution at each step, It ~ q(Fe/ Po). order to prevent sampling low probability tokens, typical approach is to restrict sampling to a subset of the vocabulary U C V at each step:
(1 po) ppo x4 U
otherwise,
where Z = = po(x) The top-k sampler restricts sampling to the & most-probable tokens; iee U is the size & subset of which maximizes Erev po(x)(e))) (Fan et al. 2018). The nucleus sampler instead restricts sampling to the smallest set of tokens with total mass above threshold e [0, 1]; i.c. U the smallest subset with Exev po(x) >= (Holtzman et al. 2019).
NEURAL TEXT DEGENERATION
In this section we discuss two degenerate properties that frequently occur in conventional neural language models trained with the maximum likelihood objective (Equation I).
First, generalgeee continuations exhibit sequence-evvlrreettito especially with deterministic decoding. The problem is seen by observing samples in Appendix Table 4] which
Repetition
3

---

shows completions from the ofate-of-the-art GPT-2 language model (Radford al. 2019). Greedy decoding as well as top-k and nucleus sampling exhibit degenerate repetition (with a certain hyper- parameter setting), although greedy decoding shows the worst degradation. Using Transformer language model trained with maximum likelihood (6.). we find that the average percentage re- peated n-grams in model continuations with greedy decoding (43%) far exceeds that of humans (0.5%), computed over prefixes drawn from a validation corpus.
Unlike previous work which only focused on degenerate sequence-level repeats (Holtzman et al. 2019). we additionally observe that neural language models exhibit substantially more repetition in next-token prediction compared human text:
Pr ($x+1 arg max po (2 N1:k) E x1:k) Pr (Ix+1 X1:k).
For instance, the Transformer language model ( predicted next-tokens that appeared in the pre- ceding 128 words 62% of the time, versus 49% in ground-truth text. This is especially concerning since the
objective focuses on optimizing next-token conditional distributions.
Token Distribution Mismatch Second, both greedy continuations and next-token predictions from conventional neural text generators have different token distributions from human text. As demonstrated by Holtzman et al. (2019), such models with greedy or beam search tend to produce high frequency tokens too often and low frequency tokens too rarely, where frequency is defined by the human token distribution. With the Transformer language model (% the set of next- token greedy predictions on a held-out validation set had roughly 40% fewer unique tokens than the ground-truth tokens (11.6k vs. 18.9k), and overproduced frequent tokens (Appendix [Figure 1). Such behavior has been linked to generations being judged as dull by humans because rare words can add engaging specificity (Weston al. 2018 See et al. 2019).
THE UNLIKELIHOOD TRAINING OBJECTIVE
We now describe unlikelihood training for neural language models, then in Section 6 demonstrate empirically that our proposal substantially improves neural text degeneration
5.1 UNLIKELIHOOD TRAINING
The key idea behind unlikelihood training is decreasing the model's 's probability of certain tokens, called negative candidates. Given a sequence (I1,..., .Ty) and a set of negative candidate tokens C (c Cm. where each Cj V. we define the unlikelihood loss for step as:
(= = Σ
log(1 po(c(x<1)).
(3)
oEC'
The loss decreases as c)( decreases. We incorporate the unlikelihood loss into a token-level unlikelihood objective which augments each time-step of maximum likelihood training:
Ci (Po( aa Σ log(1 po(c)=<))) oEC
log lo
(4)
likelihood
unlikelihood
As candidates, we use previous context tokens:
Cjrev-context {=1...... 11-13 {xz).
(5)
Intuitively, minimizing the unlikelihood loss with this candidate set makes (i) incorrect repeating tokens less likely, as the previous context contains potential repeats, and (ii) frequent tokens less likely, as these tokens appear often in the previous context. These candidates are efficient to com- pute, without requiring additional supervision.
Gradient analysis We assume pa (xx (x<e)) = softmax(a) and consider the gradient (4 with respect to the softmax input E RVV With single negative candidate, the (negative) gradient is:
(1 a Por) if + inca
1
L. x- m O p.
m, (1+a)
if = inog
4

---

where x² {0, 13 is a one-hot ground-truth vector, RV, po(-) and Pueg is the probability of the negative candidate at index 'nog (derivation in Appendix A
This unlikelihood gradient (6) differs from the likelihood gradient, (z* (2*-p) p). due to the term which varies based on the ground-truth token index i", the unlikelihood gradient is positive, increasing the ground-truth token's probability with a magnitude that grows with Pneg- Conversely, at the negative candidate index ineg the gradient is negative. At all other token indices to positive as Poeg increases. For instance, with a = 1.0 the gradient increases the probability of each token 2, when the model assigns high probability to the negative candidate (Pneg" 0.5).
a and the model's negative candidate probability, Pneg. At the
{i", ineg]. the gradient moves from negative
5.2 SEQUENCE-LEVEI UNLIKELIHOOD TRAINING
While the token-level unlikelihood objective efficiently augments maximum likelihood training with token-level penalties, it is limited to prefixes drawn from the training distribution. The resulting distribution mismatch between training sequences and generated sequences is a known issue with maximum likellhood training. motivating objectives that operate on model-generated sequences DDaméé et al. 2009 Ross et al. 2011 Ranzato et al. 2015 Yu et al. (016)..
We thus propose a sequence-level unlikelihood objective which uses unlikelihood on decoded con- tinuations. That is, given prefix (z1...., ,x)) Pa, we decode a continuation (Ix+1:.... Dk++N) n ~ po(-|z), ................................ Jx construct per-step negative candidate sets (C++1 ...., ....,,,,+N,, and define each per- step sequence-level loss for & E {k + I....,k N as:
Cirs(po((1<<)).C)))
Σ log(1 po(c(=<1)).
cECs
Intuitively, the negative candidates identify problematic tokens for the loss to penalize. choose to penalize repeating n-grams in the continuation:
We
Crepent
{ if (Zz-i,.... Lt. ,,....,,2....................... ((ts). Z for any (j - n, nn
(8)
which says that 24 is the (single) negative candidate for step t if it is part of repeating n-gram
In our experiments we apply this sequence loss in two ways: (i) using it to fine-tune a standard MLE baseline; and (ii) using to fine-tune an unlikelihood model trained at the token level, LuL-token- We refer to the former as Lur-seq and the latter as Loneennen In both cases, fine-tuning is done by equally mixing sequence-level unlikelihood updates (7) and the token-level loss from which it was initially trained (either likelihood updates ( or token-leved unlikelihood updates
Efficiency Any objective that requires explicitly decoding a sequence is constrained by sample efficiency when decoding is slow; if sample efficiency is low, the total decoding time is too large for practical use. In our experiments we show that when used for fine-tuning, the sequence-level unlike- lihood objective substantially reduced degeneration in under 1,500 updates, rendering it practical for modern large-scale neural models, even with high decoding costs.
6 EXPERIMENTS
We follow standard language modeling setup from Baevski and Auli (2099) and evaluate our method on the task of sequence completion, detailed below
Model Architecture Recent large-scale language models are based on the Transformer architec- ture, multi-layer feed-forward network with self-attention (Vaswani al., (017). We use 16-layer Transformer with 8 attention heads, embedding dimension 1024, and fully-connected dimension 4096; the architecture is based on Baevski and Auli (2019) but with standard embedding and soft- max layers. Our proposed method is architecture agnostic; we choose this one as representative of recent lcage-scale language models, e.g. Radford et al. (2019).
An alternative tried is choose penalization probability Prenalize, and 2; the single negative and no negative candidate for time otherwise; this
candidate for time when 2, Bernoulli(Ppaaliee) approach was effective but under-performed the Crepeat candidates; see Appendix ?Code and trained models are available at hhtpps://og www.b.cco.. www.coo/.accooooocc facebookresearcch
implemented with Fairseq (Ott et al. 2019)
5

---

seq-rep-4
Eyvically song excerpts of different languages including French. Spanish
EMILE
Italian, Spanish, Italian, Spanish. Italian, Spanish. Spanish. Portuguese Portuguese , Portuguese Portuguese Portuguesc. Portuguae Portuguese Portuguese Portuguese. Portuguese Portuguese Portuguese Spanish Portuguese, Portuguese- Portuguese, Portuguese Portuguese
0.744
and German In the first verse the protagonist sings about being girl who 's been in love with someone clse ", while second verse describes the relationhhip between the protagonist and lover. the third verse protugonist sings
0.063
Prefix EXLLE
starboard enginer and was going to crash. We 're going in.
he said "We 're going to crash. We going crash. We te going crush. We 're going to crash. We 're going going
0.787
to crash.
going crash. We '1o going to crash. going get back to the water "The order abandon ship was given the remaining two buttlecruisers
Hood said. "I 'm going to make sure by Beatty. who
0.000
away
Hood turned
Prefix
- ligh 27 points on 8-for slooting with three rebownds
Luns:
10 team. February 2012, he was named to the Atlantic 10 first team On Febeuary 5, 20122 he was named
January
named the
0.277
high assists against Minnesota Timberwolves... February
to 0.064
All NBA First Team. On March 7, 2012. he was named one of five finalists for the Naismith Award, which is
Table 1: Example greedy completions showing representative examples of the MLE model's de- generate single-token repetition (top), phrase-level repetition (middle), and 'structural" repetition (bottom), as well as the proposed method's s ability to fix these degenerate behaviors.
Dataset We use the Wikitext-103 dataset Merity et al. 2016). large-scale collection of Wikipedia articles containing over 100 million words and 260 thousand unique tokens. As document-level dataset, Wikitext-103 is an open-source representative of recent datasets used for large-scale language modeling (Baevski and Auli 2019; Radford et al. 2019). We perform experi- ments at the word level.
Training We train on fixed-length contiguous sequences, in our case of length 1,536, which was selected based on GPU memory constraints. For the token-level losses (LMLE. LuL-wkee). we train each model on 8 GPUs for maximum of 150k updates, evaluating on the validation set and saving the model state every 10k updates. For the experiments below, select the saved model state with the best validation perplexity.
Sequence-level fine-tuning begins with the model state selected based on the validation perplexity. Models are fine-tuned for 1,500 total updates. With probability 0.5 an update uses CULS and other- wise uses the token-level loss with which the model trained. For LULS update, we split each training sequence and greedily decode continuations (details below). The experiments use a prefix length i 50 and continuation length N 100 for fine-tuning.
Completions We evaluate a model on sequence completion by using the model to decode contin- uations of prefixes derived from the validation (or test) set. Specifically, the validation (or test) set first partitioned into sequences 1,536 tokens, as in training. Then we split each sequence into batch of prefixes of length & (discarding extra tokens), and decode continuation of length N for each prefix. The experiments below use = 50 and N = 100 for evaluation. For deterministic decoding we use greedy search and beam search with beam size 10, and for stochastic decoding we use top-k sampling with k E {3, 50} and nucleus sampling with E (0.3, 0.9).
6.1 EVALUATION METRICS
Repetition As a token-level metric for repetition, we use the fraction of next-token (top-1) predic- tions that occur in the previous é tokens (rep/(); given set 'D of length-7 sequences,
1 rep/f = IDIT
larg max E x--e-1-1).
(9)
A predicted token is called "single-token repeat" when [:] 1. Some of these single-token repeats also occur in the human- -enunrate sequences, and we thus report a variant which only counts single- token repeats that are additionally not equal to the ground-truth next-token (wrep/f).

---

Model
search
seq-rep-4
uniq-seq
ppl
acc
rep
wrep
uniq
greedy
.442
10.8k
LMLE
25.64
.395
.627
.352
11.8k
beam
.523
9.5k
greedy
.283
13.2k
LuLssden
26.91
.390
.577
.311
12.7k
beam
.336
11.7k
greedy
.177
13.1k
Lu
25.42
.399
.609
.335
12.8k
beam
.019
18.3k
greedy
.058
15.4k
LLLsstensoog
26.72
.395
.559
.293
13.8k
beam
.013
19.1k
Human
.006
19.8k
.487
19.8k
Table Results for token-level objectives (upper) and sequence-level fine-tuning (lower) according to sequence-level (left) and token-level (right) metrics using the test subset of Wikitext-100
We use the portion of duplicate n-grams (seq-rep-n) in generated sequence to measure sequence- level repetition. That is, for continuation N1+1: compute,
seq-rep-n 1.0 Jnique n-grams(X
(10)
n-grams|
and average continuations. seq-rep-n is zero when the continuation has repeating n-grams, and increases towards 1.0 as the model repeats. We compute seq-rep-n on the continuation.
quantify a model's predicted token distribution using the number of unique tokens. As a token-level metric (uniq). tions on a validation or test set D, i.. Harg p(axp(z) (<z)) T<t E D. As a sequence-leve metric (uniq-seq) we use the number of unique tokens in continuations of validation test prefixes (46).
Token Distribution
the number of unique next-token predic-
Language Modeling Quality We use perplexity (ppl), and next-token prediction accuracy (acc), defined as ( max p(xe)(ee)) xt xet ()), with N prefixes v<t and true next tokens x;.
6.2 RESULTS
Token-level and sequence-lever results on the test set are in Table 2 (valid set Appendix Table (5)
Baseline The baseline model trained with maximum likelihood (CMLe) achieved 25.64 test per- plexity, comparable to a current state-of-the-an art system (Baevski and Auli (019) (24.92). However, the greedy baseline's 's seq-level repeats (seq-rep-4 .442) and single-token repeats (rep .627) far ceed those in human text (.006, .487 respectively). The baseline continuations have far fewer unique tokens than human text (uniq-seq 11 11.8k 19.8k), with a high rate of frequent tokens (Figure
Token-Level Objective The proposed token-level unlikelihood objective (CuL-token) reduced next- token wrong repetition (wrep .311 vs. .352) while increasing the number of unique next- tokens (uniq 12.7k vs. 11.8k) compared to the baseline (CwL.)). Perplexity and accuracy were similar.
Importantly, the token-level unlikelihood objective yielded substantial improvements in sequence- level generations. With greedy search, token-level unlikelihood training improved the 4-gram repe- tition in continuations by 36% (seq-rep-4 ,283 vs. .442) while generating roughly 22% more unique tokens than the baseline (uniq-seq 13.2k 10.8k), and more favorable rate of infrequent tokens (Figure D). With beam search, unlikelihood training showed similar improvements over the baseline.
Sequence-Leve Objective ments, with 97 % reduction in 4-gram repetitions (seq-rep-4 ,013 vs. .442) from the baseline level (greedy LMLE), and 77% % more unique tokens (uniq-seq 19.1k vs. 10.8k) with beam search.
The sequence level fine-tuning (CULLtoken-seg) yielded further improve-
Compared to the token-level unlikelihood model (CuL-token) which the starting point of fine- tuning, the fine-tuned model's repetition substantially improved (seq-rep-4 .058 vs. .283), unique tokens increased (uniq-seq 15.4k 13.2k), and token-level metrics such perplexity improved
7

---

Crowdworkers
Experts
Winner
Loser
Win rate
Win rate
LtL-totenn
LALL baseline
57%
LuL-se
LMLE baseline
*71%
LuLttkenssog
beats
LMLE baseline
*82%
seg
Lul-soken
*75%
LUL-ukentssg
Cul
59%
LALE Nucleus sampling (p = 0.9)
59%
'83%
beats
Ltnttkenssog
CMLL Beam blocking (4-gram)
60%
#74%
Table 3: Human eval results. * denotes statistical significance (2-sided binomial test, p < .05).
(ppl 26.72 vs. 26.91), despite using only 1,500 updates. The token distribution improved, with in- frequent tokens produced more often than the baseline, and frequent tokens approaching the human level (Tigure[T]. Finally, after sequence-level fine-tuning, beam search out-performed greedy search.
To visualize how these improvements in metrics translate to generation quality, Table shows greedy completions that characterize the baseline's degeneration and LUL-tokenssey 's improved behavior.
GPT-2 Fine-Tuning In the preceding experiment, sequence-level fine-tuning alone (LuL-see) showed substantial improvements over the baseline using a small number of updates. This indi- cates that the proposed sequence-level fine-tuning can be a cheap, effective way to improve existing pre-trained language models. We demonstrate this by fine-tuning a pre-trained GPT-2 (Radford jet al. 2019 language model with sequence-level unlikelihood, using comparable experimental setup (details in Appendix(C) Fine-tuning with unlikelihood yiclded similar improvements in sequence-level repetition (seq-rep-4 .042 vs. .506) to those observed Table [ while maintaining language modeling quality according perplexity and accuracy (see Appendix Table 7).
Stochastic Decoding Although we have focused deterministic decoding, also confirm that model trained with the proposed unlikelihood objectives may still be used with stochastic decoders. Appendix Table 6 shows metrics for completions generated with top-k sampling (Fan et al.) 2018) and nucleus sampling (Holtzman et al.) (019).. Models trained with unlikelihood objectives maintain language modeling quality compared the baseline, but with improvements in repetition.
Human Evaluation We perform crowdworker evaluation to judge the quality of the generations of our proposed models compared to each other, the baseline, two other generation methods, and the reference. We employ pairwise setup: an evaluator is presented with prefix and shown continuations from two different models and asked to select which continuation they found more natural. Following[Li Li et al. (2019), we filter workers using quality controls (detailed in Appendix E and limit the number of annotations that they may complete. Prompts are from the Wikitext- 103 test set. All models used beam search (beam size 10) for generation, except for those that use stochastic decoding. We report the win rates for each pairwise comparison.
The main results are presented Table with additional experiments in Appendix Appendix/Tabley We find that all proposed models are preferred over the baseline, and that congruent with automatic metrics, win rates improve after adding sequence level objective. Our best model also outperforms the baseline used with either nucleus sampling or beam blocking.
We also collected limited annotations from other NLP researchers. These Expert annotators were given the same UI as the crowdworkers, and not told about models they were evaluating, but all annotators were familiar with language models. As shown in Table the LuL-wkentsey model signif- icantly outperforms both nucleus sampling and beam blocking according the experts.
CONCLUSION
We described unlikelihodd training, an approach to training neural language models. We observed that state-of-fe tiiee----therrt models trained to maximize likelihood exhibit neural text degeneration, which
8

---

characterized and quantified in terms of repetition and token distribution mismatch. Our results show that the likelihood objective is not constrained enough, in the sense that two models with the same perplexity can exhibit wildly different generation performance. We empirically showed that unlikelihood training both at the token and sequence levels substantially reduced degeneration according to automatic metrics, and outperformed likelihood-trained models with various decoding methods according to human evaluation, being superior to the current state-of-the-en approaches.

---

REFERENCES
Alexei Baevski and Michael Auli. 2019. (Adaptive input representations for neural language model- ing In International Conference on Learning Representations.
Yejin Choi. 2018. The missing representation in neural (language) models. 3rd Workshop on Rep- resentation Learning for NLP (RepLANLP).
Michael Collins. 2002. Discriminative training methods for hidden Markov models: Theory and experiments with perceptron algorithms. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002). Association for Computational Lin- guisties.
Hal Daumé, John Langford, and Daniel Marcu, 2009. Search-based structured prediction. Machine learning, 75(3):297-325.
Adji B Dieng, Kyunghyun Cho, David M Blei, and Yann LeCun. 2018. Learning with reflective likelihoods.
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2019. The second conversational intelligence challenge (convai2) arXiv preprint 1XX1:0992200099.
Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc' Aurelio Ranzato. 2017. Classical structured prediction losses for sequence sequence learning. arXiv preprint arXiv:1711.04956.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv: 880.04333. 04833.
Tianxing He and James Glass. 2019. Negative training for neural dialogue response generation. arXiv preprint arXiv:1903.02133
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018 Learning to write with cooperative discriminators In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1; Long Papers), pages 1638-1649. Association for Computational Linguistics.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint 1r0iv
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. arXiv preprint arXiv:
Ilya Kulikov, Alexander H Miller, Kyunghyun Cho, and Jason Weston. 2018. Importance of a search strategy in neural dialogue modelling arXiv preprini arXiv:1811.0090
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. 2006. A tutorial on energy- based learning. Predicting structured data.
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. A simple, fast diverse decoding algorithm for neural generation. arXiv preprint arXiv: 161 108562.
Margaret Li, Jason Weston, and Stephen Roller. 2019. Acute-eval: Improved dialogue evaluation
with optimized questions and multi-turn comparisons arXiv preprini arXiv:1909.088: 1909.03087.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mix- ture models. arXiv preprint 160.::60000777433
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv: 043044.
10

---

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl Blog, 1(8).
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. CoRR, abs/1511.0677
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635.
Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversa- tion? how controllable attributes affect human judgments) In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume I (Long and Short Papers), pages 1702-1723, Minneapolis, Min- nesota. Association for Computational Linguistics.
Shiqi Shen, Yong Cheng, Zhongjun He, Wei He. Hua Wu, Maosong Sun, and Yang Liu. 2015. Minimum risk training for neural machine translation. arXiv preprint 1rXiv:1512.02443:
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Jesse Vig. 2018. Deconstructing bert: Distilling 6 patterns from 100 million parameters. Medium.
Ashwin K Vijayakumar, Michael Cogswell, Ramprasaath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search for improved description of complex scenes. In Thirty-Second AAAI Conference on Artificial Intelligence.
Jason Weston, Emily Dinan, and Alexander H Miller. 2018. Retrieve and refine: Improved sequence generation models for dialogue. arXiv preprint
Lantao Yu, Weinan Zhang, Jun Wang, and Yingrui Yu. 2016. Seqgan: Sequence generative adver- sarial nets with policy gradient. ArXiv, abs/1 /e//166090577..
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: have dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2204 2213, Melbourne, Australia. Association for Computational Linguistics.

---

Prefix GPT2-greedy
Lyrically the song has excerpts of different languages including French, Spanish, German Italian Portuguese Spanish Portuguese Portuguese Portuguese Portuguese Portuguese Portuguese Portuguese Portuguese Portuguese Portuguese, Portuguese, Portuguese, Portuguese
GPT2-top3
German, Italian, Portuguese, , Spanish, Portuguese, Portuguese, Italian and Span- ish. is also available English, French Spanish. addition its lyries, the album features a number of original songs....
GPT2-mucleus-0.3
German, Italian, Portuguese Spanish, Portuguese Italian Portuguese Spanish Portuguese Spanish Portuguese Spanish Portuguese Spanish Portuguese, Spanish, Portuguese Spamish, Portuguese Spanish, Portuguese... ....
Prefix
The first pair of pereiopods is armed with large, asymı
GPT2-greedy
etrical, and long-range laser cannon. The second pair is armed with a large asym- metrical, and long-range laser cannon. The third pair is armed with a large asym- metrical, and long-ramg laser cannon. The fourth pair is armed with a large etrical, and highly mobile head, and the second pair has small, asymmetrical, and highly mobile head. The second pair has large and highly mobile head, and the third pair is armed with a large and highly mobile head. The first pair
GPT2-top3
GPT2-mucleus-0.3 etrical head and a large body. The first pair of pereiopods is armed with a large asymmetrical head and a large body. The first pair of pereiopods is armed with large. asymmetrical head and large body. The first pair of pereiopods is armed
Table 4: Top: Degenerate repetition in completions from state-of-the-art large-scale language model (Radford et al. 2019). The examples contain single-word repetitions, phrase-level repetitions, and structural repetitions where some tokens within a repeating phrase vary. Recently proposed stochastic samplers (top-k, nucleus) exhibit degeneration based on hyper-parameter settings.
generation frequency distribution
generation frequency distribution
0.6
0.6
Iuman
Human
Liokisee
Lookisse
Lseg
Likk
Beam Black
Limle
generstee takoro
0.4
generated takers
0.4
0.3
0.3
0.6
3.6
very rare (20%)) Toben frequency
modium (3056)
frequert (405))
very rarc (20%) modium Tolen frequency
froeuant (40%))
(a) Different combinations of unlikelihood
(b) Unlikelihood vs. stochastic decoding
Figure 1: Sequence-level token distribution using the test subset of Wikitext- 103. Nucleus sampling
(p 0.9) and beam blocking (n = 4) are used with the maximum likelihood baseline (LMLR).
A GRADIENT
Notation Let of be the true next-token (index i* E V) at step t, and let Tneg be negative candidate (index ). Let p = p(zz x<(e)= E R be the output of softmax(a) where a E IRV.
Denote the probability of an element E (1,..., probabilities of the true next-token, negative-candidate token, and any other token with $ (i i..
...,, V) as p; = p(a (a<e), and let p., Paey, and 0 be
A.1 DERIVATION
The (negative) token-level loss with a single candidate is,
L: log p(x; log(1 pl pl (x)))
(11)
12

---

and its gradient with respect to logit a; is:
3p: Oa, = (1li = [ - -p;)
ac Op,
Poog
(1li = Incgl
p4)
(12)
Preg
We consider the gradient when is the true next-token, negative-candidatte and any other token.
True Next-Token (i=i")
0L 0p.
Pneg
= - )
-p. -
(0 P.)
(13)
Pp_ da.
Paay
Pneg
= Pp
(14)
Pneg
Negative Candidate (i = ineg)
OC Opang (0 Pmag))
Pneg
(1
Pneg
(15)
Pacc Uancg
Pose
- (1 + (x).
(16)
Other Token (i & {i", )
OC Op:
(0 -p:)
Pnez
(0 Pia)
(17)
0p, Oa;
Pneg
Pneg
-p
(18)
Poeg
Combining the three cases above, we get:
VL. x² p:
(19)
where x E {0, 13V I at index i and otherwise, and m RVV
is:
(1 0t Pace
o iney
m;
(20)
(1 + i == ineg-
Multiple Candidates In general the objective considers multiple candidates (see section 5)
(po (po = -a
Σ
log(1 Pe (c(xx=)))
log ( x2))
(21)
GEC
likelihood
unlikelihood
We regroup the token-level objective to be weighted sum of per-candidate objectives:
(po (-1 (1
Σ
(1ogpo(z)(=<d) + az -og(( po po(c(x<))))
(22)
|c*
cEC
where 0g |C*).
Now the gradient can be generalized to multiple candidates, in which case the gradient takes the same form Eqn. 20 but with a, in place of
13

---

Model
search
seq-rep-4
uniq-seq
ppl
acc
rep
wrep
uniq
greedy
.499
10.6k
LMLE
24.59
.401
.619
.346
11.6k
beam
.455
9.4k
greedy
.274
12.6k
CUL-token
25.62
.396
.569
.305
12.5k
beam
.327
11.2k
greedy
.130
12.7k
LUL-seg
24.28
.406
.603
.329
12.4k
beam
.018
16.8k
greedy
.051
14.8k
25.37
.401
.551
.287
13.4k
beam
.013
17.6k
Human
.005
18.9k
.479
18.9k
Table 5: Results for token-level objectives (upper) and sequence-level fine-tuning (lower) according sequence-level (left) and token-level (right) metrics using the validation subset of wikitext-103.
Search
Model
seq-rep-4
uniq-seq
ppl
acc
rер
wrep
uniq
MLEE
.0991
14.7k
25.70
.350
.597
.355
12.6k
LLtteknn
.0491
16.4k
27.02
.344
.539
.306
13.6k
top-k-3
LuL-s0
.0068
17.9k
25.11
.353
.581
.341
13.6k
Lokeokon-seg
.0087
15.2k
26.84
.347
.524
.29
14.6k
DMLE
.0165
21.9k
25.70
.302
.511
.303
16.1k
LUL-tokon
.006
23.5k
27.02
.286
.440
.247
17.8k
top-k-50
Lul-vey
.0005
25.7k
25.11
.291
.497
.291
17.3k
LuL-tokenssey
.0009
23.7k
26.84
.289
.430
.238
18.8k
LMLE
.273
13.6k
25.70
.264
.339
.154
12.6k
LLL-token
.101
16.5k
27.02
.247
.290
.121
13.9k
top-p-0.3
LuL-seq
.0033
20.8k
25.11
.266
.327
.145
13.6k
.0041
19.1k
26.84
.250
.284
.116
14.9k
CMLE
.0154
26.9k
25.70
.288
.462
.263
18.6k
LuL-ukenn
.004
30.2k
27.02
.266
.381
.202
22.3k
top-p-0.9
Luisey
.0003
34.7k
25.11
.290
.450
.254
19.6k
Lokenwey
.0007
32.4k
26.84
.269
.376
.198
22.7k
Human
.006
19.8k
.487
19.8k
Table Stochastic decoding results according sequence-level (left) and token-level (right) met- rics using the test subset of Wikitext-10S3
B STOCHASTIC DECODING RESULTS
Table 6 provides automatic metrics for top-k and nucleus sampling (called top-p) on the Wikitext- 103 test set. These can be compared with the main results of the paper in Table In general, sam- pling methods yield worse next-token predictions than deterministic approaches (0.302 vs. 0.394 acc for top-k-50 vs. greedy MLE, where acc for stochastic decoding measures the probability that the decoding strategy chooses the ground truth word given a ground truth context). As the choice of sampling hyperparameter gets closer to greedy (i.e. lower values of k and p) next token accu- racy improves, eventually approaching the greedy MLE results. The unlikelihood-rrained sampling counterparts, but exhibit
models have similar next token accuracy (acc) to their likelihood-trail fewer repetitions. For lower values and the improvements unlikelihood training are larger, e.g. 0.277 reduced 0.0041 for 4-gram sequence repetitions (seq-rep-4) using top-p- 00.3-...3. At higher levels of p and k, for all methods the continuations contain more unique tokens than that humans, meaning those values may be too high.
14

---

Model
search
seq-rep-4
ppl
acc
rер
wтер
uniq
GPT-2
greedy
.506
20.75
.430
.589
.306
13.3k
GPT--2LLE
greedy
.460
15.82
.464
.612
.305
11.8k
GPI--uu--sa
greedy
.042
18.49
.444
.613
.317
11.3k
Human
.005
.407
17.7k
Table 7: GPT-2 results according to sequence-level and token-level metrics using the validation subset of wikitext-103. seq-rep-4 is computed on the word level; ppl, acc, rep, wrep are computed on the BPE level.
C GPT-2 FINE-TUNING
We evaluated the GPT-2 medium pre-trained model ('GPTT22" and two separate fine-tuning variants on Wikitext- 103. The first variant ('GPT-2XLE") was fine-tuned using maximum likelihood; we select the model state with the lowest validation perplexity. The second model ("GPT-2UL-as was fine-tuned using the sequence-level unlikelihood objective (:5.2). For both evaluation and sequence- level tuning, we used prefix length of 50 BPE tokens and continuation length of 100 BPE tokens. In order to train on a single GPU, we used a batch-size of 1024 tokens for MLE updates, and 300 prefix tokens for unlikelihood updates. Due the smaller batch size and single-GPU setting, used 10,000 updates during sequence-leve fine-tuning, comparable to the 1,500 updates in the main experiment (6 in terms of the total number of tokens. Results shown in Table 7
D SEQUENCE-LEVEL RANDOM CANDIDATES
In Sec. 5.2 described way penalize tokens that occurred in n-gram repetition. One al- ternative is to penalize a random subset of the generated sequence. That is, given continuation Tt-1:..... Kr now define per-step candidates (Ck+1 Ck+NN as:
{ (xL) (0
if 21 =1
Ct random-soq =
(23)
2f = O.
for each & {k: 1....., + N}. where 2, Bernoulli(Ppenatia ), and Ppenalize |0, 1] is fixed hyper-parameter. Intuitively, these candidates identify random tokens in the generated sequence (hence 'random-seq'), which then penalized by the sequence-level loss (Eqn. 77)
Results with different values of Ppenalize are shown in Table g Penalizing 10% of the generated tokens led substantial improvements in seq-rep-4 for both greedy and beam search compared to the baseline (c.g. 41% for LUL-seq greedy, 73% for CUL-tokegg greedy), though using n-gram repetition candidates yielded further improvements (:5.2) Table 5). Improvements in single-token metrics were similar to those from the n-gram repetition candidates (e.g. wrep .287). These results with random-seg candidates demonstrate that sequence fine-tuning can yield improvements without explicitly using the notion of repetition for candidate selection. We also find that penalizing 90% of the generated tokens yields substantial improvements beam search, but not greedy search; investigatin this is left future work.
15

---

Model
Prenalize
search
seq-rep-4
uniq-seq
ppl
acc
rep
wrep
uniq
greedy
.429
10.6k
LILE
24.590
.401
.619
.366
11.6k
beam
.495
9.4k
greedy
.233
9.9k
Lulvey
0.1
24.329
.404
.602
.330
12.3k
beam
.744
13.1k
greedy
.434
5.3k
Lun-soq
0.9
26.519
.399
.600
.330
12.2k
beam
.231
13.5k
greedy
.166
12.5k
LUL-tok+seg
0.1
25.518
.399
.551
.287
13.2k
beam
.166
14.2k
greedy
.423
6.7k
LuL-taktsey
0.9
26.629
.396
.551
.288
13.2k
beam
.080
16k
Human
.005
18.9k
.479
18.9k
Table 8: Results for sequence-level fine-tuning using random-seq candidates according to sequence-level (left) and token-level (right) metrics using the validation subset of wikitext-103.
E HUMAN EVALUATION DETAILS
E.1 UI SCREENSHOT
Which writer is from Wikipedia?
media aeprtartiiee
aovert seched seeest
and the rap arhters luriy'
canee
gave Atara cocon reggied
Governor
Early life and education
und dll oer
Fitmaryy 21:
perge buttar persoc appira atsr
Widddt?
Figure 2: Screen shot of the user interface used in the human evaluation.
E.2 CROWDWORKER QUALITY CONTROLS
We require workers to correctly answer both of the following quality control questions for their evaluations to be included. Both quality controls compare the true completion against a greedy baseline model.
Following Li et al. (201)). we informed workers that they must provide reasoning for their choices. We filtered workers who did not provide reasoning for least 80% of their choices.
16

---

63% of workers fail at least one of our three quality control mechanisms (2 quality control metries, and failing to give reasons). 61% fail at least one quality control question; 16% of workers fail both; 4% of workers fail to give reasoning for their choices.
E.2.1 QUALITY CONTROL
In the decades since its release, The Hustler has cemented its reputation as classic Roger Ebert echoing earlier praise for the performances direction, and cinematography and adding laurels for editor Dede Allen, cites the film as one of""
Prompt ==
Correct answer those films where scenes have such psychic weight that they grow in our memo- ries. He further cites Fast Eddie Felson the audience refers to them as touchstones. TV Guide calls the film a dark stunner offering grim world whose only bright spot is the top of the pool table, yet with characters who main- tain a shabby nobility and grace. "The four leads are again lavishly praised for their performances and the
of " only handful of movie characters real that
Incorrect answer the most influential films of the year " In his review for the Chicago Sun @-@ Times, Richard Corliss calls a film of the highest order " and a film of the highest order " In his review for the Chicago Sun @-@ Times Richard Corliss calls it film of the highest order and a film of the highest order " his review for the Chicago Sun @-@ Times, Richard Corliss calls it film of the highest order and film of
E.2.2 QUALITY CONTROL 2
Prompt which had major negative effect the state 's large merchant fleet. Gore in 1808 elected to the Massachusetts House of Representatives. where he successfully led Federalist efforts ensure the selection of Federalist slate of presidential electors. He also spearheaded actions
Correct answer drive Senator John Quincy Adams from the Federalist Party over his support of Thomas Jefferson 's foreign policy The legislature elected Adams successor nine months early and gave Adams sufficiently distasteful instructions that he resigned the post and joined with the Republicans
Gore led the Federalists to victory in 1809 against Sullivan successor, Levi Lincoln, Sr. who had taken over as acting governor upon Sullivan 's death late in 1808. During Gore 's term the principal domestic issue occupying state politics
Governor
Incorrect Answer prevent the American Revolutionary War from being fought by the British and to prevent the British from using the country to launch war against the British Gore actions in the House of Representatives were major turning point in his political career He was elected to the House of Representatives in 1811 and served until his death in 1815 = Early life and education j/s; j/s; Gore was born Boston, Massachusetts. February 22, 1798. the of Benjamin Gore and his
E.3 FULL HUMAN EVALUATION RESULTS
17

---

Crowdworkers
Experts
Winner
Loser
Win
W-L
Win rate
W-L
CUL-token
LMeE baseline
57%
17-13
CUL-soq
LMLE baseline
#71%
41-17
Lokeekenesey
beats
MLE baseline
*82%
41-9
LoL-nokenssey
LUL-woenn
#75%
56-19
sok
LuI-seq
59%
38-27
C UuLtokken-so
Nucleus
59%
47-33
*83%
30-6
beats
LUL-token-seq
Beam blocking
60%
50-34
*74%
25-9
Reference Reference
LLE baseline
*85%
17-3
Nucleus
#69%
38-17
Reference
Beam blocking
*68%
48-23
beats
Reference
LUL-token
#73%
44-16
Reference
LUL-seg
50%
30-30
Reference
LoLeekeneseg
#64%
46-26
Table 9: Full human evaluation results. Includes additional comparisons omitted for brevity, and the number of wins and loses by each comparison.
18


Reviews:
[{"review_id": "ryeEqCDG5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper targets on solving the dull and repetitive outputs in MLE training for neural text generation. The authors propose a new unlikelyhood training to avoid assigning much probability to frequent and repetitive words. The authors combine the proposed algorithms and beam search and state that the results improved over beam blocking and neculus decoding.  \n\nThe unlikelyhood training is to provide a set of negative candidates and minimize the probability of these tokens. This raises several practical issues: how to choose a reasonable $\\alpha$. \nThis set can be chosen as the previous tokens in the sequence. This is a reasonable choice, but the author does not state why the other choices are not working,e.g. Sharpening the distribution using temperature. A potential counter case is that there are similar words exists in the sequences, but the unlikely loss trends to distinguish these synonyms.  The other unlikelyhood training choice is called sequence-level set. However, it seems not sequence-level but just n-gram center.  A question would be why not chose the whole n-gram instead of just choosing the center of n-gram. Also, why a prefix is really needed is questionable. \n\n\nEq 8 seems wrong, why$i \\leq n \\leq j$\n\nTable 2 should have shown the original sequences on the repetition metrics to show it indeed make sense.ppl should be enough, acc seems redundant. It seems that unlikely training may be harmful to ppl, which is the common metric to evaluate generation quality. A better discussion should be made on this to explain why it performance or if ppl has some problem.\n\nTable 3 comparison may not be reasonable. As Nucleus sampling and beam blocking is not in training phase. This comparison is not really fair.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJeHygz6YH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nContributions:\n\nThe main contribution of this paper lies in the proposed unlikelihood training objective for open-ended text generation. The key idea is to enforce the unlikely generations to be assigned lower probability by the model. Both token and sequence-level unlikelihood training objectives are provided. Impressively, the authors show that models trained with the proposed method can generate high-quality text via only beam search, without using top-k, nucleus sampling, or beam blocking methods. \n\nStrengths:\n\n(1) Writing & Clarity: The proposed model is very well motivated, the paper is well written, and clearly presented. I enjoyed reading the paper. \n\n(2) Novelty: Though the proposed model is simple, I think it has novelty inside. The proposed model makes connection to negative sampling, and is very intuitive to reduce repetition during the training stage, instead of decoding stage. I find the gradient analysis in Section 5.1 is especially interesting. \n\n(3) Experiments: The authors did a careful job in experiments design, and conducting the experiments. Human evaluation is also provided. A lot of additional results are provided in Appendix. I feel the experiments are solid and convincing.  \n\nWeaknesses:\n\n(1) Clarity: I have three questions regarding this paper. \n\na) Using previous generated tokens as the unlikely tokens for the current generation step in the token-level unlikelihood training is intuitive, but also seems too simple. Can the authors provide some comments on this? Or, are there any better designs? \n\nb) How is the training looking like? Do we need Gumbel-softmax-like trick to backpropagate through the generated tokens in the sequence-level training? Or, this is not needed? Can the authors clarity the training process? \n\nc) The proposed model can be directly applied to dialog response generation task, which also requires diversity in generated responses. Any reason why this conditional generation task is not performed? Or, do the authors plan to also apply the proposed method to this application?\n \n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lxTev3Yr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposes training losses, unlikelihood objective, for mitigating the repetition problem of the text generated by recent neural language models. The problem is well-motivated by evidence from the existing literature. Specifically, the paper argues that the main cause of the degenerated output is the maximum likelihood objective commonly used to train language models. Their main contribution is to introduce additional objectives to penalize “unlikely” word probabilities. The proposed penalty is derived into 2 objectives: token level (previous words in context) and sentence level (future decoded words). The prior objective is used along with the MLE, while the later and more expensive is used for fine-tuning. They perform experiments on Wikitext-103 and evaluate models on the perplexity of the models, and n-gram statistics such as repetition, and uniqueness of the decoded texts. The proposed training scheme (UL-token+seq) is shown to have the closest statistics to the original corpus while the perplexity slightly suffers. The additional manual analysis shows that human annotators prefer the outputs (sentence completion) of the proposed method over the other baselines.\n\nOverall, this paper tackles a relevant problem and could propose a novel method.\n\nFor the unlikelihood objectives, there are a few clarifications on the design decision. There are some “correct” repetitions in the ground-truth text as well. However, the proposed loss minimizes all repetitions regardless. In addition, it is unclear how the proposed method mitigates the token distribution mismatch. Finally, there is a similar work where they attempt to “match” repetition (or n-gram distribution) of a reference corpus (https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16961). A discussion of how this paper distinguishes from previous work would be helpful.\n\nFor the experiments, there are some missing detail and concerns:\n\n1. The fine-tuning using UL-seq (eq 7) procedure is not well explained. For example, how many sequences are used per update? How many times that you decode in the 1,500 updates?\n\n2. the stochastic decoding results are related as the paper motivation is built on top of Holtzman et al., 2019, the results also support the claim. However, how do you explain the discrepancy between the experts and the crowd workers?\n\n3. GPT-2 results show that UL-seq does not significantly improve several evaluation metrics from MLE. This is a conflict with the result in Table 2. This could be a sign of the generalization problem of the proposed method. Additional results on different model architectures would be helpful.\n\nMinor question:\n1. It is uncertain how these repetition and uniqueness statistics translate to a downstream task (e.g. summarization or NMT). Do you have results regarding this?\n\n2. It appears that UL-token does not do much. What is the increase in training time? Do you recommend using the token loss? \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new objective for text generation with neural nets.  The main insight is that the standard likelihood objective assigns excessive probability to sequences containing repeated and frequent words.  The paper proposes an objective that penalizes these patterns.  This technique yields better text generation than alternative methods according to human evaluations.

The reviewers found the paper to be written clearly. They found the problem to be relevant and found the proposed solution method to be both novel and simple.  The experiments were carefully designed and the results were convincing.  The reviewers raised several concerns on particular details of the method.  These concerns were largely addressed by the authors in their response.  Overall, the reviewers did not find the weaknesses of the paper to be serious flaws.

This paper should be published. The paper provides a clearly presented solution for a relevant problem, along with careful experiments. 


Decision: Accept (Poster)