Neural Text Generation With Unlikelihood Training

1 INTRODUCTION . Neural text generation is a vital tool in a wide range of natural language applications . However , the standard approach – training a sequence to sequence model , e.g . Transformer ( Vaswani et al. , 2017 ) , to maximize log-likelihood and approximately decoding the most likely sequence – is known to be flawed . Generated text in open-ended applications such as language modeling or dialogue has been observed to be dull , with high frequency tokens used too often and interesting content words used too rarely ( Holtzman et al. , 2019 ; Dinan et al. , 2019 ) . Moreover , the models repeat themselves at the token , phrase , and sentence levels , and statistics comparing a set of human-generated utterances and model-generated responses indicate a discrepancy between the human and model word distributions . This does not appear to be rectified by training on more data ( Radford et al. , 2019 ) . Recent fixes involve modifying the decoding strategy using sampling or more sophisticated beam search variants . However , these decoding strategies do not address the core issue : the model ’ s underlying sequence probabilities are clearly not correct . Several reasons for exactly why neural text is degenerate have been posited , with the cause currently unknown . Possible candidates include the problem being ( i ) a by-product of the model architecture , e.g . the Transformer architecture preferring repeats ( Holtzman et al. , 2019 ; Vig , 2018 ) , ( ii ) an intrinsic property of human language ( Holtzman et al. , 2019 ) rather than a modeling deficiency , or that ( iii ) a training objective relying on fixed corpora can not take into account the real goal of using the language ( Choi , 2018 ) . Our work shows that , while the above may be factors , a primary factor is the use of the likelihood objective itself , as we demonstrate that degeneration is alleviated if we replace the likelihood objective with our proposal . While low perplexity in the limit should lead to predicting the correct next target word , there are two major flaws of the likelihood objective : ( i ) it pays relatively little attention to the argmax or the top of the ranked list of next token probabilities , instead optimizing the likelihood of the entire distribution ; ∗Equal contribution ; the ordering was decided by a coin flip . ( ii ) it is not focused on optimizing sequence generation , only on producing the next token . The first issue means that greedy or beam search decoding , which rely on the top of the list to generate , are not optimized – there is a discrepancy between maximizing the log-probability of a ground-truth token and ensuring the rank of the ground-truth token to be one . The second issue means that during sequence generation , any imperfection in next token prediction leads to error accumulation that is not addressed by likelihood training . In this work , we introduce unlikelihood training , an approach that addresses the two aforementioned issues . It combines two types of updates : a likelihood update on the true target tokens so that they are assigned high probability , and an unlikelihood update on tokens that are otherwise assigned too high a probability . We can collect these unlikely token candidates either during next-token prediction or from generated sequences , allowing us to train at both the token and sequence levels . Both token and sequence level unlikelihood training are shown to improve metrics that measure dullness and repetition of the model , while maintaining performance in other metrics such as perplexity or token accuracy compared to the maximum likelihood baseline . Finally , we assess our models using human evaluations . We find that our generations have vastly improved quality compared to likelihood trained models when both models use beam search decoding . Moreover , our approach when using beam search also significantly improves over likelihood trained models using either beam blocking or nucleus sampling , thus outperforming the current state-of-the-art . 2 RELATED WORK . Neural Text Degeneration Recently , several papers have observed various forms of neural text degeneration , especially in open-ended generation tasks . In dialogue , it has been shown that there is a mismatch between model and human word distributions , where generative models are more likely to output frequent words , but less likely to produce rare words compared to humans . For example , this was observed across all generative models submitted to the ConvAI2 NeurIPS 2018 competition ( Dinan et al. , 2019 ) . In language modeling , the work of Holtzman et al . ( 2019 ) highlighted problems with the word frequency distribution and level of repetition in model generations compared to human text . These issues are not remedied by simply increasing the amount of the training data ; e.g . largescale GPT-2 language models ( Radford et al. , 2019 ) display the same issues . Improved Decoding Algorithms Several methods have been proposed to rectify these issues . The primary ones involve changing the decoding method to a sophisticated beam search variant or to stochastic decoding , e.g . sampling . Different variants of beam search have been explored ( Li et al. , 2016 ; Vijayakumar et al. , 2018 ; Kulikov et al. , 2018 ; Holtzman et al. , 2018 ) which can decrease a model ’ s level of repetition by selecting candidates that are unlike previously chosen ones . Separately , hard or soft beam blocking has been investigated ( Paulus et al. , 2017 ; Klein et al. , 2017 ) , whereby previously generated n-grams are blocked from subsequent generation . This approach is often used in dialogue generation , fixing some token or phrase level repetitions but removing repetitions that would naturally occur in human text . The second major approach is that of sampling from the model at generation time . Top k-sampling ( Fan et al. , 2018 ) and nucleus sampling ( Holtzman et al. , 2019 ) are two methods that sample sequences based on a function of the predicted next token probability distribution given by the model . Both approaches vastly improve the repetition issue , as the randomization often reduces the number of duplicate tokens in a decoded sequence , even if highly scored paths under the model ( represented by beam search candidates ) contain repetitions . However , as the underlying model is unchanged , it often prefers semantically similar phrasing , depending on the temperature parameter of the sampling ( Holtzman et al. , 2019 ) . Furthermore , this solution is less relevant in less open-ended tasks such as machine translation , where beam search variants are the preferred method . Ideally we would like a model that can work with both beam and sampling decoding methods . Improved Learning Algorithms The proposed learning criteria are closely related to structured output prediction methods in which the goal is to increase the scores assigned by a model to true examples while decreasing those assigned to negative examples often generated by the model itself . Some representative algorithms include structured perceptron ( Collins , 2002 ) , energy-based models ( LeCun et al. , 2006 ) and more recently reflective likelihood ( Dieng et al. , 2018 ) . A particular variant in this family of algorithms , called negative training , was recently used by He and Glass ( 2019 ) to prevent generic and malicious responses in dialogue models . Similarly , these structured prediction algorithms with neural language models have been applied to machine translation in recent years by Shen et al . ( 2015 ) and Edunov et al . ( 2017 ) . 3 NEURAL TEXT GENERATION . Language Modeling In language modeling , our goal is to model a probability distribution p∗ ( x ) over variable-length text sequences x = ( x1 , . . . , x|x| ) composed of tokens from a vocabulary , xt ∈ V . We wish to find a model pθ ( x ) which resembles p∗ ( x ) , meaning that samples x̂ ∼ pθ are similar to samples from p∗ , and pθ ( x ) ≈ p∗ ( x ) for all x . When pθ ( x ) is parameterized by a neural network , we call pθ a neural language model . We assume that pθ takes the form pθ ( x ) = ∏|x| t=1 pθ ( xt|x < t ) . The de facto approach to training such a model is to find parameters θ that maximize the loglikelihood of a finite set of samples D from p∗ by minimizing : LMLE ( pθ , D ) = − |D|∑ i=1 |x ( i ) |∑ t=1 log pθ ( x ( i ) t |x ( i ) < t ) . ( 1 ) Sequence Completion A closely related problem consists of sampling a sub-sequence , or prefix , x1 : k ∼ p∗ , then using pθ to conditionally decode a continuation , x̂k+1 : N ∼ pθ ( ·|x1 : k ) . We now want the resulting completion ( x1 , . . . , xk , x̂k+1 , . . . , x̂N ) to resemble a sample from p∗ . We use sequence completion as a setting to study the behavior of neural language models due to its generality . For instance , sequence completion encompasses story generation ( Fan et al. , 2018 ) , contextual text completion ( Radford et al. , 2019 ) , language modeling ( for k = 0 ) , and dialogue modeling ( Zhang et al. , 2018 ) where x1 : k is a dialogue history and a continuation is a next utterance . Given pθ and a prefix x1 : k , finding the optimal continuation is not tractable , so in practice approximate deterministic or stochastic decoding strategies are used to generate continuations . Deterministic Decoding Two widely used deterministic decoding approaches are greedy search and beam search . The former can be seen as a special case of the latter . Greedy search selects the highest probability token at each time step : xt = arg max pθ ( xt|x < t ) . Beam search maintains a fixed-size set of partially-decoded sequences , called hypotheses . At each time step , beam search forms new hypotheses by appending each token in the vocabulary to each existing hypothesis , scoring the resulting sequences then selecting the highest scoring sequences . As we describe in Section 4 , these deterministic decoding strategies , which depend highly on underlying model probabilities , expose issues with conventionally trained neural language models . Stochastic Decoding An alternative is to sample from a model-dependent distribution at each step , xt ∼ q ( xt|x < t , pθ ) . In order to prevent sampling low probability tokens , a typical approach is to restrict sampling to a subset of the vocabulary U ⊂ V at each step : q ( xt|x < t , pθ ) = { pθ ( xt|x < t ) /Z xt ∈ U 0 otherwise , where Z = ∑ x∈U pθ ( x|x < t ) . The top-k sampler restricts sampling to the k most-probable tokens ; i.e . U is the size k subset of V which maximizes ∑ x∈U pθ ( x|x < t ) ( Fan et al. , 2018 ) . The nucleus sampler instead restricts sampling to the smallest set of tokens with total mass above a threshold p ∈ [ 0 , 1 ] ; i.e . U is the smallest subset with ∑ x∈U pθ ( x|x < t ) > = p ( Holtzman et al. , 2019 ) .

Summary: The main contribution of this paper lies in the proposed unlikelihood training objective for open-ended text generation. The key idea is to enforce the unlikely generations to be assigned lower probability by the model. Both token and sequence-level unlikelihood training objectives are provided. Impressively, the authors show that models trained with the proposed method can generate high-quality text via only beam search, without using top-k, nucleus sampling, or beam blocking methods.