Neural Text Generation With Unlikelihood Training

1 INTRODUCTION . Neural text generation is a vital tool in a wide range of natural language applications . However , the standard approach – training a sequence to sequence model , e.g . Transformer ( Vaswani et al. , 2017 ) , to maximize log-likelihood and approximately decoding the most likely sequence – is known to be flawed . Generated text in open-ended applications such as language modeling or dialogue has been observed to be dull , with high frequency tokens used too often and interesting content words used too rarely ( Holtzman et al. , 2019 ; Dinan et al. , 2019 ) . Moreover , the models repeat themselves at the token , phrase , and sentence levels , and statistics comparing a set of human-generated utterances and model-generated responses indicate a discrepancy between the human and model word distributions . This does not appear to be rectified by training on more data ( Radford et al. , 2019 ) . Recent fixes involve modifying the decoding strategy using sampling or more sophisticated beam search variants . However , these decoding strategies do not address the core issue : the model ’ s underlying sequence probabilities are clearly not correct . Several reasons for exactly why neural text is degenerate have been posited , with the cause currently unknown . Possible candidates include the problem being ( i ) a by-product of the model architecture , e.g . the Transformer architecture preferring repeats ( Holtzman et al. , 2019 ; Vig , 2018 ) , ( ii ) an intrinsic property of human language ( Holtzman et al. , 2019 ) rather than a modeling deficiency , or that ( iii ) a training objective relying on fixed corpora can not take into account the real goal of using the language ( Choi , 2018 ) . Our work shows that , while the above may be factors , a primary factor is the use of the likelihood objective itself , as we demonstrate that degeneration is alleviated if we replace the likelihood objective with our proposal . While low perplexity in the limit should lead to predicting the correct next target word , there are two major flaws of the likelihood objective : ( i ) it pays relatively little attention to the argmax or the top of the ranked list of next token probabilities , instead optimizing the likelihood of the entire distribution ; ∗Equal contribution ; the ordering was decided by a coin flip . ( ii ) it is not focused on optimizing sequence generation , only on producing the next token . The first issue means that greedy or beam search decoding , which rely on the top of the list to generate , are not optimized – there is a discrepancy between maximizing the log-probability of a ground-truth token and ensuring the rank of the ground-truth token to be one . The second issue means that during sequence generation , any imperfection in next token prediction leads to error accumulation that is not addressed by likelihood training . In this work , we introduce unlikelihood training , an approach that addresses the two aforementioned issues . It combines two types of updates : a likelihood update on the true target tokens so that they are assigned high probability , and an unlikelihood update on tokens that are otherwise assigned too high a probability . We can collect these unlikely token candidates either during next-token prediction or from generated sequences , allowing us to train at both the token and sequence levels . Both token and sequence level unlikelihood training are shown to improve metrics that measure dullness and repetition of the model , while maintaining performance in other metrics such as perplexity or token accuracy compared to the maximum likelihood baseline . Finally , we assess our models using human evaluations . We find that our generations have vastly improved quality compared to likelihood trained models when both models use beam search decoding . Moreover , our approach when using beam search also significantly improves over likelihood trained models using either beam blocking or nucleus sampling , thus outperforming the current state-of-the-art . 2 RELATED WORK . Neural Text Degeneration Recently , several papers have observed various forms of neural text degeneration , especially in open-ended generation tasks . In dialogue , it has been shown that there is a mismatch between model and human word distributions , where generative models are more likely to output frequent words , but less likely to produce rare words compared to humans . For example , this was observed across all generative models submitted to the ConvAI2 NeurIPS 2018 competition ( Dinan et al. , 2019 ) . In language modeling , the work of Holtzman et al . ( 2019 ) highlighted problems with the word frequency distribution and level of repetition in model generations compared to human text . These issues are not remedied by simply increasing the amount of the training data ; e.g . largescale GPT-2 language models ( Radford et al. , 2019 ) display the same issues . Improved Decoding Algorithms Several methods have been proposed to rectify these issues . The primary ones involve changing the decoding method to a sophisticated beam search variant or to stochastic decoding , e.g . sampling . Different variants of beam search have been explored ( Li et al. , 2016 ; Vijayakumar et al. , 2018 ; Kulikov et al. , 2018 ; Holtzman et al. , 2018 ) which can decrease a model ’ s level of repetition by selecting candidates that are unlike previously chosen ones . Separately , hard or soft beam blocking has been investigated ( Paulus et al. , 2017 ; Klein et al. , 2017 ) , whereby previously generated n-grams are blocked from subsequent generation . This approach is often used in dialogue generation , fixing some token or phrase level repetitions but removing repetitions that would naturally occur in human text . The second major approach is that of sampling from the model at generation time . Top k-sampling ( Fan et al. , 2018 ) and nucleus sampling ( Holtzman et al. , 2019 ) are two methods that sample sequences based on a function of the predicted next token probability distribution given by the model . Both approaches vastly improve the repetition issue , as the randomization often reduces the number of duplicate tokens in a decoded sequence , even if highly scored paths under the model ( represented by beam search candidates ) contain repetitions . However , as the underlying model is unchanged , it often prefers semantically similar phrasing , depending on the temperature parameter of the sampling ( Holtzman et al. , 2019 ) . Furthermore , this solution is less relevant in less open-ended tasks such as machine translation , where beam search variants are the preferred method . Ideally we would like a model that can work with both beam and sampling decoding methods . Improved Learning Algorithms The proposed learning criteria are closely related to structured output prediction methods in which the goal is to increase the scores assigned by a model to true examples while decreasing those assigned to negative examples often generated by the model itself . Some representative algorithms include structured perceptron ( Collins , 2002 ) , energy-based models ( LeCun et al. , 2006 ) and more recently reflective likelihood ( Dieng et al. , 2018 ) . A particular variant in this family of algorithms , called negative training , was recently used by He and Glass ( 2019 ) to prevent generic and malicious responses in dialogue models . Similarly , these structured prediction algorithms with neural language models have been applied to machine translation in recent years by Shen et al . ( 2015 ) and Edunov et al . ( 2017 ) . 3 NEURAL TEXT GENERATION . Language Modeling In language modeling , our goal is to model a probability distribution p∗ ( x ) over variable-length text sequences x = ( x1 , . . . , x|x| ) composed of tokens from a vocabulary , xt ∈ V . We wish to find a model pθ ( x ) which resembles p∗ ( x ) , meaning that samples x̂ ∼ pθ are similar to samples from p∗ , and pθ ( x ) ≈ p∗ ( x ) for all x . When pθ ( x ) is parameterized by a neural network , we call pθ a neural language model . We assume that pθ takes the form pθ ( x ) = ∏|x| t=1 pθ ( xt|x < t ) . The de facto approach to training such a model is to find parameters θ that maximize the loglikelihood of a finite set of samples D from p∗ by minimizing : LMLE ( pθ , D ) = − |D|∑ i=1 |x ( i ) |∑ t=1 log pθ ( x ( i ) t |x ( i ) < t ) . ( 1 ) Sequence Completion A closely related problem consists of sampling a sub-sequence , or prefix , x1 : k ∼ p∗ , then using pθ to conditionally decode a continuation , x̂k+1 : N ∼ pθ ( ·|x1 : k ) . We now want the resulting completion ( x1 , . . . , xk , x̂k+1 , . . . , x̂N ) to resemble a sample from p∗ . We use sequence completion as a setting to study the behavior of neural language models due to its generality . For instance , sequence completion encompasses story generation ( Fan et al. , 2018 ) , contextual text completion ( Radford et al. , 2019 ) , language modeling ( for k = 0 ) , and dialogue modeling ( Zhang et al. , 2018 ) where x1 : k is a dialogue history and a continuation is a next utterance . Given pθ and a prefix x1 : k , finding the optimal continuation is not tractable , so in practice approximate deterministic or stochastic decoding strategies are used to generate continuations . Deterministic Decoding Two widely used deterministic decoding approaches are greedy search and beam search . The former can be seen as a special case of the latter . Greedy search selects the highest probability token at each time step : xt = arg max pθ ( xt|x < t ) . Beam search maintains a fixed-size set of partially-decoded sequences , called hypotheses . At each time step , beam search forms new hypotheses by appending each token in the vocabulary to each existing hypothesis , scoring the resulting sequences then selecting the highest scoring sequences . As we describe in Section 4 , these deterministic decoding strategies , which depend highly on underlying model probabilities , expose issues with conventionally trained neural language models . Stochastic Decoding An alternative is to sample from a model-dependent distribution at each step , xt ∼ q ( xt|x < t , pθ ) . In order to prevent sampling low probability tokens , a typical approach is to restrict sampling to a subset of the vocabulary U ⊂ V at each step : q ( xt|x < t , pθ ) = { pθ ( xt|x < t ) /Z xt ∈ U 0 otherwise , where Z = ∑ x∈U pθ ( x|x < t ) . The top-k sampler restricts sampling to the k most-probable tokens ; i.e . U is the size k subset of V which maximizes ∑ x∈U pθ ( x|x < t ) ( Fan et al. , 2018 ) . The nucleus sampler instead restricts sampling to the smallest set of tokens with total mass above a threshold p ∈ [ 0 , 1 ] ; i.e . U is the smallest subset with ∑ x∈U pθ ( x|x < t ) > = p ( Holtzman et al. , 2019 ) .

Summary: This paper proposes training losses, unlikelihood objective, for mitigating the repetition problem of the text generated by recent neural language models. The problem is well-motivated by evidence from the existing literature. Specifically, the paper argues that the main cause of the degenerated output is the maximum likelihood objective commonly used to train language models. Their main contribution is to introduce additional objectives to penalize “unlikely” word probabilities. The proposed penalty is derived into 2 objectives: token level (previous words in context) and sentence level (future decoded words). The prior objective is used along with the MLE, while the later and more expensive is used for fine-tuning. They perform experiments on Wikitext-103 and evaluate models on the perplexity of the models, and n-gram statistics such as repetition, and uniqueness of the decoded texts. The proposed training scheme (UL-token+seq) is shown to have the closest statistics to the original corpus while the perplexity slightly suffers. The additional manual analysis shows that human annotators prefer the outputs (sentence completion) of the proposed method over the other baselines.