Straight to the Gradient: Learning to Use Novel Tokens for Neural Text Generation

1 INTRODUCTION . Text generation has been one of the most important research problems in natural language processing ( NLP ) ( Reiter & Dale , 2000 ) . Thanks to the advances in neural architectures , models are now capable of generating texts that are of better quality than before ( Brown et al. , 2020 ) . However , despite the countless efforts that have been made to improve neural architectures , models trained with the standard Maximum Likelihood Estimation ( MLE ) objective are known to prefer generating dull and highly repetitive texts . For instance , in open-ended generation tasks , such as story continuation or open dialogue generation , it has been observed that even with large pre-trained models , e.g. , GPT-2 ( Radford et al. , 2019 ) , high frequency tokens largely dominate the generation ( Welleck et al. , 2020 ; Holtzman et al. , 2020 ) . The same observation has been reported in directed generation tasks such as text summarization ( Nallapati et al. , 2016 ; See et al. , 2017 ) , image captioning ( Melas-Kyriazi et al. , 2018 ; Wang & Chan , 2019 ) and machine translation ( Tu et al. , 2016 ; Stahlberg & Byrne , 2019 ) . The methods introduced to solve the aforementioned issues with neural text generation can be primarily categorized into two groups : ( i ) training based methods , which include incorporating auxiliary losses ( See et al. , 2017 ; Welleck et al. , 2020 ) and coverage vector ( See et al. , 2017 ; Tu et al. , 2016 ) ; ( ii ) decoding based methods , such as stochastic beam search ( Kool et al. , 2019 ) , top-k sampling ( Fan et al. , 2018 ) and nucleus sampling ( Holtzman et al. , 2020 ) . Though decoding based methods , in particular nucleus and top-k sampling , perform well in practice in open-ended generation tasks , significantly reducing degeneration problem , they do not address the fundamental issue that the token-level probabilities produced by the neural model are problematic ( Welleck et al. , 2020 ) . In addition , our experiments demonstrate that sampling methods also fail to generate high-quality texts in directed generation tasks such as abstractive text summarization . In this work , based on the known observation that the model trained with MLE objective tends to generate repititive tokens or phrases , we introduce a novel method called ScaleGrad for neural text generation training , by directly maneuvering the gradients to make the model learn to use novel tokens during training . Our method lies in the training based group , which aims to address the fundamental modeling problem , that is , the token-level distribution predicted by the model . We conduct extensive experiments with different neural architectures including LSTM ( Hochreiter & Schmidhuber , 1997 ) and Transformer ( Vaswani et al. , 2017 ) across different tasks in openedended and directed text generation . Through extensive analysis we demonstrate that ScaleGrad consistently improves the generation quality according to both human evaluation and automatic metrics . Compared to other training based methods , ScaleGrad is architecturally simpler and easier to fit into current neural models ( §3.2 ) , while possessing a wider applicability to different tasks compared to decoding based methods ( §4.2 and §5.2 ) . 2 BACKGROUND . 2.1 NEURAL TEXT GENERATION . The NLP tasks involving text generation can be broadly categorized into two types : directed generation and open-ended generation ( Holtzman et al. , 2020 ) . In the former case , the output text can be seen as a constrained transformation of the input . Examples include text summarization , machine translation , and image captioning . In the later case , the input context only provides a certain degree of constraints such that the model is allowed to generate the following texts with a considerable degree of freedom . Story/text continuation and dialogue generation fall in this category . Neural models frame text generation tasks as some form of conditional language modeling , which is typically trained to maximize the log likelihood ( equivalently , minimize the negative log likelihood ) of the training data . The Maximum Likelihood Estimation or MLE objective for an input-output pair ( x , y ) can be expressed as follows . LMLE = − T∑ t=1 log pθ ( yt|y < t , x ) ( 1 ) where θ denotes model parameters , T is the length of the output sequence y , and x is the taskspecific input condition , e.g. , source document in summarization , image in image captioning , conversation history in dialogue generation and ∅ in text continuation . Teacher Forcing ( Williams & Zipser , 1989 ) , where current step ’ s target token is passed as the next input to the decoder rather than the predicted token , is usually used to train neural text generation models for faster convergence . Degeneration Degeneration has been a key problem in neural text generation models for openended tasks , where the model generates texts that are repetitive , overly generic ( dull ) , incoherent and gibberish . It can happen at different levels of granularity – token , phrase , sentence and paragraph . The problem has not been mitigated even with large-scale pre-trained models like GPT-2 Large ( Radford et al. , 2019 ; Holtzman et al. , 2020 ) . Degeneration has also been observed in directed generation tasks even though the output in these tasks is confined by the input . For instance , in text summarization , most of the advanced models such as BertSum ( Liu & Lapata , 2019 ) , BART ( Lewis et al. , 2019 ) and ProphetNet ( Yan et al. , 2020 ) make use of tri-gram blocking ( Paulus et al. , 2018 ) within beam search to remove duplicate trigrams during decoding , which improves the generation quality in terms of automatic metric . This implies that even with involvement of large-scale pretrained models , degeneration still exists . Similar issues have been reported in machine translation ( Koehn & Knowles , 2017 ; Stahlberg & Byrne , 2019 ) and image-description generation ( MelasKyriazi et al. , 2018 ; Wang & Chan , 2019 ) . 2.2 COMBATING NEURAL TEXT DEGENERATION . Out of the methods proposed to tackle neural text degeneration , top-k sampling ( Fan et al. , 2018 ) and nucleus sampling ( Holtzman et al. , 2020 ) stand out as representatives of decoding based methods and unlikelihood training ( Welleck et al. , 2020 ) as a representative training based method . During each decoding step , nucleus and top-k sampling use different functions to filter the candidate tokens , thus reformalizing the probability distribution and sample the next token from the new distribution instead of maximizing the actual likelihood . Randomness brought by these sampling methods reduces duplicate tokens in the output . However , decoding strategy solely does not solve the underlying modeling problem with MLE , as pointed out by Welleck et al . ( 2020 ) . Our analysis in §5.2 also reveals that sampling methods fail to generate high-quality texts in directed generation tasks . To address the issue with MLE , neural unlikelihood ( UL ) training has been proposed . During training , at each decoding step t , UL adds an auxiliary loss to the original cross entropy loss as follows . Lt = LtMLE + LtUL = − log pθ ( yt|y < t ) − α · ∑ c∈Ct log ( 1− pθ ( c|y < t ) ) ( 2 ) where α is a hyper-parameter and Ct is the set of negative tokens at step t , which is constructed by previous context tokens that are not the current token , Ct = { y1 , . . . , yt−1 } \ yt . The auxiliary UL loss decreases the total loss based on the “ unlikely ” probabilities of negative tokens , thus implicitly reducing the probability assigned to the repetitive tokens . UL training targets at improving the underlying modeling problem , which accords with our goal . Therefore , we mainly compare our method with UL training1 . In addition , we discuss one how our method is different from UL training from the gradient perspective in §5.4 . 3 METHODOLOGY : LEARNING TO USE NOVEL TOKENS . Training a text generation model with MLE objective treats each token in the gold ( ground truth ) sequence equally . With this approach , the model exhibits the tendency to generate repetitive tokens/phrases during inference . To mitigate this degeneration problem , we argue that the model should focus on learning to use novel tokens , rather than treating all the tokens equally . Formally , let y = ( y1 , . . . , yt , . . . , yT ) be the ground-truth token sequence that the model is learning to generate in an auto-regressive manner , one token at a time . At time step t , we define the token ỹti in the vocabulary V as a novel token , if ỹti has not been generated before , i.e. , ỹti /∈ { y1 , . . . , yt−1 } . By the definition , we have a set of novel tokens Stnovel ⊆ V at each decoding step t in training , which shrinks over time as new tokens are generated ( or observed ) in the ground-truth sequence ( see Appendix B for an illustration ) . Note that the shrinking set of novel tokens is equivalent to the negative tokens in UL except that it may contain the current target token yt , if it was observed before . To encourage the model to focus on learning to use novel tokens , we propose an architecturallysimple method that can fit into most of the auto-regressive generation models . Our method , requiring no carefully-designed components , goes straight to the gradient analysis of the loss function . 3.1 GRADIENT INFORMATION IN MLE TRAINING . Let us first consider the gradient analysis of the model trained with MLE . Let ot denote the presoftmax scores ( i.e. , logits ) over the vocabulary at time step t , where oti is the score for the token with index i . Similarly , let ptk = [ softmax ( o t ) ] k represent the probability of the ground truth token with index k in the vocabulary . The partial derivative of the MLE objective ( Eq . 1 ) at time step t with respect to the logit oti can be shown as ( omitting t and ‘ MLE ’ subscript for simplicity ) : ∇oiL = ∂L ∂pk · ∂pk ∂oi = pi − 1 ( i = k ) ( 3 ) where pi = [ softmax ( o ) ] i ( derivation is given in Appendix A ) . Specifically , the gradient of the loss w.r.t . the ground truth token logit ok is ( pk − 1 ) and for any other token logit oi is pi . As the gradient-based optimization proceeds , the gradient converges to , a number that is close enough to 0 . Another interpretation is that the gradient of the loss is supposed to be close to 0 around a ( local ) minimum . Therefore , to reach the minimum point , or to make the gradient close to 0 , the model would try to reduce the probability of non-ground truth token pi and increase the probability of ground truth token pk in the MLE training . From Eq . 3 , it is clear that the gradient that every token oi in the vocabulary receives is directly related to its generation probability pi . Therefore , we hypothesize that directly manipulating the generation probabilities of tokens , thereby controlling their gradients , can help us achieve our goal , which is to train the model so that it is encouraged to use novel tokens .

Summary: This work proposes an effective modification of language model token-level distribution during the training which prevents some forms of degeneration such as repetitions and dullness. The approach is based on the idea of encouraging the model to use tokens which were not observed in the previous context so far. In other words, this method changes softmax distribution such that unseen/novel tokens is being rescaled with a given hyper-parameter $\gamma$ (eq.4). Authors conduct several experiments using different tasks such as open-ended generation, image captioning and abstractive text summarization. As a result they confirm substantial improvement over the standard mle training and **token-level** unlikelihood training. In addition to analysis of their method, authors discuss a potential issue of unlikelihood training criterion and how their approach avoids this issue.