Straight to the Gradient: Learning to Use Novel Tokens for Neural Text Generation
Authors: Xiang Lin
SIMENG HAN
Shafiq Joty
Venue:  (2021)
Keywords: text generation
text degeneration
language model
summarization
image captioning

Abstract: Advanced large-scale neural language models have led to significant success in many natural language generation tasks. However, the most commonly used training objective, Maximum Likelihood Estimation (MLE), has been shown to be problematic, where the trained model prefers using dull and repetitive phrases. In this work, we introduce ScaleGrad, a modification straight to the gradient of the loss function, to remedy the degeneration issues of the standard MLE objective. By directly maneuvering the gradient information, ScaleGrad makes the model learn to use novel tokens during training. Empirical results show the effectiveness of our method not only in open-ended generation, but also in directed generation. With the simplicity in architecture, our method can serve as a general training objective that is applicable to most of the neural text generation tasks.

Under review as a conference paper at ICLR 2021
STRAIGHT TO THE GRADIENT: LEARNING TO USE NOVEL TOKENS FOR NEURAL TEXT GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Advanced large-scale neural language models have led to significant success in many natural language generation tasks. However, the most commonly used train- ing objective, Maximum Likelihood Estimation (MLE), has been shown to be problematic, where the trained model prefers using dull and repetitive phrases. this work, we introduce ScaleGrad, a modification straight to the gradient of the loss function, to remedy the degeneration issues of the standard MLE objective. By directly maneuvering the gradient information, ScaleGrad makes the model learn to use novel tokens during training. Empirical results show the effectiveness of our method not only in open-ended generation, but also in directed generation. With the simplicity in architecture, our method can serve as a general training objective that is applicable most of the neural text generation tasks.
INTRODUCTION
Text generation has been one of the most important research problems in natural language processing (NLP) (Reiter Dale 2000). Thanks to the advances in neural architectures. models are now capa- ble of generating texts that are of better quality than before (Brown et al. 2020). However, despite the countless efforts that have been made improve neural architectures, models trained with the standard Maximum Likelihood Estimation (MLE) objective are known to prefer generating dull and highly repetitive texts. For instance, in open-ended generation tasks, such as story continuation or open dialogue generation, it has been observed that even with large pre-trained models, e.g., GPT-2 (Radford et al. 2019). high frequency tokens largely dominate the generation (Welleck et al. 2020) Holtzman et al. 2020). The same observation has been reported in directed generation tasks such as text summarization (Nallapati et al. 2016; See et al. 2017). image captioning (Melas- Kyriazi et al,
2018
Wang & Chan 2019) and machine translation (Tu et al.) 2016 Stahlberg & Byrne (019)).
The methods introduced to solve the aforementioned issues with neural text generation can be pri- marily categorized into two groups: (i) training based methods, which include incorporating auxil- iary losses (See et al. [017 Welleck et al. 2020) and coverage vector (See et al., [201]] Tu et al.) 2016); (ii) decoding based methods, such as stochastic beam search (Kool al. 2019). top-k sam- pling Fan et al. 2018) and nucleus sampling (Holtzman et al. 2020).
Though decoding based methods, particular nucleus and top-k sampling, perform well in practice in open-ended generation tasks, significantly reducing degeneration problem, they do not address the fundamental issue that the token-level probabilities produced by the neural model are problematic (Welleck et al.) (202)). In addition, our experiments demonstrate that sampling methods also fail to generate high-quality texts in directed generation tasks such as abstractive text summarization.
In this work, based on the known observation that the model trained with MLE objective tends to generate repititive tokens or phrases, we introduce a novel method called ScaleGrad for neural text generation training, by directly mancuvering the gradients to make the model learn to use novel tokens during training. Our method lies in the training based group, which aims to address the fundamental modeling problem, that is, the token-level distribution predicted by the model.
We conduct extensive experiments with different neural architectures including LSTM (Hochreiter & Schmidhuber 1997) and Transformer (Vaswani et al. 2017) across different tasks in opened- ended and directed text generation. Through extensive analysis we demonstrate that ScaleGrad consistently improves the generation quality according to both human evaluation and automatic

---

Under review as a conference paper at ICLR 2021
metrics. Compared to other training based methods, ScaleGrad is architecturally simpler and easier to fit into current neural models (33.2) while possessing a wider applicability to different tasks compared to decoding based methods (:4.7 and (5.2).
BACKGROUND
2.1 NEURAL TEXT GENERATION
The NLP tasks involving text generation can be broadly categorized into two types: directed gener- ation and open-ended generation (Holizman et al.] (020). In the former case, the output text can be seen as a constrained transformation of the input. Examples include text summarization, machine translation, and image captioning. In the later case, the input context only provides a certain degree of constraints such that the model is allowed to generate the following texts with a considerable degree of freedom. Story/text continuation and dialogue generation fall in this category.
Neural models frame text generation tasks some form of conditional language modeling, which is typically trained to maximize the log likelihood (equivalently, minimize the negative log likelihood) of the training data. The Maximum Likelihood Estimation or MLE objective for an input-output pair (x, y) can be expressed as follows.
MLE Σ
log po po (v
where 0 denotes model parameters, T is the length of the output sequence y. and æ is the task- specific input condition, e.g,, source document in summarization, image in image captioning, con- versation history in dialogue generation and 0 in text continuation. Teacher Forcing (Williams & Zipser 1989). where current step's target token is passed as the next input to the decoder rather than the predicted token, is usually used to train neural text generation models for faster convergence.
Degeneration Degeneration has been key problem in neural text generation models for open- ended tasks, where the model generates texts that are repetitive, overly generic (dull), incoherent and gibberish. It can happen at different levels of granularity token, phrase, sentence and paragraph. The problem has not been mitigated even with large-scale pre-trained models like GPT-2 Large (Radford et al. 2019 Holtzman et al. 2020). Degeneration has also been observed in directed generation tasks even though the output in these tasks is confined by the input. For instance, in text summarization, most of the advanced models such as BertSum (Liu & Lapata 2019). BART (Lewis) let al. 2019) and ProphetNet (Yan et al. 2020) make use of tri-gram blocking (Paulus et al. 2018) within beam search to remove duplicate trigrams during decoding, which improves the generation quality in terms of automatic metric. This implies that even with involvement of large-scale pre- trained models, degeneration still exists. Similar issues have been reported in machine translation (Kochn & Knowles 2017; Stthhberg & Byrne 2019) and image-description generation (Melas-
Kyriazi et al. 2018 Wang Chan 2019).
2.2
COMBATING NEURAL TEXT DEGENERATION
Out of the methods proposed to tackle neural text degeneration, top-k sampling (Fan et al., 2018) and nucleus sampling (Holtzman et al. (0200) stand out as representatives of decoding based methods and unlikelihood training (Welleck et al. (020) as representative training based method. During each decoding step, nucleus and top-k sampling use different functions to filter the candidate tokens, thus reformalizing the probability distribution and sample the next token from the new distribution in- stead of maximizing the actual likelihood. Randomness brought by these sampling methods reduces duplicate tokens in the output. However, decoding strategy solely does not solve the underlying modeling problem with MLE, as pointed out by Welleck et al. (2020). Our analysis in $5.2 also reveals that sampling methods fail to generate high-quality texts in directed generation tasks.
To address the issue with MLE, neural unlikelihood (UL) training has been proposed. During train- ing, at each decoding step UL adds auxiliary loss to the original cross entropy loss as follows.
L Cine + Lin log po (ooll
Σ cEC
log(1 (c(g)ee))
where a is a hhper--prameee parameter and C' is the set of negative tokens at step t. which is constructed by previous context tokens that not the current token, C (v) ;-....0 y -1] } \ v The auxiliary UL.
2

---

Under review as a conference paper at ICLR 2021
loss decreases the total loss based on the "unlikely" probabilities of negative tokens, thus implicitly reducing the probability assigned to the repetitive tokens. UL training targets at improving the underlying modeling problem, which accords with goal. Therefore, we mainly compare our method with UL training In addition, we discuss one how our method is different from UL training from the gradient perspective in (5.4
METHODOLOGY: LEARNING TO USE NOVEL TOKENS
Training text generation model with MLE objective treats each token in the gold (ground truth) sequence equally. With this approach, the model exhibits the tendeney to generate repetitive to- kens/phrases during inference. To mitigate this degeneration problem, we argue that the model should focus on learning to use novel tokens, rather than treating all the tokens equally.
Formally, let y (v' "'',.....''......... ............ . be the ground-truth token sequence that the model is learning to generate in an auto-regressive manner, one token at a time. At time step t, we define the token i/ in the vocabulary V novel token, if 17, has not been generated before, i.e. 5f $ {v",....,, 1/" 1/- By the definition, we have a set of novel tokens Stevel S V/ at each decoding step / in training, which shrinks over time as new tokens are generated (or observed) in the ground-truit sequence (see Appendix B for an illustration). Note that the shrinking set of novel tokens is equivalent to the negative tokens in UL except that may contain the current target token y', if it was observed before. To encourage the model to focus on learning to use novel tokens, we propose an architecturally- simple method that can fit into most of the ruto-regressive generation models. Our method, requiring no carefully-designed components, goes straight to the gradient analysis of the loss function.
y²)
3.1 GRADIENT INFORMATION IN MLE TRAINING
Let us first consider the gradient analysis of the model trained with MLE. Let o' denote the pre- softmax scores (i.e., logits) over the vocabulary time step where of is the score for the token with index i. Similarly, let P = (ootmax((o")) represent the probability of the ground truth token with index & in the vocabulary. The partial derivative of the MLE objective (Eq. [ at time step t with respect to the logit of can be shown as (omitting and 'MLE' subscript for simplicity):
OC Opk 0o:
Vo,C C
P - 1( = k))
(3)
Opk
where P; = [s)]) (derivation is given in Appendix Ap. Specifically, the gradient of the loss w.t.. the ground truth token logit oş is (p& 1) and for any other token logit is ps. As the gradient-based optimization proceeds, the gradient converges to c, a number that is close enough to O. Another interpretation is that the gradient of the loss is supposed (local) minimum. Therefore, to reach the minimum point, or to make the gradient close to 0, the model would try to reduce the probability of non-ground truth token p; and increase the probability of ground truth token Pa in the MLE training.
be close to 0 around
From Eq. it is clear that the gradient that every token o, in the vocabulary receives is directly related to its generation probability ps. Therefore, we hypothesize that directly manipulating the generation probabilities of tokens, thereby controlling their gradients, can help us achieve our goal, which is to train the model so that it is encouraged to use novel tokens.
3.2
OUR METHOD: SCALEGRAD
To encourage the model to learn to use novel tokens for generation, we can control the gradient to force the model either increase the probability of novel tokens decrease the probability for non-novel tokens. Based on this basic idea, we propose an effective training method keeping it the simplest form. Specifically, at each decoding step of training, we re-normalize the softmax output (the probability distribution over the vocabulary) in a way such that the model is informed of the current set of novel tokens and encouraged to use them. Assuming that p' is the softmax output at step t and St nowel is the corresponding set of novel tokens at that step according to our definition, we
Note that we focus comparing our work with token-level UL in this work.
3

---

Under review as a conference paper at ICLR 2021
re-compute the probability distribution follows (again omitting for notational simplicity):
if E
Snovel
Σ j=1 Pj
MI
Ps
(4)
Pi
otherwise
Σ
VI -= Pj
where y E (0, 1) is the only encourage the model to focus on novel tokens; a smaller value of incurs more aggressive push for using novel tokens. The effect this change is that we directly modify the generation probability (after re-normalization) of the novel tokens with factor of A such that 2; X D for i E Smovel with X E (0, 1). Similarly, we have p: = a. for i & S with aq > 1 Consequently, assuming that the ground truth token is indexed with k the gradient for each token has been changed to:
in our method that controls to what degree we want to
A Ps - 1,
if i = k and E Spovel
04 Pt 1,
if i = k and & n
voL (LC=--1(iA))) 1)==
(5)
P a4 Pi:
if i A & and E Snowel
if o A and i & Spovel
We now discuss why these changes encourage the model to use novel tokens. As mentioned, during training the model tries to decrease the gradient norm to to reach local minimum. First, for a ground truth token (i.e., i k)), if it is also novel token, the gradient norm [); 04 - 1| is pushed away from so that the model has to learn to increase the probability p. further to reduce the gradient norm; if it is not novel token, |ax P: 1| is pushed slightly closer to O, which still makes the model learn to predict the ground truth but with a relatively lower strength. For non-ground truth tokens (i.e., i f k)), when it is not novel token, |a: Pal increases the gradient norm so that the model learns to assign much lower probability p. to reduce it. Similarly, when the token is novel but not a ground truth token, the resulting gradient norm |^; fos becomes smaller, for which the model only moderately learns to decrease the probability Pe to reduce the norm further.
While ScaleGrad is derived from the gradient analysis of neural generation models (supervised training), it shares some commonalities with policy gradient methods in Reinforcement Learning in the sense that both operate by scaling the gradient based on different needs learning get more reward in policy gradient and learning to generate novel tokens in ScaleGrad (A (Appendix BertixxCrraa drawws this connection). Also note that the notion of novel token set can be adapted for different purposes. For example, one can define it to be set of important tokens (e.g., based on TF-IDF scores) to promote extructiveness or factual correciness in summarization. We leave such explorations for future work.
EXPERIMENTS
We showcase the performance of ScaleGrad in both open-ended and directed generation tasks. To verify the effectiveness of our approach, for all the experiments below, we use exactly the same hyper-parameters (except for method-specific ones) and setup the corresponding baseline unless stated otherwise. All the experimental details, such as model syper-parammters, training and dataset settings regarding the reproducibility can be found in Appendix G For qualitative assessments, we show examples of generated texts in Table 4 and more in Appendix
4.1 OPEN-ENDED GENERATION
Setup We consider language modeling and text auto-completion, where we compare the perfor- mance of the model trained with ScaleGrad against the models trained with MLE and unlikelihood (UL) training (Welleck et al. 2020) introduced lately to mitigate degeneration in open-ended tasks. We follow the same setup as Welleck et al. (2020). Specifically, we fine-tune the pre-trained GPT- 2 (Radford et al. 2019) on Wikitext-103 (Merity et al.) 2017). The maximum sequence length is fixed to 300 tokens for all the models. Each model is trained for maximum of 35k iterations and evaluated based on the perplexity on the validation set after every 1k iterations. We report language modeling results on the testset for each model selected according to the perplexity the validation
Since a; P. and X;- p. are new re-normalized prohahilities, they are both naturally bounded in 00, 1].
4

---

Under review as a conference paper at ICLR 2021
Table 1: Results for open-ended generation tasks on the Wikitext-103 testset. ppl. uniq and Rep/1 are computed at BPE-level and the rest are at word-level. The "f" denotes higher value for better performance and "4" is the opposite. Number marked with are estimated based on the testset. The results are averaged over 3 runs with different random seeds. Full results with standard deviation are reported in Appendix F.T
Language Modeling
Auto Completion
Models
ppl
uniq
Rep/16 |
Rep/32 +
Rep/128
Rep-1
Rep-2 4
Rep-3 +
uniq-w f
MLE
13.241
12.54k
0.234
0.380
0.619
0.661
0.500
0.424
16.83k
UL 0.5)
14.390
12.87k
0,223
0,359
0.586
0.607
0.429
0.353
17.98k
UL (a 1.0)
16.062
13.18k
0.212
0.341
0.558
0.559
0.363
0.291
19.1
SG (1 0.2)
14.203
13.61k
0.197
0.317
0,522
0.443
0.215
0,143
22.25k
Human
18.27k
0.177
0.285
0.480
0.382*
0.096*
0.037
27.55k*
set. The same saved models are also used for text auto-complction, where 50 BPE (Sennrich al. 2016) tokens (from testset) are given as prefix and the models are to generate the continuation of apply greedy decoding in all our experiments
100 next tokens. Following Welleck et al. (2020): in this section. This allows us to evaluate the modeling capability exclusively. Later, in [5.1 we analyze how our method performs with different decoding methods in open-ended generation.
In language modeling, we measure the generation quality by the standard perplexity (ppl), and Rep// and "uniq' measures of token-level distribution Welleck et al. (2020). Rep/l measures the number of times that a token from the previous tokens is repeated, when generating the following token; in our case, 1 E {16, 32, 128). The 'uniq' is defined the number of unique next-token predictions on a test/validation set. For auto-completion, we report the repetition ratios of n-gram words (Rep-n) as well as the number of unique words (uniq-w) that are used during generation on the testset.
Results From the results in Table TT, we notice that in language modeling, the model trained with ScaleGrad (SG) yields a token distribution that is much closer to human, while maintaining lower perplexity. particular, compared the best baseline. SG achieves 1%, 2%, 4% lower repetitions in Rep/16, Rep/32 and Rep/128, respectively, while having 11 11 lower perplexity. It also uses more unique tokens compared to others (e.g., 3% more compared UL training). Overall, our method significantly improves the token-level distribution and keeps high generation quality. In auto- completion, from the quantitative perspective, SG produces texts with much fewer repetitive n- n-grams compared to MLE and UL. It uses nearly 5.5k more unique words compared to the MLE bascline.
Human evaluation We have conducted a user study to verify the quality of generated texts. The study is conducted for two pairs of systems (SG vs. UL, SG vs. MLE). For each pair, we randomly choose the same 100 prefixes for the systems to produce their own continuations and ask two native speakers of English to judge which text the better continuation of the given prefix in terms of their relevance to the prefix, grammaticality and readability, More details can be found in Appendix
From the results Table 2 we can observe that the texts produced by the models trained with ScaleGrad (SG) are preferred by the human users in most of the cases, i.e., 84.0% and 70.5%, respectively. We also compute the percentage agreement and chance-correlated Gwet's AC1/gamma coefficient (Gwet) 2008) inter-user agreement to verify the reliability of the study (details in Appendix D). We see that the agreements are substantial in both measures.
Generalizability We further verify the generalizabilitg of SG by evaluating the WikiText- 103x fine- tuned model on PTB (Marcus et al. 1993) and IMDB (Maas et al. (011)) LM datasets. From the results in Appendix [F.2 we see that SG generalizes well these domains. We also show that SG consistently improves degeneration with varied decoding lengths (Appendix F.3
Table 2: Human evaluation results for auto- completion. % Agr. is the percent agree- ment and ACI denotes Gwet's AC1/gamma coefficient. Winners are marked in bold.
Table 3: Results for image paragraph captioning.
Models
CIDEr
MLE w/o 3-block
10.51
UL w/o 3-block (a=0.5)
14.65
Win Rate
% Agr.
AC1
SG w/o 3-block (1=0.5)
19.42
SG vs MLE
84.0% 70.5%
84.0% 79.0%
0.78
MLE w/ 3-block
22.77
UL w/ 3-block (a=0.5)
22.25
SG vs UL
0.64
SG w/ 3-block (y=0.5)
24.62
5

---

Under review as a conference paper at ICLR 2021
Table 4: An example of greedy generated continuations for the same prefix.
You's patriotism, and Mei's reflections on the quotidian are few examples. More broadly, Du Fu's work transforming the lushi from mere word play into vehicle for scrious poctic utterance"
Prefix:
MLE: a major contribution to the genre. The first of the three major works of the genre, was written the early 1920s, and was the first of the three to be published in English. It was published the United States the United States, and was translated into French in the 1930s. The work was the first of the three
UL: a major influence on the modern Chinese novel. The modern Chinese novel often characterized as being "literary in nature". but not limited literary genres. The modern Chinese novel has been described as "a literary genre that has been influenced by the Western literary tradition". The modern Chinese novel has been described literary genre that has been influenced by the Western literary tradition"
SG: is powerful example of how the Chinese literary tradition can be used explore themes of nationalism and self determination. In addition to his own poetry, Du Fu wrote several works of prose. The first was a collection of short stories called "The Story of the Three Sisters". which was published in 1885. It was followed by a series of novels, including the first two volumes of the Song dynasty epic poom "The Three Sisters"
4.2 DIRECTED GENERATION
For directed generation, we consider two tasks: image paragraph captioning and text summarization.
4.2.1 IMAGE PARAGRAPH CAPTIONING
Setup We use the captioning model proposed by Melas- et al. (2018) as the baseline, which comprises a CNN encoder that is pre- trained for object detection and I-layer LSTM decoder. The models are trained and evaluated on the paragraph captioning dataset, Visual Genome (Krause et al.) 2017]. We train the model with SG and compare it to the ones trained with MLE and UI.. The performance is measured by CIDEr (Vedantam et al. 2215), which computes TF-IDF weighted n- gram overlaps between the model generated captions and the reference captions. We follow MMela- Kyriazi et al. (2018) to apply greedy inference since beam search did not yield any further gain.
Results Table 3 shows the CIDEr scores for different training methods on Visual Genome testset with and without tri-gram blocking (Paulus et al. 2018) during inference. Without tri-gram blocking, MLE produces texts that are full of repetitive phrases (see Appendix
D for examples), which leads to low CIDEr score. When UL or SG is incorporated. the performance has been notably improved from 10.51 to 14.65 and 19.42, respectively When tri-gram blocking is applied, our method is still capable of yielding 1.85 point improvement. This is because SG further improves the token-level degeneration top of tri-gram blocking. In contrast, the model trained with UL has a slightly worse CIDEr score compared to the MLE bascline. We analyze n- n-gram level degeneration further in |5.2
4.2.2 ABSTRACTIVE TEXT SUMMARIZATION
Setup We use the abstractive summa- rization model BertSum (Liu & Lapata 2019) as our baseline, which adopts a Transformer architecture to take advan- tage of pre-trained BERT (Devlin et al., 2019) as the encoder. At the first stage, the encoder is trained with an extractive summarization objective (binary clas- sification for sentence selection). At the second stage, it initializes the de- coder randomly and traits the entire encoder-decoder model with abstrac- tive (or generative) objective. For our experiments, we take the encoder that was trained at the first stage and train the entire (abstractive) model with different training methods (MLE, UL and SG)
Table 5: Experimental results for text summarization on CNN/DM and NYT50 testsets.
Models
R-1
R-2
R-L
WMD-1
CNN/DM
BertSum w/ MLE
41.87
19.42
38.93
19.89
BertSum w/ UL (a 0.5)
42.03
19.36
39.09
20.21
BertSum w/ SG (7 0.8)
42.19
19.53
39.25
20.23
NYT50
BertSum w/ MLE
48.73
31.00
45.23
28.73
BertSum w/ UL. (a = 0.5)
48.54
30.73
44.99
28.50
BertSum w/ SG (7 = 0.8)
49,29
31.30
45.78
29.14
using the default training setup on two benchmark datasets: CNN/DM (Hermann et al. 2015 Nai-
lapati al. 2016) and NYT50 (Durrett et al. 2016). During inference, length normalization (Wu
et al. 2016), tri- tri-gram blocking and beam search (beam size = 5) are used as in (Liu & Lapata 2019).
Athhughh UI. was originally proposed for open-ended generation, it is applicable to directed generation. We did the same scale of haper-parameter searching for UT.. Details can be seen in Appendix F

---

Under review as a conference paper at ICLR 2021
We evaluate performance of the models with the standard F1-based ROUGE Lin 2004) scores (R- 1, R-2, R-L) and a model- based evaluation MoverScore (Zhao et al. Word Mover Distance (WMD) between the reference summary and generated summary based on the representations from BERT. We report 1-gram MoverScore (W MD-1), 1), which has been proven to have higher correlation with human than other metries (Zhao et al. 2019).
2019). which computes the
Results From Table 5] we notice that on CNN/DM, the model trained with SG outperforms the models trained with MLE and UL when measured by ROUGE. In WMD-1, UL yields similar perfor- mance as ours, Both SG and UL further improve over the MLE baseline. The improvements imply that token-level degeneration may still exist even when tri-gram blocking is applied. On NYT-50, UL underperforms MLE, while our method improves in all measures. We discuss the possible reason why UL underperforms from gradient perspective in 5.4
5 ANALYSIS
In this section, we perform a series of analysis to gain more insights about our method.
5.1 OPEN-ENDED GENERATION
Compatibility with decoding strate- gies One advantage of our method is that it is compatible with decoding- based methods. One can choose dif- ferent decoding strategies based on the specific needs. Table [ provides the results of different decoding strategies used along with our SG training for text nuto-completion (results for more varia- tions are in Appendix[H] (H). We observe that beam search, even with larger beam size, is not effective in mitigating the degeneration issue, which accords with the observation in (Holtzman et al. (2020). As expected, stochastic decoding, top-k and nucleus (top-p) sampling, help further reduce repetition. This sets good examples of combining training and decoding strategies for the task in hand.
Table 6: Results of different decoding strategies with
ScaleGrad training for auto-complction.
Approaches
Rep-1
Rep-2
Rep-3
uniq-w
SG+Greedy Search
0.441
0.214
0.144
22,23k
SG+Beam Search (4=
0.453
0.250
0.171
8.32k
SG+Top-p (p= 0.3)
0.356
0.107
0.049
30.48k
SG+Top-k (k = 40)
0.254
0.039
0.012
39.50k
Table 7: Summarization results (F1-based ROUGE-1 and MoverScore) for stochastic de- coding on NYT50 testset.
Table 8: Degeneration analysis for image para- graph captioning with/without tri-gram block- ing. Numbers in bold are closest to human.
Models
ROUGE-1
WMD-1
Models
Rep-1
Rep-2
Rep-3
Top-p (p=0.3)
45.44
24.61
MLE
0.723
0.587
0.530
Top-p (p=0.9)
42.33
21.67
SG
0.500
0.270
0.195
Top-k (k=40)
41.23
20.70
MLE w/ 3-block
0.575
0.271
0,094
Top-k (k=100)
40.86
20.38
SG w/ 3-block
0.440
0.146
0.037
Baseline
48.73
28.73
Human
0.421
0.123
0.042
5.2 DIRECTED GENERATION
Comparison with stochastic decoding Although top-p and top-k sampling have been proven successful in open-ended generation, to our knowledge, none has tested them directed generation tasks. In order to see if they could lead to the same improvements as ScaleGrad, we conduct addi- tional experiments with the BertSum summmarization model, whose underlying language model is more mature due to the involvement of BERT, compared to the image paragraph captioning model. For the interested readers, we also provide the results of stochastic decoding on image paragraph captioning in Appendix!!
Table 7 shows the performance of BertSum trained with MLE on NYT50 testset when stochastic decoding is applied during inference. Since ROUGE-1 measures the exact 1-gram overlaps between reference and generated summaries. it may not be sufficient to evaluate the performance of stochastic decoding methods, which may generate more diverse output while conveying the same meaning. Therefore, also report the MoverScore that is capable of considering the semantic similarity
7

---

Under review as a conference paper at ICLR 2021
MLE
145
MLE
50
14.
1455
14.-
130
12.5
01
32
03
04
0.5
0.5
07
04
0.9
01
02
03
04
0.5
0.5
0.7
0.
09
33
32
03
34
05
06
67
00
09
(a) Perplexity
(b) Rep/l
(c) # of unique tokens
Figure 1: Hyper-parameter (7) sensitivity in the language modeling task on Wikitext-109 develop- ment set. Rep/l is computed the average of Rep/16, Rep/32 and Rep/128. Detailed results for Rep/l can be found in Appendix K
rather than just n- gram overlaps. However, both the ROUGE and MoverScore in Table 7 lead the conclusion that stochastic decoding methods significantly lower the performance compared to the standard beam search. This implies that they may not be good fit for directed generation tasks. In contrast, our method possesses wider applicability in mitigating degeneration issues.
n-gram degeneration To investigate further how SG minimizes degeneration and helps to im- prove the performance in automatic measures, we compute the n-gram repetition ratios the outputs from the image captioning model (Melas- Kyriazi et al. 2018) and report the numbers in Table & Compared to human, the MLE baseline has significantly higher repetitions, thus having the lowest CIDEr score (Table 3. With SG, the model yields much better repetition ratio, which explains the notable performance boost in CIDEr. Tri-gram blocking resolves the issue of 3- or higher n-gram degeneration in a hard-coded way, improving CIDEr significantly. However, the token and 2-gram repetitions still remain high and improvable in MLE with tri- tri-gram blocking. When both tri-gram blocking and SG are applied, the generated texts have the lowest and most human-like repetitions.
5.3
SENSITIVITY
Towards better usage and understanding of ScaleGrad, we show how the key metrics in language modeling change with the hyper-parameter y in Figure As discussed, a smaller value of incurs a stronger push
novel tokens, giving higher perplexity and more unique tokens. In general, y can be chosen based on the performance of the baseline model. If the baseline produces many repetitive tokens/phrases (e.g, image paragraph captioning experiments), a smaller value of 7 should be used. Conversely, in tasks with less degeneration (e.g., summarization experiments), a larger 7 can be used further improve the unigram and bigram level degeneration without affecting the perplexity much.
5.4 DISCUSSION ON THE UNLIKELIHOOD TRAINING FROM GRADIENT PERSPECTIVE
Experimental results in the directed generation tasks empirically reveal that unlikelihood (UL) train- ing could not bring about improvements consistently. In this section, we analyze UL. from the perspective of its gradients and contrast this with ours. For UL, the gradient of the total loss (Eq. 2 with a single negative token w.t.t the logit o;
(1 1 Peeg
Paes
if incg
Vo = mi 1(i= = k), where
(6)
(1+a)
if inog
where p; [softmax(o)], Pneg is the probability of the negative-candidat token with index 'ugg and 1 = k) is the indicator function with & being the index of the ground truth token (see the original paper for derivation). From our previous discussion in $3.1 know that as the gradient-based optimization proceeds, the gradient converges to a number that is close enough to 0. Therefore, with preset paper-parameter, the probability of the ground truth token p& should always increase decreases despite the ground truth
as the gradient norm of the loss w.r.. its logit (i.e., IV Vo,Cl C)) token being repetitive (negative) not. Should this be the case, i.e., the generation probability of the ground truth token Pk decreases as the gradient V L decreases, the model is not to learn
*Since Melas-Kyriard Kyriazi et al. 2018 used soft tri-gram blocking, some of the duplicate tri-grams still remain.
8

---

Under review as a conference paper at ICLR 2021
to predict the ground truth tokens correctly (since the pk has decreased), which in turn hurts the generation quality.
Since the ground truth is always non-negative token by definition (i.e., i = = o inog). the gradient Pes We see that when Pneg norm from Eq. is V C] = 14 Pk 1| where 14: (1-aaja a 1 Pag
(e.g., when a = and Pneg > 0.5), μik becomes negative, having the gradient norm Vo Cl = -1pp Pk - = |px] Pk + 1. In this case, Ps decreases as the gradient norm decreases, which contradicts with the optimization principle we mentioned earlier. To be more specific, in order to decrease the gradient norm as the training proceeds. the model will have to reduce the value of pa- which goes against the goal of learning. Thus, UL. becomes less effective in such special cases (subject to the choice of the value of a). In contrast, the gradient analysis in Eq. 5 shows that ScaleGrad does not have such properties in learning to predict ground truth tokens. our earlier exploration, we modeled the novel tokens as an auxiliary loss, which also has the similar properties as UL (Appendix Dp.
CONCLUSION
We have introduced novel training method, called ScaleGrad, directly modifying the gradient of the standard MLE objective to remedy the text degeneration issues. The improvement verified by both automatic metric and human evaluation against the baselines in extensive experiments across different tasks in open-ended and directed generation and different architectures (i.e. LSTM and Transformer) demonstrate the effectiveness and generalizability of our method. Further analysis shows that ScaleGrad yields token distributions that are much closer to duman--witter texts com- pared to the baselines. Our method brings a good alternative to current training strategies.
REFERENCES
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv, 2020.
Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19 1423. URL https:
7/ wwww.wc. wwwwwb.wwwb... w/w.acl.ee..rgg/aata...... wttp.///tttnoooooo/o/-----------
Greg Durrett, Taylor Berg-Kirkpatrick, and Dan Klein. Learning- single-document sum- marization with compression and anaphoricity constraints. In Proceedings of the 54th An- mual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1998-2008, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/PI5 1188 URL RtLhhttps: /ttp::::/wwww... www. http://www.wwww....... aclweb. http://wwwww..../..a..o.ooo.ooo. http:/aahhollogyp16-1111
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers). pp. 889-898, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi:
1082. UtL : www. acwweb........c...... org/ antho /P18 1082
Kilem Li Gwet. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29 48, 2008
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Pro- cessing Systems 28, pp. 1693 Curran Associates, Inc., 2015. URL http:://ppppp..tt nips. httpp//cc/papee/550// teach in wnn--maahhhh nes to read- www.anddccmeeeeeed.ppp

---

Under review as a conference paper at ICLR 2021
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8): 1735-1780, November 1997. ISSN 0899-7667. doi: 11116///co..997....88
URL
httpss:/doi.. org. 11 http://o..1121/ooooooo/
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case neural text degeneration. In International Conference on Learning Representations, 2020. URL https:
//openreview.. http://www.eeee/oommmmmmeeoo/eee wwwoooommddidgggooe
Philipp Koehn and Rebecca Knowles. Six challenges for neural machine translation. In Pro- ceedings of the First Workshop on Neural Machine Translation, pp. 28-39, Vancouver, Au- gust 2017. Association for Computational Linguistics. doi: 10.18653/w1/W17-32222 URL
https:
www. www.ww.lwbb.co org/ anthology/ httpr//noooooy//-332220000 (it-oooggy/1132204
Wouter Kool, Herke Van Hoof, and Max Welling. Stochastic beams and where to find them: The Gumbel-top-k trick for sampling sequences without replacement. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3499-3508, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http: http:::/ppxoccecingg.o mlr press/ v97 1077/ool1100 9t..aa.ht....
Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. A hierarchical approach for generating descriptive image paragraphs. In Computer Vision and Patterm Recognition (CVPR), 2017.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zeutlemoyer. Bart: Denoising eqquence-to-sqqeene pre-training for natural language generation, translation, and comprehension, 2019.
Chin-Yeu Lin. Rouge: package for automatic evaluation of summaries. In Proc. ACL workshop
on Text Summarization Branches Out, pp. 10, 2004. com/~eyle httl///wwlllaae http://paprrss//e wttps//AA2200000 P000 pdf
http:://eeeearch.iiccooo
Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP-IJCNLB)) pp. 3730- 3740, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10.18653/v 1/119-1377 URL wttphhtpss/// htwp::://wwwww.... www. aclweb. www./eb...rrr//atttooooo///bbb/.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http:
7/www. www.waawwwbbbooom www.ebborrggntto.oooo htt.l//www.rr//aaaaoooooo//ll---
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):11-330, 1993. URL
https::/ http:://wwwwwcclee.. org/ wttprr//nnoolog/g99999 2004
Luke Melas-Kyriazi, Alexander Rush, and George Han. Training for diversity in image paragraph captioning. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 757-761, Brussels, Belgium, October-November 2018. Association for Com- putational Linguistics. doi: 10.18653/v1/DI8- 1084. URL https: www. ac web org/
BitcooooyyDDB--11
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR, 2017. URL
net wttp://wwwww.../e//////////a///e
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Guîtlşehre, and Bing Xiang. Abstrac- tive text summarization using sequence-to-sequee Computational Natural Language Learning, pp. 280-290, Berlin,
RNNs and beyond. In Proceedings of The
20th SIGNLL Conference Germany, August 2016. Association for Computational Linguistics. doi: 10.186633/1/KI 1028.
URL https:: / www. a e. web. www.orrg/attholoogogooo//ooo ogy KK 6- 6-1028
10

---

Under review as a conference paper at ICLR 2021
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive net htt.://eeeee/aaaaaaaa-a........
summarization, In ICLR, 2018. URL
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. Open-Al Blog, 2019.
Ehud Reiter and Robert Dale. Building Natural Language Generation Systems. Cambridge Univer- sity Press, USA, 2000. ISBN 0521620368.
Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 118836/1//111 1099. URL https:
//www. acwweb. wwg. wnthaooggy//11-1111 1099
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/11/P11--116. URL httph:://///... www aclweb.
vegg/anholl woghoooooyyy/eee /P16-//P16--- 1162
Felix Stahlberg and Bill Byrne. On NMT search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3356 3356-3362, Hong Kong, China, November 2019. Association for Computational I Linguistics. doi: 10. 10.18653/v//D19-133 URL. htt.h:ttp:///t/ttm /www www ac web. o wtt.g//attlllggg//oooooooooo
Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. Modeling coverage for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 76-85, Berlin, Germany, Au- gust 2016. Association for Computational Linguistics. doi: 10. 10.18653/11/P16-1003 URL
https: www www.wwwwwww.b........ o org/ wwwaaooooyy/II-1110000
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998 6008. Curran Associates, Inc., 2017. URL http::/papee/ autpp.ccccpaaae/BBatttttaaooooaa ---------e----------e--eee pdf
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-basse image description evaluation. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 4566-4577 2015.
Q. Wang and A. B. Chan. Describing like humans: On diversity in image captioning. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 4190-4198, 2019.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In International Conference on Learning Rep- resentations, 2020. URL http::ttt////ttttt /tttpp:://opnnreveeee net wtt.m/frummmm Pic SJeYe
R. J. Williams and D. Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1 1989.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin John- son, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang. Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick. Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's 's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/16009006 44, 2016. URL / wwwp:wvooooogoogooooo org/ abs 609. 081 44
11

---

Under review as a conference paper at ICLR 2021
Yu Yan, Weizhen Qi. Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou, Prophetmet: Predicting future n-gram for sequence-to-sequene pre-training. arXiv preprint arXiv:2001.04063 04066,, 2020.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. Mover- Score: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp- 578, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: N0.18653/v1/D1999 1053. URL https: : www. ac web. wwt..//aantooolggyii---------- 1-10
12

---

Under review as a conference paper at ICLR 2021
A DERIVATIONS
Derivation of the gradient of loss w.r.f. logit We follow the same notation as in the main paper. At time step l, assuming that the pre-softmax scores (i.e., logits) are denoted as of over the vocabulary V. where of denotes the score for the token with index i in the vocabulary. Similarly, we have pf ssoft Let & denote the index of the ground truth token step
The cross entropy loss at step I. is given as (we omit for notational simplicity):
C =
Σ
log
where 3/4 = 1ff k. otherwise ys 0. Thus the loss function can be rewritten as:
E log(
C log Pk log(
Σ, Eje )
Ok
(8)
Therefore, we can derive the partial derivative of the loss w.r.t the logit o, as follows.
V., Co Vo, log( Σ (e Vo
j
E,00 V Σ e 1(i
(9)
E
1 k)
Eje0 60
P: 1 = k:)
B NOVEL TOKEN SET ILLUSTRATION
Figure 2 shows an example of how the novel token set changes when the model is learning to predict the sentence "people who are interested ..". At beginning. the novel token S is equivalent to the vocabulary V. The size of the novel token set shrinks as the decoding proceeds.
who
who
who
who
the people the in in space
space
people the in space
space
are people the
interested interested Interested people ese N Sworel S interested
interested
S Novel V
Target:
people
who
are
Decoder:
Input:
SOS
people
who
are
Figure 2: An illustration of how the novel token set changes decoding proceeds for the sentence "people who are interested ... The words marked in purple are the target words that the model is learning to predict at each decoding step.
13

---

Under review as a conference paper at ICLR 2021
C CONNECTION WITH POLICY GRADIENT OBJECTIVE IN REINFORCEMENT LEARNING
The text generation agent can also be trained with a policy gradient method with the objective of maximizing the expected reward (or minimizing the expected negative reward) per time-step.
fiu Bo
Σ MEV
(10)
where r(y!) the reward for token yf sampled from the vocabulary (i.e., action space) using the current policy po (o 2) and æ is the input text. The policy gradient w.r.f. the logit On can be expressed as follows (omitting superscript t).
V CRL = Σ
Vo. om x)
= Σ (pm 1(m = i))
(1I)
B:EV
WEE
Under the reinforcement learning setup, the (sampled) tokens with higher rewards will be "pushed up", or increased in probability, while tokens resulting in lower rewards will be suppressed. From the perspective of gradient analysis, Eq. [1] shows that a higher reward leads to a larger value of the gradient norm (pm 1mm i))l. which in turn forces the model to learn to assign higher probability Pm the the sampled token (i.e.,
in order to reduce the norm (r(; (pm 1)|- Meanwhile, the model also learns to assign lower probabilities to other tokens in the vocabulary (i.e,, m # i) reduce the norm (r(μ) pm
In this specific example, reinforcement learning essentially works by scaling the gradient based on the rewards for each sampled tokens. While our method (Eq. 5) scales the gradient for each token based on the information of the novel tokens. Both of the methods share the same fundamental idea that we can have the model trained serve the specific needs by scaling the gradient.
D HUMAN EVALUATION DETAILS
We conduct the human evaluation for two pairs of systems i.e., SG vs. MLE and SG vs. UL. For each pair, the models generate their own continuations based the same 100 randomly chosen prefixes. Two native speakers English are then asked to evaluate the generated texts independently. During the study, users are instructed to judge which generated text is better continuation of the prefix based on the overall quality (e.g., readability, relevance to the prefix, grammar, and fluency).
The Win Rate in Table 2 is calculated as the total number of times that two users prefer the texts produced by the winner divided by the total number of cases in the evaluation (2 x 100 = 200). To get reliable human study, we also compute the percentage agreement and the chance corre- lated measure, Gwet's ACI/gamma coefficient (Gwet 2008) as the inter-rater agreement. Gwet's AC1/gamma coefficient overcomes the issue where traditional measures, such as Cohen's Kappa, are not robust to skewed distributions of rankings. Figure 3 shows the interface for human evaluation study.
E HYPER-PARAMETER SEARCH DOMAIN FOR DIRECTED GENERATION
In the experiments with the directed generation tasks, we conduct the same seale of hyper-parameter search for unlikelihood training (UL) our proposed ScaleGrad (SG) on the validation set. Specifically, for the hyper-parameter in length normalization (beam search decoding), (0.0,0.1,0.2) 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, 1.0} for image paragraph captioning. For the model- specific hyper-parameterr, a in UL is chosen from (0.5, 1.0 I and y SG chosen from (0.5, 0.8).
8 E (0.0, 0.5, 1.0, 1.5, 2.0} for text summarization and B
'I open-ended generation, directed generation, we tried other values then found that these two reduce degeneration in reasonable diverse degrees.
1 is recommended by the author. While in our initial exploration for
14

---

Under review as a conference paper at ICLR 2021
Evaluation # 4 Evaluations of Generated Texts.
Help
Given the same prefix text, please rank the generated texts from general aspect. Please look at which text more readablefis more coherent with the prefix/is more grammatically correct, etc. Do note that you may see the word 'unk' appearing in texts due to the truncated vocabulary, try to ignore this if does not affect the general quality very much.
Prefix Text: In addition, Kiss You received gold certifications from the IFPI Norway and Denmark associations, signifying collective shipments 20
TEXT1:, 000 copies. The song was released as single in the United Kingdom April 2008, and as a single on May 2008. It was also released as digital download on iTunes on June 10, 2008. The song was as a digital download on Amazon.com July 24, song was released digital
August
TEXT2: 000 copies. Music video The music video You was directed
and was directed by unk video begins girl red
a young girl dress, who kissing a man. video
kissing
General Rankling
Text better.
Text better.
Next
Figure 3: Human evaluation interface
F EXPERIMENTAL RESULTS ON OPEN-ENDED GENERATION
F.1 FULL EXPERIMENTAL RESULTS ON WIKITEXT-103
We present the full experimental results on WikiText-103 (Merity et al. 2017) for open-ended gen- crations in Table 9 All the numbers are averaged over runs with different randoms seeds and shown together with standard deviations.
Table 9: Results for open-ended generations. ppl. uniq and Rep/l are computed at BPE-level and the rest are at word-level. The "t" denotes higher value for better performance and "I" is the opposite. Number marked with are estimated based on the testset.
Langnage Modeling
Anto Completion
Models
ppl
uniq
Rep/16
Rep/32
Rep/128
Rep-1
Rep-2
Rep-3
uniq-w
13.24_3x-4
pa-k
55
0.380
0661922-6
1e-5
La--
To-5
1--1
UL 0.5)
14.39
12.87 12.87a
0.228
0.359
2te-7
0.586
0.607 =
0.429 soc-6
0.353
17 17.98k 17.88 6e-22
UL-a
1.0)
16.0642--2
13 18k
0.212
0.341
0.558
0.559-te-1
0.363
D.291
19. 19.11k_7e--
(y
14.20
1361k (a-3
-
L5
0.522 -4--6
22.25k 12
Human
18:27k
0.177
0.285
0.450
0.382
0.096
0.037
27.58k
F.2 ON GENERALIZABILITY OF SCALEGRAD
To further verify the generalizability (i.e., different datasets and domains) of our method in open- ended generation, apart from WikiText- 103 (Merity et al.) 2017). we evaluate the models on two other language modeling datasets: Penn TreeBank or PTB MMarcus et al., 1993) and IMBD (Maas al.) 2011). In particular, after fine-tuning GPT-2 with different training strategies (MLE, SG and UT) on WikiText-103 training data, we test the language modeling and auto-completion performance with the same setting described in $4.1 For PTB, randomly sample 500 movie reviews from the dataset.
the standard testset, while for IMDB,
Table shows the experimental results the PTB testset, from which tently improves over the MLE baseline in degeneration while possessing an acceptable increase in perplexity, and it outperforms UL. consistently.
that SG consis-
15

---

Under review as a conference paper at ICLR 2021
Table 10: Results for open-ended generations on PTB testset. ppl. uniq and Rep/l are computed at BPE-level and the rest are word-level. The "f" denotes higher value for better performance and "V" is the opposite. Number marked with are estimated based on the PTB testset.
Language Modeling
Auto Completion
Models
ppl 4
uniqq
Rep/16 +
Rep/32
Rep/128 4
Rep-1 4
Rep-2 .
Rep-3 !
uniq-w
MLE
33,952
5.60k
0.157
0.292
0.530
0.652
0.493
0.424
6.46k
UL (a 1.0)
41.232
5.96k
0.139
0.260
0.476
0.533
0.333
0.259
7.60k
SG (y 0.2)
40.731
6.15k
0.126
0.231
0.426
0.417
0.198
0.131
8.42k
Human
8.84k
0.118
0.222
0.421
0.362*
0.089
0.033*
11.32k*
Table 11: Results for open-ended generations on movie reviews from IMDB dataset. ppl, uniq and Rep/l computed BPE-level and the rest at word-level. Number marked with are estimated based on the extracted movie reviews from IMDB.
Language Modeling
Auto Completion
Models
ppl
uniy
Rep/16
Rep/32
Rep/128
Rep-1
Rep-2
Rep-3
uniq-w
MLE
100.764
7 7,48k
0.153
0.254
0.449
0.662
0.499
0.429
7.70k
UL (a 1.0)
108. 334
8.09 0999
0.123
0.205
0.373
0.545
0.346
0.274
9.31k
SG (y = 0.2)
110.451
8.14k
0.114
0.187
0.344
0.383
0.142
0,081
10.42k
Human
14.49k
0.118
0.208
0.378
0.329*
0.084*
0.009
*19.11k
we show the experimental results on IMDB movic reviews and observe similar perfor- mance trending in the experiment PTB testset. From the two experiments, can draw the conclusion that our method, SG, is capable of generalizing well to different datasets and domains. Examples of generated text for auto completion task can be found in Appendix
In Table []
F.3 AUTO COMPLETION WITH DIFFERENT DECODING LENGTHS
In Figure 4 we show the Rep-1 of generated text from the auto completion task with the constraints different decoding (continuation) lengths. observe that compared MLE counterpart, SG yields consistent improvements Rep-1, token-level degeneration, regardless the different de- coding lengths, which again verifies the effectiveness and generalizability of our method.
G EXPERIMENTAL DETAILS
In this section, present the details of the datasets used in our experiments well as the necessary experimental setup. All the experiments were conducted with a single GPU on our machine (CPU: Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz; GPU: NVIDIA RTX 2080Ti).
For each task in our experiments, we use the same model architecture and train with different objectives (i.e., MLE, ScaleGrad and unlikelihood). The hyper- parameerss that used for different training objectives in the same task are exactly same, except for the ones described in Appendix E We list the key hyper-parameters in this section. Though they may not be exhaustive, all the hyper- parameters are clearly presented in our source code. In addition, all the hyper-parameters that are not listed in this section remain unchanged from their corresponding default setup.
G.1 OPEN-ENDED GENERATION
Dataset The WikiText- 103 Merity et al. 2017 is collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia. The training, validation and test sets contain 104m, 218k and 245k tokens, respectively.
16

---

Under review as a conference paper at ICLR 2021
Rep-1(%)
MLE SSG
58
53
48
43
38
33
Length 25
50
75
100
Figure 4: Box plot for Rep-1 in auto completion with different decoding lengths. All the numbers are computed based on the results from runs with different random seeds.
Experiments For all the experiments, we use the same setup and the same hlper-parameteen as listed in Table except for the -peeiid-peecifc syper-parameters We load the GPT-2 medium and fine-tune it on WikiText-103 with a maximum of 35k iterations and select the model based on the validation perplexity.
Table 12:
Hyper-parameter for open-ended generation. parameters. Ir, is initial learning rate.
M denotes the model-specific hyper-
Models
Iro
M
batch
MLE
x 10-5
300
UL
2 x 10-5
0.5/1.0
300
ScaleGrad
2 10-5
0.2
300
G.2 SUMMARIZATION
Dataset We use CNN/DM (Hermann al. 2015 Nallapati et al. 2016) and NYT50 Durrrttt al., 2016) in our experiments for text summarization. Table 13 shows the dataset statistics in details.
Table 13: Dataset statistics for summarization.
Dataset
Training Size
Validation Size
Test Size
CNN/DM
287,227 96,834
13,368 4,000
11,490
NYT50
3,452
Experiments The models are taken from (Liu & Lapata 2019) and train the models for the ab- stractive summarization with MLE, unlikelihood training and ScaleGrad on CNN/DM and NYT50. We list the hyper-parameters that used in Table 14
17

---

Under review as a conference paper at ICLR 2021
Table 14: Hyper-parameter lists for summarization. M denotes the model-specific hyper- parameters. Ir and Irge stand for initial learning rate for BERT and Transformer decoder. is the hyper-paramett in length normalization.
Models
I
Irdec
M
batch
8
Beam Size
CNN/DM
MLE
0.002
0.2
140
1.0
UL
0.002
0.2
0.5
140
2.0
ScaleGrad
0.002
0.2
0.8
140
1.5
5
NYT50
MLE
0.002
0.2
140
1.5
UL
0.002
0.2
0.5
140
2.0
5
ScaleGrad
0.002
0.2
0.8
140
1.5
G.3 IMAGE PARAGRAPH GENERATION
Dataset We use the image paragraph captioning corpus Visual Genome dataset, introduced by Krause et al. (2017). The dataset contains 14,575 training, 2,487 validation, and 2,489 testing im- ages. The average length of description paragraph 67.50 tokens.
Experiments We follow the experimental setup in (Melas- Perlas-yrrizzi et al. 2018). We train the model with different objectives and choose the model for testing based on the validation loss. During generation, tri-gram blocking and length-normalization applied. Hyper-parameters that are used in our experiments are listed Table 15
Table 15: hyper-parameters. Ir, is initial learning rate.
lists for image paragraph captioning. denotes the model-specific
Models
Iro
batch
3 (w/o w/ 3-blocking)
MLE
5 x 10-4
10
0.0/0.2
UL
5 5 10
0.5
10
0.0/0.3
ScaleGrad
x 10
0.5
10
0.6/0.6
H EXPERIMENTAL RESULTS OF DIFFERENT DECODING STRATEGIES FOR AUTO-COMPLETION.
Table 16: Results of different decoding strategies for auto-completion.
Approaches
Rep-1
Rep-2
Rep-3
uniq-w
Greed Search
0.441
0.214
0.144
22.23k
Beam Search
0.422
0.210
0.134
8.75k
Beam Search (b 6)
0.453
0.250
0.171
8.32k
Beam Search 10)
0.489
0.298
0.214
8.00k
Top-p (p 0.3)
0.356
0.107
0.049
30.48k
Top-p (p = 0.9)
0,217
0.027
0.008
52.76k
Top-k (k 40)
0.254
0,039
0,012
39.50k
Top-k (k = 100)
0.234
0.031
0.010
44.27k
Table 16 shows the results for the auto-completion task when we train the model with ScaleGrad and infer with different decoding strategies.
18

---

Under review as a conference paper at ICLR 2021
I STOCHASTIC DECODING FOR IMAGE PARAGRAPH CAPTIONING
We apply different stochastic decoding strategies for the MLE baseline on image paragraph caption- ing and report the results in Table 17 The experimental results demonstrate that stochastic decoding strategies do not work well in directed generation tasks, which is consitent with our findings in summarizaiton experiments.
Table 17: Image paragraph captioning results for stochastic decoding on Visual Genome testset.
Models
CIDEr
Top-p (p=0.3)
19.54
Top-p (p=0.9)
18.67
Top-k (k=40)
18.73
Top-k (k=100)
18.05
MLE w/ 3-block
22.77
NEURAL NOVEL LIKELIHOOD TRAINING
In our earlier exploration, experimented with regularization loss based on the novel tokens, which is similar to UL. We can call it novel likelihood (NL). The total loss at time step t can be expressed as follows.
== C.IL + S.u log loglo( - a log po (clvvc)
(12)
oEC
where a is hyper-parameter and Ct is the set of novel tokens at time step t. which the same in ScaleGrad (33.2). i.e., C = V ,..'"...''"" y" with V being the vocabulary. The NL loss LrL boosts the probabilities of novel tokens. In earlier empirical evaluation on language model, yielded similar performance
UL. We thus also analyze the method from a gradient perspective. According to Eq. I2 it is easy to show that at time step t, the gradient of the overall loss w.zt. the logit o; for one single novel token is (omitting for simplicity):
V C
1(i k)
(p. L(i i,)) (1 + a) p. 1(i k) 1(i in) (13)
where p: = [softmax ( in is the index of the novel token and ls is the index of the target token. We can see that when the target token not novel token, i.e., becomes V., L] = [1 + a:) Pi 1/. When pt I-a the norm decreases contradicts the gradient optimization principle. Thus, the NL loss has the similar property as the UL loss.
+ i, the gradient norm
increases, which
K HYPER-PARAMETER SENSITIVITY
To fully present the sensitivity Rep/l to the hyper-parameter, we further show how the Rep/l (i.e., 1=16, 32 and 128) change with in Figure
L EXAMPLES
In the following, we show the examples of generated texts in three tasks: auto-completion (Table [8
and Table 19, image paragraph captioning (Table 20 and Table (1] and text summarization (Ta-
ble 22 Table 23 Table 24 and Table 25). In addition, Table 26 and Table 27 show the example of auto completion on PTB testset and movic reviews from IMDB dataset.
19

---

Under review as a conference paper at ICLR 2021
023-
022
0.31
0.5
0.11
0.56
020
0.5
038
818
032
01
02
05
04
06
077
04
80
02
06
(a) R
(b) Rep/32
(c) Rep/128
Figure 5:
(7) sensitivity the language modeling task on Wikitext- 103 develop-
ment set.
Table 18: Example of generated continuations for different training objectives.
Prefix
Robert Boulter is an English film, television and theatre actor. He had a guest starring role on the television series The Bill 2000. This was followed by a starring
MLE
role in the film The Last of the Mohicans in 2002. He has appeared in the television series The X Factor, The X Factor Australia, The X Factor New Zealand, The Factor South Africa, The Factor New Zealand. The X Factor South Africa, The X Factor New Zealand, The X Factor New Zealand, The X Factor New Zealand, The Factor New Zealand, The X Factor New Zealand, The X Factor New Zealand, The X Factor New Zealand, The X Factor New Zealand
Unlikelihood
role the film The Last of the Mohicans, which was released in 2002. He also appeared in the television series The X Factor, where he played the role of DJ. In 2005, he starred in the film The Last of the Mohicans, which was released in 2006. He also appeared in the television series The X Factor, where he played the role of DJ. In 2008, he starred in the television series The X Factor, where he played the role
ScaleGrad
role the BBC's comedy series The Office in 2002, In 2003, he appeared the BBC's crime drama series The Yard. playing a police officer who is caught up in a murder investigation. He also appeared in the BBC's crime drama series The X Factor, playing a man who is accused of murdering his wife. 2004, he starred in the BBC's comedy series The Office, playing fictional version of himself. He has appeared in several
20

---

Under review as a conference paper at ICLR 2021
Table 19: Example 2 of generated continuations for different training objectives.
Prefix
You's patriotism, and Mei's reflections on the quotidian few examples. More broadly, Du Fu's work in transforming the lushi from mere word play into "a vehicle for serious poetic utterance"
MLE
a major contribution to the genre. The first of the three major works of the genre, was written in the early 1920s, and the first of the three to be published in English. It was published in the United States in the United States, and was translated into French in the 1930s. The work was the first of the three
Unlikelihood
a major influence on the modern Chinese novel. The modern Chinese novel is often charac- terized being "literary in nature", but it is not limited to literary genres. The modern Chinese novel has been described as "a literary genre that has been influenced by the Western literary tradition". The modern Chinese novel has been described as "a literary genre that has been influ- enced by the Western literary tradition"
ScaleGrad
is powerful example of how the Chinese literary tradition can be used to explore themes of nationalism and self determination. addition to his own poetry, Du Fu wrote several works of prose. The first was collection of short stories called "The Story of the Three Sisters", which was published in 1885. was followed by a series novels, including the first two volumes of the Song dynasty epic poem "The Three Sisters"
Table 20: Example I of generated paragraphs for different training objective on Visual Genome testset.
Reference
Three adult horses are field and a small horse. A horse color light brown has the neck low and is eating grass. This horse has the mane and the tail color tan. Two dark brown horses also are eating grass from the ground. The baby horse has a white spot on the front and tiny. back there wall of stones.
MLE w/o tri-gram blocking
There are two horses in field. The horses are brown and white. The horses are brown and white. The horses are brown and white. The horses are brown and white. The horses are brown and white. The horses are brown and white. The horses are brown. The horses are brown. The horses brown. The horses are brown. The horses are brown. The horses are brown. The horses are brown. The horses are brown. The horses are brown The horses are brown. The horses brown. The horses are brown. The horses are brown. The horses brown. The horses brown.
MLE w/ tri-gram blocking
There are two horses in a field. The horses are brown and white. The horse on the right is brown and brown. The horse brown. The other horse white. The grass is green and green. The sky is very and green. There trees in the background. The trees green and healthy. The horses are black. The sun is shining brightly.
SealeGrad w/ tri-gram blocking
Two horses are standing in field. The field is covered with green grass and weeds. The horse is brown and white. The horses are brown and white. There a large tree behind the horses. The sky is blue and clear. There are many trees in the background. The grass is green. The sun is shining brightly. The sky mostly cloudy.
21

---

Under review as a conference paper at ICLR 2021
Table 21: Example 2 of generated paragraphs for different training objectives on Visual Genome testset.
Reference
There is white and yellow train with red doors next to red and yellow train. They are on top of the railroad tracks. There is a brick walkway next to the trains.
MLE w/o tri-gram blocking
A train is on the tracks. The train is yellow and yellow. The train is red and white The train red and white. The train red and white The train is on the train. The train is yellow and yellow. The train is blue and white. The train is on the tracks. The train is yellow and yellow The train is blue and white. The train is blue and white. The train is on the tracks The train yellow. The train is blue. The train is blue. The train is blue. The train is blue. The train is blue. The train is blue. The train is blue. The train blue. The train blue. The train is blue.
MLEE w/ tri-gram blocking
A train is on the tracks. The train is yellow and yellow. The train has a yellow stripe on the front. The front of the train is red. The sky blue. There are many windows on the train. There is a train on the platform. The platform is made of metal. The is platform on the side of the train. The are many tracks on the train. There are wires on the ground. There is a building behind the train tracks. There is large building behind the train.
ScaleGrad w/ tri-gram blocking
train is on the tracks. There are two sets of tracks next the train. The train white and yellow. There is large white building behind the trains.
Table 22: Example of summaries generated by the model trained with different training objective on CNN/DM testset.
Reference
report by dick marty, lead investigator for council of europe, gives bleak description of secret prisons run by cia in poland and romania from 2003 to 2006. marty says he relied heavily on information from cia agents. critics poland and romania attack marty's use of anonymous sources.
MLE
swiss sen dick marty reports that central intelligence agency operates secret prisons run by amer- ican in poland and romania from 2003 to 2006. says prison were operated exclusively by ameri- cans in poland and.
Unlikelihood
swiss sen dick marty reports that secret prisons run by central intelligence agency in eastern europe, with information he says is gleaned from anonymous intelligence agents. report is pre- pared by swiss senator investigating cia operations for council of curope, 46 nation rights group. scathing report says prison were operated exclusively by americans poland and romania from 2003 to 2006.
ScaleGrad
dick marty, swiss senator investigating cia operations for council of europe, gives bleak descrip- tion of secret prisons run by central intelligence agency in eastern europe, with information he says is gleaned from anonymous intelligence agents. report says prisons were operated exclu- sively by americans in poland and romania from 2003 to 2006.
22

---

Under review as a conference paper at ICLR 2021
Table 23: Example 2 of summaries generated by the model trained with different training objective on CNN/DM testset.
Reference
middleton wrote about eating whale for national newspaper travel story. she described eating it at the #123-a-night juvet landscape hotel. whale and dolphin conservation group said the news was 'disappointing' the wde said ms. middleton should have gone whale watching instead.
MLE
the 3 -l-ycar-old has been slammed for 'promoting the cruel and unnecessary whaling industry', the 31-ycar-old boasted of cating minke whale norway, the horror of environmental cam- paigners who say she is helping promote an 'unimaginably cruel' trade. the hunting and impor- tation of whale meat banned in britain, as it is in most parts of the world.
Unlikelihood
the 3 3--year-lld boasted of eating minke whale at juvet landscape hotel in norway. the 31 31-year-old has been slammed for "promoting the cruel and unnecessary whaling industry 'iqzenvironmental campaigners say she is helping promote an unimaginably cruel trade.
ScaleGrad
duchess of cambridge's brother-in- has led personal crusade against poaching and protecting wildlife. pippa middleton boasted of eating minke whale norway, conservation group said she is helping promote 'promoting the cruel and unnecessary whaling industry'.
Table 24: Example of summaries generated by the model trained with different training objective on NYT50 testset.
Reference
protesters angry over bribery scandal involving state-run oil company petrobras. brazilian presi- dent dilma rousseff also is struggling with economic downturn.
MIEE
protesters are calling for president dilma rousseff to impeached. rousseff's supporters call for the president to be im say there hasn't 't been any evidence she was involved in the corruption scandal.
Unlikelihood
protesters are calling for president dilma rousseff to be impeached. there is number of issues play. one of the biggest an investigation into a multimillion-doll kickback scheme the state-run oil company petrobras.
ScaleGrad
president dilma rousseff is facing an economic downturn and massive bribery scandal. rousseff has defended her right to protest and acknowledged the need to clean up corruption at petrobras.
23

---

Under review as a conference paper at ICLR 2021
Table 25: Example 2 of summaries generated by the model trained with different training objective on NYT50 testset.
Reference
wollemi pine, which flourished 200 million years ago, is available to gardeners and can be grown indoors outdoors. thought to extinct, it discovered in 1994 in wollemi national park in australia. botanists took cuttings from wild colony of trees and began propagating baby pines and sending them to botanic gardens around world.
MLE
new york city plant - ramachan pine, which grew among dinosaur 200 million years ago, named after dr david noble, who rappelled into deep gorge in blue mountains of wollemi national park, ten miles west of sydney, australia, photos. map. pine was discovered until 1994, when bush delay and national parks officer david noble tried rappelled into woods of wollemi, which he had never seen before. scientists had evidence of wollemi pine's 's existence, but only from fossils.
Unlikelihood
nicholas d kristof op-ed column plight of wollemi pine, one of oldest on earth who grew among dinosaur 200 million years ago. notes that was not discovered until 1994, when bush rescue officer david noble rappelled into deep gorge in blue mountains of sydney, australia, and came back up with branch of tall tree that he never saw before.
SealeGrad
nicholas d kristof op-ed column on new species of wollemi pine, which grew among dinosaur 200 million years ago and was discovered until 1994, when bush delay and national parks officer david noble rappelled into deep gorge in blue mountains of wollemi national park, 120 miles west of sydney, australia, and discovered new species, wollemia nobilis, named after noble.
Table 26: Example of generated continuations for different training objectives on PTB testset.
Prefix
last resort who were criticized after the 1987 crash once again couldn't handle the selling pressure. Big investment banks refused step up the plate to support the beleaguered floor traders by buying big blocks of stock, traders say. Heavy selling
MIE
pressure also felt by the Wall Street banks. The Wall Street banks were forced sell more stock to hedge funds and other investors, and to buy more stock to hedge against losses. The Wall Street banks were forced sell more stock hedge funds and other investors, and to buy more stock to hedge against losses. The Wall Street banks were forced to sell more stock to hedge funds and other investors, and to buy more stock to hedge against losses.
Unlikelihood
pressure was also felt by the banks themselves. In 1987, the New York Federal Reserve Bank, which had been the largest bank in the United States since the 1930s, was forced to sell $1.5 billion worth stock. The bank's stock price fell to $0.001, and it was forced to close its New York branch. The bank's stock price fell to $0.001, and it was forced to close its New
ScaleGrad
by the big banks led to a sharp drop in prices, and the market collapsed. In 1992, the Federal Reserve Bank of New York issued a report the financial crisis, which concluded that the Fed's actions had not been sufficient to prevent the collapse of the housing market. The report recommended that the Fed increase its balance sheet by $1.5 trillion (equivalent to about $2.5 trillion today), but this was not implemented.
24

---

Under review as a conference paper at ICLR 2021
Table 27:
Example of generated continuations for different training objectives movie reviews from IMDB dataset.
Prefix
One of the other reviewers has mentioned that after watching just Oz episode you'll 'Ill be hooked. They are right, as this is exactly what happened with me. The first thing that struck me about Oz was its brutality and unflinching scenes of violence.
MLE
I was expecting anything like that in show about family. was expecting show about a family, but I was expecting something more. I was expecting show about a family, but I was expecting something more. I was expecting show about a family, but I was expecting something more. was expecting a show about family, but was expecting something more. I was expecting show about family, but was expecting something more. was expecting show
Unlikelihood
I was immediately hooked. The violence how it was handled. was also struck by the fact that the show was so well-written that it was almost impossible
graphic that I was completely taken aback by
be entertained. The second thing that struck me about Oz was its ability to make me laugh. was taken aback by the violence that couldn 't help but laugh. The violence was so graphic that was completely taken aback by how
ScaleGrad
It was also very funny, but not funny enough to make me laugh out loud. The second thing that struck me the fact that the show's characters were well developed that they were able carry their own weight without being annoying annoying. In its original broadcast, "Oz" was watched by 9.5 million viewers, according to Nielsen ratings. was the highest rated
25


Reviews:
[{"review_id": "hZDHvFAk2G", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to modify a language model's token-level distributions by rescaling the output probability of tokens that do no appear in the context ('novel tokens'). The authors show improvements over MLE and token-level unlikelihood in terms of repetition, with increases in perplexity. \n\n\n#### Clarity and significance\n- **ScaleGrad motivation**. There are many different ways to change the gradient, e.g. any regularization function, any scaling of the output probabilities, or even gradient clipping modifies the gradients that are used to update the model. As a result, the presentation of their method as \"a modification straight to the gradient of the loss function\" seems odd, and the name ScaleGrad suggests that they are proposing the general notion of rescaling gradients. Instead, they propose to scale a novel set's output probabilities then renormalize.\n\n- **Specific solution**. The method is specifically designed around the 'novel set', which could limit its significance. The authors speculate that they can alter the novel set (e.g. for importance or factual correctness), but this appears to be nontrivial.\n\n- **Unlikelihood discussion**. The discussion in 5.4 deals with the case of $p_{neg}>0.5$, meaning that the probability of the ground-truth token $p_*$ is $<0.5$ (due to normalization). If I'm understanding their argument, the authors argue that the resulting gradient contradicts the fact that the gradient should go to zero at an optimum. However, *the model is not at an optimum* if $p_<0.5$. Could the authors clarify the statements \"the model is not to learn to predict the ground truth tokens correctly\", \"contradicts the optimization principle\", and \"essentially rejects the ground truth token\"? \n\n- **Method for promoting novelty**. It's unclear why this specific method (renormalizing over the novel set) is the best or simplest method for promoting novelty. A downside is that we no longer know which objective the model is optimizing. In the appendix, the authors discuss a variant that uses an additional loss (Section I), yet do not perform an empirical comparison with that or other 'novelty promoting' variations. They argue in Section I that this suffers 'the similar issue as the UL loss', but that issue was unclear (see point above).\n\nOverall I'm borderline on this paper: the authors do perform a lot of experiments and show improvements, but I'm hesitant that scaling novel tokens and renormalizing the model's output distribution is significant.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y7_SCEsK8og", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**I have updated this review after noting the authors’ detailed response.**\n\nThis paper focuses on the problem of “Neural Text Degeneration”—where text sampled from a language model can either be too repetitive and bland or too random and nonsensical. The authors focus largely on the former problem, proposing a finetuning loss that specifically incentivizes the use of tokens that have not yet been decoded in the given document. The authors test whether this improves repetition and unique token coverage with greedy decoding in open-ended generation. A small human study is conducted and the proposed method, ScaleGrad, is found to outperform MLE and Unlikelihood Training (UT). Similarly good results are obtained on Image Captioning with and without trigram repetition blocking. On Abstractive Summarization BeamSearch is used and again outperform MLE and UT. Analysis attempts to make comparisons across different decoding strategies, though coverage of different variations is limited. The authors argue that stochastic decoding is outperformed by ScaleGrad, though they note that trigram blocking still helps ScaleGrad. Multiple hyperparameter settings are shown, with some analysis on how gamma can be chosen to get a desired behavior. Finally, the authors analyze why UT may not be as effective: it penalizes gold repetitions too much and does little for other tokens.\n\nStrength:\n- The results are good for greedy decoding\n- The method is well motivated and well explained\n- The analysis regarding Unlikelihood Training is interesting\n\nWeaknesses\n-  The results shown do not make proper comparisons across models, baselines, and hyperaparameters over all tasks.\n- Results for stochastic decoding should have been shown across tasks.\n- Despite citing the need for awkward rules such as trigram repetition blocking as a reason to propose ScaleGrad, trigram repetition blocking still helps significantly.\n- Some details are hidden away in the appendix, which I had to read thoroughly in order to fully understand the comparison.\n\nI recommend to reject this paper, because the experimental comparisons are not quite fair and because of implicit argumentation about what Greedy decoding can or should do that is never made explicit.\n\nThe following two paragraphs are obsolete, because the authors shared experimental results from a larger set of experiments.\n> The results in Table 1—which show the main metrics of interest on open-ended generation—are missing two key points of comparison: ScaleGrad is only show with gamma=0.2, even though gamma=0.5 & gamma=0.8 are used for the rest of the experiments, giving us little idea of how these metrics change over hyperparameter settings. This is despite the fact that two hyperparameter options for Unlikelihood Training are shown. In a footnote on page 6, for directed generation, the authors state “Although UL was originally proposed for open-ended generation, it is applicable to directed generation. We did  the same scale hyper-parameter search for UL. Details can be seen in Appendix E.” However, in Appendix E two hyperparameter settings for alpha are shown, the same two as used in Table 1, but two hyperparameter settings for gamma in ScaleGrad are shown _neither of which are shown in Table 1_ nor are repetition or uniqueness numbers shown for these hyperparameters settings anywhere in the paper or the appendices. This makes me question whether the improvements shown in Table 1 hold across hyperparameter settings as the authors claim in their analysis of Figure 1.\n\n> However, Figure 1 is missing necessary data points and comparison. First of all gamma=0.2 is not shown, though at least gamma=0.1,0.3 are so it can be somewhat inferred. That is suboptimal, but this graph does not even go up to gamma=0.8, which is what is used in the Abstractive Text Summarization experiment! Furthermore, the number in Figure 1 (b) cannot be directly compared to other decoding methods, because they are an average of repetition metrics shown in Table 1. Luckily, Figure 1 (c) can be compared, and if cross-referenced with Table 1, shows that Unlikelihood Training does better than ScaleGrad with a higher gamma. However, Figure 1 has no data on either Unlikelihood Training or a human baseline. It really should not be necessary to go looking through Table 1, Figure 1, and Appendix F to see that Unlikelihood Training is outperforming ScaleGrad on some metrics. Worse, the data presented in Figure 1 (b) actually makes comparison impossible, which makes me uncomfortable about the universally positive results in Table 1.\n\nOn page 4 the authors write “Following Welleck et al. (2020), we apply greedy decoding in our experiments in this section. This allows us to evaluate the modeling capability exclusively.” We will get into the matter of comparison to Welleck et al. 2020, but I would like to begin by addressing whether Greedy Decoding is a neutral choice that only tests modeling capability, because it is clearly not. There is a spectrum of generation algorithms between probability maximization and straight-forward sampling. Greedy is closer to probability maximization, but it only maximizes local probabilities (Meister et al., 2020) and inevitably comes-up with lower probability outputs than Beam Search or Bound & Branch (Stahlberg & Byrne, 2019). Welleck et al., 2020 show that Greedy Decoding results in better text along their proposed metrics for open-ended generation.\n\nSince Greedy Decoding is not a “neutral” choice, I do not believe it is appropriate to exclude stochastic decoding baselines from the given comparisons. Stochastic decoding algorithms such as sampling, top-k sampling, and Nucleus Sampling usually do very well on repetition and uniqueness metrics. Indeed, they can be seen to outperform all the other models on Table 16 in Appendix H.\n\nIn the analysis section, tables are quite limited in their coverage. In Table 6 no comparisons are made to systems that have not been trained with ScaleGrad, and these algorithms were not reported on in Table 1 so no comparison can properly be made even if the reader goes searching for the data.  In Table 8, Unlikelihood Training is not included in the comparison even though it does very similarly to ScaleGrad on the same task in Table 5. Finally, Table 5 shows that trigram blocking still helps significantly on ScaleGrad trained systems. This is understandable, but disappointing since getting rid of these kind of rules is described as the reason for proposing ScaleGrad.\n\nAltogether, I feel the comparisons made in this paper are not quite convincing and the argument about why Greedy decoding, a deterministic algorithm, should even be able to match the properties of a large, noisy distribution is not properly fleshed-out.\n\nMeister, Clara, Ryan Cotterell, and Tim Vieira. \"If Beam Search Is the Answer, What Was the Question?.\" Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.\n\nStahlberg, Felix, and Bill Byrne. \"On NMT Search Errors and Model Errors: Cat Got Your Tongue?.\" Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "84s-GIHdgFU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a  technique to encourage generating certain tokens (i.e. non-repetitive ones) in text generation. The idea is to scale the softmax probability  for certain words (in the novel set) by a factor of gamma. The authors show how this affects learning by deriving the effect on the gradient. \n\nMany experiments are presented both on open ended generation (language modeling, story telling) as well as abstractive text summarization to justify the method. The model seems to encourage more diversity than unlikelihood training in open ended generation (while still maintaining a lower perplexity). The gains in summarization are marginal. \n\nI have two questions:\n(1) How are lambda and alpha connected to \\gamma in Section 3? This make the method section clearer. \n(2) How much is the model discouraged from generating stop words like \"the\" or \"a\" (and how does this affect fluency)\n\nPros:\n-Well justified and simple method to solve a relevant problem in text generation. \n-Lots of experiments, gains in open ended generation seem decent. \n\nCons:\n-Gains on summarization are marginal / non-existent suggesting that this is not as large of a problem for more constrained tasks.\n-Some clarity on the questions above would be helpful. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3_jEVP1T5t", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThis work proposes an effective modification of language model token-level distribution during the training which prevents some forms of degeneration such as repetitions and dullness. The approach is based on the idea of encouraging the model to use tokens which were not observed in the previous context so far. In other words, this method changes softmax distribution such that unseen/novel tokens is being rescaled with a given hyper-parameter $\\gamma$ (eq.4). Authors conduct several experiments using different tasks such as open-ended generation, image captioning and abstractive text summarization. As a result they confirm substantial improvement over the standard mle training and **token-level** unlikelihood training. In addition to analysis of their method, authors discuss a potential issue of unlikelihood training criterion and how their approach avoids this issue.\n\n## Strong points\n\n1. Main method is easy to understand and I believe is easy to implement: this work can be a motivating example for future research towards degenerative text generation. From my understanding there are some interesting future work such as setting individial $\\gamma$ for some tokens, specific masks for novel token sets etc.\n\n2. Large-scale experiments **with some human evaluation**. I enjoyed seeing improvements on multiple tasks including summarization (with automatic metrics at least). In addition, analysis of stochastic decoding used in directed generation is meaningful and highlights the importance of this work. Appendix includes detailed description of each experiment protocol including the protocol of human evaluation. Convincing examples of generated continuations are given in the appendix.\n\n3. Code with implemented method and experiments is provided: code is based on fairseq custom module which makes it relatively easy to extend and do more research with it.\n\n## Weak points\n\n1. Misleading comparison choice: authors claim to compare their approach with unlikelihood training (UL) and choose **token-level** UL loss without even mentioning the existence of **sequence-level** UL loss which works better based on the original paper. In fact, the whole narrative looks like sequence-level version does not exist. Simply stating that ScaleGrad is being compared with token-level UL (which works worse than sequence-level) would make future conclusions more clear.\n\n2. Some relevant work got completely ignored. I am not aware of the full variety of prior work for this popular problem these days, but there is one i am aware of: https://arxiv.org/pdf/2003.11963.pdf, where authors *do similar gradient analysis* as here. If this one is missed, I wonder what else may be missing in the related work here.\n\n3. No human evaluation for text summarization. Given known weakness of automatic metrics in text summarization task and the fact that authors did human eval for text completion, I wounder why they decided to exclude it from here (I can totally see budget limitation as one of the factors, and saying this explicitly would be helpful).\n\n4. The potential issue of UL (sec. 5.4) does not look convincing. From my understanding the main line there is \"*UL essentially rejects the ground truth token in such special cases (subject to the choice of the value of $\\alpha$).*\". This statement on its own is not clear to me and seems to be disconnected from the previous one: \"*In this case, the norm increases as pk increases, which contradicts with the optimization principle.*\" I agree that in this case ($\\alpha=1, p_\\text{neg}>0.5$) the norm increases as $p_k$ (prob of ground truth token) increases, but I don't see any problem or contradiction here. From my understanding when $p_\\text{neg}$ goes above some threshold, then norm of the gradient of $p_k$ is growing proportionally to $p_k$. Keep in mind that as $p_k$ is increasing, $p_\\text{neg} > \\frac{1}{1+\\alpha}$ is eventually dissatisfied (because of softmax property), i.e. I don't see any issue. Would be great if authors can elaborate more about this.\n\n## Recommendation\n\nOverall I vote for accepting this work as long as main concerns will be addressed/discussed. This is a decent approach with a strong experimental evidence and it will be useful for text generation community in the future research. I would be even more satisfied if authors can discuss / clarify / address the points I highlighted above.\n\n## Comments and questions\n\n1. stochastic beam search was mentioned as one of the efforts to solve text generation issue, but I believe it is more about doing sampling without replacement on the sequence-level. I am curious if authors may provide some perspective on how stochastic beam may reduce the degeneracy (e.g. compared to simple beam search).\n\n2. In sec. 2.1 teacher forcing is described as being used \"*used to train neural text generation models for faster convergence.*\". I wonder how can one use MLE criterion on the token level without teacher forcing? E.g. if we use predicted token as the context in the next time step, then we have no target truth token for the next time step (since the context is changed). In other words I think that teacher forcing is essential if we aim to maximize the probability of training sequences.\n\n3. In section 2.2: \"*thus reformalizing the probability distribution*\", this wording reformalizing sounds a bit weird to me, but it is clear what authors had in mind.\n\n4. Did you think about combining UL loss (both seq level and token level) with ScaleGrad? From my understanding it is possible since ScaleGrad emphasize novel tokens, and this softmax from scalegrad may be used in the UL loss, which may help even further! Importantly, sequence-level UL would allow to use ScaleGrad on the sequence-level, since there is no need for ground truth target there, and while penalizing the repeating words with UL, ScaleGrad would emphasize the novel words (sounds promising to me). Overall the paper narrative looks as combating the UL method (with some misleading gaps about token vs. sequence level), but to me it looks like they may work together!", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes ScaleGrad, a simple technique to encourage generating non-repetitive tokens for text generation tasks. The key idea is to modify a language model's token-level distributions by rescaling the softmax probability for certain words (in the novel set) by a factor of $\gamma$. Experiments show that ScaleGrad outperforms MLE and Unlikelihood Training (UT).

This paper receives 2 reject and 2 accept recommendations. Most of the reviewers have provided very detailed comments, and the authors have also provided very long and detailed responses. On one hand, all the reviewers agree that the experiments are comprehensive, and the motivation of the proposed method is clear. 

On the other hand, several concerns still exist after the rebuttal, namely, hand-wavy arguments and inconsistent experimental protocol. (i) The empirical evidence in the experiments is not convincing enough. It makes reviewers more reluctant about the approach after seeing more experimental results during the discussion. That is, for different tasks, different $\gamma$s are used, while the hyper-parameters used for other methods seem to be default values. This makes reviewers hesitant that scaling novel tokens and renormalizing the model's output distribution is really significant. (ii) Another minor concern is that the discussion on the potential issue of UL (sec. 5.4) does not look convincing. (iii) After reading the paper, the AC also feels that the novelty of the proposed method might be a little bit limited. 

The rebuttal unfortunately did not fully address the reviewers' main concerns. On balance, the AC regrets that the paper cannot be recommended for acceptance at this time. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
Decision: Reject