Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation 
Authors: Jin Xu
Xiaojiang Liu
Jianhao Yan
Deng Cai
Huayang Li
Jian Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: Repetition
Language Model
Analyses
Mitigating

Abstract: While large-scale neural language models, such as GPT2 and BART,
have achieved impressive results on various text generation tasks, they tend to get stuck in undesirable sentence-level loops with maximization-based decoding algorithms (\textit{e.g.}, greedy search). This phenomenon is counter-intuitive since there are few consecutive sentence-level repetitions in the human corpus (e.g., 0.02\% in Wikitext-103). To investigate the underlying reasons for generating consecutive sentence-level repetitions, we study the relationship between the probability of repetitive tokens and 
their previous repetitions in context. Through our quantitative experiments, we find that 1) Models have a preference to repeat the previous sentence; 2) The sentence-level repetitions have a \textit{self-reinforcement effect}: the more times a sentence is repeated in the context, the higher the probability of continuing to generate that sentence; 3) The sentences with higher initial probabilities usually have a stronger self-reinforcement effect. Motivated by our findings,  we propose a simple and effective training method \textbf{DITTO} (Pseu\underline{D}o-Repet\underline{IT}ion Penaliza\underline{T}i\underline{O}n), where the model learns to penalize probabilities of sentence-level repetitions from synthetic repetitive data.  Although our method is motivated by mitigating repetitions, our experiments show that DITTO not only mitigates the repetition issue without sacrificing perplexity, but also achieves better generation quality. Extensive experiments on open-ended text generation (Wikitext-103) and text summarization (CNN/DailyMail)  demonstrate the generality and effectiveness of our method.

Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation
Jin Xu Xiaojiang Liu*, Jianhao Yan?, Deng Caf Huayang Li Jian Li' Institute for Interdisciplinary Information Sciences, Tsinghua University School of Engineering, Westlake University
The Chinese University of Hong Kong *Apple
xujin210mails htw.:/000malss.ssnnnuuuuee edu. cn, iiaojiang_Liu@applee www.llppulppe...m com, elliottyan3@@gmall.comm thisisjcykcdd@gaal com, h11466apple. com, 1ijian83dmail tsinghua eertshuuueeuu.ee
Abstract
While large-scale neural language models, such as GPT2 and BART, have achieved impressive results various text generation tasks, they tend to get stuck in unde- sirable sentence-level loops with maximization-basee decoding algorithms (e.g., greedy search). This phenomenon is counter-intuitive since there are few consecu- tive sentence-level repetitions in human corpora (e.g., 0.02% Vikitext-103)) To investigate the underlying reasons for generating consecutive sentence-leve repeti- tions, we study the relationship between the probabilities the repetitive tokens and their previous repetitions in the context. Through quantitative experiments, we find that 1) Language models have preference to repeat the previous sentence; 2) The sentence-level repetitions have self-reinforccemet effect: the more times a sentence is repeated in the context, the higher the probability of continuing to generate that sentence; 3) The sentences with higher initial probabilities usually have stronger self-reinforcement effect. Motivated by our findings, we propose a simple and effective training method DITTO where the model learns to penalize probabilities of sentence-level repetitions from pseudo repetitive data. Although our method is motivated by mitigating repeti- tions, experiments show that DITTO not only mitigates the repetition issue without sacrificing perplexity, but also achieves better generation quality. Extensive ex- periments on open-ended text generation (Wikitext- 103) and text summarization (CNN/DailyMail) demonstrate the generality and effectiveness of our method. Code is released at https::/// / ggithu.. com/ Jxu- Thu/DITTO
PenalizaTiOn),
Introduction
Recently, large-scale Transformer-bes based (34 neural language models (c.g., GPT2 29], BART
and OPT (39]) have shown remarkable capability generate text and achieved better performance than before, such as open-ended generation (29 5 and summarization tasks [17 39 However, models with standard maximization-bann based decoding repetitions [3]. As shown in Figure [] the model has a stronger preference for consecutive sentence- level repetitions than the word- phrase-level while human language has fewer consecutive sentence- level repetitions, which shows a discrepancy between human language and the generated texts. Existing approaches mitigate repetitions can be categorized into decoding-based and training-based methods. Decoding- methods rectify these issues by soft or hard n-gram blocking [26 [6]
known get stuck in redundant consecutive
"The work was conducted Apple.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

rim slightly raised about surrounding surface an outer rampart that has been only lightly impacted. No significant craters intersect the
35.0
Consecutive Repetition (%)
Human
sloped about @8. 5 toward the direction
from the Earth.
28.0
Model
Greedy Decoding:
The age crater about 3 g-@ billion years and has been prosimity the south lunar pole for at loast 10 9.00 onn years. The South Crater located on the southern edge the northern highlands The South northern highlands. Crater is located on the southern adge of the northern highlands.. The South Crster located on the southern edge the northern highlands The South
21.0
Crater located on souther edge
South
14.0
Crater located on southern edge
northern highlands. South
7.0
Crater located on the souther adge of the northern highlands.. The South
Crster located on the southern edge the northern highlands The South Crater on southern edge northern highlands... ....
0.0
Word Phrase#2 Phrase#3 Phrase#4 Sentence
Figure 1: Statistics of human sentences versus model generation on the dev set of Wikitext- 103. We train a Transformer model (750M parameters, similar to GPT-2 Large) on large-scale human corpus Wikitext-100 (over 100 million words). Left: Greedy decoding gets stuck in consecutive sentence-level repetition. Right: The percent of consecutive repetition of the word-, phrase (# number of words)- and sentence-level (see Appendix A for formulations of consecutive repetition). The model results are the average of generated texts given different prefixes from the Wikitext-100 103 dev set. Specifically, given 50 tokens the prefix, the model greedily generates the next 200 tokens. Compared to human language, the model has substantially more consecutive sentence-level repetition.
perplexityfinformation controlling B 21 and stochastic sampling 7 based methods minimize the probabilities of tokens that already are generated in the previous context 36 19]. Despite their effectiveness, the reasons why the model prefers repetitions and how repetition occurs during decoding are still unclear.
Training-
Fu et al. (9] is the first to analyze the repetition problems from theoretical perspective by assuming that the language models can be approximated by short-sighted first-order Markov models. However, Holtzman al. [13] observe the cases of positive feedback loop of repetitions, which indicates that language models do look at the long-distance context and may not be simply viewed as first- order Markov models. The cases also reveal that the probabilities of repetitive tokens have certain relationships with previous repetitions in context. However, they do not analyze why the model prefers consecutive repetitions, In this paper, we further dig into the problem and conduct quantitative experiments to analyze the underlying issue of the repetition.
For quantitative investigation consecutive repetitions, compare the probabilities of the same tokens in repetitive sentences. For example, given a sequence, 'I love oranges. I love oranges ", we compare the probability Po ('oranges" love oranges. love') with (""rannges") love'), ). The difference between them is that, for the second 'oranges', there is already token 'oranges' that shares the same sentence-level context love'. We manually repeat the sentence n times as the context so that the next 'oranges' has repetitions in context. In this way, we can investigate the relationship between the probability of the token and the number of repetitions in context. For example, we can first construct the context by repeating the sentence 'I love oranges !' times plus 'I love", and then obtaín the probability that the model outputs 'orange' at the current step.
Through our quantitative investigation across the different corpus, we find that 1) The model tends to raise the probability of repeating the previous sentence. Specifically, even if there is only one sentence-level context repetition, the probability of repetition at the current step increases in most cases. The cause of the phenomenon may be that the model is confident with its context, when there is a previous token (i.e., 'oranges') ) sharing the same sentence-level context (i.e., 'I love") ))." that the model learns shortcut to directly copy the token; 2) Self-reinforcement effeet: the probability of repetition increases almost monotonically with the number of historical repetitions. Finally, the probability of repetition stabilizes around a certain ceiling value. As shown in Figure as the number of repetitions increases, the probability of the word 'rounds" and 'general' both increase almost monotonically and finally stabilize; 3) Sentences with higher initial probabilities usually have stronger self-reinforeement effect. For example, in Figure we can find that the sentence with a higher initial probability (i.e., the red bar at '0" of x-axi)) grows faster and can reach an extremely high value with few repetitions. Furthermore, the sentences with higher initial likelihood (e.g.. sentences generated by the model itself with maximization-bn based decoding algorithms) may have a stronger self- rei-rreefrree effect.

---

pinlked
I.,, In., is the 1-th token in the n-th repetition of the sentence & and Tag is the number of tokens in sentence Denote the sentence-level context token max pt prijk.))
0.20
Pobelbiltyy
0.10
20
0.00
15
20
25
39
Repetttion
Ropetitio
Figure 2: Manually repeat given sentence several times, feed the model and observe the same token's 's probability p(aa(a<<) (in red) and maximum probability max p(- (x()) (in blue) as sentence repetition times increase. Left: Repeat a normal sentence ('She is named a proxy general under Gaara.) .) and present the probability of the token 'general'. Right: Repeat random sentence ('fría backed rounds Manganiello Benzedrine Magruder Crego Stansel Zemin compressus. .') where tokens randomly sampled from the vocabulary, and present the probability of the token 'rounds'. The probability repetition (in red) has a self-reinforcement effect: the probability of repetition (y-axis) increases almost monotonically with the number of historical repetitions (x-axis). Best viewed in color and zoomed desktop monitor.
According to findings, the reasons why the model tends repeat themselves sentence repetition occurs since the previous sentence generated by the maximization- based decoding algorithms has a relatively high probability, and the model tends to further increase the probability of repeating that sentence. Once the model generates one repetitive sentence, the probability of the repetitive sentence would be further enhanced since there more repetitions sharing the same sentence-level context to support the choice of copying. As result, the model gets stuck in the sentence-level loops due to the self-reiiforcemment effect.
follows: The
To mitigate the tendency to repeat previous sentences and overcome the self-reinforeement effect, aim make the model aware that the probabilities of generating redundant sentences should be reduced. In addition, the more sentence-level repetitions exist, the lower the probability of the redundant sentence should be. In this paper, propose simple and effective training- training-based method, SeeuDo-RepetlTee PenalizaTiQn (DITTO). We first manually construct pseudo data by repeating a sentence sampled from the training corpus. Then, design repetition penalization loss function on these pseudo data for the model to learn to exponentially decay the repetition probability as the number of sentence repetitions increases. The experimental results show that DITTO not only significantly reduces repetitions without sacrificing language model capability in terms of perplexity, but also achieves better generation quality. In the Wikitext-103 22 open-ended generation task, our methods can achieve lower perplexity and higher MAUVE scores than the strong competitors. With stochastic decoding strategies such as top-k 7 and nucleus sampling [R], generated texts by our methods are the closest to human- wrrtten texts, as measured by MAUVE score and human evaluation.
In the commonly- used CNN/DailyMail summarization benchmark [12 23, our methods also achieve the best performance among the strong competitors.
2 Analyzing Repetition
In this section, we would like to quantitatively investigate the relationship between the probabilities of repetitive sentences and previous repetitions in context. We first define several metrics to measure how the probability changes as the number of repetitions increases. Then, we summarize our findings from the experimental results, and discuss how sentence-level repetitions occur during decoding.
2.1 Experiment Design
Formally, we have a sentence s from a corpus D and repeat times to construct a sample sequence = (s s .... .....))))) where s = (In,1,
End <I (In,1 In.-1)). We define that there sentence-level context repetition for the

---

IP: Rate Increasedd Prob
TP: Average Token Prob
WR: Winner
0.90
Repetition 90 (b)
10 20 30 # Repetition
90100
Repetition (a)
(c)
Figure 3: Construct sample by repeating a sentence N = 100 times, manually feed it to the model and calculate TP, (average token probability), IP, (rate of increased probability) and WR, (winner rate) for 1,... N. The results average sentences from different corpus Drandom book and Dwiki respectively.
token In, if and only there is another same token Zi,1 context such that X;,<! = Xn, and < n. For example, for the sequence 'I love oranges love oranges aforementioned, the second token "oranges' has a sentence-lever context repetition. By feeding the sequence x to pre-trained model Ps. obtain the probability distribution Pe (x( (<n..)) where X<n.I (s ........ s T 1--1__ <t,)().
To study whether Po (2 Xen.t) increases as n increases, we define several metrics as follows:
Average Token Probability: TP(s") I. E/= Po (o which is the average token probability of n-th repetitive sentence s" 'PP(s") is the initial probability of tokens in the first
sentence.
Rate of Increased Token Probability: IP(s")= ł. Σ (PPo( > Po (x,,)))
where is the indicator function. We use IP(s") to calculate that, how many probabilities of tokens increase in s" compared to the initial ones in s
Winner Rate: We say In is winner Po(In.)(xen.)) > Pa(2o,1(xxol and Dnat = arg max max P(, Then, we define the winner rate as WR(s")- = T. Σte, 1 (Tn, is winner). A higher winner rate means that the repetitions are more likely to be generated by a maximization- based decoding algorithm such as greedy decoding.
Following the previous work 33 19], the pre-trained model is a 16-layer Transformer decoder trained on Wikitext- 103 [22]. The details are introduced Sec. [4.]] Given the corpus D, we can calculate the average values of TP, IP and WR of the n-th repetitive sentence s" as
TP
1
Σ TP(s"), SED
IP,
1 IP((s") MED
WR,
WR(s")
D
D
D
SED
by enumerating all sentences s in D. To further study the effect of different corpus, we construct three corpus 1) D Prandom: the tokens of sentences are randomly sampled from the vocabulary of the model; 2) D the sentences are randomly sampled from BookCorpus 44), and 3) Dwiki: the sentences are randomly sampled from the dev set of Wikitext-103. The size of all different corpus is 1,000.
2.2 Results and Analyses
In the analyses, we attempt to answer: 1) Why sentence repetitions occur? 2) Why the model gets stuck into the repetition loop? 3) What kinds of sentences are more likely to be repeated?
Why sentence repetitions occur? As shown in Figure 3 (b). IP; higher than 90% across the various corpus, meaning that even if there is only one sentence-level context repetition, the probability of repetition increases in cases. That indicates the model has a strong preference to repeat the previous sentence. Note that the token repetition has not occurred at the current prediction step, and there is only the same sentence-leve context. For example, model assigns higher probability to Po( "oranges"] love oranges love") than Pa ("oranges") love') since the model has seen the pattern 'I love oranges' in the previous context. Thus, the model may be too confident with its previous context repetition and learn 'cheap' shortcut where it directly copies the next token, "oranges". It quite different from human language because human would try to avoid full sentence repetition, the repeated sentence has no new information in most cases.

---

Why the model gets stuck into the repetition loop? Furthermore, as shown in Figure 3 as the number of repetitions increases, TP, IP and WR increase monotonically. That means, the sentence- level repetition has a relffreinforceement effect: the more times sentence has been repeated in the context, the higher the probability of continuing to generate that sentence. Finally, IP, WR, TP converge around certain ceiling values.
What kinds of sentences more likely to be repeated? We can find that the sentences with high initial sentence probability TPo, such as sentences sampled from Dwits and Drook. have a stronger self- reinforcement effect: TP and WR grows faster and reach the high ceiling values with few repetitions (See Appendix D.I for more experiments at sentences with different initial probabilities). The higher TP and WR, the more likely the sentence be repetitively generated by maximization-bann based decoding algorithms. Note that sentences generated by maximization-bel based decoding algorithm have higher initial likelihoods. Thus, the model prefers repeat itself.
Analyses During decoding with maximization based algorithms, the sentence-level repetition occurs because 1) Previous generated sentences have high likelihoods and thus have more potential to repeated; 2) Given previously generated sentences, the model is more likely to generate the repetitive sentence since the model is confident on the historically generated context and tries to find hints context for the current generation; 3) Once the model repeats the sentence for several times, would get stuck in the sentence loop due to self- reieforeiforrtttnttt effect. The effectiveness of stochastic sampling approaches such as top-k [7] and nucleus sampling [13] may rely on 1) compared to the maximization-basee based decoding algorithms, the likelihood of previously generated sentences is lower and thus the sentences have less risk of being repeated; 2) current tokens are generated stochastically rather than chosen with maximum probability which can avoid the reinforeiforttttt effect.
3 Pseudo-repetition Penalization Training
According to our analyses in Sec. 2 it can be clearly seen that the core issue of sentence-level repetition for the model is the tendency to repeat previous sentences and the self-reinforcemmnt effect. In this section, we propose a simple and effective method, named SeeuDo-RpetlTin PenalizaTiOn (DITTO). It first manually feeds repetitive sentences the model and then explicitly teaches the model to be averse to such repetitions.
Pseudo Repetitive Data Construction. To construct pseudo repetitive sample x, we first ran- domly pick a sentence s from the training corpus. Then, the pseudo data x = (s"..... .......... N (zu,0:-.. 21,0 ,2,00
IN,L,) is constructed by repeating the sentence by N 1 times. The sentences are repeated until they reaches the maximum input sequence length of the model (e.g.. 1,536 tokens in the Transformer for open-end generation).
ZN,U." ,
Sentence-level Repetition Penalization. To define the per-step penalization loss for token x e {z1,0. ... LN.L. J. we define the training objective for the 1-th token in the n-th repetition of the sentence s as
CDiTro (Po(In.)Xen.t) = - log(1 - (Po - A Pā (
(1)
where Pa(.) (-) means that the value is excluded for gradient backpropgation, which can implemented by tensordetach in pytorch [25]. X is the penalization factor. When À = 1, the loss function is minimized when the probability of token xn, in n-tth repetition is same as that in the (n (n-1)-th repetition to avoid the reffrreinforemment effect; when À < 1, the probability of token 2n,/ in n-th repetition should be smaller than that in the (n 1) repetition to make model averse to sentence-level repetition. In other words, the probability of tokens in repetitive sentence should decay exponentially with factor of X, where À is
In our experiments, we apply the sentence-level repetition penalization by fine- tine-tunigg a standard MLE baseline. Fine-tuning is done by equally mixing the sentence-level repetition penalization update and normal MLE loss update. We find that, appending the previous context of the repetitive sentence as the prefix to the pseudo repetitive data x can achieve better and more stable performance. We have tried other alternative loss functions, but they do not perform better. See Appendix C for the details.
5

---

Discussion In human language, there are necessary token-level and phrase-level repetitions that naturally occurs. For example, names of people, city and lemma, set phrases and proverbs appear many times in long documents such as Wikipedia. Thus, given the prefix, the model should have the ability to copy these words or phrases from the prefix and increase their repetition probabilities. These useful repetitions can be viewed as positive samples, and our pseudo repetitive data can be viewed as negative samples. Combining them for training, the model should learn to distinguish between useful and useless contexts. Thus, although our method is motivated by overcoming sentence-level repetition issues, may improve the learning ability and generalizability of the model. Unlike previous methods (e.g., token-level unlikehood training [36] and n-gram blocking [26] [6] that put hard constraints to penalize any repetitions (regardless of the necessity of repetitions), our training objective explicitly encodes the intuition that the model should be inclined to avoid over-repetitions. Therefore, it may
alleviate the drawbacks of over-penalization.
4 Experiments
4.1 Setup
Dataset and Model Training We train models on the benchmark dataset Wikitext-103 [22] to evaluate the performance of open-ended generation. The dataset contains over 100 million words. The experiments are conducted at the word level. The model architecture and training hyper-parameters exactly follow the implementations of Welleck et al. [36 Specifically, we use a 16-layer Transformer with 8 attention heads, hidden size 1024 and fonnecconnected dimension 4096. There are total of 750 million parameters, similar to GPT-2 Large (29]. We first train the baseline model with standard maximum likelihood (MLE) for maximum of 150k updates with a batch size of 3 samples. The maximum length of each training sample is 1,536 tokens. Then, we fine-tune the model that has the best validation perplexity with DITTO training for 10k steps. Unless otherwise mentioned, À is set as 0.5. Baseline models are trained following their official implementations. All models are trained on 8 NVIDIA Tesla V100 GPUs. The experiments are implemented based on fairseq codebase (24
Evaluation Following common practices 33 [9], we evaluate the quality of open-ended sequence completions on the test set Wikitext-103, where the prefix is 50 tokens, and models autoregressively generate the next 100 tokens. Models including various baselines are selected with the best perplexity on the validation set for fair comparison. The evaluation metrics are listed as follows:
MAUVE: MAUVE 27 is metric to measure how close model-generated text is to human
language by approximately calculating the KI. divergence between the distribution of model generated sequences and human language. The range of MAUVE is from 0 to 1. A higher MAUVE score indicates better generated sequences.
Given the prefix and true next token, we use perplexity, and next-token prediction accuracy measure the language modeling ability.
Perplexity and Accuracy:
Repetition: Following previous work [36 19], we calculate the portion of duplicate 4- grams (Repetition-4) in a generated sequence to measure phrase-level repetition, defined 1.0 Junique
and average over completions. Similarly, we use por- tion of duplicate sentences (Repetition-Sen) to measure sentence-level repetition, defined as 1.0 junique sentences)/ /ssentences). Since there are natural repetitions in human language, the optimal model should produce text whose repetition metrics are close to that of the gold text.
4.2 Results of Open-ended Generation
Method Comparison We compare with training- based methods since decoding- based methods
can be readily applied to models with our training method. Following (36 19, we first compare different training-basse algorithms based on greedy decoding. The results are presented Table Our DITTO can significantly reduce the phrase-level and sentence-level repetitions and achieve the best MAUVE score of 0.77. Although UL-token+seq can generate fewer repetitions, its improvement in the general quality of the model measured by MAUVE is limited since its generations usually not relevant to the prefix (See Appendix F for examples). Compared to the standard baseline MLE,

---

Table 1: Results of different training-
craining-based based methods on the test set of Wikitext- 103 for open- ended generation. The results are reported based three runs with different random seeds. The best value is bolded and the second best underlined.
Model
MAUVE
Perplexity
Accuracy
Repetition-4
Repetition-Sen
MLE
29
0.04
25.68
40.00
10.00
44.20+1.438 %
14.50 1.59 %
UL-token 36
0.57
26.98
0.39 +0.00
28.30+0.78 %
7.40 %
UL-token+seq [36]
0.48
25.95
0.40+0000
7.60+0.46 %
0.05 +0.03 %
SG [9]
0.74
25.84
0..00
23.0010.22s %
5.24.10.75 10.75 %
DITTO (ours)
0.77
24.33
0.42
22.00 +0.31 %
+8.74 %
Human
1.10%
0.01%
Table 2: Results of different training-hased methods the test set of Wikitext- 103 under different stochastic decoding algorithms. l = 50 and top-p (p = 0.9) for nucleus sampling. The numbers
closest to human scores are in bold except for MAUVE (27].
Search
Model
MAUVE
Repetition-4
Repetition-Sen
MLE
29
0.94 10.00
10.09 %
0.25 10.06 %co
UL-token
36
0,95
0.70 %
0.00
Top-k
UL-token+seq
36
0.93
0.09 +0.1 %
0.06 %c
SG 19
0.93
0.50 %
0.00
DITTO
0.96
1.00 %
0.09 %
MLE
0.94
1..0 %
0.01
UL-token
36
0.94
0.47 10.00 %
0.00
Nucleus
Ul-token++seg
36
0.94
O. +..05
0.02 %c
SG [1
0.93
0.40 %
0.06 %o
DITTO
0.96
4.98 %
0.08
Human
1.10%
0.10%o
other competitors reduce repetitions by sacrificing perplexity while DITTO achieves lower perplexity and higher accuracy. These results show that DITTO can not only mitigate repetitions but also help the model improve the language modeling and generation quality.
Compatibility with Stochastic Decoding Strategies Sampling-based decoding algorithms such as top-k (7) and top-p (nucleus sampling) [3 are widely used in the various applications (1 [30 and large models 22 5). We also confirm that our DITTO is compatible with these popular stochastic decoding strategies. The results are presented in Table 2 It can be seen that models trained with DITTO can reduce repetitions compared to the MLE baseline and achieve the closest results to human in repetition metrics. For the quality of generated sequences, DITTO achieves the highest MAUVE of 0.96 with top-k or nucleus sampling.
Human Evaluation We conduct pairwise crowdworker evaluation to judge the quality of the generations of DITTO compared to other baselines. For each pair of methods, the model generates texts based on the same 100 random pre- fixes from the test set of Wikitext- 1033. Evaluators are asked to independently judge which generation is better terms of their grammaticality, relevance, coherence and informa- tiveness. The evaluation interface and more details are Appendix E. As shown in Table 3 DITTO consistently and significantly outperforms other competitors across different decoding strategics.
Table 3: Human Evaluation Results. * means the results are statistically sig- nificant (2-sided binomial test, p<.05).
Win Rate
Greedy Search
DITTO vs MLE DITTO vs UL-token DITTO vs UL-token+seq DITTO vs SG
*84%
*80%
*65%
*68%
Nucleus Sampling p 0.9
DITTO vs MLE DITTO vs UL-token
*65%
Self-reinforeemmett Effect As mentioned in Sec. 22, the
*71
sentence-level repetition has a self-reinforcemment effect,
DITTO vs UL-token+sse
*62%
which is the core issue of sentence-level repetition. We further study when the model trained with different methods. The settings of experiments follow those in Sec. 2
DITTO vs SG
*63%
As shown in Figure 4 we can find 1) TP,
7

---

1.00
TP: Average token Prob
0.600
0.30
0.60
DITTO Mi-trieen
Winner
DFTTO
0.20
to
0.40
UL EoiEn
0.20
0:10
0.20
40
80
100
1.0
40 50 60
90100
10 20
30
40 50
90100
Repetition
Repetition
Repetition
Figure 4: Results of different training-based methods by feeding repetitive sentences described in
Sec. 2] average the results on sentences from Dwiki
IP and WR increase almost monotonically UL and SG, which shows that the self-reinforcement effect has not been solved in these training-based methods; 2) When the model is trained with DITTO, these metrics drop rapidly the number of repetitions grows, showing the effectiveness of DITTO in overcoming the self-reinforcement effect.
4.3 Results of Directed Generation
Table 4: Abstractive summarization results on CNN//ailyMaii..
Model
ROUGE-1
ROUGE-2
ROUGE-L
Pointer-generator Coverage
:2
39.53
17.28
36.38
Mask Attention Network
40.98
18.29
37.88
BertSum 20
42.13
19.60
39.18
UniLM 6
43.08
20.43
40.34
UniLM V2
43.16
20.42
40.14
ERNIE-GER Jarge 37
44,02
21.17
41.26
PEGASUS 38
44.17
21.47
41.11
ProphetNet 28
44.20
21.17
41.30
PALM 4
44.30
21.12
41.14
BART- large w.t. MLE 17
44. 44.11:00.33
21.21110001
40.83:10.02
large w.t. UL-token
36
44.1710.04
21.20110.02
40.83-10.03
BART-large w.t. UL-token+seq
36
44.13-10007
21.1510.111
40.71:10.09
BART-large w.t. SG [9
44.18+00.06
21.17+0.07
40.89+00.05
BART-large DITTO
44.41=:0033
21.45:0.01
41.16:10.02
Setup We further conduct experiments on the directed abstractive summarization task MNii [3] given an input document, the model generates several sentences as summa- rization. We adopt the state-of-the-an art model BART-large [17] as our baseline, which is large-scale encoder-decoder Transformer architecture trained on 160Gb data. For DITTO training, given document and its summarization, from the summarization and then repeating until reaching the maximum summarization length of the decoder model while leaving the document unchanged. We follow the official implementa- tions 17) to train the BART-large model on the CNN/DailyMaii and then fine-tune the model with the best validation perplexity using DITTO training. The X set as 0.9. During inference, tri-gram blocking 32 23 and beam search (beam size 5) are used as Lewis et al.
construct the pseudo data by first random sampling sentence
Results We evaluate the performance of the model with the standard F1 -asdd ROUGE 331 scores (ROUGE-I, ROUGE-2, ROUGE-L.. shown in Figure our training method consistently outperforms UL and SG with a large margin on ROUGE scores and outperforms other competitive baselines. The results also show that DITTO is compatible with the n-gram blocking, a decoding- based method for mitigating repetitions, demonstrating the generality of our approach.
4.4 Analyses
Auto-completion with different decoding lengths From a practice point view, we analyze DITTO in different decoding lengths. As shown in Figure 5(a) and (b), the models trained with DITTO consistently and significantly outperform those trained with MILE with the constraints of different decoding lengths, which shows the effectiivnees and robustness of our method.

---

16.1
Hague 1ag4400oon San
0.94
46.6
0.64
44.4
0.16
Wilfod CoN
42.4
0.18
80 130
320
325
350
100
200
300
120
340
300
0.00
43.0
(8]
Figure 5: Results of DITTO in different decoding lengths and hyper-parametert À is the penalization factor and y the mix ratio of pseudo data and true data. Results are average of three runs with different random seeds.
Hyper-parameter Study Towards better usage and understanding of DITTO, we study the two hyper-parameters: 1) mix ratio of pseudo data and actual data, denoted as and 2) penalization factor X. As is shown in Figure[5cc models achieve the best performance with ? 0.5 on both datasets. Thus, equally mixing the DITTO update and the MLE training loss update is recommended in practice. As for X, we can observe that the optimal value varies different tasks. In open-ended generation task Wikitext-100 103, stronger penalization with À 0.5 is preferred. However, mild penalization with À 0.9 is better the summarization task. We conjecture that the repetition issue is more severe in generation tasks with more freedom, such as the open-ended generation.
5 Related Work
Repetition in Neural Text Generation. Repetition has been key problem in neural text generation for various tasks including open-ended generation (e.g., text continuation
36 19 and direct observes that, with maximization--assd decoding strategies such as greedy search, language models prefer to generate bland and consecutive repetitions at the word-level, phrase-level, and sentence-level. Consecutive word-level and phrase-level repetition are lexical repetitions [10] where the sentence structure is incorrect, while sentence-level repetition is semantic repetition that indicates the expression is not informative and has a discrepancy with human language [0. Recently, large-scale pre-training with Transformer architecture [3 such as GPT-2 (2 and BART [17, have greatly improved the ability of language modeling and can generate fluent sentences similar to human language. However, the generations still have a large number of unexpected consecutive sentence-level repetitions 22 in practice, which remains a severe problem in neural text generation.
[3
generation (e.g., summarization 32 2 Previous work [1
36
18
15
He al. [1 find that, when given the ground-truth context as the prefix repetitive sentences generated by the model itself as the prefix, the language model can generate high quality texts which shows the model has the self-recovery ability. However, they adopt ancestral sampling to generate continuations rather than maximization-Be based decoding methods. When the model generates tokens by ancestral sampling such that the current sentence structure different from previous sentences, the model stops repeating the sentence since their sentence-level context are different. Different from their findings, our analyses reveal that the probability of generation repetitive tokens will increase they share the same sentence-level context.
Approaches for Mitigating Repetition. Approaches for mitigating repetition neural text gen-
eration can be categorized into training-based (36 19 33 and decodding-used [32 7
13 33
approaches. The representative training- approaches are unlikelihood training (UL) 36 and straight gradient (SG) [19]. UL reduces the probability of negative candidate tokens, and SG improves the probability of tokens that do not belong to negative tokens, where the negative to- kens are a set of tokens that appear the previous context. However, our experiments in Sec. 4.2 show that they cannot solve self-reinforcemmnn issue. In contrast, our work analyzes why models prefer repeat themselves and quickly get stuck into the sentence-level loop, and mitigate the self-ren renfooreifrr issue. Many decoding- -aasddd methods have been proposed to rectify these issues in various tasks. In summarization, n-gram blocking [32 23 17 is often used to block repetitive graamms from subsequent generations. In open-ended tasks, - ( and nucleus sampling [13] 35 truncate unreliable tail and employ sampling according to token distribution for generating fluent and informative texts. SimCTG (3 proposes contrastive search to reduce repetitions by first calibrating the model's representation space and then encouraging a trade-off between diversity and coherence the generated text.

---

Analyses of Repetition Problem. Although previous work has noticed the repetition issue and many approaches have been proposed to mitigate it at the model training or decoding stages, little has been discussed about the causes of the repetition. Fu el al. [ theoretically analyzes the problem by assuming the language models can approximated to first-order Markov models. However, the cases of positive feedback loop of repetitions presented by Holtzman et al. [13] indicate the repetition probability has complex relationships with a quite long-range context and language models may not be simplified as first-order Markov models. However, there is a lack of detailed numeric and statistical results across various sentences and analyses their work. In contrast, we further dig into the problem and conduct quantitative experiments to analyze underlying issues of repetition.
Conclusion and Future Work
We quantitatively investigate why language models with maximization-bana decoding prefer consec- utive sentence-level repetitions. Our observations and analyses show that the issue of the repetition is the tendency to repeat previous sentences and the self-reinforcement effect. Guided by our analyses, we propose simple and effective method named DITTO by constructing pseudo data and teaching model to learn to gradually decay the repetition probability as the number of sentence repetitions grows. The experiments the Wikitext-103 open-ended generation task and the CNN//ailyMai summarization task demonstrate the superiority of our methods.
This work investigates the relationship between the probability of the token and the number of repetitions in context. However, there should be deeper reasons for why the model raises the probability of repeating tokens from the perspective of model embedding, neural network architecture or intrinsic characteristics of language. Our current analysis has not touched these aspects, which we leave for future work. We hope our quantitative analyses and approaches to mitigate sentence-level repetitions can help the NLP community better understand, resolve the repetition issue and improve the fundamental language modeling ability of neural language models.
7 Acknowledgements
The authors thank three anonymous reviewers and meta reviewer for their comments, which greatly improved the article. Jin Xu and Jian Li are supported in part by the National Natural Science Foundation of China Grant 62161 146004. Turing AI Institute of Nanjing and Xi' an Institute for Interdisciplinary Information Core Technology.
References
[1] Daniel Adiwardana, Minh-Thame Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppi- lan, Zi Yang, Apoory Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. Towards a buman--ike open-domain chatbot. arXiv preprint (XXi:22011009977, 2020.
[2] Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, et al. Unilmv2: Pseudo-masked language models for unified language model pre-training. In International Conference on Machine Learning, pages 642-552. PMLR, 2020.
[3] Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R Varshney. Mirostat: A neural text decoding algorithm that directly controls perplexity. In International Conference on Learning Representations, 2020.
[4] Bin Bi, Chenliang Li, Chen Wu, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. Palm: Pre-training
language model for context-conditioned generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language 2020.
Processing (EMNLP), pages
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 2020.
10

---

[6] Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre-training for natural language under- standing and generation. Advances in Neural Information Processing Systems, 32, 2019.
[7] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pages 889-898, 2018.
[8] Zhihao Fan, Yeyun Gong, Dayiheng Liu, Zhongyu Wei, Siyuan Wang, Jian Jiao, Nan Duan, Ruofei Zhang, and Xuan-Jing Huang. Mask attention networks: Rethinking and strengthen transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1692-1701, 2021.
[9] Zihao Fu, Wai Lam, Anthony Man-Cho So, and Bei Shi. A theoretical analysis of the repetition problem in text generation. arXiv preprint arXiv:2012.14666 2020.
[10] Jian Guan, Xiaoxi Mao, Changjie Fan, Zitao Liu, Wenbiao Ding, and Minlie Huang. Long text generation by modeling sentence-level and discourse-level coherence. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6379-6393, 2021.
[11] Tianxing He, Jingzhao Zhang, Zhiming Zhou, and James Glass. Exposure bias versus self- recovery: Are distortions really incremental for autoregressive text generation? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5087- 5102, 2021.
[12] Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines read and comprehend. Advances in neural information processing systems, 28, 2015.
[13] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2019.
[14] Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2018.
[15] Andrej
Karpathy and Li Fei-Fei. Deep visual-semantee alignments for generating image descriptions. In Proceedings the pages 3128-3137, 2015.
the IEEE conference on computer vision and pattern recognition,
[16] Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. Opennmt: Open- toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages 67-72, 2017.
[17] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to toeenceet--seuueee pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 7871-7880, 2020.
[18] Jiwei Li, Will Monroe, and Dan Jurafsky. A simple, fast diverse decoding algorithm for neural generation. arXiv preprint 161.
2016.
[19] Xiang Lin, Simeng Han, and Shafiq Joty. Straight to the gradient: Learning to use novel tokens for neural text generation. In International Conference on Machine Learning, pages 6642-6653. PMLR, 2021.
[20] Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP-IICNIIP), pages 3730-3740, 2019.
11

---

[21] Clara Meister, Gian Wiher, Tiago Pimentel, and Ryan Cotterell. On the probability paradox language generation. arXiv preprint arXiv:2203.172117 2022.
[22] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: wet::::099..440 2016.
[23] Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. Abstractive text summarization using sequence-to-sequene mms and beyond. In Yoav Goldberg and Stefan Riezler, editors, Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-12, 2016, pages 280-290 ACL, 2016.
[24] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
[25] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vitito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and Garnett, editors, Advances in Neural Information Processing Systems 32, Curran Associates, Inc., 2019.
pages
[26] Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations, 2018.
[27] Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34, 2021.
[28] Weizhen Qi, Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. Prophetnet: Predicting future n-gram for Findings of the Association for Computational Linguisties: EMNLP 2020, pages 2401-2410, 2020.
In
[29] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
[30] Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-7, 2021.
[31] Lin CY ROUGE. A package for automatic evaluation of summaries. In Proceedings of Workshop on Text Summarization of ACL, Spain, 2004.
[32] Abigail See, Peter J Liu, and Christopher D Manning, Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, 2017.
[33] Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. arXiv preprint 2rXiv:2202.06411 2022.
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[35] Sean Welleck, Ilia Kulikov, Jaedeok Kim, Richard Yuanzhe Pang, and Kyunghyun Cho. Consis- tency of recurrent language model with respect to incomplete decoding. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 2020.
12

---

[36] Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training, In International Conference on Learning Representations, 2019.
[37] Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie- gen: an enhanced multi-flow pre- pre-training and fine-tuning framework for natural language generation. In Proceedings of the Twenty-Ninth International Conference International Joint Conferences on Artificial Intelligence, pages 3997-4003, 2021.
[38] Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11228-11339 PMLR, 2020.
[39] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: open pre-trained transformer language models. CoRR, 0as/2205.01066 2022.
[40] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference Computer Vision (ICCV), December 2015.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (cither in the supplemental material or as a URL)? [Yes] See Section 4.1
The code is attached in supplemental materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)? [Yes] See reported results in Table 1 and
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 41
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes] See Appendix ??.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "szhBPP31hZ", "reviewer": "Reviewer_95eQ", "summary": "The paper investigates the repetition issue in natural text generation. The described loop issue was caused by the maximization-based decoding algorithm we use. Machine generated text has higher chance to be repetitive compared to human corpus. The paper starts with analysis on individual cases and then some more scalable experiments with a few standard metrics. It introduces a “self-reinforcement effect” where the model will be more likely to repeat if the similar context has shown before. The more times of previous occurrences, the more likely it will repeat. To tackle the problem, the paper proposes a simple and intuitive training objective, DITTO, to reduce the repetition. The paper conducts experiment on open ended text generation and text summarization on CNNDM. \n", "questions": "The equation 1 is designed to penalize “one-hop” similarity. Let’s say you have a sequence: The orange is blue. The orange is red. The orange is blue. The orange is red. [repetition] \nIn this case, every two sentences are different, but 2-sentence repetition is not a problem here. Is it possible to boost the equation so you can cover all the previous sentences?", "limitations": "The DITTO equation tackles limited scenario of repetition. Some data augmentation or contrastive learning techniques can be applied to make the training less “synthetic”. \n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper tackles a core challenge in NLG. The “loop” of the paper is complete and convincing. The proposed technique is inspired by experiments and analysis. The idea of the paper is generally clear and easy to follow. \nThe proposed technique is relatively simple to implement and use. \n\nIt compares with some prior work and shows its effectiveness. Human evaluation is also provided to validate the proposed technique. \n\nThe empirical results are good. The proposed method is competitive compared to prior work including unlikelihood training and StraightGradient.\n\nWeakness\n\nI have a major question regarding the function of DITTO (Equation 1).\nEquation 1: \n-math.log( 1 - abs(cur - alpha * prev))\nLet’s set alpha = 1 for now. \nIf the model is well-trained to avoid repetition, the current probability for a target token v P(x_{n,l} = v) is lower than the probability if shows before P(x_{n-1,l} = v), then this is an ideal scenario that the model did great. However, the equation actually penalizes this situation. \nHere is my experiment:\n(Good case) If cur=0.1, prev=0.5, alpha=1, the loss is 0.51.\n cur=0.1;prev=0.5;alpha=1\n-math.log( 1 - abs(cur - alpha * prev ))\n0.5108256237659907\n\n(Bad case) If cur=0.9, prev=0.5, alpha=1, the loss is still 0.51\ncur=0.9;prev=0.5;alpha=1\n-math.log( 1 - abs(cur - alpha * prev ))\n0.5108256237659907\n\nThis doesn’t make sense to me that the loss is the same for both cases. In the “good” case, the model decrease the chance of repetition so it should be rewarded rather than penalized. \n\nIf the alpha is set to be <1, it’s better than the case described when alpha=1, but still I think it’s unfair to penalize a model to not be repetitive. \ncur=0.1;prev=0.5;alpha=0.5\n-math.log( 1 - abs(cur - alpha * prev ))\n0.16251892949777494\ncur=0.9;prev=0.5;alpha=0.5\n-math.log( 1 - abs(cur - alpha * prev ))\n1.0498221244986778\n\n\nIf I understand the equation correctly, should it be max(_, 0) instead of abs(_)?\n\ncur=0.1;prev=0.5;alpha=1\n-math.log( 1 - max(cur - alpha * prev,0 ))\n-0.0\n\ncur=0.9;prev=0.5;alpha=1\n-math.log( 1 - max(cur - alpha * prev,0 ))\n0.5108256237659907\nTo conclude, the designed equation penalizes the model even the current pred has less “overlap” with previous pred. I am not sure if there is some misunderstanding of the equation. I apologize if there is, and I am open to discussion. Authors and reviewers are welcomed to comment on this. \n\n\nThe coverage of the experiment could be further improved. There are only two datasets used in this paper, Wiki103 for LM and CNNDM for summarization. It would be better if the paper can provide more evidence on more datasets. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IAquLGyuuF3", "reviewer": "Reviewer_tE1F", "summary": "This paper presents an empirical study showing the problem of repetitions in text generation (especially sentence repetitions), and proposed a training method called DITTO to address this problem where they add a training objective to penalize the probabilities of sentence level repetitions for the synthetically created examples (repeated sentences). Authors evaluated their method on different tasks and datasets, open domain text generation and summarization, and showed their method has much less repetition while still achieves competitive performance on other metrics. \n", "questions": "In sec 3:\nAuthors mentioned \"We find that, appending another sentence sampled from the training corpus randomly as the prefix to the pseudo repetitive data x can achieve better and more stable performance.\" \nCan you clarify this?  is this providing some context to the repetitive sentences? but if so, why do you need to randomly sample another sentence, rather than just the previous sentence? \n\nFig 4:  I’m a bit confused by the results for UL. \nEarlier tables showed it has much lower repetition rate than other methods, which seems to be inconsistent with the results presented here. \n\n\n", "limitations": "Authors didn't discuss this. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \nThe paper presents a good empirical analysis for the repetition problem in text generation.  Authors defined a few metrics, and evaluated on different datasets demonstrating the problem.  Though the community is aware of such problems, this is the first time I see such an analysis systematically showing the empirical results. \n\nBased on the repetition pattens authors identified (higher probabilities when repetitive sentences are included in the context), authors proposed a training method to penalize such repetition. The proposed training objective is well motivated. \n\n\nWeakness: \n\nFor the repetition problem, though the empirical analysis is good, I feel the authors are overclaiming slightly. They do show the sentence repetition happens, and models has a self-reinforcement effect. However, they say that’s the reason that sentence repetition occur (I.e., the model has seen the previous sentence in the context). I feel this is not exactly the explanation yet. The underlying cause of this phenomenon still needs additional analysis, probably more inside the neural networks. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2kas10y2IOs", "reviewer": "Reviewer_HDLP", "summary": "This work proposes a novel way of mitigating sentence-level repetition in autoregressive sequence generation: they form an artificial set of negative samples by repeating the same sentence from training corpus up to the max length of prediction. After that they use a loss function to penalize those instances from being produced during fine-tuning. They compared their method with multiple variants from previous work and concluded the superior quality of generated sequences. They have done both automatic and human evaluation for open-ended generation and automatic evaluation for directed generation.", "questions": "Here I will write both my comments and questions:\n\nLine 30: [30] is also training based method.\n\nLine 40-41: How does your analysis differ from one in Holtzman et al. work? I can see the introduction of several metrics to measure the rate of self-reinforcing loop, but other than that it doesn't look different. If you want to stress the point about analyzing sentence-level repeat, then it is a good place to do so.\n\nLine 52-56: When you say \"The cause ... may be that ...\", then the overall claim does not sound strong at all. If you do not have sufficient evidence (e.g. by showing how this effect does not hold on some other test sets without repeats or so, then I am not sure how this is different from all other existing observations about n-gram level repeat.\n\nLine 61, 63, 64: Please avoid using \"can\" or \"may\". if it is not always the case, then quantify how often this happens. After rewriting these sentences the overall reading will be improved.\n\nLine 63-65: Where is the justification for this statement? It sounds incomplete without any reasoning afterwards.\n\nLine 89: what does it mean to \"statistically investigate\"? Other wording will improve this sentence.\n\nLine 123: \"To our surprise\": why is it surprising? All previous work was showing that this is indeed a large problem. I agree that random token sentence repeat is more surprising, but authors don't say that.\n\nLines 125-132: I do not see the answer to the question \"Why sentence repetitions occur?\". I see observations, but no verified hypothesis of the reason behind it. For instance, Contrastive Generation framework paper [30] specifically introduce the hypothesis of very similar embeddings of repeating tokens which they further verified. I do not see any similar process here. Perhaps authors had some valid answer, but it is not presented here.\n\nLines 133-137: Where is the answer for the question? The observation of self-reinforcing loop on artificial corpus is not an answer. Moreover, there is previous work showing that LM has the ability to break such a loop: https://arxiv.org/pdf/1905.10617.pdf I would be happy to see if authors can elaborate more about this.\n\nLines 144-153: Due to the unclear claims in the section 2.2 the statements in \"Analyses\" paragraph do not sound strong.\n\nLine 167: why did you choose the loss function to look like this? It is interesting to see if there is some theoretical benefit of using such construction of log(1-|p1-p2|). For instance, the gradient of the loss may show some benefits. Alternatively you could have optimized the difference between log-probabilities: max(margin, logp(x_n) - logp(x_{n-1})) . Some justification around the loss function design would help readers to understand the motivation better.\n\nLines 190-193: At least for UL-seq it is not true that it penalize any repetitions. In fact it is specified by the length of ngram. In other words, if n=15, then it will only penalize those sentence-a-like ngrams. \n\nLines 256-258: DITTO is specifically designed to mitigate repeats up to max generation lengths, while other methods you compared to did not. In other words, other methods could have a chance to show better performance if they match the design of negative samples similarly to DITTO. And the same holds in opposite direction I believe: if DITTO would be trained for e.g. 150 tokens only, then it may fall in the same repetition loop again after generation that number of tokens. I think this is very important point to mention in this section.\n\nLines 305-207: As I said earlier, this can be a result of not 100% fair comparison between training and evaluation protocols. ", "limitations": "Relevant related work which is missing (sorry if it is there and I missed it):\n\nhttps://arxiv.org/pdf/1905.10617.pdf\n\nhttps://arxiv.org/abs/2007.14966\n\nhttps://www.semanticscholar.org/paper/High-probability-or-low-information-The-paradox-in-Meister-Wiher/49a7515b1c0250a12c0e6b6fa0a8b6d46fc42adf\n\nhttps://arxiv.org/abs/2002.02492", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Originality\n\n## Strengths\n\n* Authors proposed a novel way of viewing inter-sentence repetition which is a more global way compared to n-gram repetition (which can also be inter-sentence for large enough n-gram length)\n\n## Weaknesses\n\n* Many claims around idea justification are written in a vague way, which I will touch later in the questions section.\n\n# Quality\n\n## Strengths\n\n* Authors uploaded a well-done code base with experiments and instruction of how to run it. The code is based on fairseq which makes it not hard to use in the future work by other researchers.\n\n* Authors composed comparison with other methods which avoids this type of degeneracy in addition to the comparison with the baseline method.\n\n## Weaknesses\n\n* Related work section could be improved. For instance, there is a recent work (not related with myself in any way) about the ability of the LM to break the reinforcing loop: https://arxiv.org/pdf/1905.10617.pdf . In other words, there exists previous work directly related and opposite to observations and claims for this work. I want to stress that this is totally fine, but authors should consider discussing such results and providing some explanation of how the setup is different etc.\n\n* Experiment comparison is not very transparent. For instance, sequence-level UL models are trained to penalize continuations generated by up to 150 tokens. It was observed earlier that after such 150 tokens it may tend to repeat again (looks like this is exactly what happens in Fig.4). In other words, the length of generation influences the ability of the model to not repeat up to such extent. In DITTO the model is being penalized by the max length of generation. You may have done the experiment where DITTO would be trained on negative sentences of similar length to match the UL training protocol. If you are not going to do that, then at least clarifying this is necessary, because it addresses your claim that UL does not break the reinforcing loop. Because it may do so if you train UL model in similar fashion (with longer generated seqs).\n\n# Clarity\n\n## Strengths \n\n* The experimental protocol is clear.\n\n## Weaknesses\n\n* The section I was very curious about 2.2 is extremely vague and does not provide any answers (I will touch this later in the questions)\n\n# Significance\n\n## Strengths\n\n* I believe that this general method provides a significant contribution for future work beyond this specific use case: using an external set of negative samples which are easy to form and optimize. \n\n## Weaknesses\n\n* On the other hand from the the strong point #1, the loss function is not well justified (I touch this again in the questions) as in why such construction (with modulo difference) is better than eg the margin based version.\n\n* Even though Contrastive framework for text generation was discussed (it was mistakenly categorized as decoding based method while in fact it is training based too), it was not used for experimental comparison. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept