Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective
Authors: Huayang Li
Tian Lan
Zihao Fu
Deng Cai
Lemao Liu
Nigel Collier
Taro Watanabe
Yixuan Su
Venue: NeurIPS 2023 poster (2023)
Keywords: language modeling
text generation
natural language processing

Abstract: There are a number of diverging hypotheses about the neural text degeneration problem, i.e., generating repetitive and dull loops, which makes this problem both interesting and confusing. In this work, we aim to advance our understanding by presenting a straightforward and fundamental explanation from the data perspective. Our preliminary investigation reveals a strong correlation between the degeneration issue and the presence of repetitions in training data. Subsequent experiments also demonstrate that by selectively dropping out the attention to repetitive words in training data, degeneration can be significantly minimized. Furthermore, our empirical analysis illustrates that prior works addressing the degeneration issue from various standpoints, such as the high-inflow words, the likelihood objective, and the self-reinforcement phenomenon, can be interpreted by one simple explanation. That is, penalizing the repetitions in training data is a common and fundamental factor for their effectiveness. Moreover, our experiments reveal that penalizing the repetitions in training data remains critical even when considering larger model sizes and instruction tuning.

Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective
Huayang Li
Tian Lan
Zihao Fu
Deng Cai*
Lemao Liu
Nigel Collier
Taro Watanabe
Yixuan Su
Nara Institute of Science and Technology
Tencent AI Lab
University of Cambridge
Cohere
(li. huayanu 1h6, taro]@is.. narossis.naist..i jpp lantiangmf contiangmftby@gmai. tby@gmail. com
(jcykcai, redmoonlluj@teencent. (zf268, nhc30, ys484]0cam, ac.
Abstract
There are number of diverging hypotheses about the neural degeneration problem, i.., generating repetitive and dull loops, which makes this problem both interesting and confusing. this work, aim advance our understanding by presenting straightforward and fundamental explanation from the data per- spective. Our preliminary investigation reveals strong correlation between the degeneration issue and the presence of repetitions in training data. Subsequent experiments also demonstrate that by selectively dropping out the attention repetitive words in training data, degeneration can be significantly minimized. Furthermore, our empirical analysis illustrates that prior works addressing the degeneration issue from various standpoints, such as the high-inflow words, the likelihood objective, and the self-reinforcement phenomenon, can be interpreted by simple explanation. That is, penalizing the repetitions in training data is a common and fundamental factor for their effectiveness. Moreover, our experiments reveal that penalizing the repetitions in training data remains critical even when considering larger model sizes and instruction tuning. Our code available at https://gi thub. com/ gmf tey BeayyM//TBYyRRe-Drrpuuttt
Introduction
The emergence of neural language models (LM) has led to significant achievements in various text generation tasks, such as machine translation [3, 24, 31], summarization [22]. and open-ended text generation [17, 20]. However, in open open-ended text generation, neural LMs exhibit strikingly severe degeneration issue, producing unreasonably repetitive texts, particularly when employing maximum posteriori (MAP) decoding algorithms. As illustrated in Fig. 1, even a well-trained LM [20] may suffer from severe degeneration issue.
There have been numerous attempts to explain the phenomenon of neural text degeneration, with many attributing this problem flaws in the learning process. Fu al. [8] claim that high-inflow increase the probability of generating repetitions. A collection of studies [11, 15, 26, 34] argue that the likelihood objective is the primary factor, because has the problem of exposure bias and focuses more on next-token prediction rather than sequence generation. Meanwhile, both Chiang and Chen [4] and Xu al. [36] assert that the relforeinforeement mechanism can make neural LMs fall into the repetitive loops. Intriguingly, despite the divergence in these explanations, all
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

<: was not possible to know which grade of general an officer was by his collar insignia However, major generals and lieutenant generals wore two rows of nine buttons in groups of three down the of the overcoat, and brigadier generals wore two rows of cight buttons in groups of two.
It not possible know which grade of general an officer would be required to be given to a particular officer The Army was not aware of the Army's role in the conflict, and the Army was not aware of the Army's role in the conflict. The Army's role in the conflict was limited to the training of
Figure 1: Illustration of repetitions in human and generated text. The human text is from Wikitext- 103, and generated text is by greedy search using the GPT-2 model trained on Wikitext-103, The underlined text the prompt for generation. The blue words and red words indicate the repetitions in human text and machine-
machine-generated generated text, respectively.
corresponding methods proposed have been shown to effectively alleviate the text degeneration issue. This observation leads us to ponder: could there exist more fundamental factors that can explain the degeneration issue?
In this study, we strive to provide straightforward and fundamental interpretation for the degeneration problem from the data perspective. Our preliminary investigation reveals that repetitive words in the training data play a crucial role in the issue. It also worth noting that repetitions are not necessarily of low quality, because it is natural and common phenomenon in human writing [1, 12, 28]. Across five datasets with different domains, we observe strong correlation between repetition rates in training and generated text. This finding encourages us to further assess the impact of repetitive words in training data by randomly dropping out attention them during training. Employing repetition dropout, we discover that the repetition rate in generated text can be substantially reduced. Lastly, we reconcile many previous hypotheses with our single explanation, asserting that penalizing repetitions in training data is key factor to the success of alleviating the degeneration issue.
As large language models (LLMs) gain popularity, it appears that the degeneration issue has been somewhat solved. We investigate the impact of various factors associated with LLMs on degeneration, including increasing model size and training models using instruction-tuning data. Our experiments reveal that penalizing repetitions in training data continues to be crucial in the context of LLMs.
Our contributions are threefold:
We demonstrate that the proportion of repetitive words in training data has a significant influence on the degeneration issue. Inspired by this finding, we also propose method, namely, repetition dropout, to mitigate the degeneration.
We find that penalizing repetitions in data is a more fundamental factor for reducing repeti- tions, which also provides a unified explanation for various cxisting hypotheses, such as the attributions to high-inflow words, likelihood objectives, and self relfrreinfrreement phenomenon.
We investigate the influence of factors associated with large language models reducing repetitions, including model scale and instruction tuning.
2 Related Work
The degeneration (or repetition) issue in neural language models (LM), particularly in open-ended text generation, has garnered significant attention in recent years. Previous works have proposed disparate interpretations and solutions for this issue
Many researchers hypothesize that factors within the learning process contributes the degeneration issue. Fu et al. [8] assert that the high-inflow words in training data may increase the probability of generating repetitions, where the inflow of a word is defined as the probability sum of all its preceding words. Another factor identified is the likelihood objective. Welleck et al. [34] argue that likelihood objective has discrepancy with the MAP decoding and focuses more on next-token prediction rather than sequence generation. Thus, they proposed an unlikelihood objective to address the two flaws. Based on the same principle, Lin et al. [15] propose to scale the gradient of specified non-novel tokens, i.e., words in the prefix context at each time step, to alleviate the degeneration issue. Su et al. [26] and Jiang et [11] pointed out that the degeneration issue may caused by the high similarity between token representations. Thus, they leveraged contrastive learning to leam more distinct

---

representation for each token. In addition, both Xu et al. [36] and Chiang and Chen [4] argue that degeneration caused by the self-reinforcementt phenomenon and Xu al. [36] address this issue by penalizing the repetitions in pseudo repetitive data.
Apart from issues within the learning process, other explanations for degeneration have been proposed. Many researchers contend that decoding methods [7, 10, 16, 20, 37] is the primary factor. Holtzman al. [10] argue that word probabilities in buman-generated text exhibit high variance and randomness, while high-probability text produced by MAP decoding methods tends to be repetitive and dull. This observation explains why sampling-bas based decoding methods [7, 10, 16, 20] can substantially mitigate the degeneration issue. Riley and Chiang [21] find that tasks with lower constraints, or larger solution spaces, suffer from more severe degeneration issue. The model architecture [8, 32, 34] and size [14] may also contribute, but the two factors have not been quantitatively evaluated.
However, with many explanations, understanding the primary becomes increasingly challenging. our work, we strive to provide a fundamental explanation for the previous hypotheses in the learning process. While our work may not encompass all the existing explanations, we believe it lays a solid foundation for further understanding of the degeneration issue.
of the degeneration issue
3 Backkround
Language model (LM) aims to estimate the probability of a sentence in natural language according to the chain rule of probability:
P(x) ((oll))))) ... P(ZZ(TLL-1
= P(x1) II P(a((11--)
(1)
where <3,,,..,, is a sequence of words with length and æ is the previous words, i.e., the context, for predicting word Zz.-
Model Since attention- LM [20, 31] has became the backbone of many tasks, we will use GPT-2 model [20] as the main architecture for our empirical studies. The core of GPT-2 model is to use the attention mechanism to update the representation of words æ::
QK + M
Attention( Q, V) softmax
V.
B
where 3 is a scalar to control the scale of the attention score, Q, K, and V are the linear transformation of h1. E g..xd using matrices W W,. and W. E [x respectively. hIL the current hidden representation of words æ1:L. and d is the hidden size. M is masking matrix that makes sure only the information of h1s is accessible for h, at each time step the word x; can perceive the information of zj, then M [i]j] equals to 0, otherwise -00
Training Generally, neural LMs are trained by optimizing the likelihood objective:
C -
log P(x; ((1,((111111))
mED i=2
log (oftmax(((H),))))))))
(3)
where h E Rdd is the hidden vector of æ;-1 output by the last layer of neural LM, e.g., the GPT-2 model [20]. The [z:] is defined as taking the probability regarding to x; in the distribution got from softmax. The o(-) is a linear layer that transforms the h to logits.
'GPT-2 also incorporates essential sub- layers such as residual connections and layer normalization. While the model employs multi-head attention mechanism, we have omitted details for brevity. For a comprehensive explanation of these sub-layers. refer to Vaswani al. [31] and Radford al. [20].

---

80
Rep-2 of Gen. Text (%)
Rep-2 Gen. Text (%)
60
60
40
40
Open Wiki-103
20
20
FreeLaw
PubMed ArXiv
0
4.5
13.5
18
0
17.5
35
52.5
70
Rep-2 of Human Text (%)
#Params. of LM (Billion) (b)
(a)
Figure 2: (a) Relationship between rep-2 scores of human and generated text. (b) Relationship between the scale of an LM and the rep-2 score of generated text. Note that results with the same symbol in Fig. 2(a) are from models trained on different shards of the corresponding dataset. The rep-2 score is defined in Eq. (4). We use the GPT-2 and OPT LMs for Fig. 2(a) and 2(b), respectively, and use greedy search as the decoding method.
Inference Since neural LMs are trained to maximize the likelihood objective, one intuitive practice for inference is to use the MAP decoding method, e-g., greedy search beam search. However, in open-ended text generation, this tactic will cause an extremely severe degeneration issue on vanilla neural LMs trained by likelihood objective [10].
Evaluation
The main focus of this empirical study is to investigate the reason for the repetition issue. Therefore, the rep-n is an important evaluation metric in our work, following previous works [8, 25, 27, 34]:
rep-n 1.0 UniqueNgrams((, r2)
(4)
T. where n is the length of n-gram, and UniqueNgganii a function find all unique n-grams in sentence æ. For the corpus-level evaluation, we report the averaged rep-n scores of instances in the dataset. To ensure that our results in main experiments results of rep-w and rep-r in previous works [8, 34]. The rep-w = + Et= {ar, E 11t=w-1=-1}- which measures the word-level repetition in prefix window with length w. In our experiments, we set w to 16. following Fu et al. [8]. The rep-r = + (il(,, = A r++1 x.+1. / i) (æ; = I -1 -4-1, 3k H. It is for the portion of repetitive snippets measured by sentence length.
biased rep-n, we also report the
In addition the measurement of repetition, we also consider the perplexity (PPL) on the real data, which demonstrates the performance in language modeling [2]. Although LMs with lower PPL may not consistently lead better generation results, able reflect trivial solutions for the degeneration issue, e.g., random or over-fitting models.
4 Preliminary Study: Rethinking Neural Text Degeneration from Data Perspective
The propensity of neural LMs with impressive performance to fall into naive repetitive loops is puzzling. Although numerous hypotheses have been proposed, many of them approach this issue from divergent aspects, and some are not intuitive for understanding. According to Ockham's 's Razor, a simpler explanation is often preferable. Therefore, in our preliminary study, we start by evaluating one elementary factor for most AI systems, the training data. Concretely, repetition is natural and common phenomenon human languages for various reasons [1, 12, 28]. It is intriguing to investigate whether there are connections between valid repetitions in natural language and incorrect repetitions in generated language. Moreover, it is important to note that data containing repetitions not necessarily of low quality, shown in Fig. 1.

---

Model
Rep-2 of FT data(%)
Rep-2(%)
Rep-3(%)
Rep-4(%)
LLAMA2 W/O FT
47.79
41.97
38.52 38.52
FT LIAMM2 ALPACA
5.54
15.08
10.91
8.93
FT LLAMA2 ALPACA + WT-103 50K
9.67
41.63
35.64
32.29
FT LLAMA2 ON WT-103 50K
10.31
54.10
49.77
36.80
Table I: Results of LLAMA2-7B on instruction-tuning data. The "FT" means fine-tuning. The
column "Rep-2 of FT Data" indicates the rep-2 score of the training data. The rest Rep-n scores are evaluated on the generated text. The ALPACA is the Instruction-tuning dataset used in [29], "WT-103 50K" is the instruction-tuniin
dataset we constructed based on Wikitext-103 (Appx. A.2), and "ALPACA+ + WT- 50K" is the mixture of both.
Setup To assess the correlation between repetitions in generated text and those in human text, we propose to train GPT-2 models [20] on data with varying rep-2 scores and then evaluate the rep-2 scores of the text generated by the corresponding models. Specifically, we sorted the training instances in each dataset D based their rep-2 scores. We then divided the sorted training data into six shards, each containing an equal number of words but a varying percentage of repetitions. and trained GPT-2 model on each shard. Notably, we will use the full test set of dataset 'D to evaluate the models trained on different shards of D. More implementation details are in Appx. A.
Our preliminary study investigates five datasets across various domains. Wikitext- 103 is a widely used dataset for language modeling [2. 5, 13] and open-ended generation [8. 34J. We adopt the standard split for training, validation, and test sets. The remaining four datasets, Open WebText2, FreeLaw, PubMed, and ArXiv, are part of the Pile dataset [9]. To ensure consistent analysis, we sample equivalent number of words as the Wikitext-100 103 dataset from each of the four Pile datasets. For the validation and test sets, we randomly sample 2,000 sentences from each of the four Pile datasets. The training, validation, and test sets non-overlapping across all five datasets.
Findings Fig. 2(a) demonstrates a strong correlation between the rep-2 scores of human and generated text each dataset, indicating that the degeneration issue becomes more severe as the percentage of repetitions in training data increases. However, due to varying data distributions across datasets from different domains, the rep-2 score of generated text may vary even when trained on data with the same rep-2 score. Another intriguing observation is that neural LM will amplify the repetition in data by more than 10 times, similar the bias amplification found in Zhao et al. [39].
Investigations Related to Large Language Models As e.g., ChatGPT [17] and LLAMA gain popularity, it appears that the degeneration issue has been somewhat solved. In this section, we will investigate the impact of various factors associated with LLMs on degeneration, including increasing model size and training models with iestruction-tuingg data.
Many amazing model abilities emerge when scaling up the model and data size [33]. An interesting question is, whether the degeneration issue will be solved by simply scaling up? We use set of OPT models [38] with different model sizes to investigate this question. As shown in Fig. 2(b), the 2 score of generated text sharply drops before increasing the model size 6.7 billion parameters, indicating that increasing the model size does alleviate the repetition issue to some extent. However, the gains achieved by increasing the model size diminish over time. The ОРT-66B model still generates text with high rep-2 score. This observation shows that increasing the model size is not an efficient way to alleviate the degeneration.
Degeneration in certain instruction-tuneet LLMs, such as ChatGPT [17], is relatively rare, leading to the hypothesis that the instruction- eustruction-tuning -uning phase, which trains LLMs on instruction-response pairs, could alleviate this issue. This conjecture implies that the utilization of instructioo-tuningg data vital for mitigating degeneration. To explore this, we fine-tune the LLAMA2 model [30] using three instruction- tuning datasets: ALPACA [29], ALPACA + WT-103 103 50K, and WT-103 50K, as demonstrated in Table The r scores for these datasets are 5.54, 9.67, and 10.31, respectively. Our experiments reveal that LLAMA2 fine-tuned on ALPACA exhibits less repetitions, while training on WT-103 50K and ALPACA + WT-103 50K still displays significant degeneration. This finding is consistent with our prior observations, where repetition issues strongly correlate with the presence of repetitions in the training data. It suggests that the low repetition rate in instruction -tuning data may contribute to the decreased degeneration.
5

---

These investigations reveal that our primary finding, namely, the degeneration issue has strong correlation with repetitive patterns in training data, remains vital the context of LLMs. We anticipate that this finding will offer valuable insights for understanding the degeneration issue and contribute the development of more effective large language models.
Method
According to our observation the preliminary study, we hypothesize that learning the patterns of good repetitions in natural language is crucial factor of the degeneration issue. To better evaluate our hypothesis, we propose simple method to control the percentage of repetitions that model can perceive during training, namely, repetition dropout. To this end, we can directly measure the impact of the repetition in natural language by training on the full dataset.
5.1 Repetition Dropout
Inspired by dropout [23], which prevents the Algorithm 1 Repetition Dropout, Python-like model from over-fitting on specific combina- tions of parameters, we propose the repetition dropout to avoid the model over-relying on repet- itive words in the training phase. Since atten- tion is the core component of Transformer based LMS to perceive other context words, we apply our repetition dropout to the attention sub-layer', which defined Eq. (2). More concretely, method replaces the triangular matrix M in Eq. (2) by M':
x: ListLint], input sequence float, repetition dropout rate int, length ngram
def find_ find_ngranssx, n):
# find start and end index of each n-gran ngran_dict t dict()
for in range(n-1, len(x)):
ngram tuple(x[i-m-1:111]
ngran in ngram_dict:
i+1))
12
else:
agran_dict(ngrnn [(i-n+1, i+1)]
14
return ngran_dict
M M Mrep:
(5)
16
def gen_nask_rep(x, P, n):
where Mrep is the masking matrix for repetition
mask_rep Find_ngrams(s n)
len(x)
18
ngran_dict for idx in agram_dict.items.
dropout. The algorithm for generating the mask- ing vector for each input sentence in Algorithm 1. In line 5-14 of this algorithm, we define a function find_ngrams to collect n-grams in The function sen_mask_res in line 16-26 to randomly drop out those repeti-
19
shown
if len(idx)
21
# mask repetetive n-grams according dropout rate p
23
if randon. formee 1) < p:
24
for rask inn
idx:
26
return mask_rep
tive n-grams according pre-specified repetition
dropout rate p. which is between [0, 1]. A higher repetition dropout rate indicates that more repetitive n-grams
accessible for the model. We concatenate the masking vectors for sentences in batch as the Myep- To avoid the model over-fitting on a specific Myep. we generate Mrep for each layer of the LM independently. It should be noted that repetition dropout only employed in the training phase, analogous to the conventional dropout method [23].
In this method, compel the model to predict subsequent words without depending repetitions present in training data. Repetition dropout can be viewed as a means to regulate the quantity of repetitions within the training dataset. If learning repetition dropout will substantially mitigate the degeneration issue. We want to emphasize that this approach is designed more effectively assess the significance of repetitions in training data.
repetitions in training data is vital aspect,
Experiments
Setup To evaluate the effect of repetition dropout, conduct empirical studies the Wikitext- 103, WebTeee,, and Freelavw datasets, which have been introduced in section 4. Compared with section 4, train the GPT-2 model the complete datasets rather than splitting them into smaller shards. Please refer to Appx. A for more training details about the GPT-2 model".
2 Notice that the repetition dropout not restricted attention mechanism. can also easily extended the hidden representation repetitive words. Phttps://githuun http:///ggghuuu..coo///gggg////e tras tronsrorerrss

---

Model
(cp-2(%)
Rep-3(9%
Re)
Rep-w (%)
Rep-r (%)
PPL
Wikitext 103
HI-RE
41.91
33.82
28.35
38.60
66.22
SCALEGRAD
12.49
6,85
4,44
18.31
28.98
24.72
UL
36.77
28.22
22.88
39.13
61.89
21.93
MLEE
47.05
38.46
32,64
46.42
72.39
21.98
+RAND-DROPOUT
38.33
28.84
22.66
37.76
66.19
23.50
PRE-ODROPOUT
9.78
4.34
2.14
22.56
25.45
28.26
HUMAN
3.36
0.84
0.28
10.64
3.82
FreeLaw
MLE
51.74
46.19
42.22
39.22
73.06
16.13
38.82
32.19
27.78
31.31
61.30
18.85
-REP-DROPOUT
10.15
5.60
3.49
17.55
23.21
20.68
HUMAN
2.77
0.89
0.50
10.61
8.10
Open WebTex22
MLE
73.96
70.61
67.91
67.28
88.27
80.37
FRAN--RROOOS
63.43
57.16
52.30
58.92
82.76
91.75
-REP-DROPOUT 25.24
16.14
11.10
34.73
49.80
107.02
HUMAN
4.53
1.39
0.61
12.41
13.09
Table 2: Performances of language models three datasets. Both RAND-DROPOUT and REP- DROPOUT use dropout rate 0.6. Rep-n is defined in Eq. (4). The decoding method for all models is greedy search. PPL the perplexity score on the real test data. Note that we do not report the PPL for HI-RE, because its vocabulary is different from other baselines.
The main baseline method is MLE, which indicates the vanilla GPT-2 model trained by likelihood objective, as defined in Eq. (3). The + REP- NEP-DROPOUT the proposed method, i.c. repetition dropout. We also have several baseline methods. The first one is the + LAND-DROPoR dropout on random tokens instead of repetitive tokens. Note that the number of random tokens selected for dropout for this baseline is constrained to the same the repetitive tokens. We also compare with three previous works that also conducted experiments on Wikitext- 103: re-encoding of high-inflow tokens (HI-RE) [8], training with scaled gradient (SCALEGRAD) [15], and unlikelihood
which applies
objective at token level (UL) [34]. During inference we use greedy search as the decoding method to generate 128 tokens, using the first 32 tokens of each line in the test set as the prompt. More details about those baseline methods are also shown in the Appx.
6.1 Main Results
O Rep-2
PPL
As shown in Tab. 2, we evaluate the baseline methods and our method on three datasets. On the Wikitext-109 dataset, both the MLE and DROPOUT methods suffer from
50
30
Rep-2 in Gen. Text (%)
MLE the severe degeneration issue. More than 33%
30
18
percent 4-grams in the generated sentences of the two baseline methods are repetitive, which different from the patterns in natural language. Nevertheless, MIEE + REP-DROPOUT can sig- nificantly reduce the repetition issue in gener- ated sentences. This observation indicates that repetition in the training data is crucial for the degeneration issue. Compared with methods in previous works in Tab. i.e., HI-RE, SCALE- GRAD, and UL, the REP- DROPOUT also show better performance on alleviating the degener- ation issue. The results FreeLaa and Wikitext- 103. We also conduct quantitative and qualitative experiments evaluate REP-DROPOUT on larger LMs, e.g., GPT-XL[20], which are shown in Appx. B and C, respectively.
20
12
10
0.0
0.4
0.6
Repetition Dropout Rate
Figure 3: Rep-2 score and perplexity of MLE REP-DROPOUT on the test set of Wikitext- 103. The brown dash line is the human-level rep-2 score.
datasets are consistent with those on
7

---

We also evaluate the effect of MLE REP- REP-DROPOUT with different dropout rates for the repetitions in Wikitext- 103, which can be regarded as controlling the percentage of repetitions shown in Fig. 3, increasing the dropout rate of our method will reduce the number of repetitions in generated sentences continuously. Moreover, we observe clear trade-off between the rep-2 score of generated sentences and the perplexity on the real data in test set. This suggests that learning on the repetitions in training data can be beneficial for language modeling. Consistent results are also shown on FreeLaw and OpenWebText datasets.
dataset. As
6.2 Relation to Previous Hypotheses
Previous research has proposed various hypotheses and approaches to understanding and solving the problem of degeneration in neural text generation. However, we argue that many of these proposals can be explained by simple explanation. That is, penalizing repetitions in data is common and fundamental factor for their success.
The core idea of many previous works is to penalize specific set of data, c.g., all the prefix words F1.t-1 at timestep t, to alleviate the degeneration. However, as shown in Fig. 4, many of them have implicit connections with the repetitions in data. For example, the set of high-inflow words (<)) penalized Fu et al. [8] has a noticeable interaction with that of repetitive words (O), and the set of prefix words (m) penalized in [34] and [15] is the super-set of the repetitive words (O). In Xu et al.
[36], they directly penalize the repetitions in pseudo repetitive data (O). In this section, will show that repetitive words play an important role in previous works.
High-Inflow Words Some researchers [8] at- tribute the degeneration issue to high-inflow words, whose probability sum of all the poten- tial preceding words is higher than threshold. Thus, they propose that merging high-inflow word pairs can alleviate the problem (HI-RE). However, we find that 26% of high-inflow word pairs are repetitive in each sentence of Wikitext- 103, and merging these pairs can significantly reduce the rep-2 score of real data. Therefore, we argue that merging actually an alternative way of reducing repe- titions in training data.
Prefix Words
Pscudo Rep. Sents
High-Inflow Words
Rep. Words
Figure 4: Relationship between the penalized data in previous works. We use 1, and O to repre- sent the sets of high-inflow words [8], prefix words [11. 15, 26, 34], and pseudo repetitive sentences [36], respectively. We also demonstrate the set of repetitive words in real data by O.
inflow word pairs
We evaluate the argument by controlling single variable, the type of high-inflow words to be merged, in line 4-6 of Tab. 3. The vanilla HI-RE method (Line 4) merges all the high-inflow words (O), which takes 31.1% of the total training words. We find that the method (Line 5), which only merges repetitive high-inflow pairs, i.e., the intersection between high-inflow words and repetitive words (c n O), achieves performance comparable to the original HI-RE method (Line 4). Note that the method in line only merges 8.1% of the total training words. which much less than the vanilla method. In contrast, the HI-RE method that merges random high-inflow pairs (Line 6), which has the same number as repetitive high-inflow words (<> n D)) cannot alleviate the degeneration. This suggests that penalizing repetitions in data is critical in the success of Fu et al. [8].
Likelihood Objective Many researchers [11, 15, 26, 34] think that the likelihood objective is the main factor for the degeneration issuc. All of those works share the same principle that words in the prefix context (=) cause the degeneration issue. Therefore, Welleck et al. [34] and Lin et al. [15] propose to reduce the probabilities of repetitions appearing in x1:t-1 at time step t. Su et al. [26] and Jiang et al. [11] leverage the contrastive learning to ensure that the hidden representation at time step t distinctive to those of T1:1-1-
However, the attribution to the likelihood objective merely serves as superficial explanation, and the core of this issue lies in the fact that the model inevitably learns repetitive behavior when conducting maximum likelihood estimation on repetitive data. As shown in section 6, the GPT-2 model with simple repetition dropout, which also optimized by likelihood objective, achieves an extremely low rep-4 score generated text. This indicates that likelihood objective might not be the most
8

---

Model
Penal, Scope
Word Percent (%)
Rep-2(%)
Rep-3(%%
Rep-4(%)
PPL
MLE
N/A
0.0
47.05
38.46
32.64
21.98
PREP-DROPOUT
Subset of
11.4
9.78
4.34
2.14
28.26
DITTO
25.0
44,24
34.74
28.34
4 HI-RE
3T.1
41.91
33.82
28.35
HI-RE
ono
8.1
43.62
33.67
27.12
HI-RE
Subset of
8.1
52.41
43.72
37.56
SCALEGRAD
TO0.0
12.49
6.35
4.44
24.72
SCALEGRAD
19.0
17.53
10.38
6.94
23.33
SCALEGRAD
Suhset of
19.0
22.97
15.22
10.94
23.18
Table 3: Impact of penalizing repetitions in different kinds of data on Wikitext-103. The meanings
of the symbols O O. are illustrated in Fig. The "Subset of SHapE" means subset randomly sampled from SHAPE. The word percent is # Penalizzed Words # Words in Data
important factor in the degeneration issue. Nevertheless, all methods that penalize tokens in the prefix context ( break the reliance repetitions (O). Thus, we hypothesize that preventing the model from learning on repetitions a key factor in their success.
To evaluate the impact of repetitions on these methods, we conduct experiments following the same principle used to analyze high-inflow words. We choose the SCALEGRAD [15] method our baseline, which penalizes non-nover tokens by scaling the gradient [15]. The non-novel tokens are the entire prefix context #1 () at time step I, i.e., 100% of the training words, for the vanilla SCALEGRAR method. We also propose two variants of SCALEGRADD the first uses the repetitive words within prefix (LNC) (D) as the non-novel tokens, while the second randomly samples subset, which has the same number of words as the first one (L O), from the prefix data (Subset of [). The two methods only penalize 19.0% the total training words. As shown in Tab. 3, the SCALEGRAD method repetitive words (Line 8) achieves performance close the standard SCALEGRAD method (Line 7) in terms of the rep-n metric. In contrast, the SCALEGRAD method on random subset (Line 9) is not effective in alleviating the degeneration issue the the other two methods. Other works [11, 26, 34] that attribute the degeneration issue to likelihood objective also penalize tokens in prefix the SCALEGRAD [15], but with different techniques. Thus, think that penalizing repetitions in data is also a crucial factor for the success of these methods.
Seff-reinforcement Phenomenon Both Xu al. [36] and Chiang and Chen [4] find that degenera- tion is always accompanied by the self-reinforeement phenomenon, i.e., the probability of a predicted word becomes higher when it is repeated more times. Thus, they hypothesize that degeneration is (partially) caused by self reiffrriffore To mitigate this issue, [36] proposed a data- aatta-aggmentttion method, namely DITTO (Line 3 of Tab. 3). which constructs pseudo data by repeating training sentence multiple times and penalizes the probabilities of repetitive tokens in the pseudo data (O).
We argue that, as the degeneration issue, the senf-reinforcement is also a by-product when neural LMs learning on repetitive patterns in real data. First, we observe similar self-reinforcement phenomenon on the repetitive words in real data. For instance, the probabilities of the second appearances of the theme-related words generally higher than their first appearances, shown in Fig. 5(b). Second, we find that the model trained by repetition dropout can break the reif-reiinorremment loop, shown by the examples in Appx. C. Although the text generated by GPT-2 REP- REP-DROPOUT on Wikitext- 103 may contain a few inappropriate n-gram repetitions, it will not fall into the infinite repetition loop that frequently appear in the text generated by the vanilla GPT-2.
6.3 Why LMs Learn the Repetition Patterns?
In the last empirical study, we make an attempt to investigate the reasons behind LMs learning repetition patterns, specifically the role these repetitions play in neural LMs. To address this inquiry, we examine 300 randomly selected instances, each featuring repetitive bi-grams within a 256-word sentence. These cases are broadly categorized into three groups, as per Altmann and Köhler [1] and Tannen [28]:
Repetitions for grammatical purposes, such as determiners, conjunctions, etc. * Theme: Repetitions closely associated with the subject matter of the text.
Grammar:
Limited inventory: Repetitions resulting from a language's restricted means of expressing a particular concept, leading to high- nigh-frequency occurrences, c.g., the phrase "pair of" in English.

---

50
0.5
Percentage (%6
AVG. Probubility
0.4
w/ mask
30
0.3
20
0.2
10
0.1
0
Grammar
Theme
Lid. Inventory
Grammar
Theme
Tad. Inventory
Figure 5: (a) Percentages of three types of repetitions in Wikitext-103. (b) Probabilities of three types of words with and without masking their repetitions in context using the vanilla GPT-2 model.
We assign each case to the earliest category it satisfies if it meets the criteria for multiple categories. Detailed guidelines for human evaluators to classify repetitive words can found in Appx. D. As demonstrated in Fig. 5(a), almost 50% of the repetitions in Wikitext- 103 fulfill grammatical functions, while both theme-related and inventory-related repetitions constitute around 25% each.
To understand the impact of different types of repetitions on a vanilla neural LM, we measure the probability change of prediction when masking the information of its repetitions in the context through attention mechanism. For example, given the human repetitions in Fig. 1, we will determine how the probability of the last appearance of "generals wore..." changes when masking its first appearance of "generals wore..." in the context.
As shown in Fig. 5(b), masking the theme-related repetitions will cause noticeable drop of the prediction probabilities. However, the prediction grammatical and inventory-relate repetitions not affected by masking their previous appearances in context. This observation indicates that neural LMs spend its effort on optimizing the prediction accuracy of those theme-related words by implicitly repeating words in context.
7 Conclusion & Limitations
In this work, find that the repetition in training data is fundamental factor for the degeneration (or repetition) problem. First, training data is an integral part of most Al systems, and our preliminary study demonstrates strong correlation between the repetitions in training data and the degeneration issue. Second, we find that simply dropping out the attention to repetitions in training data can significantly reduce the degeneration issue, which is more effective than other baselines. Finally, we conduct extensive empirical analyses to demonstrate that penalizing repetitions in data is the key success factor for many previous works, such as those attribute degeneration issue to high-inflow words, the likelihood objective, and the self-reinforcement mechanism. Experiments also show that our findings are critical even the context of large language models. We hope that the viewpoint we provide to understanding the degeneration can inspire more principled research in the future.
Our work also has some limitations. First, despite the excellent performance on redueing repetitions, the repetition dropout method may hurt the perplexity of the language model. Second, we examine our work mostly on standard benchmark of the language generation task, following previous works [8, 10, 26, 34]. It also deserves to extend our work to large-scale data and model.
8
Acknowledgement
The authors would like express their sincere gratitude to the three anonymous reviewers and the meta reviewer for their insightful comments and constructive feedback. Furthermore, this work was partial supported by the JSPS KAKENHI Grant, under the number 23KJ
10

---

References
[1] Gabriel Altmann and Reinhard Köhler. 2015. Forms and degrees of repetition in texts: detection and analysis, volume 68. Walter de Gruyter GmbH & Co KG.
[2] Alexei Baevski and Michael Auli. 2019. Adaptive input representations for neural language modeling. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
[3] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representa- tions, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
[4] Ting-Rut Baig-Rui Chiang and Yun-Nung Chen. 2021. Relating neural text degeneration exposure bias. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 228-239, Punta Cana, Dominican Republic. Association for Computational Linguistics.
[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XI: Attentive language models beyond fixed-length context. In Pro- ceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, Florence, Italy, Association for Computational Linguisties.
[6] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zeulemoyer. 2023. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305. 14314.
[7] Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
[8] Zihao Fu, Wai Lam, Anthony Man-Cho So, and Bei Shi. 2021. A theoretical analysis of the repetition problem in text generation. In Thirty-Fifth AAA/ Conference on Artificial Intelligence.
[9] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00022
[10] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, OpenReview.net.
[11] Shaojie Jiang. Ruqing Zhang, Svitlana Vakulenko, and Maarten de Rijke. 2022. A sim- ple contrastive learning objective for alleviating neural text degeneration. arXiv preprint arXiv:2205.02517.
[12] Toshiki Kawamoto, Hidetaka Kamigaito, Kotaro Funakoshi, and Manabu Okumura. 2022. Generating repetitions with appropriate repeated words. In P Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 852-859, Seattle, United States. Association for Computational Linguistics.
[13] Huayang Li, Deng Cai, Jin Xu, and Taro Watanabe. 2022. Residual learning of neural text generation with n-gram language model. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1523-1533, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
[14] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2022. Contrastive decoding: Open-ended text generation as optimization. CoRR, abs/2210.15097.
[15] Xiang Lin, Simeng Han, and Shafiq Joty. 2021. Straight to the gradient: Learning to use novel tokens for neural text generation. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages
6642-6653. PMLR.
11

---

[16] Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. 2023. Locally Typical Sampling. Transactions of the Association for Computational Linguistics, 11:102-121.
[17] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744
[18] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary VeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-PPeformance Deep Learning Library. In Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc.
[19] Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Measuring the gap between neural text and human text using divergence frontiers. In Advances in Neural Information Processing Systems, volume 34, pages 4816 4828. Curran Associates, Inc.
[20] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
[21] Darcey Riley and David Chiang. 2022. A continuum of generation tasks for investigating length bias and degenerate repetition. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 426 440, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
[22] Abigail See, Peter Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083.
[23] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 19999
[24] Yixuan Su, Deng Cai, Yan Wang, David Vandyke, Simon Baker, Piji Li, and Nigel Collier. 2021. Non-autoreeeessive text generation with pre-trained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 234-243, Online. Association for Computational Linguistics.
[25] Yixuan Su and Nigel Collier. 2023. Contrastive search is what you need for neural text generation. Transactions on Machine Learning Research.
[26] Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. In Advances in Neural Information Processing
Systems.
[27] Yixuan Su and Jialu Xu. 2022. An empirical study contrastive search and contrastive decoding for open-ended text generation. arXiv preprint arXiv:2211.10797
[28] Deborah Tannen. 1987. Repetition in conversation: Toward a poetics of talk. Language, pages 574-605.
[29] Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Perey Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction- following model. Stanford Center for Research on Foundation Models. hipp://erfm. slanford. cdu/2023/03/11aalpaar html, 366)7.
[30] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. 2023. Ilama Open foundation and fine-tuned chat models. arXiv preprint arX.22230..09288
12

---

[31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
[32] Jesse Vig. 2018. Deconstructing bert: Distilling patterns from 100 million parameters. Medium, December.
[33] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837.
[34] Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- veew.net.
[35] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jemnite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-Arr Natural Language Processing. pages 38 45. Association for Computational Linguistics.
[36] Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. 2022. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation. In Advances in Neural Information Processing Systems.
[37] Haoran Yang, Deng Cai, Huayang Li, Wei Bi, Wai Lam, and Shuming Shi. 2023. A frustratingly simple decoding method for neural text generation. arXiv preprint 2rXiv:2305.12673
[38] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0r1 220.00080.00
[39] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai- Kei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pages 2979-2989, Copenhagen, Denmark. Association for Computational Linguistics.
13

---

A Implementation Details
A.1 Preliminary Study
The basic GPT-2 model' is trained from scratch each corpus, which has 12 transformer blocks and 12 attention heads with 768 hidden dimensions. The Huggingface transformers [35] and Pytorch toolkit [18] are used to train the GPT-2 model in the distributed manner on A100 GPU server. The hyper-parame pper-pparaeters during training are shown in Tab.
Hyper-parameter
Value
Optimization steps
100K
Test interval
10K
Dropout rate
0.1
Grad clipping
1.0
Learning rate
5e
Batch size
128
Maximum sequence length
256
Warmup steps
10K
Learning scheduler
I Linear decay
Random seed
Number GPUs
4
Learning objective
Cross-Entropy Loss
Table 4: The hyper papere-parmeters during GPT-2 training procedure.
A.1.1
Our Method
Most of the ssper-parameters for our proposed method are the same that in Tab. 4 for better variable controlling. The specific haper-parameters for our proposed method are the length of repetitive n-gram and its repetition dropout rate p, which are set as 2 and 0.6, respectively.
A.1.2 Baselines
In this subsection, the specific hyper-parameter for three baselines are described, and most of the hyper-parameters are the same as that in Tab.
Hyper-pand paper-parammter
Value
Re-encoding of High-inflow Tokens (HI-RE)
Re-encoding
0.03
Scaled Gradient (SCALEGRAD)
Scale grade y
0.2
Token-lever Unlikelihood Training (UL)
Rank alpha
1.0
Table 5: The hyper-parameten of three baselines in this paper.
A.2 Instruction Tuning
To evaluate the effect of instruction tuning, conduct experiments three datasets:
1. ALPACA: The instruction-tuning dataset used by Alpaca [29].
2. WT-113 50K: We randomly sample 50k non-title sentences from Wikitext- 103 and convert them to the instruction-tuning format, following [29].
3. ALPACA WT-103 50K: The mixture of both.
Model details can be found at https://huggingffcc.coo
14

---

We use the QLoRA [6] to fine-tune the LLAMA2-7B model [30], due to the limited computational resources. The decoding method for generating text is greedy search. We use the test set of Wikitext- 103, which is also converted to the instruction-tuning format, to evaluate the performance of different models. Below is the template used for converting Wikitext-100 to the instruction-tuning format:
"instruction": "Please continue writing based on the following prefix. The text be continued should be relevant, fluent and informative." " "input": PREFIX, prefix of a sentence "output": COMPLETION the completion of the prefix
B Additional Experiments
In addition to the repetition measurements (Tab. 2), we also evaluate our method on additional metrics, e.g., MAUVE [19]. As shown in Tab. 6, most of the performances of method and baseline methods are consistent with the observation in Tab. 2.
MLE((%
HI-RE
SCALEGRAD
UL.
REP-DROPOUT
MAUVE 49.70
35.83
52.80
50.06
52.20
Table Evaluation on MAUVE. The settings are the same as those Tab. 2.
Model
Rep-2(%)
Rep-3(%)
Rep-4(%)
Rep-w (%)
Rep-r (%)
MLE
54.26
49.
45.84
66.10 10
37.72
+ REP-DROPOUT
11.36
5.80
3.67
24.39
18.19
Table 7: Experiments on GPT-XL (1.5 Billion parameters)
In addition to GPT-2 model, we also conducted experiments on model with larger size. We directly apply repetition dropout method to the GPT-XL model, which has 1.5 billion parameters. Since is difficult to train GPT-XL model from scratch, we fine-tuned it on Wikitext-103 for epochs. Results in Tab. 7 demonstrate that our method can also alleviate the degeneration of LLMs after fine-tuning on larger model.
C Qualitative Analysis
In Fig. 6, we show some generated results of the GPT-XL model and GPT-XL-REP-DROOOU The training and inference settings the two models are described in Appx. We can observe that the repetitions in generated text were reduced significantly after employing the REP-DROPOUT method. In addition, the GPT-XL trained by MLE easily falls into sentence-level repetition, because the self-reinforeement mechanism [36]. Nevertheless, the GPT-XL+ OPT-XL+REP-A REP-DROPOUT does not suffer from this issue.
"ntts:://ugggnngffcccoo//
15

---

Input Case 1:
Traditional Chinese literary criticism emphasized the life of the author when interpreting work, practice which Burton Watson attributes to the close links that traditional Chinese thought posits
GPT-XL:
between the author and the reader", ". The author's life was also central theme in the works of the early modernist poets, such as the poets of the Ming dynasty, who were influenced by the writings of the early modernist poets. The carly modernist pocts were also influenced by the writings of the carly modernist pocts, such as the pocts of the Song dynasty, who were influeneed by the writings of the carly modernist pocts. The carly modernist pocts were also influenced by the writings of the early modernist poets, such as the poets of the Ming dynasty, who were influenced by the writings of the early modernist...
GPT-XT,+ Rep-Dropout:
between the individual and the collective" The work also source of inspiration for the later works of Chinese @-@ American authors such as John W. Campbell, who wrote his memoirs that he leammed much from the book ". In the early 20th century, the Chinese scholar and translator Wang Hui 1879 1940 wrote that the book is the most important work of Chinese literature Burton's translation of the text published in 1891, and was translated into English by the American scholar and translator John W. D. Rockefeller in 1904. It was the first English translation of...
Input Case 2:
In May 1951, he announced he was retiring from professional football to become sports director radio station in Buffalo. He was hired sportscaster by
GPT-XL::
WLUP @-@ AM in Buffalo, and was promoted to the position of sports director 1952. He was also radio announcer for the Buffalo Bills and the Buffalo Braves. In 1953, he named the radio play @-@ by @-@ play announcer for the Buffalo Bills. He
also color commentator for the Bills. In 1954, he named the radio play @- by @--@ play announcer for the Buffalo Braves. He was also color commentator for the Braves. In 1955, he named the radio play @-@ by @--@ play announcer for the Buffalo Bills. He was...
GPT-XL++ Rep-Dropout:
WLUP @-@@) FM Buffalo, New York, 1952. In 1953, joined the CBS radio network color commentator: call the NFL championship game between the New York Giants and the Chicago Bears, The Giants won the game, 24- 0. After the game, he asked by reporter if he had ever seen game like that. "1 have," he replied." " I've seen lot of football. " He was also color commentator for the CBS television series The NFL in 1954, During the 1950s, the NFL was the dominant league in America...
His first assignment
Figure 6: Example of generated text. The input text is in bleu, while the repetitions are in red. Both of the two models are fine-tuned on the Wikitext- 103 for 3 epochs, as described in section B. We use greedy search for decoding.
D Classification of Repetition Words
We categorize repetitions into three groups, as outlined by Altmann and Köhler [1] and Tannen [28]: grammar, theme, and limited inventory. For each sampled instance, we initially determine whether the repetitive n-gram falls under the grammar category, meaning any word in the n-gram determiner, preposition, conjunction, etc. Next, if the repetitive n-gram does not belong to the grammar category, assess whether any words closely related to the text's subject matter, thereby placing it in the theme category. For instance, "H. gammarus" is considered part of the theme category when repetitively used in article about Homarus gammarus. The third category encompasses repetitions stemming from a language's limited means of expressing a specific concept, known as limited inventory. Popular phrases such as "pair of" are examples of repetitive n-grams in this category.
In cases where multiple repetitive n-grams appear within a 256-word sentence, we only take one into account, If a repetitive n-gram satisfies the criteria for more than one category, particularly theme and limited inventory, we allocate it to the earliest applicable category.
16


Reviews:
[{"review_id": "qGwG6DiDZw", "reviewer": "Reviewer_6H9c", "summary": "This paper explains the repetition in model generated text from a data standpoint, pointing out that there is a strong correlation between the degeneration issue and the presence of repetitions in training data. The authors find out that penalizing repetitions in data can alleviate degeneration, and propose a method, repetition dropout, that is to apply dropout in transformers’ sublayers during training time. The proposed method achieve significant improvement in terms of REP-n scores.  ", "questions": "NA", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This is a very well written paper! The preliminary study does a great job in introducing and clearly motivating the problem.\n- The proposed repetition dropout is also a very simple technique, yet still improve the performance in REP-n stats.\n- The comparison between different methods and objectives in section 6 is very interesting and thorough.", "weaknesses": "Currently it seems that the repetition dropout mask is applied on each instance, but what about repeated text in different instance? How should we apply it in current LM training paradigm? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: Currently it seems that the repetition dropout mask is applied on each instance, but what about repeated text in different instance?**\n\nThanks for the question. To ensure that we understand your query correctly, we would like to confirm that by \"repeated text\" you are referring to repetitive n-grams, and by \"instance\" you mean a sentence. If our interpretation is accurate, in our work, the decision to drop an n-gram through the attention mechanism is entirely dependent on its context, i.e., a sentence with 256 words. In other words, if an n-gram appears only once in the context, we will not drop it. However, if it occurs multiple times within the context, there is a possibility that we will mask them according to the dropout rate. Please let us know if we have misunderstood your question or if you require further clarification!\n\n**Q: How should we apply it in current LM training paradigm?**\n\nTable 1: Experiments on GPT-XL (1.5 Billion parameters)\n\n\n|                   | rep-2 | rep-3 | rep-4 | rep-w | rep-r |\n|-------------------|-------|-------|-------|-------|-------|\n| MLE               | 54.26 | 49.21 | 45.84 | 66.10 | 37.72 |\n| MLE + Rep-Dropout | 11.36 | 5.80  | 3.67  | 24.39 | 18.19 |\n\n\n\nThanks for this good question. The only difference between current LM training paradigm and ours lies in the repetition dropout mask, which could be easily pre-computed before pretaining. Thus, our proposed repetition dropout technique is compatible with most of LM pertaining paradigm. For example, it is easy for us to extend our proposed repetition dropout technique to the larger language model. In Table 1, we directly apply our repetition dropout method to the GPT-XL model, which has 1.5 billion parameters. Since it is difficult to trained GPT-XL model from scratch, we fine-tuned it on Wikitext-103 for 3 epochs. Results in Table 1 demonstrate that our method can also alleviate the degeneration of LLMs after fine-tuning."}, {"review_id": "2WvYGUXngw", "reviewer": "Reviewer_6VM2", "summary": "The paper explores the issue of degeneration in text generation, which refers to the generation of repetitive words and dull loops by neural language models. The authors focus on the impact of repetition in the training data and propose a method to address this issue. Specifically, they suggest dropping out repetitive words during the training data pre-processing stage. The main conclusion drawn from their investigation is that penalizing repetitive n-grams in the training data is crucial for the effectiveness of existing methods in preventing degeneration. By highlighting the importance of addressing repetition in the training data, the paper contributes to understanding the factors influencing degeneration in text generation.", "questions": "Does dropping some repetitive n-grams sometimes distort the semantics of the document?", "limitations": "There are other potential factors that can impact the degeneration. The paper should have a more comprehensive discussion about all the factors.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is generally well-written and is easy to follow. The problem of generating repetitive words and dull loops has been widely observed and extensively discussed in the field, which makes this problem important and worth exploring. The conclusion drawn by the authors, that training data plays a pivotal role in the occurrence of degeneration, is logical and well-supported.", "weaknesses": "The paper primarily focuses on data pre-processing rather than novel training techniques. This limited scope may diminish its technical novelty. Many researchers working on text generation may have already explored similar data pre-processing ideas, making the proposed method less impactful or original. Also, the conclusion of the paper lacks novel insights and seems to affirm an intuitive understanding. While it is reasonable to assume that data quality has a direct impact on degeneration, this insight does not offer any new or surprising findings. As a result, the conclusion may be considered weak in terms of offering novel contributions or expanding the current understanding of the issue.\n\nExplore additional factors that could contribute to degeneration in text generation, such as the number of training data and model size. Investigating these factors may provide deeper insights into the phenomenon and allow for a more comprehensive understanding of degeneration. This expansion could strengthen the paper and enhance its impact within the field.\n\nFor me, it would be more interesting to study other factors that impact degeneration, such as the number of training data, the model size and the model architecture. Investigating these factors may provide deeper insights into the phenomenon and allow for a more comprehensive understanding of degeneration. With more training data and more parameters, large language models can capture semantics and thus prevent degeneration. For example, degeneration has been greatly resolved for ChatGPT due to its expressiveness and more general understanding of texts,  In my opinion, tackling degeneration in data pre-processing is not very meaningful because the root cause is still the model lacks of capability to understand and generate reasonable texts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: While it is reasonable to assume that data quality has a direct impact on degeneration, this finding is not surprising**\n\nWe'd like to emphasize that the goal of our work is to investigate the relationship between repetition in training data and degeneration. It is important to note that data containing repetitions is not necessarily of low quality. In fact, the repetition of certain words in natural language is sometimes necessary [1,12,23] (cited in paper). The problem is that LMs will learn and amplify the repetition in training data, which has not been studied in previous works. In addition, we give a unified interpretation about the reason of success of many previous works from the data perspective. \n\nIn recent years, data-centric AI has garnered the attention of numerous researchers, as many model behaviors may be linked to patterns in the data. By establishing a clear connection between repetitions in training data and the degeneration issue, we believe that researchers can gain a better understanding of degeneration and develop more targeted and effective methods.\n\n\n**Q: Explore additional factors**\n\nTable 1: Rep-2 score of text generated by OPT models on five datasets using greedy search.\n\n| Dataset\\Models | opt-125m | opt-350m | opt-1.3b | opt-2.7b | opt-6.7b | opt-13b | opt-30b | opt-66b |\n|----------------|----------|----------|----------|----------|----------|---------|---------|---------|\n| OpenWeb        | 69.66    | 67.74    | 58.80    | 54.75    | 51.68    | 50.46   | 46.17   | 47.52   |\n| Wiki-103       | 73.77    | 70.50    | 61.47    | 58.24    | 54.62    | 53.73   | 50.29   | 51.70   |\n| FreeLaw        | 72.80    | 69.90    | 60.37    | 56.90    | 51.95    | 50.18   | 48.45   | 47.44   |\n| PubMed         | 72.68    | 69.28    | 61.52    | 57.33    | 54.98    | 52.52   | 51.46   | 51.21   |\n| ArXiv          | 76.25    | 75.16    | 66.46    | 62.67    | 59.75    | 58.25   | 56.49   | 54.92   |\n\nThank you for your valuable suggestion. In the early version of our paper, we did evaluate factors beyond the data, such as the impact of model architecture and model size.\n\nWe obtained some interesting findings. For instance, in Table 1, we assessed the rep-2 score of OPT models with parameters ranging from 125M to 66B on the five datasets used in our final paper. Our results show that increasing the model size does alleviate the repetition issue to some extent. However, the gains achieved by increasing the model size diminish over time. The OPT-66B model still generates text with high rep-2 score. We also evaluated the impact of various model architectures, such as enc-dec Transformer models, dec-only Transformer models, LSTM models, etc. All models trained by MLE exhibit severe repetition issues, with no clear indication of which architecture suffers more from this problem.\n\nNevertheless, we decided to remove those sections from final submission, which was a difficult decision. The reason behind this is that it is challenging to thoroughly analyze and explain all these factors within a 9-page paper. As a result, we chose to focus on one critical factor, the repetition in training data.\n\nWe can reintroduce the discussion about other factors in the appendix of the revised paper.\n\n**Q: The paper primarily focuses on data pre-processing rather than novel training techniques.**\n\nWe appreciate your concern and would like to clarify that while our analysis paper includes experiments with data pre-processing to clearly demonstrate the research problem, our method to alleviate degeneration is a learning algorithm, not a data pre-processing method. Specifically, inspired by dropout, we propose an attention-based repetition dropout method to encourage the model to make predictions without relying on repetitive n-grams in the context. As demonstrated in our paper, the LM trained using our method achieves exceptionally low rep-n scores at a much lower cost than scaling up the model and data size.\n\n**Q: With more training data and more parameters, LLMs can prevent degeneration, e.g., ChatGPT...**\n\nWe agree that increasing the number of data and model size is likely to improve the performance. However, as shown in Table 1, OPT-60B still suffers from severe degeneration and increasing the size can only alleviate the degeneration issue to a certain extent. E.g., the rep-2 improves from 70 to 47 when the model size increases from 125M to 60B. This fact shows that the degeneration issue can not be mainly attributed to the small scale of model size and training dataset size. Moreover, in our preliminary experiments, lots of LLMs, including GPT-3 (e.g., text-davinci-002, 175B parameters), continue to suffers from the degeneration issue. Fortunately, simply dropping out the attentions relating to repetitions in training data can effectively reduce the repetition to an extremely level even with relatively small scale of model size and training data. Therefore, we think the repetitions in training data is a crucial factor for the degeneration issue.\n\nAs you pointed out, we also observed that ChatGPT exhibits less degeneration. However, ChatGPT is a product-level system got from a long development pipeline. To the best of our knowledge, there is no convincing literature within our community explaining how ChatGPT achieved this. In fact, this is a research topic that we are currently exploring.\n\nWe will add this discussion to the revised paper to clarify our motivation.\n\n**Q: Does dropping some repetitive n-grams sometimes distort the semantics of the document?**\n\nYes, it is possible. This is a common potential issue of many previous works, e.g., BERT, which also masks part of the text. That’s why we only apply the repetition dropout in the training time. In other words, the word at each time step can access all the prefix words during the inference time. When conducting human evaluation on the generated results, we also find that the text quality of our method is much higher than the MLE method."}, {"review_id": "saoDamb0pp", "reviewer": "Reviewer_jpnv", "summary": "In this paper, the authors demonstrate that repetition in the training data is a major cause of the neural text repetition problem. They first show a strong correlation between the repetition ratio of training data and generated text. Based on the observation, they propose repetition dropout to prohibit the model from learning repetition in data during training. Experimental results show that repetition dropout significantly addresses the repetition problem compared to previous approaches and penalizing repetition in training data is a key factor for reducing the problem. In analysis, the authors demonstrate that their method specifically reduces repetition caused by subject matter rather than grammar or highly frequent phrases.", "questions": "- Is there a possible reason for a relatively high repetition ratio in OpenWebText2? ", "limitations": "As written in the Limitation section, experiments on various scales could be conducted in future work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method and supporting experiments are well-motivated and the findings are interesting.\n- They significantly reduce the neural text repetition problem compared to previous work.", "weaknesses": "- To demonstrate that the distribution of generated texts is close to that of human texts, authors could further utilize metrics such as MAUVE [1].\n- Experiments on various parameter sizes would be beneficial for further understanding such as \"Does increasing model size enhance the robustness to the repetition problem\" or \"How are the previous and proposed methods effective as the model size increases?\"\n- A case study on samples generated from the models would provide further insights.\n\n[1] Pillutla, Krishna, et al. \"Mauve: Measuring the gap between neural text and human text using divergence frontiers.\" Advances in Neural Information Processing Systems 34 (2021): 4816-4828.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: metrics such as MAUVE**\n*Table 1: MAUVE scores on Wikitext-103*\n\n|             | Rep-4 ⬆️| MAUVE ⬆️| PPL  ⬇️ |\n|-------------|-------|-------|-------|\n| MLE         | 32.64 | 49.70 | 21.98 |\n| HI-Re       | 28.35 | 35.83 | --    |\n| ScaleGrad   | 5.01  | 52.80 | 39.11 |\n| UL          | 22.88 | 50.06 | 21.93 |\n| Rep-Dropout | 2.14  | 52.20 | 28.26 |\n|             |       |       |       |\n\nThanks for your kind suggestion. Most results on MAUVE score are consistent with those on rep-n score. As shown in Table 1, our method and ScaleGrad achieve the best performance on Wikitext-103. One exception is the High-Inflow Re-encoding baseline, which achieves the worst performance in terms of MAUVE.\n\n**Q: Does increasing model size enhance the robustness to the repetition problem? How are the previous and proposed methods effective as the model size increases?**\n\n*Table 2: Rep-2 score of text generated by OPT models on five datasets using greedy search.*\n| Dataset\\Models | opt-125m | opt-350m | opt-1.3b | opt-2.7b | opt-6.7b | opt-13b | opt-30b | opt-66b |\n|----------------|----------|----------|----------|----------|----------|---------|---------|---------|\n| OpenWeb        | 69.66    | 67.74    | 58.80    | 54.75    | 51.68    | 50.46   | 46.17   | 47.52   |\n| Wiki-103       | 73.77    | 70.50    | 61.47    | 58.24    | 54.62    | 53.73   | 50.29   | 51.70   |\n| FreeLaw        | 72.80    | 69.90    | 60.37    | 56.90    | 51.95    | 50.18   | 48.45   | 47.44   |\n| PubMed         | 72.68    | 69.28    | 61.52    | 57.33    | 54.98    | 52.52   | 51.46   | 51.21   |\n| ArXiv          | 76.25    | 75.16    | 66.46    | 62.67    | 59.75    | 58.25   | 56.49   | 54.92   |\n|                |          |          |          |          |          |         |         |         |\n\nThank you for your valuable suggestion. We did discuss this aspect in an earlier version of our paper but decided to remove it from the final submission, as we felt it might detract from the paper's focus. As demonstrated in Table 1, we evaluated the rep-2 score of OPT models with parameters ranging from 125 Million to 66 Billion on the five datasets used in our final paper. Our results show that increasing the model size does alleviate the repetition issue to some extent. However, the gains achieved by increasing the model size diminish over time, and the OPT-66B model continues to generate text with an extremely high rep-2 score.\n\n*Table 3: Experiments on GPT-XL (1.5 Billion parameters)*\n\n|                   | rep-2 | rep-3 | rep-4 | rep-w | rep-r |\n|-------------------|-------|-------|-------|-------|-------|\n| MLE               | 54.26 | 49.21 | 45.84 | 66.10 | 37.72 |\n| MLE + Rep-Dropout | 11.36 | 5.80  | 3.67  | 24.39 | 18.19 |\n\nBecause of the time limit of the rebuttal period, we only evaluated our method and one baseline method on GPT-XL, which has 1.5 billion parameters. We fine-tuned the GPT-XL using MLE and our method for 3 epochs. As shown in Table 3, we find that fine-tuning a larger model with our method can also significantly alleviate the repetition issue. \n\n**Q: A case study on samples generated from the models would provide further insights.**\n\nThanks for the useful suggestion. In the revised version, we will add a case study section to show the characteristics of our method and the baseline method. \n\n**Q: Is there a possible reason for a relatively high repetition ratio in OpenWebText2?**\n\nThanks for your good question. The data in OpenWebText2 comes from Reddit where each thread is discussing a particular topic with diversity in text style and many people are involved. In the section 6.3, we find that LMs would spend more effort on learning the repetition of theme-related n-grams to reduce the model’s PPL. Consequently, we hypothesize that the Reddit data may encourage the repetition behavior in the learning process. That’s also why we conduct experiments on multiple datasets in section 4."}, {"review_id": "3jBBOi0Lie", "reviewer": "Reviewer_Pdys", "summary": "This paper studies the cause of neural text degeneration, i.e. language models tend to generate repetitive loops. They design an experiment showing that text degeneration is correlated with the amount of repetitive text in the training data. Motivated by this finding, they propose repetitive dropout, which applies dropout on the attention weights over repetitive context. They experiment with 3 datasets and demonstrate that their proposed method can greatly reduce repetition. They also re-inspect previous hypotheses:\n\n1. high in-flow words: they show that merging repetitive words contributes to a large portion of the effect of merging high in-flow words.\n2. the maximum likelihood objective: they show that even when the model is trained with the maximum likelihood objective, as long as repetitions are penalized, the model does not degenerate much. Thus, the maximum likelihood objective is not the cause of neural text degeneration.\n3. self-reinforcement of repetition: the argue that\n    a. It is also caused by the repetitive words in the training data.\n    b. The self-reinforcement loops are broken by their proposed method.\n \nFinally, they categorize repetitions into three categories and study their frequency along with their effect on models’ degeneration behavior. They find that the *theme* category has the greatest effect.\n", "questions": "1. If repetition in the training data is the cause of degeneration, why not simply remove those repetitive text from the training data? This seems to me a more straightforward method than applying attention dropout.\n2. If the authors promise to address the issues in the weakness section above, I would be happy to increase my score, because I think the main message is clear and interesting enough.\n3. Having some analyses on large language models would make this work more impactful, e.g.\n    1.  If I understood correctly, the analyses in Section 6.3 can also be done for LLMs.\n    2.  I also wonder what would be an effective way to fine-tune an existing (large) model so it doesn't generate repetitive text.\n", "limitations": "Limitations are discussed in Section 7.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Most parts of the paper are easy to follow.\n2. Neural text degeneration has been discovered since 2019, but its cause is still unclear. Finding the root cause may interest audiences in this field.\n3. Despite the hypothesis being simple, as far as I know, it hasn’t been studied. The experiment design in Section 4 is clever.\n4. They proposed a simple method that can reduce repetition effectively.\n5. They inspected a few previous hypotheses and the results are aligned with their hypothesis (in general).\n6. They inspected the effect of three types of repetitions in the training data, which in my opinion is insightful.\n\nIn sum, in my opinion, their analyses of the cause of neural text degeneration is comprehensive and the conclusion is convincing.\n", "weaknesses": "In my opinion, because the main message of this paper is clear enough, the following issues may not be very crucial.\n\n1.  The authors only compare their method with one baseline in Table 1. There are other mitigations for text degeneration, e.g. [14, 22, 11] cited in the paper. \n2. Section 6.2 is relatively hard to follow. \n    1. Line 248, “The core idea of many previous works is to penalize a specific set of data to alleviate the degeneration.” is vague, though I can roughly figure out what it means after reading the following parts.\n    2. Having a brief introduction for the definition of high-inflow words will make the paper more self-contained.\n    3. For the Likelihood Objective part, the paragraph at line 283 needs a clearer topic sentence. The connection between the experiment design in paragraph at line 288 and the argument in the previous paragraph is not very clear to me either.\n3. At Line 312, the author mentioned “we find that the model trained by repetition dropout can break the self-reinforcement loop”. Some evidence should be provided.\n4. At Line 223, “we hypothesize that the exposure bias issue is an important factor for the left repetition, because we find that lower-order n -gram2…”, I couldn't understand the explanation. Some details should be provided too.\nrepetitions generally appear earlier than higher-order repetitions\n4. Given the popularity of large language models, the study on GPT-2-sized models may be less useful for the general audience.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: The authors only compare their method with one baseline in Table 1.**\n\n*Table 1: Added experiments of ScaleGrad on FreeLaw*\n\n| | Rep-2 | Rep-3 | Rep-4 | Rep-w | Rep-r |\n|-------------|-------|-------|-------|-------|-------|\n| MLE  | 51.74 | 46.19 | 42.22 | 39.22 | 73.06 |\n| ScaleGrad   | 15.82 | 10.28 | 7.52  | 15.97 | 32.40 |\n| Rep-Dropout | 10.15 | 5.60  | 3.49  | 17.55 | 23.21 |\n\n*Table 2: Added experiments of ScaleGrad on OpenWebText2*\n\n|| Rep-2 | Rep-3 | Rep-4 | Rep-w | Rep-r |\n|-------------|-------|-------|-------|-------|-------|\n| MLE| 73.96 | 70.61 | 67.91 | 67.28 | 88.27 |\n| ScaleGrad   | 26.61 | 21.26 | 17.98 | 26.08 | 45.21 |\n| Rep-Dropout | 25.24 | 16.14 | 11.10 | 34.73 | 49.80 |\n\nThank you for your feedback. In our original paper, we conducted experiments on three datasets, as presented in Table 1. For the Wikitext-103 dataset, we compared our method against four baseline approaches: MLE, ScaleGrad, HI-Re, and UL. However, for the FreeLaw and OpenText2 datasets, we only compared our method with the MLE baseline to demonstrate its effectiveness, because in their original works they didn’t evaluate on the FreeLaw and OpenText2 datasets.\n\nAs per your suggestion, we have now included additional baselines for the FreeLaw and OpenWebText2 datasets. Due to the time limit of the response period, we have incorporated the performance of the most effective baseline, ScaleGrad, for these two datasets.\n\nThe updated results can be found in Tables 1 and 2. Our methods consistently outperform both MLE and ScaleGrad across the majority of Rep-X metrics. The findings on the two datasets align with those observed on the Wikitext-103 dataset. We will report these additional experiments in the revised paper.\n\n**Q: Section 6.2 is relatively hard to follow.**\n\nThanks for your valuable suggestions. We will follow your comments and elaborate those parts with more details in the revised version.\n\n**Q: At Line 312...**\n\nThanks for your valuable suggestion! This conclusion was made by inspecting the generated results of MLE and our method. In the revised paper, we promise to attach a case study to support this conclusion. \n\n**Q: At Line 223**\n\nWe appreciate your feedback and will clarify this aspect in the revised version. The statement in question aims to explain why our method's Rep-n score did not reach human-level performance. Upon examining the challenging cases encountered by our method, we identified an error accumulation phenomenon, where higher-order repetitions typically occur following lower-order repetitions.\n\nAs discussed in previous works, an LM is more prone to fall into repetitive patterns when facing unseen states, which is referred to as the exposure bias issue. Consequently, when our model encounters lower-order repetitions that were not observed in training, it tends to generate more severe degeneration. \n\n**Q: ... Study on GPT-2-sized models may be less useful for the general audience.**\n\n*Table 3: Rep-2 score of text generated by OPT models on five datasets using greedy search.*\n\n| Dataset\\Models | opt-125m | opt-350m | opt-1.3b | opt-2.7b | opt-6.7b | opt-13b | opt-30b | opt-66b |\n|----------------|----------|----------|----------|----------|----------|---------|---------|---------|\n| OpenWeb        | 69.66    | 67.74    | 58.80    | 54.75    | 51.68    | 50.46   | 46.17   | 47.52   |\n| Wiki-103       | 73.77    | 70.50    | 61.47    | 58.24    | 54.62    | 53.73   | 50.29   | 51.70   |\n| FreeLaw        | 72.80    | 69.90    | 60.37    | 56.90    | 51.95    | 50.18   | 48.45   | 47.44   |\n| PubMed         | 72.68    | 69.28    | 61.52    | 57.33    | 54.98    | 52.52   | 51.46   | 51.21   |\n| ArXiv          | 76.25    | 75.16    | 66.46    | 62.67    | 59.75    | 58.25   | 56.49   | 54.92   |\n\nWe appreciate your valuable suggestion. Indeed, we initially discussed this aspect in an earlier version of our paper. However, we decided to remove it from the final submission to maintain focus and due to the inability to thoroughly investigate all factors within a single paper. We will reintroduce this discussion in the revised paper.\n\nAs shown in Table 1, we evaluated the Rep-2 score of OPT models with parameters ranging from 125M to 66B on the five datasets. The results indicate that increasing the model size does help alleviate the repetition issue to some extent. However, the gains from increasing the model size diminish as the size grows. Notably, the OPT-66B model still generates text with high Rep-2 score.\n\n**Q: ... why not simply remove those repetitive text from the training data?...**\n\nThanks for the interesting question. Actually, our preliminary study in section 4 is based on  a similar idea. However, it is important to note that human text with repetitive n-grams doesn’t indicate low quality. Discarding such data points could have negative effects on the model training. To address this concern, we proposed the repetition dropout method. This approach selectively masks parts of repetitive n-grams, enabling more efficient utilization of the training data.\n\n **Q: ... the analyses in Section 6.3 can also be done for LLMs.**\n\nThanks for your suggestion. Yes, the analyses in section 6.3 can also be conducted on LLMs. We will leave the through analysis about LLMs in our future work.\n\n**Q: ... an effective way to fine-tune an existing (large) model ...**\n\n*Table 2: Experiments on GPT-XL (1.5 Billion parameters)*\n|| rep-2 | rep-3 | rep-4 | rep-w | rep-r |\n|-------------------|-------|-------|-------|-------|-------|\n| MLE| 54.26 | 49.21 | 45.84 | 66.10 | 37.72 |\n| MLE + Rep-Dropout | 11.36 | 5.80  | 3.67  | 24.39 | 18.19 |\n\nThanks for this good question. Our method can be directly extended to LLMs. In Table 2, we use our method to fine-tune the GPT-XL model for three epochs. As shown in Table 2, our model can also significantly alleviate the repetition issue of GPT-XL, which further validates the effectiveness of our proposed approach.."}]

Author rebuttal:

Meta-review: All reviewers have agreed that the problem this paper focuses on is important and well-motivated, the paper is well-written, the approach is simple but effective, and the experiments are comprehensive.

After discussion, all reviewers voted to accept this paper, and I believe that this paper can provide some insights about LLM control. 
The camera-ready version can be better by including more experiments and explanations shown during the discussion so that the readers can better understand the paper.
Decision: Accept (poster)