SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Model	Rep-2(%)	Rep-3(%)	Rep-4(%)	Rep-w (%)	Rep-r (%)	PPL
Wikitext-103
HI-RE	41.91	33.82	28.35	38.60	66.22	-
SCALEGRAD	12.49	6.85	4.44	18.31	28.98	24.72
UL	36.77	28.22	22.88	39.13	61.89	21.93
MLE	47.05	38.46	32.64	46.42	72.59	21.98
+RAND-DROPOUT	38.33	28.84	22.66	37.76	66.19	23.50
+REP-DROPOUT	9.78	4.34	2.14	22.56	25.45	28.26
HUMAN	3.56	0.84	0.28	10.64	5.82	-
FreeLaw
MLE	51.74	46.19	42.22	39.22	73.06	16.13
+RAND-DROPOUT	38.82	32.19	27.78	31.31	61.30	18.85
+REP-DROPOUT	10.15	5.60	3.49	17.55	23.21	20.68
HUMAN	2.77	0.89	0.50	10.61	8.10	-
OpenWebText2
MLE	73.96	70.61	67.91	67.28	88.27	80.37
+RAND-DROPOUT	63.43	57.16	52.30	58.92	82.76	91.75
+REP-DROPOUT	25.24	16.14	11.10	34.73	49.80	107.02
HUMAN	4.53	1.39	0.61	12.41	13.09	-
Table 2: Performances of language models on three datasets. Both RAND-DROPOUT and REP-DROPOUT use a dropout rate 0.6. Rep-  $n$  is defined in Eq. (4). The decoding method for all models is greedy search. PPL is the perplexity score on the real test data. Note that we do not report the PPL for HI-RE, because its vocabulary is different from other baselines.
The main baseline method is MLE, which indicates the vanilla GPT-2 model trained by likelihood objective, as defined in Eq. (3). The + REP-DROPOUT is the proposed method, i.e., repetition dropout. We also have several baseline methods. The first one is the + RAND-DROPOUT, which applies dropout on random tokens instead of repetitive tokens. Note that the number of random tokens selected for dropout for this baseline is constrained to the same as the repetitive tokens. We also compare with three previous works that also conducted experiments on Wikitext-103: re-encoding of high-inflow tokens (HI-RE) [8], training with scaled gradient (SCALEGRAD) [15], and unlikelihood objective at token level (UL) [34]. During inference we use greedy search as the decoding method to generate 128 tokens, using the first 32 tokens of each line in the test set as the prompt. More details about those baseline methods are also shown in the Appx. A.
As shown in Tab. 2, we evaluate the baseline methods and our method on three datasets. On the Wikitext-103 dataset, both the MLE and MLE + RAND-DROPOUT methods suffer from the severe degeneration issue. More than  $33\%$  percent 4-grams in the generated sentences of the two baseline methods are repetitive, which is different from the patterns in natural language. Nevertheless, MLE + REP-DROPOUT can significantly reduce the repetition issue in generated sentences. This observation indicates that repetition in the training data is crucial for the degeneration issue. Compared with methods in previous works in Tab. 2, i.e., HI-RE, SCALE-GRAD, and UL, the REP-DROPOUT also show better performance on alleviating the degener
Figure 3: Rep-2 score and perplexity of MLE + REP-DROPOUT on the test set of Wikitext-103. The brown dash line is the human-level rep-2 score.
ation issue. The results on FreeLaw and OpenWebText2 datasets are consistent with those on Wikitext-103. We also conduct quantitative and qualitative experiments to evaluate  $^+$  REP-DROPOUT on larger LMs, e.g., GPT-XL[20], which are shown in Appx. B and C, respectively.
We also evaluate the effect of MLE + REP-DROPOUT with different dropout rates for the repetitions in Wikitext-103, which can be regarded as controlling the percentage of repetitions in a dataset. As shown in Fig. 3, increasing the dropout rate of our method will reduce the number of repetitions in generated sentences continuously. Moreover, we observe a clear trade-off between the rep-2 score of generated sentences and the perplexity on the real data in test set. This suggests that learning on the repetitions in training data can be beneficial for language modeling. Consistent results are also shown on FreeLaw and OpenWebText2 datasets.
6.2 Relation to Previous Hypotheses
Previous research has proposed various hypotheses and approaches to understanding and solving the problem of degeneration in neural text generation. However, we argue that many of these proposals can be explained by a simple explanation. That is, penalizing repetitions in data is a common and fundamental factor for their success.
The core idea of many previous works is to penalize a specific set of data, e.g., all the prefix words  $x_{1:t-1}$  at timestep  $t$ , to alleviate the degeneration. However, as shown in Fig. 4, many of them have implicit connections with the repetitions in data. For example, the set of high-inflow words  $(\diamondsuit)$  penalized in Fu et al. [8] has a noticeable interaction with that of repetitive words  $(\bigcirc)$ , and the set of prefix words  $(\square)$  penalized in [34] and [15] is the super-set of the repetitive words  $(\bigcirc)$ . In Xu et al. [36], they directly penalize the repetitions in pseudo repetitive data  $(\bigcirc)$ . In this section, we will show that repetitive words play an important role in previous works.
High-Inflow Words Some researchers [8] attribute the degeneration issue to high-inflow words, whose probability sum of all the potential preceding words is higher than a threshold. Thus, they propose that merging high-inflow word pairs can alleviate the problem (HI-RE). However, we find that  $26\%$  of high-inflow word pairs are repetitive in each sentence of Wikitext103, and merging these pairs can significantly reduce the rep-2 score of real data. Therefore, we argue that merging high-inflow word pairs is actually an alternative way of reducing repetitions in training data.
We evaluate the argument by controlling a single variable, the type of high-inflow words to be
Figure 4: Relationship between the penalized data in previous works. We use  $\diamondsuit$ ,  $\square$ , and  $\bigcirc$  to represent the sets of high-inflow words [8], prefix words [11, 15, 26, 34], and pseudo repetitive sentences [36], respectively. We also demonstrate the set of repetitive words in real data by  $\bigcirc$ .
merged, in line 4-6 of Tab. 3. The vanilla HI-RE method (Line 4) merges all the high-inflow words  $(\diamond)$ , which takes  $31.1\%$  of the total training words. We find that the method (Line 5), which only merges repetitive high-inflow pairs, i.e., the intersection between high-inflow words and repetitive words  $(\diamond \cap \bigcirc)$ , achieves performance comparable to the original HI-RE method (Line 4). Note that the method in line 5 only merges  $8.1\%$  of the total training words, which is much less than the vanilla method. In contrast, the HI-RE method that merges random high-inflow pairs (Line 6), which has the same number as repetitive high-inflow words  $(\diamond \cap \bigcirc)$ , cannot alleviate the degeneration. This suggests that penalizing repetitions in data is critical in the success of Fu et al. [8].
Likelihood Objective Many researchers [11, 15, 26, 34] think that the likelihood objective is the main factor for the degeneration issue. All of those works share the same principle that words in the prefix context (□) cause the degeneration issue. Therefore, Welleck et al. [34] and Lin et al. [15] propose to reduce the probabilities of repetitions appearing in  $x_{1:t-1}$  at time step  $t$ . Su et al. [26] and Jiang et al. [11] leverage the contrastive learning to ensure that the hidden representation at time step  $t$  is distinctive to those of  $x_{1:t-1}$ .
However, the attribution to the likelihood objective merely serves as a superficial explanation, and the core of this issue lies in the fact that the model inevitably learns repetitive behavior when conducting maximum likelihood estimation on repetitive data. As shown in section 6.1, the GPT-2 model with simple repetition dropout, which is also optimized by likelihood objective, achieves an extremely low rep-4 score on generated text. This indicates that likelihood objective might not be the most
#	Model	Penal. Scope	Word Percent (%)	Rep-2(%)	Rep-3(%)	Rep-4(%)	PPL
1	MLE	N/A	0.0	47.05	38.46	32.64	21.98
2	+REP-DROPOUT	Subset of○	11.4	9.78	4.34	2.14	28.26
3	DITTO	○	25.0	44.24	34.74	28.34	-
4	HI-RE	◇	31.1	41.91	33.82	28.35	-
5	HI-RE	◇∩○	8.1	43.62	33.67	27.12	-
6	HI-RE	Subset of◇	8.1	52.41	43.72	37.56	-
7	SCALEGRAD	□	100.0	12.49	6.85	4.44	24.72
8	SCALEGRAD	□∩○	19.0	17.53	10.38	6.94	23.33
9	SCALEGRAD	Subset of□	19.0	22.97	15.22	10.94	23.18
Table 3: Impact of penalizing repetitions in different kinds of data on Wikitext-103. The meanings of the symbols  $\bigcirc$ ,  $\diamond$ ,  $\square$  are illustrated in Fig. 4. The "Subset of SHAPE" means a subset randomly sampled from SHAPE. The word percent is  $\frac{\# \text{Penalized Words}}{\# \text{Words in Data}}$ .
important factor in the degeneration issue. Nevertheless, all methods that penalize tokens in the prefix context  $(\square)$  break the reliance on repetitions  $(\bigcirc)$ . Thus, we hypothesize that preventing the model from learning on repetitions is a key factor in their success.
To evaluate the impact of repetitions on these methods, we conduct experiments following the same principle used to analyze high-inflow words. We choose the SCALEGRAD [15] method as our baseline, which penalizes non-novel tokens by scaling the gradient [15]. The non-novel tokens are the entire prefix context  $x_{1:t-1}$  at time step  $t$ , i.e., 100% of the training words, for the vanilla SCALEGRAD method. We also propose two variants of SCALEGRAD: the first uses the repetitive words within prefix  $\square \cap \square$  as the non-novel tokens, while the second randomly samples a subset, which has the same number of words as the first one  $\square \cap \square$ , from the prefix data (Subset of  $\square$ ). The two methods only penalize 19.0% of the total training words. As shown in Tab. 3, the SCALEGRAD method on repetitive words (Line 8) achieves performance close to the standard SCALEGRAD method (Line 7) in terms of the rep- $n$  metric. In contrast, the SCALEGRAD method on random subset (Line 9) is not effective in alleviating the degeneration issue as the other two methods. Other works [11, 26, 34] that attribute the degeneration issue to likelihood objective also penalize tokens in prefix as the SCALEGRAD [15], but with different techniques. Thus, we think that penalizing repetitions in data is also a crucial factor for the success of these methods.
Self-reinforcement Phenomenon Both Xu et al. [36] and Chiang and Chen [4] find that degeneration is always accompanied by the self-reinforcement phenomenon, i.e., the probability of a predicted word becomes higher when it is repeated more times. Thus, they hypothesize that degeneration is (partially) caused by self-reinforcement. To mitigate this issue, [36] proposed a data-augmentation method, namely DITTO (Line 3 of Tab. 3), which constructs pseudo data by repeating a training sentence multiple times and penalizes the probabilities of repetitive tokens in the pseudo data  $(\bigcirc)$ .
We argue that, as the degeneration issue, the self-reinforcement is also a by-product when neural LMs learning on repetitive patterns in real data. First, we observe a similar self-reinforcement phenomenon on the repetitive words in real data. For instance, the probabilities of the second appearances of the theme-related words are generally higher than their first appearances, as shown in Fig. 5(b). Second, we find that the model trained by repetition dropout can break the self-reinforcement loop, as shown by the examples in Appx. C. Although the text generated by GPT-2 + REP-DROPOUT on Wikitext-103 may contain a few inappropriate  $n$ -gram repetitions, it will not fall into the infinite repetition loop that frequently appear in the text generated by the vanilla GPT-2.
6.3 Why LMs Learn the Repetition Patterns?
In the last empirical study, we make an attempt to investigate the reasons behind LMs learning repetition patterns, specifically the role these repetitions play in neural LMs. To address this inquiry, we examine 300 randomly selected instances, each featuring repetitive bi-grams within a 256-word sentence. These cases are broadly categorized into three groups, as per Altmann and Kohler [1] and Tannen [28]:
- Grammar: Repetitions for grammatical purposes, such as determiners, conjunctions, etc.
- Theme: Repetitions closely associated with the subject matter of the text.
- Limited inventory: Repetitions resulting from a language's restricted means of expressing a particular concept, leading to high-frequency occurrences, e.g., the phrase "pair of" in English.
(b)
Figure 5: (a) Percentages of three types of repetitions in Wikitext-103. (b) Probabilities of three types of words with and without masking their repetitions in context using the vanilla GPT-2 model.
We assign each case to the earliest category it satisfies if it meets the criteria for multiple categories. Detailed guidelines for human evaluators to classify repetitive words can be found in Appx. D. As demonstrated in Fig. 5(a), almost  $50\%$  of the repetitions in Wikitext-103 fulfill grammatical functions, while both theme-related and inventory-related repetitions constitute around  $25\%$  each.
To understand the impact of different types of repetitions on a vanilla neural LM, we measure the probability change of a prediction when masking the information of its repetitions in the context through attention mechanism. For example, given the human repetitions in Fig. 1, we will determine how the probability of the last appearance of "generals wore..." changes when masking its first appearance of "generals wore..." in the context.
As shown in Fig. 5(b), masking the theme-related repetitions will cause a noticeable drop of the prediction probabilities. However, the prediction of grammatical and inventory-related repetitions are not affected by masking their previous appearances in context. This observation indicates that neural LMs spend its effort on optimizing the prediction accuracy of those theme-related words by implicitly repeating words in context.
In this work, we find that the repetition in training data is a fundamental factor for the degeneration (or repetition) problem. First, training data is an integral part of most AI systems, and our preliminary study demonstrates a strong correlation between the repetitions in training data and the degeneration issue. Second, we find that simply dropping out the attention to repetitions in training data can significantly reduce the degeneration issue, which is more effective than other baselines. Finally, we conduct extensive empirical analyses to demonstrate that penalizing repetitions in data is the key success factor for many previous works, such as those attribute degeneration issue to high-inflow words, the likelihood objective, and the self-reinforcement mechanism. Experiments also show that our findings are critical even in the context of large language models. We hope that the viewpoint we provide to understanding the degeneration can inspire more principled research in the future.
Our work also has some limitations. First, despite the excellent performance on reducing repetitions, the repetition dropout method may hurt the perplexity of the language model. Second, we examine our work mostly on standard benchmark of the language generation task, following previous works [8, 10, 26, 34]. It also deserves to extend our work to large-scale data and model.
The authors would like to express their sincere gratitude to the three anonymous reviewers and the meta reviewer for their insightful comments and constructive feedback. Furthermore, this work was partial supported by the JSPS KAKENHI Grant, under the number 23KJ1594.
[31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
[32] Jesse Vig. 2018. Deconstructing bert: Distilling 6 patterns from 100 million parameters. Medium, December.
[33] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837.
[34] Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
[35] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. pages 38-45. Association for Computational Linguistics.
[36] Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. 2022. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation. In Advances in Neural Information Processing Systems.
[37] Haoran Yang, Deng Cai, Huayang Li, Wei Bi, Wai Lam, and Shuming Shi. 2023. A frustratingly simple decoding method for neural text generation. arXiv preprint arXiv:2305.12675.
[38] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.
[39] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2979–2989, Copenhagen, Denmark. Association for Computational Linguistics.
A.1 Preliminary Study
The basic GPT-2 model<sup>4</sup> is trained from scratch on each corpus, which has 12 transformer blocks and 12 attention heads with 768 hidden dimensions. The Huggingface transformers [35] and Pytorch toolkit [18] are used to train the GPT-2 model in the distributed manner on A100 GPU server. The hyper-parameters during training are shown in Tab. 4.
Hyper-parameter	Value
Optimization steps	100K
Test interval	10K
Dropout rate	0.1
Grad clipping	1.0
Learning rate	5e-5
Batch size	128
Maximum sequence length	256
Warmup steps	10K
Learning scheduler	Linear decay
Random seed	0
Number of GPUs	4
Learning objective	Cross-Entropy Loss
A.1.1 Our Method
Most of the hyper-parameters for our proposed method are the same as that in Tab. 4 for better variable controlling. The specific hyper-parameters for our proposed method are the length of repetitive  $n$ -gram and its repetition dropout rate  $p$ , which are set as 2 and 0.6, respectively.
A.1.2 Baselines
In this subsection, the specific hyper-parameters for three baselines are described, and most of the hyper-parameters are the same as that in Tab. 5.
Table 4: The hyper-parameters during GPT-2 training procedure.
Hyper-parameter	Value
Re-encoding of High-inflow Tokens (HI-RE)
Re-encoding γ	0.03
Scaled Gradient (SCALEGRAD)
Scale grade γ	0.2
Token-level Unlikelihood Training (UL)
Rank alpha α	1.0
Table 5: The hyper-parameters of three baselines in this paper.
A.2 Instruction Tuning
To evaluate the effect of instruction tuning, we conduct experiments on three datasets:
1. ALPACA: The instruction-tuning dataset used by Alpaca [29].
2. WT-103 50K: We randomly sample 50k non-title sentences from Wikitext-103 and convert them to the instruction-tuning format, following [29].
3. ALPACA + WT-103 50K: The mixture of both.
4Model details can be found at https://huggingface.co/gpt2
We use the QLoRA [6] to fine-tune the LLAMA2-7B model [30], due to the limited computational resources. The decoding method for generating text is greedy search. We use the test set of Wikitext-103, which is also converted to the instruction-tuning format, to evaluate the performance of different models. Below is the template used for converting Wikitext-103 to the instruction-tuning format:
"instruction": "Please continue writing based on the following prefix. The text to be continued should be relevant, fluent and informative."
    "input": Prefix, # prefix of a sentence
    "output": COMPLETION # the completion of the prefix
In addition to the repetition measurements (Tab. 2), we also evaluate our method on additional metrics, e.g., MAUVE [19]. As shown in Tab. 6, most of the performances of our method and baseline methods are consistent with the observation in Tab. 2.
MLE(%)	HI-RE	SCALEGRAD	UL	REP-DROPOUT
MAUVE	49.70	35.83	52.80	50.06	52.20
Table 6: Evaluation on MAUVE. The settings are the same as those in Tab. 2.
Model	Rep-2(%)	Rep-3(%)	Rep-4(%)	Rep-w (%)	Rep-r (%)
MLE	54.26	49.21	45.84	66.10	37.72
+ REP-DROPOUT	11.36	5.80	3.67	24.39	18.19
Table 7: Experiments on GPT-XL (1.5 Billion parameters)
In addition to GPT-2 model, we also conducted experiments on model with larger size. We directly apply our repetition dropout method to the GPT-XL model, which has 1.5 billion parameters. Since it is difficult to train GPT-XL model from scratch, we fine-tuned it on Wikitext-103 for 3 epochs. Results in Tab. 7 demonstrate that our method can also alleviate the degeneration of LLMs after fine-tuning on larger model.
C Qualitative Analysis
In Fig. 6, we show some generated results of the GPT-XL model and GPT-XL+REP-DROPOUT. The training and inference settings of the two models are described in Appx. B. We can observe that the repetitions in generated text were reduced significantly after employing the REP-DROPOUT method. In addition, the GPT-XL trained by MLE easily falls into a sentence-level repetition, because of the self-reinforcement mechanism [36]. Nevertheless, the GPT-XL+REP-DROPOUT does not suffer from this issue.
1   
2   
3   
4   
5
5https://huggingface.co/gpt2-x1
Input Case 1:
Traditional Chinese literary criticism emphasized the life of the author when interpreting a work, a practice which Burton Watson attributes to "the close links that traditional Chinese thought posits
GPT-XL:
between the author and the reader". The author's life was also a central theme in the works of the early modernist poets, such as the poets of the Ming dynasty, who were influenced by the writings of the early modernist poets. The early modernist poets were also influenced by the writings of the early modernist poets, such as the poets of the Song dynasty, who were influenced by the writings of the early modernist poets. The early modernist poets were also influenced by the writings of the early modernist poets, such as the poets of the Ming dynasty, who were influenced by the writings of the early modernist...
GPT-XL + Rep-Dropout:
between the individual and the collective". The work was also a source of inspiration for the later works of Chinese @-@ American authors such as John W. Campbell, who wrote in his memoirs that he "learned much from [ the ] book ". In the early 20th century, the Chinese scholar and translator Wang Hui ( 1879 - 1940 ) wrote that the book " is the most important work of Chinese literature ". Burton's translation of the text was published in 1891, and was translated into English by the American scholar and translator John W. D. Rockefeller in 1904. It was the first English translation of...
Input Case 2:
In May 1951, he announced he was retiring from professional football to become sports director at a radio station in Buffalo. He was hired as a sportscaster by
WLUP @-@ AM in Buffalo, and was promoted to the position of sports director in 1952. He was also a radio announcer for the Buffalo Bills and the Buffalo Braves. In 1953, he was named the radio play @-@ by @-@ play announcer for the Buffalo Bills. He was also a color commentator for the Bills. In 1954, he was named the radio play @-@ by @-@ play announcer for the Buffalo Braves. He was also a color commentator for the Braves. In 1955, he was named the radio play @-@ by @-@ play announcer for the Buffalo Bills. He was...