Title: Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective
Venue: NeurIPS 2023
Accepted: yes
Keywords: language modeling

text generation

natural language processing
Abstract: There are a number of diverging hypotheses about the neural text degeneration problem, i.e., generating repetitive and dull loops, which makes this problem both interesting and confusing. In this work, we aim to advance our understanding by presenting a straightforward and fundamental explanation from the data perspective. Our preliminary investigation reveals a strong correlation between the degeneration issue and the presence of repetitions in training data. Subsequent experiments also demonstrate that by selectively dropping out the attention to repetitive words in training data, degeneration can be significantly minimized. Furthermore, our empirical analysis illustrates that prior works addressing the degeneration issue from various standpoints, such as the high-inflow words, the likelihood objective, and the self-reinforcement phenomenon, can be interpreted by one simple explanation. That is, penalizing the repetitions in training data is a common and fundamental factor for their effectiveness. Moreover, our experiments reveal that penalizing the repetitions in training data remains critical even when considering larger model sizes and instruction tuning.

Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective

Abstract

There are a number of diverging hypotheses about the neural text degeneration problem, i.e., generating repetitive and dull loops, which makes this problem both interesting and confusing. In this work, we aim to advance our understanding by presenting a straightforward and fundamental explanation from the data perspective. Our preliminary investigation reveals a strong correlation between the degeneration issue and the presence of repetitions in training data. Subsequent experiments also demonstrate that by selectively dropping out the attention to repetitive words in training data, degeneration can be significantly minimized. Furthermore, our empirical analysis illustrates that prior works addressing the degeneration issue from various standpoints, such as the high-inflow words, the likelihood objective, and the self-reinforcement phenomenon, can be interpreted by one simple explanation. That is, penalizing the repetitions in training data is a common and fundamental factor for their effectiveness. Moreover, our experiments reveal that penalizing the repetitions in training data remains critical even when considering larger model sizes and instruction tuning. Our code is available at https://github.com/gmftbyGMFTBY/Rep-Dropout .

1 Introduction

The emergence of neural language models (LM) has led to significant achievements in various text generation tasks, such as machine translation [3, 24, 31], summarization [22], and open-ended text generation [17, 20]. However, in open-ended text generation, neural LMs exhibit a strikingly severe degeneration issue, producing unreasonably repetitive texts, particularly when employing maximum a posteriori (MAP) decoding algorithms. As illustrated in Fig. 1, even a well-trained LM [20] may suffer from a severe degeneration issue.

There have been numerous attempts to explain the phenomenon of neural text degeneration, with many attributing this problem to flaws in the learning process. Fu et al. [8] claim that high-inflow words increase the probability of generating repetitions. A collection of studies [11, 15, 26, 34] argue that the likelihood objective is the primary factor, because it has the problem of exposure bias and focuses more on next-token prediction rather than sequence generation. Meanwhile, both Chiang and Chen [4] and Xu et al. [36] assert that the self-reinforcement mechanism can make neural LMs fall into the repetitive loops. Intriguingly, despite the divergence in these explanations, all

! : It was not possible to know which grade of general an officer would be required to be given to a particular officer . The Army was not aware of the Army's role in the conflict , and the Army was not aware of the Army's role in the conflict . The Army's role in the conflict was limited to the training of

" : It was not possible to know which grade of general an officer was by his collar insignia . However , major generals and lieutenant generals wore two rows of nine buttons in groups of three down the of the overcoat , and brigadier generals wore two rows of eight buttons in groups of two .

corresponding methods proposed have been shown to effectively alleviate the text degeneration issue. This observation leads us to ponder: could there exist more fundamental factors that can explain the degeneration issue?

In this study, we strive to provide a straightforward and fundamental interpretation for the degeneration problem from the data perspective. Our preliminary investigation reveals that repetitive words in the training data play a crucial role in the issue. It is also worth noting that repetitions are not necessarily of low quality, because it is a natural and common phenomenon in human writing [1, 12, 28]. Across five datasets with different domains, we observe a strong correlation between repetition rates in training and generated text. This finding encourages us to further assess the impact of repetitive words in training data by randomly dropping out attention to them during training. Employing repetition dropout, we discover that the repetition rate in generated text can be substantially reduced. Lastly, we reconcile many previous hypotheses with our single explanation, asserting that penalizing repetitions in training data is a key factor to the success of alleviating the degeneration issue.

As large language models (LLMs) gain popularity, it appears that the degeneration issue has been somewhat solved. We investigate the impact of various factors associated with LLMs on degeneration, including increasing model size and training models using instruction-tuning data. Our experiments reveal that penalizing repetitions in training data continues to be crucial in the context of LLMs.

Our contributions are threefold:

We demonstrate that the proportion of repetitive words in training data has a significant influence on the degeneration issue. Inspired by this finding, we also propose a method, namely, repetition dropout , to mitigate the degeneration.

We find that penalizing repetitions in data is a more fundamental factor for reducing repetitions, which also provides a unified explanation for various existing hypotheses, such as the attributions to high-inflow words, likelihood objectives, and self-reinforcement phenomenon.

We investigate the influence of factors associated with large language models on reducing repetitions, including model scale and instruction tuning.

2 Related Work

The degeneration (or repetition) issue in neural language models (LM), particularly in open-ended text generation, has garnered significant attention in recent years. Previous works have proposed disparate interpretations and solutions for this issue.

Many researchers hypothesize that factors within the learning process contributes to the degeneration issue. Fu et al. [8] assert that the high-inflow words in training data may increase the probability of generating repetitions, where the inflow of a word is defined as the probability sum of all its preceding words. Another factor identified is the likelihood objective. Welleck et al. [34] argue that likelihood objective has a discrepancy with the MAP decoding and focuses more on next-token prediction rather than sequence generation. Thus, they proposed an unlikelihood objective to address the two flaws. Based on the same principle, Lin et al. [15] propose to scale the gradient of specified non-novel tokens, i.e., words in the prefix context at each time step, to alleviate the degeneration issue. Su et al. [26] and Jiang et al. [11] pointed out that the degeneration issue may caused by the high similarity between token representations. Thus, they leveraged contrastive learning to learn a more distinct

representation for each token. In addition, both Xu et al. [36] and Chiang and Chen [4] argue that degeneration is caused by the self-reinforcement phenomenon and Xu et al. [36] address this issue by penalizing the repetitions in pseudo repetitive data.

Apart from issues within the learning process, other explanations for degeneration have been proposed. Many researchers contend that decoding methods [7, 10, 16, 20, 37] is the primary factor. Holtzman et al. [10] argue that word probabilities in human-generated text exhibit high variance and randomness, while high-probability text produced by MAP decoding methods tends to be repetitive and dull. This observation explains why sampling-based decoding methods [7, 10, 16, 20] can substantially mitigate the degeneration issue. Riley and Chiang [21] find that tasks with lower constraints, or larger solution spaces, suffer from more severe degeneration issue. The model architecture [8, 32, 34] and size [14] may also contribute, but the two factors have not been quantitatively evaluated.

However, with so many explanations, understanding the primary cause of the degeneration issue becomes increasingly challenging. In our work, we strive to provide a fundamental explanation for the previous hypotheses in the learning process. While our work may not encompass all the existing explanations, we believe it lays a solid foundation for further understanding of the degeneration issue.

3 Background

Language model (LM) aims to estimate the probability of a sentence in natural language according to the chain rule of probability:

where x = <x 1 , . . . , x L > is a sequence of words with length L , and x 1: i -1 is the previous i -1 words, i.e., the context, for predicting word x i .

Model Since attention-based LM [20, 31] has became the backbone of many tasks, we will use GPT-2 model [20] as the main architecture for our empirical studies. The core of GPT-2 model 1 is to use the attention mechanism to update the representation of words in x :

where β is a scalar to control the scale of the attention score, Q , K , and V are the linear transformation of h 1: L ∈ R L × d using matrices W q , W k , and W v ∈ R d × d , respectively. h 1: L is the current hidden representation of words x 1: L , and d is the hidden size. M is masking matrix that makes sure only the information of h 1: i is accessible for h i at each time step i . If the word x i can perceive the information of x j , then M [ i ][ j ] equals to 0 , otherwise -∞ .

Training Generally, neural LMs are trained by optimizing the likelihood objective:

where h i -1 ∈ R d is the hidden vector of x i -1 output by the last layer of an neural LM, e.g., the GPT-2 model [20]. The [ x i ] is defined as taking the probability regarding to x i in the distribution got from softmax . The ϕ ( · ) is a linear layer that transforms the h i -1 to logits.

(a)

(b)

80

(%)

of Gen. Text (

60

40

Rep-2 (

OpenWeb

20

口

Wiki-103

FreeLaw

☆

PubMed

☆

ArXiv

0

0

4.5

6

13.5

18

Rep-2 of Human Text (%)

80

Rep-2 of Gen. Text (%)

60

40

20

0

0

17.5

35

52.5

70

#Params. of LM (Billion)

Inference Since neural LMs are trained to maximize the likelihood objective, one intuitive practice for inference is to use the MAP decoding method, e.g., greedy search or beam search. However, in open-ended text generation, this tactic will cause an extremely severe degeneration issue on vanilla neural LMs trained by likelihood objective [10].

Evaluation The main focus of this empirical study is to investigate the reason for the repetition issue. Therefore, the repn is an important evaluation metric in our work, following previous works [8, 25, 27, 34]:

̸

L -n +1 where n is the length of n -gram, and UniqueNgrams is a function to find all unique n -grams in a sentence x . For the corpus-level evaluation, we report the averaged repn scores of instances in the dataset. To ensure that our results in main experiments are not biased to repn , we also report the results of repw and repr in previous works [8, 34]. The repw = 1 L ∑ L t =1 1 { x t ∈ x t -w -1: t -1 } , which measures the word-level repetition in a prefix window with length w . In our experiments, we set w to 16, following Fu et al. [8]. The repr = 1 L |{ i | ( x i = x j ∧ x i +1 = x j +1 , ∃ j = i ) ∨ ( x i = x k ∧ x i -1 = x k -1 , ∃ k = i ) }| . It is for the portion of repetitive snippets measured by sentence length.

̸

In addition to the measurement of repetition, we also consider the perplexity (PPL) on the real data, which demonstrates the performance in language modeling [2]. Although LMs with lower PPL may not consistently lead to better generation results, it is able to reflect trivial solutions for the degeneration issue, e.g., random or over-fitting models.

4 Preliminary Study: Rethinking Neural Text Degeneration from Data Perspective

The propensity of neural LMs with impressive performance to fall into naive repetitive loops is puzzling. Although numerous hypotheses have been proposed, many of them approach this issue from divergent aspects, and some are not intuitive for understanding. According to Ockham's Razor, a simpler explanation is often preferable. Therefore, in our preliminary study, we start by evaluating one elementary factor for most AI systems, the training data. Concretely, repetition is a natural and common phenomenon in human languages for various reasons [1, 12, 28]. It is intriguing to investigate whether there are connections between valid repetitions in natural language and incorrect repetitions in generated language. Moreover, it is important to note that data containing repetitions is not necessarily of low quality, as shown in Fig. 1.

Setup To assess the correlation between repetitions in generated text and those in human text, we propose to train GPT-2 models [20] on data with varying rep2 scores and then evaluate the rep2 scores of the text generated by the corresponding models. Specifically, we sorted the training instances in each dataset D based on their rep2 scores. We then divided the sorted training data into six shards, each containing an equal number of words but a varying percentage of repetitions, and trained a GPT-2 model on each shard. Notably, we will use the full test set of a dataset D to evaluate the models trained on different shards of D . More implementation details are in Appx. A.

Our preliminary study investigates five datasets across various domains. Wikitext-103 is a widely used dataset for language modeling [2, 5, 13] and open-ended generation [8, 34]. We adopt the standard split for training, validation, and test sets. The remaining four datasets, OpenWebText2, FreeLaw, PubMed, and ArXiv, are part of the Pile dataset [9]. To ensure consistent analysis, we sample an equivalent number of words as the Wikitext-103 dataset from each of the four Pile datasets. For the validation and test sets, we randomly sample 2,000 sentences from each of the four Pile datasets. The training, validation, and test sets are non-overlapping across all five datasets.

Findings Fig. 2(a) demonstrates a strong correlation between the rep-2 scores of human and generated text on each dataset, indicating that the degeneration issue becomes more severe as the percentage of repetitions in training data increases. However, due to varying data distributions across datasets from different domains, the rep-2 score of generated text may vary even when trained on data with the same rep-2 score. Another intriguing observation is that neural LM will amplify the repetition in data by more than 10 times, similar to the bias amplification found in Zhao et al. [39].

Investigations Related to Large Language Models As LLMs, e.g., ChatGPT [17] and LLAMA[30], gain popularity, it appears that the degeneration issue has been somewhat solved. In this section, we will investigate the impact of various factors associated with LLMs on degeneration, including increasing model size and training models with instruction-tuning data.

Many amazing model abilities emerge when scaling up the model and data size [33]. An interesting question is, whether the degeneration issue will be solved by simply scaling up ? We use a set of OPT models [38] with different model sizes to investigate this question. As shown in Fig. 2(b), the rep-2 score of generated text sharply drops before increasing the model size to 6.7 billion parameters, indicating that increasing the model size does alleviate the repetition issue to some extent. However, the gains achieved by increasing the model size diminish over time. The OPT-66B model still generates text with high rep-2 score. This observation shows that increasing the model size is not an efficient way to alleviate the degeneration.

Degeneration in certain instruction-tuned LLMs, such as ChatGPT [17], is relatively rare, leading to the hypothesis that the instruction-tuning phase, which trains LLMs on instruction-response pairs, could alleviate this issue. This conjecture implies that the utilization of instruction-tuning data is vital for mitigating degeneration. To explore this, we fine-tune the LLAMA2 model [30] using three instruction-tuning datasets: ALPACA [29], ALPACA + WT-103 50K, and WT-103 50K, as demonstrated in Table 1. The rep-2 scores for these datasets are 5.54, 9.67, and 10.31, respectively. Our experiments reveal that LLAMA2 fine-tuned on ALPACA exhibits less repetitions, while training on WT-103 50K and ALPACA + WT-103 50K still displays significant degeneration. This finding is consistent with our prior observations, where repetition issues strongly correlate with the presence of repetitions in the training data. It suggests that the low repetition rate in instruction-tuning data may contribute to the decreased degeneration.

These investigations reveal that our primary finding, namely, the degeneration issue has a strong correlation with repetitive patterns in training data, remains vital in the context of LLMs. We anticipate that this finding will offer valuable insights for understanding the degeneration issue and contribute to the development of more effective large language models.

5 Method

According to our observation in the preliminary study, we hypothesize that learning the patterns of good repetitions in natural language is a crucial factor of the degeneration issue. To better evaluate our hypothesis, we propose a simple method to control the percentage of repetitions that the model can perceive during training, namely, repetition dropout . To this end, we can directly measure the impact of the repetition in natural language by training on the full dataset.

5.1 Repetition Dropout

Inspired by dropout [23], which prevents the model from over-fitting on specific combinations of parameters, we propose the repetition dropout to avoid the model over-relying on repetitive words in the training phase. Since attention is the core component of Transformer-based LMs to perceive other context words, we apply our repetition dropout to the attention sub-layer 2 , which is defined in Eq. (2). More concretely, our method replaces the triangular matrix M in Eq. (2) by M ′ :

where M rep is the masking matrix for repetition dropout. The algorithm for generating the masking vector for each input sentence x is shown in Algorithm 1. In line 5-14 of this algorithm, we define a function find_ngrams to collect all n -grams in x . The function gen_mask_rep in line 16-26 is to randomly drop out those repetitive n -grams according a pre-specified repetition

Algorithm 1 Repetition Dropout, Python-like

dropout rate p , which is between [0 , 1] . A higher repetition dropout rate indicates that more repetitive n -grams are not accessible for the model. We concatenate the masking vectors for sentences in a batch as the M rep . To avoid the model over-fitting on a specific M rep , we generate M rep for each layer of the LM independently. It should be noted that repetition dropout is only employed in the training phase, analogous to the conventional dropout method [23].

In this method, we compel the model to predict subsequent words without depending on repetitions present in training data. Repetition dropout can be viewed as a means to regulate the quantity of repetitions within the training dataset. If learning on repetitions in training data is a vital aspect, repetition dropout will substantially mitigate the degeneration issue. We want to emphasize that this approach is designed to more effectively assess the significance of repetitions in training data.

6 Experiments

Setup To evaluate the effect of repetition dropout, we conduct empirical studies on the Wikitext-103, OpenWebText2, and FreeLaw datasets, which have been introduced in section 4. Compared with section 4, we train the GPT-2 model on the complete datasets rather than splitting them into smaller shards. Please refer to Appx. A for more training details about the GPT-2 model 3 .

The main baseline method is MLE, which indicates the vanilla GPT-2 model trained by likelihood objective, as defined in Eq. (3). The + REP-DROPOUT is the proposed method, i.e., repetition dropout. We also have several baseline methods. The first one is the + RAND-DROPOUT, which applies dropout on random tokens instead of repetitive tokens. Note that the number of random tokens selected for dropout for this baseline is constrained to the same as the repetitive tokens. We also compare with three previous works that also conducted experiments on Wikitext-103: re-encoding of high-inflow tokens (HI-RE) [8], training with scaled gradient (SCALEGRAD) [15], and unlikelihood objective at token level (UL) [34]. During inference we use greedy search as the decoding method to generate 128 tokens, using the first 32 tokens of each line in the test set as the prompt. More details about those baseline methods are also shown in the Appx. A.

6.1 Main Results

As shown in Tab. 2, we evaluate the baseline methods and our method on three datasets. On the Wikitext-103 dataset, both the MLE and MLE + RAND-DROPOUT methods suffer from the severe degeneration issue. More than 33% percent 4 -grams in the generated sentences of the two baseline methods are repetitive, which is different from the patterns in natural language. Nevertheless, MLE + REP-DROPOUT can significantly reduce the repetition issue in generated sentences. This observation indicates that repetition in the training data is crucial for the degeneration issue. Compared with methods in previous works in Tab. 2, i.e., HI-RE, SCALEGRAD, and UL, the REP-DROPOUT also show better performance on alleviating the degener-

Rep-2

☆

PPL

50

30

. Text (%)

40

24

Gen.

30

18

PPL

20

12

.n

2

10

6

R

0

0

0.0

0.2

0.4

0.6

Repetition Dropout Rate

ation issue. The results on FreeLaw and OpenWebText2 datasets are consistent with those on Wikitext-103. We also conduct quantitative and qualitative experiments to evaluate + REP-DROPOUT on larger LMs, e.g., GPT-XL[20], which are shown in Appx. B and C, respectively.

We also evaluate the effect of MLE + REP-DROPOUT with different dropout rates for the repetitions in Wikitext-103, which can be regarded as controlling the percentage of repetitions in a dataset. As shown in Fig. 3, increasing the dropout rate of our method will reduce the number of repetitions in generated sentences continuously. Moreover, we observe a clear trade-off between the rep2 score of generated sentences and the perplexity on the real data in test set. This suggests that learning on the repetitions in training data can be beneficial for language modeling. Consistent results are also shown on FreeLaw and OpenWebText2 datasets.

6.2 Relation to Previous Hypotheses

Previous research has proposed various hypotheses and approaches to understanding and solving the problem of degeneration in neural text generation. However, we argue that many of these proposals can be explained by a simple explanation. That is, penalizing repetitions in data is a common and fundamental factor for their success.

The core idea of many previous works is to penalize a specific set of data, e.g., all the prefix words x 1: t -1 at timestep t , to alleviate the degeneration. However, as shown in Fig. 4, many of them have implicit connections with the repetitions in data. For example, the set of high-inflow words ( ) penalized in Fu et al. [8] has a noticeable interaction with that of repetitive words ( ), and the set of prefix words ( ) penalized in [34] and [15] is the super-set of the repetitive words ( ). In Xu et al. [36], they directly penalize the repetitions in pseudo repetitive data ( ). In this section, we will show that repetitive words play an important role in previous works.

High-Inflow Words Some researchers [8] attribute the degeneration issue to high-inflow words, whose probability sum of all the potential preceding words is higher than a threshold. Thus, they propose that merging high-inflow word pairs can alleviate the problem (HI-RE). However, we find that 26% of high-inflow word pairs are repetitive in each sentence of Wikitext103, and merging these pairs can significantly reduce the rep2 score of real data. Therefore, we argue that merging high-inflow word pairs is actually an alternative way of reducing repetitions in training data.

We evaluate the argument by controlling a single variable, the type of high-inflow words to be

High-Inflow

Words

Rep.

Words

Prefix Words

Pseudo Rep.

Sents

merged, in line 4-6 of Tab. 3. The vanilla HI-RE method (Line 4) merges all the high-inflow words ( ), which takes 31 . 1% of the total training words. We find that the method (Line 5), which only merges repetitive high-inflow pairs, i.e., the intersection between high-inflow words and repetitive words ( ∩ ), achieves performance comparable to the original HI-RE method (Line 4). Note that the method in line 5 only merges 8 . 1% of the total training words, which is much less than the vanilla method. In contrast, the HI-RE method that merges random high-inflow pairs (Line 6), which has the same number as repetitive high-inflow words ( ∩ ), cannot alleviate the degeneration. This suggests that penalizing repetitions in data is critical in the success of Fu et al. [8].

Likelihood Objective Many researchers [11, 15, 26, 34] think that the likelihood objective is the main factor for the degeneration issue. All of those works share the same principle that words in the prefix context ( ) cause the degeneration issue. Therefore, Welleck et al. [34] and Lin et al. [15] propose to reduce the probabilities of repetitions appearing in x 1: t -1 at time step t . Su et al. [26] and Jiang et al. [11] leverage the contrastive learning to ensure that the hidden representation at time step t is distinctive to those of x 1: t -1 .

However, the attribution to the likelihood objective merely serves as a superficial explanation, and the core of this issue lies in the fact that the model inevitably learns repetitive behavior when conducting maximum likelihood estimation on repetitive data. As shown in section 6.1, the GPT-2 model with simple repetition dropout, which is also optimized by likelihood objective, achieves an extremely low rep4 score on generated text. This indicates that likelihood objective might not be the most

important factor in the degeneration issue. Nevertheless, all methods that penalize tokens in the prefix context ( ) break the reliance on repetitions ( ). Thus, we hypothesize that preventing the model from learning on repetitions is a key factor in their success.

To evaluate the impact of repetitions on these methods, we conduct experiments following the same principle used to analyze high-inflow words. We choose the SCALEGRAD [15] method as our baseline, which penalizes non-novel tokens by scaling the gradient [15]. The non-novel tokens are the entire prefix context x 1: t -1 ( ) at time step t , i.e., 100% of the training words, for the vanilla SCALEGRAD method. We also propose two variants of SCALEGRAD: the first uses the repetitive words within prefix ( ∩ ) as the non-novel tokens, while the second randomly samples a subset, which has the same number of words as the first one ( ∩ ), from the prefix data (Subset of ). The two methods only penalize 19 . 0% of the total training words. As shown in Tab. 3, the SCALEGRAD method on repetitive words (Line 8) achieves performance close to the standard SCALEGRAD method (Line 7) in terms of the repn metric. In contrast, the SCALEGRAD method on random subset (Line 9) is not effective in alleviating the degeneration issue as the the other two methods. Other works [11, 26, 34] that attribute the degeneration issue to likelihood objective also penalize tokens in prefix as the SCALEGRAD [15], but with different techniques. Thus, we think that penalizing repetitions in data is also a crucial factor for the success of these methods.

Self-reinforcement Phenomenon Both Xu et al. [36] and Chiang and Chen [4] find that degeneration is always accompanied by the self-reinforcement phenomenon, i.e., the probability of a predicted word becomes higher when it is repeated more times. Thus, they hypothesize that degeneration is (partially) caused by self-reinforcement. To mitigate this issue, [36] proposed a data-augmentation method, namely DITTO (Line 3 of Tab. 3), which constructs pseudo data by repeating a training sentence multiple times and penalizes the probabilities of repetitive tokens in the pseudo data ( ).

We argue that, as the degeneration issue, the self-reinforcement is also a by-product when neural LMs learning on repetitive patterns in real data. First, we observe a similar self-reinforcement phenomenon on the repetitive words in real data. For instance, the probabilities of the second appearances of the theme-related words are generally higher than their first appearances, as shown in Fig. 5(b). Second, we find that the model trained by repetition dropout can break the self-reinforcement loop, as shown by the examples in Appx. C. Although the text generated by GPT-2 + REP-DROPOUT on Wikitext-103 may contain a few inappropriate n -gram repetitions, it will not fall into the infinite repetition loop that frequently appear in the text generated by the vanilla GPT-2.

6.3 Why LMs Learn the Repetition Patterns?

In the last empirical study, we make an attempt to investigate the reasons behind LMs learning repetition patterns, specifically the role these repetitions play in neural LMs. To address this inquiry, we examine 300 randomly selected instances, each featuring repetitive bi-grams within a 256-word sentence. These cases are broadly categorized into three groups, as per Altmann and Köhler [1] and Tannen [28]:

Grammar : Repetitions for grammatical purposes, such as determiners, conjunctions, etc.

Theme : Repetitions closely associated with the subject matter of the text.

Limited inventory : Repetitions resulting from a language's restricted means of expressing a particular concept, leading to high-frequency occurrences, e.g., the phrase "pair of" in English.

Percentage (%)

0

10

20

30

40

50

Theme

Ltd. Inventory

(a)

AVG. Probability

0

0.1

0.2

0.3

0.4

0.5

Theme

Ltd. Inventory

w/o mask

w/ mask

(b)

We assign each case to the earliest category it satisfies if it meets the criteria for multiple categories. Detailed guidelines for human evaluators to classify repetitive words can be found in Appx. D. As demonstrated in Fig. 5(a), almost 50% of the repetitions in Wikitext-103 fulfill grammatical functions, while both theme-related and inventory-related repetitions constitute around 25% each.

To understand the impact of different types of repetitions on a vanilla neural LM, we measure the probability change of a prediction when masking the information of its repetitions in the context through attention mechanism. For example, given the human repetitions in Fig. 1, we will determine how the probability of the last appearance of 'generals wore...' changes when masking its first appearance of 'generals wore...' in the context.

As shown in Fig. 5(b), masking the theme-related repetitions will cause a noticeable drop of the prediction probabilities. However, the prediction of grammatical and inventory-related repetitions are not affected by masking their previous appearances in context. This observation indicates that neural LMs spend its effort on optimizing the prediction accuracy of those theme-related words by implicitly repeating words in context.

7 Conclusion & Limitations

In this work, we find that the repetition in training data is a fundamental factor for the degeneration (or repetition) problem. First, training data is an integral part of most AI systems, and our preliminary study demonstrates a strong correlation between the repetitions in training data and the degeneration issue. Second, we find that simply dropping out the attention to repetitions in training data can significantly reduce the degeneration issue, which is more effective than other baselines. Finally, we conduct extensive empirical analyses to demonstrate that penalizing repetitions in data is the key success factor for many previous works, such as those attribute degeneration issue to high-inflow words, the likelihood objective, and the self-reinforcement mechanism. Experiments also show that our findings are critical even in the context of large language models. We hope that the viewpoint we provide to understanding the degeneration can inspire more principled research in the future.

Our work also has some limitations. First, despite the excellent performance on reducing repetitions, the repetition dropout method may hurt the perplexity of the language model. Second, we examine our work mostly on standard benchmark of the language generation task, following previous works [8, 10, 26, 34]. It also deserves to extend our work to large-scale data and model.

References

Gabriel Altmann and Reinhard Köhler. 2015. Forms and degrees of repetition in texts: detection and analysis , volume 68. Walter de Gruyter GmbH & Co KG.

Alexei Baevski and Michael Auli. 2019. Adaptive input representations for neural language modeling. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings .

Ting-Rui Chiang and Yun-Nung Chen. 2021. Relating neural text degeneration to exposure bias. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP , pages 228-239, Punta Cana, Dominican Republic. Association for Computational Linguistics.

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2978-2988, Florence, Italy. Association for Computational Linguistics.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314 .

Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 889-898, Melbourne, Australia. Association for Computational Linguistics.

Zihao Fu, Wai Lam, Anthony Man-Cho So, and Bei Shi. 2021. A theoretical analysis of the repetition problem in text generation. In Thirty-Fifth AAAI Conference on Artificial Intelligence .

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 .

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net.

Shaojie Jiang, Ruqing Zhang, Svitlana Vakulenko, and Maarten de Rijke. 2022. A simple contrastive learning objective for alleviating neural text degeneration. arXiv preprint arXiv:2205.02517 .

Toshiki Kawamoto, Hidetaka Kamigaito, Kotaro Funakoshi, and Manabu Okumura. 2022. Generating repetitions with appropriate repeated words. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 852-859, Seattle, United States. Association for Computational Linguistics.

Huayang Li, Deng Cai, Jin Xu, and Taro Watanabe. 2022. Residual learning of neural text generation with n-gram language model. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 1523-1533, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2022. Contrastive decoding: Open-ended text generation as optimization. CoRR , abs/2210.15097.

Xiang Lin, Simeng Han, and Shafiq Joty. 2021. Straight to the gradient: Learning to use novel tokens for neural text generation. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 6642-6653. PMLR.

Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. 2023. Locally Typical Sampling. Transactions of the Association for Computational Linguistics , 11:102-121.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc.

Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Measuring the gap between neural text and human text using divergence frontiers. In Advances in Neural Information Processing Systems , volume 34, pages 4816-4828. Curran Associates, Inc.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.

Darcey Riley and David Chiang. 2022. A continuum of generation tasks for investigating length bias and degenerate repetition. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP , pages 426-440, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1073-1083.

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research , 15(56):1929-1958.

Yixuan Su, Deng Cai, Yan Wang, David Vandyke, Simon Baker, Piji Li, and Nigel Collier. 2021. Non-autoregressive text generation with pre-trained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , pages 234-243, Online. Association for Computational Linguistics.

Yixuan Su and Nigel Collier. 2023. Contrastive search is what you need for neural text generation. Transactions on Machine Learning Research .

Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. In Advances in Neural Information Processing Systems .

Yixuan Su and Jialu Xu. 2022. An empirical study on contrastive search and contrastive decoding for open-ended text generation. arXiv preprint arXiv:2211.10797 .

Deborah Tannen. 1987. Repetition in conversation: Toward a poetics of talk. Language , pages 574-605.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instructionfollowing model. Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html , 3(6):7.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 5998-6008.

Jesse Vig. 2018. Deconstructing bert: Distilling 6 patterns from 100 million parameters. Medium, December .

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems , 35:24824-24837.

Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. pages 38-45. Association for Computational Linguistics.

Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. 2022. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation. In Advances in Neural Information Processing Systems .

Haoran Yang, Deng Cai, Huayang Li, Wei Bi, Wai Lam, and Shuming Shi. 2023. A frustratingly simple decoding method for neural text generation. arXiv preprint arXiv:2305.12675 .

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 .

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages 2979-2989, Copenhagen, Denmark. Association for Computational Linguistics.

A Implementation Details

A.1 Preliminary Study

The basic GPT-2 model 4 is trained from scratch on each corpus, which has 12 transformer blocks and 12 attention heads with 768 hidden dimensions. The Huggingface transformers [35] and Pytorch toolkit [18] are used to train the GPT-2 model in the distributed manner on A100 GPU server. The hyper-parameters during training are shown in Tab. 4.

A.1.1 Our Method

Most of the hyper-parameters for our proposed method are the same as that in Tab. 4 for better variable controlling. The specific hyper-parameters for our proposed method are the length of repetitive n -gram and its repetition dropout rate p , which are set as 2 and 0.6, respectively.

A.1.2 Baselines

In this subsection, the specific hyper-parameters for three baselines are described, and most of the hyper-parameters are the same as that in Tab. 5.

A.2 Instruction Tuning

To evaluate the effect of instruction tuning, we conduct experiments on three datasets:

ALPACA: The instruction-tuning dataset used by Alpaca [29].

WT-103 50K: We randomly sample 50k non-title sentences from Wikitext-103 and convert them to the instruction-tuning format, following [29].

ALPACA + WT-103 50K: The mixture of both.

We use the QLoRA [6] to fine-tune the LLAMA2-7B model [30], due to the limited computational resources. The decoding method for generating text is greedy search. We use the test set of Wikitext103, which is also converted to the instruction-tuning format, to evaluate the performance of different models. Below is the template used for converting Wikitext-103 to the instruction-tuning format:

B Additional Experiments

In addition to the repetition measurements (Tab. 2), we also evaluate our method on additional metrics, e.g., MAUVE [19]. As shown in Tab. 6, most of the performances of our method and baseline methods are consistent with the observation in Tab. 2.

In addition to GPT-2 model, we also conducted experiments on model with larger size. We directly apply our repetition dropout method to the GPT-XL model, which has 1.5 billion parameters. Since it is difficult to train GPT-XL model 5 from scratch, we fine-tuned it on Wikitext-103 for 3 epochs. Results in Tab. 7 demonstrate that our method can also alleviate the degeneration of LLMs after fine-tuning on larger model.

C Qualitative Analysis

In Fig. 6, we show some generated results of the GPT-XL model and GPT-XL+REP-DROPOUT. The training and inference settings of the two models are described in Appx. B. We can observe that the repetitions in generated text were reduced significantly after employing the REP-DROPOUT method. In addition, the GPT-XL trained by MLE easily falls into a sentence-level repetition, because of the self-reinforcement mechanism [36]. Nevertheless, the GPT-XL+REP-DROPOUT does not suffer from this issue.

Input Case 1:

Traditional Chinese literary criticism emphasized the life of the author when interpreting a work, a practice which Burton Watson attributes to " the close links that traditional Chinese thought posits

GPT-XL:

between the author and the reader ". The author's life was also a central theme in the works of the early modernist poets, such as the poets of the Ming dynasty, who were influenced by the writings of the early modernist poets. The early modernist poets were also influenced by the writings of the early modernist poets, such as the poets of the Song dynasty, who were influenced by the writings of the early modernist poets. The early modernist poets were also influenced by the writings of the early modernist poets, such as the poets of the Ming dynasty, who were influenced by the writings of the early modernist…

GPT-XL + Rep-Dropout:

between the individual and the collective ". The work was also a source of inspiration for the later works of Chinese @-@ American authors such as John W. Campbell, who wrote in his memoirs that he " learned much from [ the ] book ". In the early 20th century, the Chinese scholar and translator Wang Hui ( 1879 - 1940 ) wrote that the book " is the most important work of Chinese literature ". Burton's translation of the text was published in 1891, and was translated into English by the American scholar and translator John W. D. Rockefeller in 1904. It was the first English translation of…

Input Case 2:

In May 1951, he announced he was retiring from professional football to become sports director at a radio station in Buffalo. He was hired as a sportscaster by

GPT-XL:

WLUP @-@ AM in Buffalo, and was promoted to the position of sports director in 1952. He was also a radio announcer for the Buffalo Bills and the Buffalo Braves. In 1953, he was named the radio play @-@ by @-@ play announcer for the Buffalo Bills. He was also a color commentator for the Bills. In 1954, he was named the radio play @-@ by @-@ play announcer for the Buffalo Braves. He was also a color commentator for the Braves. In 1955, he was named the radio play @-@ by @-@ play announcer for the Buffalo Bills. He was…

GPT-XL + Rep-Dropout:

WLUP @-@ FM in Buffalo, New York, in 1952. In 1953, he joined the CBS radio network as a color commentator. His first assignment was to call the NFL championship game between the New York Giants and the Chicago Bears. The Giants won the game, 24 - 0. After the game, he was asked by a reporter if he had ever seen a game like that. " I have, " he replied. " I've seen a lot of football. " He was also a color commentator for the CBS television series The NFL in 1954. During the 1950s, the NFL was the dominant league in America…

Figure 6: Example of generated text. The input text is in bleu, while the repetitions are in red. Both of the two models are fine-tuned on the Wikitext-103 for 3 epochs, as described in section B. We use greedy search for decoding.

D Classification of Repetition Words

We categorize repetitions into three groups, as outlined by Altmann and Köhler [1] and Tannen [28]: grammar , theme , and limited inventory . For each sampled instance, we initially determine whether the repetitive n -gram falls under the grammar category, meaning any word in the n -gram is a determiner, preposition, conjunction, etc. Next, if the repetitive n -gram does not belong to the grammar category, we assess whether any words are closely related to the text's subject matter, thereby placing it in the theme category. For instance, "H. gammarus" is considered part of the theme category when repetitively used in an article about Homarus gammarus. The third category encompasses repetitions stemming from a language's limited means of expressing a specific concept, known as limited inventory. Popular phrases such as "pair of" are examples of repetitive n -grams in this category.

In cases where multiple repetitive n -grams appear within a 256-word sentence, we only take one into account. If a repetitive n -gram satisfies the criteria for more than one category, particularly theme and limited inventory, we allocate it to the earliest applicable category.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the cause of neural text degeneration, i.e. language models tend to generate repetitive loops. They design an experiment showing that text degeneration is correlated with the amount of repetitive text in the training data. Motivated by this finding, they propose repetitive dropout, which applies dropout on the attention weights over repetitive context. They experiment with 3 datasets and demonstrate that their proposed method can greatly reduce repetition. They also re-inspect previous hypotheses:\n\n1. high in-flow words: they show that merging repetitive words contributes to a large portion of the effect of merging high in-flow words.\n2. the maximum likelihood objective: they show that even when the model is trained with the maximum likelihood objective, as long as repetitions are penalized, the model does not degenerate much. Thus, the maximum likelihood objective is not the cause of neural text degeneration.\n3. self-reinforcement of repetition: the argue that\n    a. It is also caused by the repetitive words in the training data.\n    b. The self-reinforcement loops are broken by their proposed method.\n \nFinally, they categorize repetitions into three categories and study their frequency along with their effect on models’ degeneration behavior. They find that the *theme* category has the greatest effect.\n\nSTRENGTHS:\n1. Most parts of the paper are easy to follow.\n2. Neural text degeneration has been discovered since 2019, but its cause is still unclear. Finding the root cause may interest audiences in this field.\n3. Despite the hypothesis being simple, as far as I know, it hasn’t been studied. The experiment design in Section 4 is clever.\n4. They proposed a simple method that can reduce repetition effectively.\n5. They inspected a few previous hypotheses and the results are aligned with their hypothesis (in general).\n6. They inspected the effect of three types of repetitions in the training data, which in my opinion is insightful.\n\nIn sum, in my opinion, their analyses of the cause of neural text degeneration is comprehensive and the conclusion is convincing.\n\nWEAKNESSES:\nIn my opinion, because the main message of this paper is clear enough, the following issues may not be very crucial.\n\n1.  The authors only compare their method with one baseline in Table 1. There are other mitigations for text degeneration, e.g. [14, 22, 11] cited in the paper. \n2. Section 6.2 is relatively hard to follow. \n    1. Line 248, “The core idea of many previous works is to penalize a specific set of data to alleviate the degeneration.” is vague, though I can roughly figure out what it means after reading the following parts.\n    2. Having a brief introduction for the definition of high-inflow words will make the paper more self-contained.\n    3. For the Likelihood Objective part, the paragraph at line 283 needs a clearer topic sentence. The connection between the experiment design in paragraph at line 288 and the argument in the previous paragraph is not very clear to me either.\n3. At Line 312, the author mentioned “we find that the model trained by repetition dropout can break the self-reinforcement loop”. Some evidence should be provided.\n4. At Line 223, “we hypothesize that the exposure bias issue is an important factor for the left repetition, because we find that lower-order n -gram2…”, I couldn't understand the explanation. Some details should be provided too.\nrepetitions generally appear earlier than higher-order repetitions\n4. Given the popularity of large language models, the study on GPT-2-sized models may be less useful for the general audience.\n\nQUESTIONS:\n1. If repetition in the training data is the cause of degeneration, why not simply remove those repetitive text from the training data? This seems to me a more straightforward method than applying attention dropout.\n2. If the authors promise to address the issues in the weakness section above, I would be happy to increase my score, because I think the main message is clear and interesting enough.\n3. Having some analyses on large language models would make this work more impactful, e.g.\n    1.  If I understood correctly, the analyses in Section 6.3 can also be done for LLMs.\n    2.  I also wonder what would be an effective way to fine-tune an existing (large) model so it doesn't generate repetitive text.\n\nLIMITATIONS:\nLimitations are discussed in Section 7.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors demonstrate that repetition in the training data is a major cause of the neural text repetition problem. They first show a strong correlation between the repetition ratio of training data and generated text. Based on the observation, they propose repetition dropout to prohibit the model from learning repetition in data during training. Experimental results show that repetition dropout significantly addresses the repetition problem compared to previous approaches and penalizing repetition in training data is a key factor for reducing the problem. In analysis, the authors demonstrate that their method specifically reduces repetition caused by subject matter rather than grammar or highly frequent phrases.\n\nSTRENGTHS:\n- The proposed method and supporting experiments are well-motivated and the findings are interesting.\n- They significantly reduce the neural text repetition problem compared to previous work.\n\nWEAKNESSES:\n- To demonstrate that the distribution of generated texts is close to that of human texts, authors could further utilize metrics such as MAUVE [1].\n- Experiments on various parameter sizes would be beneficial for further understanding such as \"Does increasing model size enhance the robustness to the repetition problem\" or \"How are the previous and proposed methods effective as the model size increases?\"\n- A case study on samples generated from the models would provide further insights.\n\n[1] Pillutla, Krishna, et al. \"Mauve: Measuring the gap between neural text and human text using divergence frontiers.\" Advances in Neural Information Processing Systems 34 (2021): 4816-4828.\n\nQUESTIONS:\n- Is there a possible reason for a relatively high repetition ratio in OpenWebText2?\n\nLIMITATIONS:\nAs written in the Limitation section, experiments on various scales could be conducted in future work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the issue of degeneration in text generation, which refers to the generation of repetitive words and dull loops by neural language models. The authors focus on the impact of repetition in the training data and propose a method to address this issue. Specifically, they suggest dropping out repetitive words during the training data pre-processing stage. The main conclusion drawn from their investigation is that penalizing repetitive n-grams in the training data is crucial for the effectiveness of existing methods in preventing degeneration. By highlighting the importance of addressing repetition in the training data, the paper contributes to understanding the factors influencing degeneration in text generation.\n\nSTRENGTHS:\nThe paper is generally well-written and is easy to follow. The problem of generating repetitive words and dull loops has been widely observed and extensively discussed in the field, which makes this problem important and worth exploring. The conclusion drawn by the authors, that training data plays a pivotal role in the occurrence of degeneration, is logical and well-supported.\n\nWEAKNESSES:\nThe paper primarily focuses on data pre-processing rather than novel training techniques. This limited scope may diminish its technical novelty. Many researchers working on text generation may have already explored similar data pre-processing ideas, making the proposed method less impactful or original. Also, the conclusion of the paper lacks novel insights and seems to affirm an intuitive understanding. While it is reasonable to assume that data quality has a direct impact on degeneration, this insight does not offer any new or surprising findings. As a result, the conclusion may be considered weak in terms of offering novel contributions or expanding the current understanding of the issue.\n\nExplore additional factors that could contribute to degeneration in text generation, such as the number of training data and model size. Investigating these factors may provide deeper insights into the phenomenon and allow for a more comprehensive understanding of degeneration. This expansion could strengthen the paper and enhance its impact within the field.\n\nFor me, it would be more interesting to study other factors that impact degeneration, such as the number of training data, the model size and the model architecture. Investigating these factors may provide deeper insights into the phenomenon and allow for a more comprehensive understanding of degeneration. With more training data and more parameters, large language models can capture semantics and thus prevent degeneration. For example, degeneration has been greatly resolved for ChatGPT due to its expressiveness and more general understanding of texts,  In my opinion, tackling degeneration in data pre-processing is not very meaningful because the root cause is still the model lacks of capability to understand and generate reasonable texts.\n\nQUESTIONS:\nDoes dropping some repetitive n-grams sometimes distort the semantics of the document?\n\nLIMITATIONS:\nThere are other potential factors that can impact the degeneration. The paper should have a more comprehensive discussion about all the factors.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper explains the repetition in model generated text from a data standpoint, pointing out that there is a strong correlation between the degeneration issue and the presence of repetitions in training data. The authors find out that penalizing repetitions in data can alleviate degeneration, and propose a method, repetition dropout, that is to apply dropout in transformers’ sublayers during training time. The proposed method achieve significant improvement in terms of REP-n scores.

STRENGTHS:
- This is a very well written paper! The preliminary study does a great job in introducing and clearly motivating the problem.
- The proposed repetition dropout is also a very simple technique, yet still improve the performance in REP-n stats.
- The comparison between different methods and objectives in section 6 is very interesting and thorough.

WEAKNESSES:
Currently it seems that the repetition dropout mask is applied on each instance, but what about repeated text in different instance? How should we apply it in current LM training paradigm?

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-09