SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Huayang Li $^{\text{心}}$  Tian Lan $^{\text{♣}}$  Zihao Fu $^{\text{♣}}$  Deng Cai $^{\text{♣}}$
Lemao Liu\* Nigel Collier\* Taro Watanabe Yixuan Su\*,
Nara Institute of Science and Technology Tencnt AI Lab
$\spadesuit$ University of Cambridge  $\diamond$ Cohere
{li.huayang.lh6, taro}@is.naist.jp lantiangmftby@gmail.com {jcykcai, redmondliu}@tencent.com {zf268, nhc30, ys484}@cam.ac.uk
There are a number of diverging hypotheses about the neural text degeneration problem, i.e., generating repetitive and dull loops, which makes this problem both interesting and confusing. In this work, we aim to advance our understanding by presenting a straightforward and fundamental explanation from the data perspective. Our preliminary investigation reveals a strong correlation between the degeneration issue and the presence of repetitions in training data. Subsequent experiments also demonstrate that by selectively dropping out the attention to repetitive words in training data, degeneration can be significantly minimized. Furthermore, our empirical analysis illustrates that prior works addressing the degeneration issue from various standpoints, such as the high-inflow words, the likelihood objective, and the self-reinforcement phenomenon, can be interpreted by one simple explanation. That is, penalizing the repetitions in training data is a common and fundamental factor for their effectiveness. Moreover, our experiments reveal that penalizing the repetitions in training data remains critical even when considering larger model sizes and instruction tuning. Our code is available at https://github.com/gmftbyGMFTBY/Rep-Dropout.
The emergence of neural language models (LM) has led to significant achievements in various text generation tasks, such as machine translation [3, 24, 31], summarization [22], and open-ended text generation [17, 20]. However, in open-ended text generation, neural LMs exhibit a strikingly severe degeneration issue, producing unreasonably repetitive texts, particularly when employing maximum a posteriori (MAP) decoding algorithms. As illustrated in Fig. 1, even a well-trained LM [20] may suffer from a severe degeneration issue.
There have been numerous attempts to explain the phenomenon of neural text degeneration, with many attributing this problem to flaws in the learning process. Fu et al. [8] claim that high-inflow words increase the probability of generating repetitions. A collection of studies [11, 15, 26, 34] argue that the likelihood objective is the primary factor, because it has the problem of exposure bias and focuses more on next-token prediction rather than sequence generation. Meanwhile, both Chiang and Chen [4] and Xu et al. [36] assert that the self-reinforcement mechanism can make neural LMs fall into the repetitive loops. Intriguingly, despite the divergence in these explanations, all
It was not possible to know which grade of general an officer was by his collar insignia. However, major generals and lieutenant generals wore two rows of nine buttons in groups of three down the of the overcoat, and brigadier generals wore two rows of eight buttons in groups of two.
It was not possible to know which grade of general an officer would be required to be given to a particular officer. The Army was not aware of the Army's role in the conflict, and the Army was not aware of the Army's role in the conflict. The Army's role in the conflict was limited to the training of
Figure 1: Illustration of repetitions in human and generated text. The human text is from Wikitext-103, and generated text is by greedy search using the GPT-2 model trained on Wikitext-103. The underlined text is the prompt for generation. The blue words and red words indicate the repetitions in human text and machine-generated text, respectively.
corresponding methods proposed have been shown to effectively alleviate the text degeneration issue. This observation leads us to ponder: could there exist more fundamental factors that can explain the degeneration issue?
In this study, we strive to provide a straightforward and fundamental interpretation for the degeneration problem from the data perspective. Our preliminary investigation reveals that repetitive words in the training data play a crucial role in the issue. It is also worth noting that repetitions are not necessarily of low quality, because it is a natural and common phenomenon in human writing [1, 12, 28]. Across five datasets with different domains, we observe a strong correlation between repetition rates in training and generated text. This finding encourages us to further assess the impact of repetitive words in training data by randomly dropping out attention to them during training. Employing repetition dropout, we discover that the repetition rate in generated text can be substantially reduced. Lastly, we reconcile many previous hypotheses with our single explanation, asserting that penalizing repetitions in training data is a key factor to the success of alleviating the degeneration issue.
As large language models (LLMs) gain popularity, it appears that the degeneration issue has been somewhat solved. We investigate the impact of various factors associated with LLMs on degeneration, including increasing model size and training models using instruction-tuning data. Our experiments reveal that penalizing repetitions in training data continues to be crucial in the context of LLMs.
Our contributions are threefold:
- We demonstrate that the proportion of repetitive words in training data has a significant influence on the degeneration issue. Inspired by this finding, we also propose a method, namely, repetition dropout, to mitigate the degeneration.
- We find that penalizing repetitions in data is a more fundamental factor for reducing repetitions, which also provides a unified explanation for various existing hypotheses, such as the attributions to high-inflow words, likelihood objectives, and self-reinforcement phenomenon.
- We investigate the influence of factors associated with large language models on reducing repetitions, including model scale and instruction tuning.
The degeneration (or repetition) issue in neural language models (LM), particularly in open-ended text generation, has garnered significant attention in recent years. Previous works have proposed disparate interpretations and solutions for this issue.
Many researchers hypothesize that factors within the learning process contributes to the degeneration issue. Fu et al. [8] assert that the high-inflow words in training data may increase the probability of generating repetitions, where the inflow of a word is defined as the probability sum of all its preceding words. Another factor identified is the likelihood objective. Welleck et al. [34] argue that likelihood objective has a discrepancy with the MAP decoding and focuses more on next-token prediction rather than sequence generation. Thus, they proposed an unlikelihood objective to address the two flaws. Based on the same principle, Lin et al. [15] propose to scale the gradient of specified non-novel tokens, i.e., words in the prefix context at each time step, to alleviate the degeneration issue. Su et al. [26] and Jiang et al. [11] pointed out that the degeneration issue may caused by the high similarity between token representations. Thus, they leveraged contrastive learning to learn a more distinct
representation for each token. In addition, both Xu et al. [36] and Chiang and Chen [4] argue that degeneration is caused by the self-reinforcement phenomenon and Xu et al. [36] address this issue by penalizing the repetitions in pseudo repetitive data.
Apart from issues within the learning process, other explanations for degeneration have been proposed. Many researchers contend that decoding methods [7, 10, 16, 20, 37] is the primary factor. Holtzman et al. [10] argue that word probabilities in human-generated text exhibit high variance and randomness, while high-probability text produced by MAP decoding methods tends to be repetitive and dull. This observation explains why sampling-based decoding methods [7, 10, 16, 20] can substantially mitigate the degeneration issue. Riley and Chiang [21] find that tasks with lower constraints, or larger solution spaces, suffer from more severe degeneration issue. The model architecture [8, 32, 34] and size [14] may also contribute, but the two factors have not been quantitatively evaluated.
However, with so many explanations, understanding the primary cause of the degeneration issue becomes increasingly challenging. In our work, we strive to provide a fundamental explanation for the previous hypotheses in the learning process. While our work may not encompass all the existing explanations, we believe it lays a solid foundation for further understanding of the degeneration issue.
Language model (LM) aims to estimate the probability of a sentence in natural language according to the chain rule of probability:
$$
\begin{array}{l} P (\boldsymbol {x}) = P (x _ {1}) P (x _ {2} | x _ {1}) \dots P (x _ {L} | \boldsymbol {x} _ {1: L - 1}) \\ = P \left(x _ {1}\right) \prod_ {i = 2} ^ {L} P \left(x _ {i} \mid \boldsymbol {x} _ {1: i - 1}\right), \tag {1} \\ \end{array}
$$
where  $\pmb{x} = <x_{1},\dots,x_{L}>$  is a sequence of words with length  $L$ , and  $x_{1:i-1}$  is the previous  $i-1$  words, i.e., the context, for predicting word  $x_{i}$ .
Model Since attention-based LM [20, 31] has become the backbone of many tasks, we will use GPT-2 model [20] as the main architecture for our empirical studies. The core of GPT-2 model is to use the attention mechanism to update the representation of words in  $\mathbf{x}$ :
$$
\operatorname {A t t e n t i o n} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) = \operatorname {s o f t m a x} \left(\frac {\mathbf {Q} \mathbf {K} ^ {T} + \mathbf {M}}{\beta}\right) \mathbf {V}, \tag {2}
$$
where  $\beta$  is a scalar to control the scale of the attention score,  $\mathbf{Q}$ ,  $\mathbf{K}$ , and  $\mathbf{V}$  are the linear transformation of  $h_{1:L} \in \mathbb{R}^{L \times d}$  using matrices  $\mathbf{W}_q$ ,  $\mathbf{W}_k$ , and  $\mathbf{W}_v \in \mathbb{R}^{d \times d}$ , respectively.  $h_{1:L}$  is the current hidden representation of words  $x_{1:L}$ , and  $d$  is the hidden size.  $\mathbf{M}$  is masking matrix that makes sure only the information of  $h_{1:i}$  is accessible for  $h_i$  at each time step  $i$ . If the word  $x_i$  can perceive the information of  $x_j$ , then  $\mathbf{M}[i][j]$  equals to 0, otherwise  $-\infty$ .
Training Generally, neural LMs are trained by optimizing the likelihood objective:
$$
\begin{array}{l} \mathcal {L} = - \sum_ {\boldsymbol {x} \in \mathcal {D}} \sum_ {i = 2} ^ {L} \log P (x _ {i} | \boldsymbol {x} _ {1: i - 1}; \theta) \\ = - \sum_ {\boldsymbol {x} \in \mathcal {D}} \sum_ {i = 2} ^ {L} \log \operatorname {s o f t m a x} \left(\phi \left(\mathbf {H} _ {i - 1}\right)\right) \left[ x _ {i} \right], \tag {3} \\ \end{array}
$$
where  $\pmb{h}_{i-1} \in \mathbb{R}^d$  is the hidden vector of  $x_{i-1}$  output by the last layer of an neural LM, e.g., the GPT-2 model [20]. The  $[x_i]$  is defined as taking the probability regarding to  $x_i$  in the distribution got from softmax. The  $\phi(\cdot)$  is a linear layer that transforms the  $\pmb{h}_{i-1}$  to logits.
$^{1}$ GPT-2 also incorporates essential sub-layers such as residual connections and layer normalization. While the model employs a multi-head attention mechanism, we have omitted details for brevity. For a comprehensive explanation of these sub-layers, refer to Vaswani et al. [31] and Radford et al. [20].
(b)
Figure 2: (a) Relationship between rep-2 scores of human and generated text. (b) Relationship between the scale of an LM and the rep-2 score of generated text. Note that results with the same symbol in Fig. 2(a) are from models trained on different shards of the corresponding dataset. The rep-2 score is defined in Eq. (4). We use the GPT-2 and OPT LMs for Fig. 2(a) and 2(b), respectively, and use greedy search as the decoding method.
Inference Since neural LMs are trained to maximize the likelihood objective, one intuitive practice for inference is to use the MAP decoding method, e.g., greedy search or beam search. However, in open-ended text generation, this tactic will cause an extremely severe degeneration issue on vanilla neural LMs trained by likelihood objective [10].
Evaluation The main focus of this empirical study is to investigate the reason for the repetition issue. Therefore, the rep-  $n$  is an important evaluation metric in our work, following previous works [8, 25, 27, 34]:
$$
\operatorname {r e p} - n = 1. 0 - \frac {\left| \text {U n i q u e N g r a m s} (\boldsymbol {x} , n) \right|}{L - n + 1} \tag {4}
$$
where  $n$  is the length of  $n$ -gram, and UniqueNgrams is a function to find all unique  $n$ -grams in a sentence  $\pmb{x}$ . For the corpus-level evaluation, we report the averaged rep- $n$  scores of instances in the dataset. To ensure that our results in main experiments are not biased to rep- $n$ , we also report the results of rep- $w$  and rep- $r$  in previous works [8, 34]. The rep- $w = \frac{1}{L} \sum_{t=1}^{L} \mathbb{1}\{x_t \in \pmb{x}_{t-w-1:t-1}\}$ , which measures the word-level repetition in a prefix window with length  $w$ . In our experiments, we set  $w$  to 16, following Fu et al. [8]. The rep- $r = \frac{1}{L} |\{i| (x_i = x_j \land x_{i+1} = x_{j+1}, \exists j \neq i) \vee (x_i = x_k \land x_{i-1} = x_{k-1}, \exists k \neq i)\}|$ . It is for the portion of repetitive snippets measured by sentence length.
In addition to the measurement of repetition, we also consider the perplexity (PPL) on the real data, which demonstrates the performance in language modeling [2]. Although LMs with lower PPL may not consistently lead to better generation results, it is able to reflect trivial solutions for the degeneration issue, e.g., random or over-fitting models.
4 Preliminary Study: Rethinking Neural Text Degeneration from Data Perspective
The propensity of neural LMs with impressive performance to fall into naive repetitive loops is puzzling. Although numerous hypotheses have been proposed, many of them approach this issue from divergent aspects, and some are not intuitive for understanding. According to Ockham's Razor, a simpler explanation is often preferable. Therefore, in our preliminary study, we start by evaluating one elementary factor for most AI systems, the training data. Concretely, repetition is a natural and common phenomenon in human languages for various reasons [1, 12, 28]. It is intriguing to investigate whether there are connections between valid repetitions in natural language and incorrect repetitions in generated language. Moreover, it is important to note that data containing repetitions is not necessarily of low quality, as shown in Fig. 1.
#	Model	Rep-2 of FT data(%)	Rep-2(%)	Rep-3(%)	Rep-4(%)
1	LLAMA2 w/o FT	-	47.79	41.97	38.52
2	FT LLAMA2 ON ALPACA	5.54	15.08	10.91	8.93
3	FT LLAMA2 ON ALPACA + WT-103 50K	9.67	41.63	35.64	32.29
4	FT LLAMA2 ON WT-103 50K	10.31	54.10	49.77	36.80
Table 1: Results of LLAMA2-7B on instruction-tuning data. The "FT" means fine-tuning. The column "Rep-2 of FT Data" indicates the rep-2 score of the training data. The rest Rep- $n$  scores are evaluated on the generated text. The ALPACA is the instruction-tuning dataset used in [29], "WT-103  $50\mathrm{K}$ " is the instruction-tuning dataset we constructed based on Wikitext-103 (Appx. A.2), and "ALPACA + WT-103  $50\mathrm{K}$ " is the mixture of both.
Setup To assess the correlation between repetitions in generated text and those in human text, we propose to train GPT-2 models [20] on data with varying rep-2 scores and then evaluate the rep-2 scores of the text generated by the corresponding models. Specifically, we sorted the training instances in each dataset  $\mathcal{D}$  based on their rep-2 scores. We then divided the sorted training data into six shards, each containing an equal number of words but a varying percentage of repetitions, and trained a GPT-2 model on each shard. Notably, we will use the full test set of a dataset  $\mathcal{D}$  to evaluate the models trained on different shards of  $\mathcal{D}$ . More implementation details are in Appx. A.
Our preliminary study investigates five datasets across various domains. Wikitext-103 is a widely used dataset for language modeling [2, 5, 13] and open-ended generation [8, 34]. We adopt the standard split for training, validation, and test sets. The remaining four datasets, OpenWebText2, FreeLaw, PubMed, and ArXiv, are part of the Pile dataset [9]. To ensure consistent analysis, we sample an equivalent number of words as the Wikitext-103 dataset from each of the four Pile datasets. For the validation and test sets, we randomly sample 2,000 sentences from each of the four Pile datasets. The training, validation, and test sets are non-overlapping across all five datasets.
Findings Fig. 2(a) demonstrates a strong correlation between the rep-2 scores of human and generated text on each dataset, indicating that the degeneration issue becomes more severe as the percentage of repetitions in training data increases. However, due to varying data distributions across datasets from different domains, the rep-2 score of generated text may vary even when trained on data with the same rep-2 score. Another intriguing observation is that neural LM will amplify the repetition in data by more than 10 times, similar to the bias amplification found in Zhao et al. [39].
Investigations Related to Large Language Models As LLMs, e.g., ChatGPT [17] and LLAMA[30], gain popularity, it appears that the degeneration issue has been somewhat solved. In this section, we will investigate the impact of various factors associated with LLMs on degeneration, including increasing model size and training models with instruction-tuning data.
Many amazing model abilities emerge when scaling up the model and data size [33]. An interesting question is, whether the degeneration issue will be solved by simply scaling up? We use a set of OPT models [38] with different model sizes to investigate this question. As shown in Fig. 2(b), the rep-2 score of generated text sharply drops before increasing the model size to 6.7 billion parameters, indicating that increasing the model size does alleviate the repetition issue to some extent. However, the gains achieved by increasing the model size diminish over time. The OPT-66B model still generates text with high rep-2 score. This observation shows that increasing the model size is not an efficient way to alleviate the degeneration.
Degeneration in certain instruction-tuned LLMs, such as ChatGPT [17], is relatively rare, leading to the hypothesis that the instruction-tuning phase, which trains LLMs on instruction-response pairs, could alleviate this issue. This conjecture implies that the utilization of instruction-tuning data is vital for mitigating degeneration. To explore this, we fine-tune the LLAMA2 model [30] using three instruction-tuning datasets: ALPACA [29], ALPACA + WT-103 50K, and WT-103 50K, as demonstrated in Table 1. The rep-2 scores for these datasets are 5.54, 9.67, and 10.31, respectively. Our experiments reveal that LLAMA2 fine-tuned on ALPACA exhibits less repetitions, while training on WT-103 50K and ALPACA + WT-103 50K still displays significant degeneration. This finding is consistent with our prior observations, where repetition issues strongly correlate with the presence of repetitions in the training data. It suggests that the low repetition rate in instruction-tuning data may contribute to the decreased degeneration.
These investigations reveal that our primary finding, namely, the degeneration issue has a strong correlation with repetitive patterns in training data, remains vital in the context of LLMs. We anticipate that this finding will offer valuable insights for understanding the degeneration issue and contribute to the development of more effective large language models.
According to our observation in the preliminary study, we hypothesize that learning the patterns of good repetitions in natural language is a crucial factor of the degeneration issue. To better evaluate our hypothesis, we propose a simple method to control the percentage of repetitions that the model can perceive during training, namely, repetition dropout. To this end, we can directly measure the impact of the repetition in natural language by training on the full dataset.
5.1 Repetition Dropout
Inspired by dropout [23], which prevents the model from over-fitting on specific combinations of parameters, we propose the repetition dropout to avoid the model over-relying on repetitive words in the training phase. Since attention is the core component of Transformer-based LMs to perceive other context words, we apply our repetition dropout to the attention sub-layer, which is defined in Eq. (2). More concretely, our method replaces the triangular matrix  $\mathbf{M}$  in Eq. (2) by  $\mathbf{M}'$ :
$$
\mathbf {M} ^ {\prime} = \mathbf {M} + \mathbf {M} _ {r e p}, \tag {5}
$$
where  $\mathbf{M}_{rep}$  is the masking matrix for repetition dropout. The algorithm for generating the masking vector for each input sentence  $x$  is shown in Algorithm 1. In line 5-14 of this algorithm, we define a function find_ngrams to collect all  $n$ -grams in  $x$ . The function gen_mask_rep in line 16-26 is to randomly drop out those repetitive  $n$ -grams according to a pre-specified repetition
dropout rate  $p$ , which is between [0, 1]. A higher repetition dropout rate indicates that more repetitive  $n$ -grams are not accessible for the model. We concatenate the masking vectors for sentences in a batch as the  $\mathbf{M}_{rep}$ . To avoid the model over-fitting on a specific  $\mathbf{M}_{rep}$ , we generate  $\mathbf{M}_{rep}$  for each layer of the LM independently. It should be noted that repetition dropout is only employed in the training phase, analogous to the conventional dropout method [23].
In this method, we compel the model to predict subsequent words without depending on repetitions present in training data. Repetition dropout can be viewed as a means to regulate the quantity of repetitions within the training dataset. If learning on repetitions in training data is a vital aspect, repetition dropout will substantially mitigate the degeneration issue. We want to emphasize that this approach is designed to more effectively assess the significance of repetitions in training data.
Setup To evaluate the effect of repetition dropout, we conduct empirical studies on the Wikitext-103, OpenWebText2, and FreeLaw datasets, which have been introduced in section 4. Compared with section 4, we train the GPT-2 model on the complete datasets rather than splitting them into smaller shards. Please refer to Appx. A for more training details about the GPT-2 model<sup>3</sup>.
Algorithm 1 Repetition Dropout, Python-like
1 # x: List[int], input sequence  
2 # p: float, repetition dropout rate  
3 # n: int, length of ngram  
4  
5 def find_ngrams(x, n):  
6 # find start and end index of each n-gram  
7 ngram_dict = dict()  
8 for i in range(n-1, len(x)):  
9 ngram = tuple[x[i-n+1:i+1]  
10 if ngram in ngram_dict:  
11 ngram_dict[ngram].append((i-n+1, i+1))  
12 else:  
13 ngram_dict[ngram] = [(i-n+1, i+1)]  
14 return ngram_dict  
15  
16 def gen_mask_rep(x, p, n):  
17 mask_rep = [0] * len(x)  
18 ngram_dict = find_ngrams(x, n)  
19 for _, idx in ngram_dict.items():  
20 if len(idx) > 1:  
21 # mask repetitive n-grams according  
22 # to the dropout rate p  
23 if random.uniform(0, 1) < p:  
24 for i, j in idx:  
25 mask_rep[i:j] = -inf  
26 return mask_rep
2Notice that the repetition dropout is not restricted to attention mechanism. It can also be easily extended to the hidden representation of repetitive words.
<sup>3</sup>https://github.com/huggingface/transformers