Mitigating the Learning Bias towards Repetition by Self-Contrastive Training for Open-Ended Generation
Year: 2023

Abstract:
Despite the huge progress in myriad generation tasks, pretrained language models (LMs) such as GPT2 still tend to generate repetitive texts with maximization-based decoding algorithms for open-ended generation. We attribute their overestimation of token-level repetition probabilities to the learning bias: LMs capture simple repetitive patterns faster with the MLE loss. We propose self-contrastive training to penalize the output of a premature checkpoint of the same model when it incorrectly predicts repetition, which is shown to mitigate repetition effectively while maintaining fluency on two datasets. Furthermore, we find that LMs use longer-range dependencies to predict repetitive tokens than non-repetitive ones, which may be the cause of sentence-level repetition loops1.

heading: 1 Introduction
text: Existing LMs prefer to generate repetitive texts for open-ended generation with greedy decoding or beam search (Welleck et al., 2020a). Even largescale pretrained LMs such as GPT3 (Brown et al., 2020) still generate redundant sentences (Dou et al., 2022). Despite many solutions proposed from the perspective of both training (Welleck et al., 2020b) and decoding (Holtzman et al., 2020), the cause of preference for repetition still needs to be clarified.
By analyzing the training dynamics of LMs regarding (non-)repetitive tokens, we reveal the learning bias towards repetition: LMs capture simple repetitive patterns first, which dominate the output distribution throughout the input space, and then learn more non-repetitive patterns during training. We show that the repetition problem can be mitigated by only training more steps (i.e., allowing over-fitting), although the coherence with inputs will be impacted. Conversely, when trained insuf-
∗Corresponding author 1The code is available at https://github.com/
thu-coai/SelfCont
ficiently, LMs will overestimate repetition probabilities even for golden prefixes. We propose selfcontrastive training (SELFCONT), which exploits the contrast with a premature checkpoint of the same model by penalizing its output when it incorrectly predicts repetition. Experiments on two datasets show that SELFCONT effectively alleviates repetition while maintaining fluency by factoring out the undesired repetition behaviors highlighted by the premature checkpoint.
Besides the above analysis about overestimating token-level repetition probabilities during training, we also find that LMs use longer-range dependencies to predict repetitive tokens than non-repetitive ones. It may explain why LMs tend to fall into repetition loops (Xu et al., 2022). The problem may be solved by improving the modeling of long-range dependencies (e.g., increasing model sizes), which are left to future work.

heading: 2 Related Work
text: Regarding the cause of the repetition problem, Fu et al. (2021) theoretically derived bounds of repetition probabilities of the first-order Markov LM, although it is difficult to extend the bounds to general LMs. Another line of works attributed repetition to error accumulation during generation (Welleck et al., 2020b; Arora et al., 2022), while LMs still prefer repetition given golden prefixes.
We divide recent works that alleviate repetition into training- and decoding-based methods: (1) Training-based Methods. Welleck et al. (2020b) proposed unlikelihood training (UL) to reduce the probabilities of repetitive generations. Lin et al. (2021) and Xu et al. (2022) further extended the framework at the token and sequence level, respectively. SELFCONT focuses on token-level modeling, which is orthogonal with sequence-level methods. Xi et al. (2021) adopted additional modules to learn repetition patterns and control repetition explicitly. (2) Decoding-based Methods.
6897
One straightforward solution to repetition is blocking repetitive n-grams generations (Paulus et al., 2018) or penalizing probabilities of repetitive candidates (Keskar et al., 2019). Li et al. (2022) selected candidates that maximize the probability difference between different-sized models. Sampling-based decoding methods are also shown effective in avoiding repetition, such as temperature sampling (Ficler and Goldberg, 2017), Top-k sampling (Fan et al., 2018), nucleus sampling (Holtzman et al., 2020), and typical sampling (Meister et al., 2022). Although these methods reduce superficial repetition, it is unclear whether they utilize the underlying long-range dependencies to maintain coherence.

heading: 3 Empirical Analysis
text: Neural networks (NNs) are highly expressive to approximate arbitrary input-output mappings. Using Fourier analysis, Rahaman et al. (2019) showed the spectral bias of NNs: they learn low-frequency components faster during training, which are less complex and vary globally without local fluctuation. Our key hypothesis is that simple repetitive patterns may be such low-frequency components and learned by LMs early. In this section, we first formulate LMs (§3.1), and then investigate the training dynamics (§3.2) and the ability to model long-range dependencies (§3.3) of LMs.

heading: 3.1 Language Models
text: LMs aim to fit the mapping xt = f(x1:t−1) defined by a training corpus, where x1:t is a sequence from the corpus. To this end, they are usually trained by minimizing the following cross-entropy loss:
L = −xTt · log [ softmax ( fθ(x1:t−1) )] , (1)
where xt ∈ {0, 1}|V| is the one-hot representation of xt indicating its index in the vocabulary V , and fθ(x1:t−1) ∈ R|V| is the output logits of the LM parameterized by θ. Predictably, with more training steps, argmax(fθ) is closer to the target function f . Early stopping (Morgan and Bourlard, 1989) is a commonly used regularization technique to avoid over-fitting, e.g., stopping training when the validation loss reaches the minimum. Since NNs prioritize learning low-complexity components, early stopping may result in unexpected generations. We are inspired to investigate whether simple repetitive patterns in human-written texts are learned first, thus dominating the generations.

heading: 3.2 Training Dynamics
text: We randomly sample 1k sequences containing 512 tokens from the Wikitext-103 dataset (Merity et al., 2016) and train GPT2base from scratch for 100 epochs2. Given a golden prefix x1:t−1, we regard the model prediction x̂t = argmax ( fθ(x1:t−1) ) as
correct if x̂t = xt. We call xt or x̂t repetitive if it is included in x1:t−1, and non-repetitive otherwise.
kens, in model-generated texts with greedy decoding.
Figure 1 plots the training curves, revealing the learning bias of the LM: (1) The initially learned components prefer to copy input tokens throughout the input space, as indicated by predicting repetitive tokens at ∼90% of positions for both golden and generated prefixes. (2) With golden prefixes, at those positions where xt is repetitive, the LM predicts repetition almost constantly during training. When xt is non-repetitive, the LM predicts more non-repetitive tokens with more training steps. The repetition ratio also gradually decreases in modelgenerated texts. (3) The token prediction accuracy improves faster when xt is repetitive, indicating that the LM learns repetitive patterns more easily. Moreover, we notice that the validation loss rises at the 1,500th step, where the LM predicts much more repetitive tokens than the ground truth. At the end of the training, the generation has a closer token repetition ratio to the ground truth. But manual
2We use only 1k samples because we expect to over-fit these samples to observe how repetition in generated texts changes with the fitting degree, considering that it will be very time-consuming to fit the whole Wikitext-103 dataset.

heading: 3.3 Modeling Long-Range Dependencies
text: Figure 2 indicates (1) The LM only learns dependencies within ∼100 tokens overall. When the prefix length is larger than 100, the perplexity on golden tokens no longer drops significantly (p ⩾ 0.05). (2) The LM learns and utilizes longer-range dependencies to predict repetitive tokens than non-repetitive ones. The perplexity on golden repetitive/non-repetitive tokens plateaus when the prefix length is larger than 160/50, respectively. The case is similar for generated texts. (3) The LM uses short-range contexts to predict non-repetitive tokens regardless of decoding algorithms. Contexts beyond 100 tokens hardly help predict non-repetitive tokens, implying samplingbased decoding reduces repetition through randomness instead of using long-range dependencies.
Based on the above observation, we conjecture that the LMs keep repeating the same sentence with maximization-based decoding (Xu et al., 2022) because they rarely learn long-range non-repetitive patterns beyond the sentence level. When generating long texts, LMs may struggle to maintain non-repetitive within a long range. To test the idea,
erated by LMs trained on different training sets.
As shown in Table 1, compared to Doriginal, the LM trained on Drandom has higher repetition ratios because it learns shorter-range non-repetitive patterns only within one sentence. Besides, although sentences in each Drandom example are unrelated, they can contain repetitive tokens3, making the LM learn spurious long-range repetitive patterns to get into repetition loops. In contrast, the LM trained on Dnorept rarely gets into loops since it learns both repetitive and non-repetitive patterns almost within one sentence. Specifically, any adjacent five sentences in each Dnorept example are unrelated and hardly share tokens. These findings empirically support our hypothesis. Appendix A.2 shows more details.
3The ratios of tokens that have appeared in previous 128 tokens are 12.52% and 32.05% for the training sets of Doriginal and Drandom, respectively. Drandom has even more repetition than Doriginal possibly because random sentences repeat highfrequency words than human-written sentences.

heading: 4 Self-Contrastive Training
text: We denote the premature checkpoint as fθ0 , which frequently predicts repetitive tokens. Formally, the SELFCONT algorithm is formulated as follows:
fθ = fθ1 + sg(wfθ0), (2)
w = λ1(xt ̸∈ x1:t−1)1(x̂t ∈ x1:t−1) (3) x̂t = argmax ( fθ0(x1:t−1) ) , (4)
where sg(·) means stopping back-propagation of gradients, λ is a tunable hyper-parameter to control the extent of repetition penalty, and 1 is the indicator function. fθ1 is the target LM initialized from fθ0 , and we optimize fθ using Eq. 1 until the validation loss converges to the minimum. The gradient for each token u ∈ V has changed to:
∇uL = exp(fθ1 |u)∑
v∈V wv,uexp(fθ1 |v) − 1(u = xt), (5)
wv,u =exp ( w(fθ0 |v − fθ0 |u) ) , (6)
where fθ1 |u is the output of fθ1 at the u-th dimension. If w is 0, wv,u is always 1 and ∇uL degenerates to the same as the vanilla LM. If w is not 0 and u is not xt, tokens with high logits under fθ0 will receive larger gradients than the vanilla LM since wv,u is mostly smaller than 1 with different v. As for u = xt (w ̸= 0), it may also be penalized with a positive gradient if fθ0 |u is large enough, which usually means a dull token. By penalizing components that excessively prefer repetitive or dull tokens highlighted by fθ0 , fθ1 can utilize more complex patterns learned later to generate texts.

heading: 5 Experiments
text: Datasets We conduct experiments on Wikitext103 (Merity et al., 2016) and WritingPrompts (Fan
et al., 2018). The prompt and story in each WritingPrompts example are concatenated as a sequence. We set the maximum sequence length to 512 and take the first 50 tokens as input to generate the rest. Table 3 presents the detailed statistics.
Baselines We compare SELFCONT to three baselines: MLE, token-level UL (Welleck et al., 2020b) and ScaleGrad (Lin et al., 2021). Since SELFCONT focuses on token-level modeling, we do not compare it to sentence-level methods that directly penalize repetition loops, e.g., DITTO (Xu et al., 2022).
Implementation All baselines are implemented based on GPT2base. We set the batch size to 16, the learning rate to 1e-4, and λ in Eq. 3 to 4.0. For SELFCONT, we fine-tune GPT2base for one epoch using MLE and take the checkpoint as fθ0 for both datasets. We use different p for different models based on the performance on the validation set. Appendix B shows more details.
Metrics We use perplexity (PPL) under GPT2xl to evaluate fluency, MAUVE (Pillutla et al., 2021) to measure the similarity between golden and generated distributions, the token repetition ratios (R-l) to measure the ratio of tokens that appear in previous l tokens (Welleck et al., 2020b), and distinct (Dn) (Li et al., 2016) to evaluate the n-gram diversity. The closer scores to the ground truth mean better quality for all metrics.
Results As shown in Table 2, SELFCONT outperforms baselines in all metrics using greedy decod-
ing. However, the high R-128 score shows it can still generate repetition loops due to the disability of small-scale LMs to model long-range dependencies. Using nucleus decoding, we see that different baselines can achieve similar repetition ratios and diversity to the truth by tuning p, while SELFCONT has better fluency and higher MAUVE scores.

heading: 6 Conclusion
text: We present empirical studies on LMs’ preference for repetition by analyzing the training dynamics, which highlights their learning bias towards simple repetitive patterns. We propose penalizing outputs of a premature checkpoint during training, which effectively mitigates repetition while maintaining fluency. We also provide insight into why LMs easily fall into repetition loops by showing their disability to model long-range dependencies. Sampling-based decoding reduces repetition through randomness but not utilizing long-range dependencies. We believe that maximization-based decoding can also generate coherent texts without repetition by improving the modeling of long-range dependencies, which is left to future work.

heading: Acknowledgments
text: This work was supported by the National Science Foundation for Distinguished Young Scholars (with No. 62125604) and the NSFC projects (Key project with No. 61936010). This work was also supported by the Guoqiang Institute of Tsinghua University, with Grant No. 2020GQG0005.

heading: 7 Limitations
text: The limitations of this paper mainly lie in the following folds: (1) We do not provide any theoretical analysis for the correlation between long-range dependencies and repetition loops, as well as solutions to avoid repetition loops with maximizationbased decoding. (2) We do not discuss the source of LMs’ learning bias, which may be caused by multiple factors, such as the Transformer architecture (Vaswani et al., 2017), the MLE loss, or the auto-regressive generation manner. (3) We conduct experiments based on GPT2 due to resource limitations. The conclusions may differ for extra-large LMs (such as GPT3). (4) We do not experiment with RNN-based models, which are also shown to prefer repetition (Elman, 1990). (5) We do not perform the manual evaluation to compare SELFCONT with baselines since we focus on repetition in this
paper, which can be automatically evaluated reliably. Perplexity and mauve scores are also shown to correlate highly with manual evaluation for evaluating fluency and overall quality, respectively.

heading: A Details for Empirical Analysis
text: A.1 Training Dynamics
Table 4 shows several cases generated by the LM with greedy decoding at different training steps. We summarize the findings as follows: (1) In the beginning, the LM keeps repeating the high-frequency word “<eos>,” indicating that it does not capture phrase-level dependencies yet. (2) At the 1500th step, the LM first generates a few fluent sentences and then gets stuck into the repetition of “the building,” showing that it learns long-range dependencies conditioned on the golden prefix while the repetitive patterns dominate the probability distributions conditioned on the generated prefix. This case suggests the global tendency towards repetition for out-of-distribution inputs. (3) At the 6000th step, the LM can generate long, fluent texts without repetition. However, it is difficult for the LM to maintain coherence with inputs due to over-fitting. For example, in the generated first sentence, “she had begun in 1962,” “she” conflicts with “he” in the input.
A.2 Long-Range Dependencies
Observation For the experiment in Figure 2, we generate texts with three decoding algorithms conditioned on the first 50 tokens on the test set. Ancestral decoding means directly sampling tokens from the original probability distribution. For nucleus decoding, we set p to 0.9. Figure 3 shows the performance of GPT2large, which shows similar results with GPT2base in Figure 2.
Verification For the experiment in Table 1, we use the same approach to construct the corresponding validation sets of 480 examples for Doriginal, Drandom and Dnorept, and train three LMs until the best validation performance. Table 5 shows several generation cases with greedy decoding. The LMs trained on Doriginal and Drandom fall into repetition loops. Although the LM trained on Dnorept also generates sentences that have previously appeared, it does not get stuck into loops. We further investigate whether the three LMs show the selfreinforcement effect: the more times a sentence is repeated in the context, the higher the probability of continuing to generate that sentence (Holtzman et al., 2020; Xu et al., 2022). Figure 4 indicates that the LMs trained on Doriginal and Drandom show the above effect, while the LM trained on Dnorept
does not. The results suggest that longer-range repetitive patterns biased LMs to fall into repetition loops through the self-reinforcement effect whether such patterns are true or spurious. The LM trained on Dnorept always generate sentences in a limited set due to greedy decoding which aims to find the global maxima of probability distributions, instead of the preference for repetition loops.

heading: B Hyper-Parameters
text: We decide the hyper-parameters λ in Eq. 3 and p for nucleus sampling by searching for the value that makes the R-64 score of generated texts closest to the ground truth on the validation set. We search λ in the range {1.0, 2.0, 3.0, 4.0, 5.0, 6.0}, and p in the range {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}. Table 6 shows the settings of p for different models.
As for baselines, we follow the original papers to set α to 1.0 for UL and γ to 0.2 for ScaleGrad.
As for the choice of fθ0 , we empirically choose
the checkpoint after training for one epoch, which allows enough training steps for self-contrastive training. We use the premature checkpoint of the same model instead of other models since different models may have different biases. It costs about 24 hours to train SELFCONT on Wikitext-103 (∼10 epochs) or CNN News (∼6 epochs). The results are based on one NVIDIA Tesla V100 (32GB memory) with a random single run.

heading: C Modeling Token-Level Repetition
text: We compare SELFCONT with baselines in terms of the performance for modeling token-level repetition. As shown in Table 7, SELFCONT achieves higher overall accuracy, higher F1 score on nonrepetitive tokens, and comparable F1 score on repetitive tokens.

heading: D Case Study
text: Table 8 and Table 9 show the cases generated by different models on Wikitext-103 with greedy decoding and nucleus decoding, respectively. We see that SELFCONT can still get stuck into loops with greedy decoding since it hardly learns longer-range dependencies than standard LMs. Although sam-
Models Acc Repetitive Non-RepetitiveP R F1 P R F1
MLE 78.9 67.4 87.0 75.9 90.2 73.9 81.2 UL 80.5 70.9 83.0 76.5 88.2 78.9 83.3 ScaleGrad 81.6 75.9 76.1 76.0 85.2 85.0 85.1
SELFCONT 82.3 78.5 73.9 76.1 84.4 87.4 85.9
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 7
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3 and Section 5.
3 B1. Did you cite the creators of artifacts you used? Section 3 and Section 5.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section 3 and Section 5.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 3 and Section 5.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 5.
C 3 Did you run computational experiments? Section 3 and Section 5.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 3, Section 5, Appendix Section B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 3, Section 5, Appendix Section B
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section B
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We believe that maximization-based decoding can also generate coherent texts without repetition by improving the modeling of long-range dependencies, which is left to future work