Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loop

## Abstract

Abstract Large Language Models (LLM) are already widely used to generate content for a variety of online platforms. As we are not able to safely distinguish LLM-generated content from human-produced content, LLM-generated content is used to train the next generation of LLMs, giving rise to a self-consuming training loop. From the image generation domain we know that such a self-consuming training loop reduces both quality and diversity of images finally ending in a model collapse. However, it is unclear whether this alarming effect can also be observed for LLMs. Therefore, we present the first study investigating the self-consuming training loop for LLMs. Further, we propose a novel method based on logic expressions that allows us to unambiguously verify the correctness of LLM-generated content, which is difficult for natural language text. We find that the self-consuming training loop produces correct outputs, however, the output declines in its diversity depending on the proportion of the used generated data. Fresh data can slow down this decline, but not stop it. Given these concerning results, we encourage researchers to study methods to negate this process.

## 1 Introduction

Transformer-based language models have received much attention in the machine learning community in recent years. Especially large language models (LLM) trained on massive amounts of data from the internet became state of the art in many benchmarks Brown et al. () and specialized conversational LLM applications like ChatGPT(^1^11[https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt)) have a massive influence on society already. LLMs can be used for multiple tasks ranging from code generation Chen et al. (); Sobania et al. (); Fan et al. () and automated program repair Sobania et al. () to text summarization Yang et al. () and teaching assistance Baidoo-Anu and Ansah ().

Due to their convincing generated outputs, LLMs can be used to generate a large amount of content that is posted online to coding platforms like GitHub and Stackoverflow(^2^22[https://www.microsoft.com/en-us/Investor/events/FY-2023/Morgan-Stanley-TMT-Conference](https://www.microsoft.com/en-us/Investor/events/FY-2023/Morgan-Stanley-TMT-Conference)), social media platforms like Reddit(^3^33[https://www.vice.com/en/article/jg5qy8/reddit-moderators-brace-for-a-chatgpt-spam-apocalypse](https://www.vice.com/en/article/jg5qy8/reddit-moderators-brace-for-a-chatgpt-spam-apocalypse)) and other platforms on the internet. Even academic writing is already being influenced by LLM outputs Geng and Trotta (); Liang et al. (). Such LLM-generated text is often hard to distinguish from human-generated content Sadasivan et al. () and in turn might unwillingly be used to train the next generation of LLMs. Even paying for human-generated content might not be an option in the future, as workers at paid services like Amazon’s Mechanical Turk also use LLMs to produce content Veselovsky et al. ().
Consequently, a self-consuming training loop emerges in which future models are trained repeatedly on LLM-generated data from previous generations.
This process was first observed for generative models in the image domain Martínez et al. (); Alemohammad et al. (); Shumailov et al. (); Bertrand et al. (). These studies found that this self-consuming training loop leads to a decline in quality and diversity of generated images, ultimately resulting in a so called model collapse.
However, it is unclear what happens with LLMs that are trained in such a self-consuming training loop from scratch, as usually done in real-world applications like ChatGPT.

Therefore, we present the first study analyzing the behavior of LLMs trained over many generations in a self-consuming training loop. We conduct experiments on a GPT-style model in different settings and measure both quality and diversity of samples from the trained model at each generation in the self-consuming training loop. The settings differ in the way a dataset is created for each generation (so called data cycles) as well as the proportion of original real and LLM-generated synthetic data samples.
To better analyze the behavior of the trained models we conduct our experiments on a dataset consisting of logic expressions. In contrast to natural language, this logic expressions can be evaluated unambiguously. This allows us to analytically and accurately measure correctness and diversity of the generated samples.

We find that repeatedly training new models with synthetic data from previous models initially improves quality. However, diversity degenerates and the learned distribution inevitably collapses to a single point. In extreme cases this happens already after less than $10$ generations. Additionally, we find that the speed at which diversity degenerates depends on the data cycle as well as on the proportion of real and synthetic data. Fresh real data added during the data cycle slows down but can not negate the effects of a self-consuming training loop.

In summary, our main contributions are as follows:

- •
The first comprehensive empirical study of the self-consuming training loop for LLMs trained from scratch,
- •
A novel method to unambiguously evaluate quality/correctness and diversity of LLM-generated outputs,
- •
An in-depth analysis and discussion of the effects and implications of this self-consuming training loop for LLMs.

Following this introduction, Sect. 2 describes the self-consuming training loop and gives an overview of related work. In Sect. 3, we describe our experimental setting. Section 4 presents the experimental results, followed by a discussion in Section 5. Section 6 concludes the paper.

## 2 The Self-Consuming Training Loop

Figure: Figure 1: Self-consuming training loop: In the first generation $t=1$ a model $\mathcal{M}_{t}$ is initially trained on a real dataset $\mathcal{D}_{0}$. From this model $\mathcal{M}_{t}$ a sample $\mathcal{S}_{t}$ is drawn to build a new dataset $\mathcal{D}_{t}$. The new dataset $\mathcal{D}_{t}$ is in turn used to train a new model $\mathcal{M}_{t+1}$ from scratch for the next generation $t+1$. This process is repeated iteratively until the maximum number of generations $T$ is reached.
Refer to caption: x1.png

Figure: Algorithm 1 Self-Consuming Training Loop

Current generations of LLMs are usually trained on large amounts of unstructured data like text and code gathered from the Internet Brown et al. (). Due to their generative nature and capacity those models can in turn be used to generate new synthetic data, often indistinguishable from the original real data Sadasivan et al. ().
This synthetic data ends up back on the internet and thus in the next large dataset, which is used to train the next generation of LLMs.
These LLMs in turn produce new content and data, setting in motion a repetitive cycle in which new generations of models are trained each time with a higher proportion of synthetic data from previous generations.
We call this process a self-consuming training loop, depicted in Figure 1.

More specifically, consider a dataset $\mathcal{D}_{0}$ consisting of real data points $x\in X$ sampled from the original distribution $P_{X}$.
A generative model $\mathcal{M}_{t}$ is trained on this original dataset $\mathcal{D}_{0}$ until the data is sufficiently fitted, producing the first generation $t=1$ of generative models.
In a self-consuming training loop,
a new set of $m$ synthetic data points $\mathcal{S}_{t}$ is now sampled from the previous generation of generative models $\mathcal{M}_{t}$.
The next generation of generative models $\mathcal{M}_{t+1}$ is then trained from scratch on the new dataset $\mathcal{D}_{t}$.
This self-consuming training loop is repeated until the maximum number of generations $t=T$ is reached.
Algorithm 1 presents a pseudo-code describing this self-consuming training loop.

Related Work Current research has analyzed the self-consuming training loop mostly in the context of image generating models.
Martínez et al. () first studied the self-consuming training loop for diffusion models and find that the self-consuming training loop leads to a collapse in diversity of the generated images.
Other work have given theoretical frameworks for this phenomenon and look at more complex data cycles in the context of image generation Alemohammad et al. (); Shumailov et al. (); Bertrand et al. (). Both Alemohammad et al. () and Bertrand et al. () find that fresh real data can lead to stability within the self-consuming training loop.

Shumailov et al. () observe a degeneration of diversity for Variational Autoencoders and Gaussian Mixture Models if some of the output is used again as input. Additionally, the authors perform experiments for iteratively fine-tuning LLMs in a self-consuming way and observe a degradation in quality. They find that degradation is less strong than in the imagine generation context and that keeping some original data helps mitigating this phenomenon. Other contemporary work also analyzed repeated fine-tuning of LLMs in a self-consuming way and observed a decrease in lexical diversity Guo et al. ().

However, new generations of LLMs are usually trained from scratch with web scraped datasets while fine-tuning is mainly done with curated datasets. So the analysis of the self-consuming training loop when training from scratch is of pressing concern. To the best of our knowledge, we are the first to study the behavior of LLMs trained in a self-consuming loop from scratch. Furthermore, self-consuming loops in LLMs have not yet been analyzed with a method that allows to unambiguously evaluate the quality and diversity of generated model output.

Figure: (a) Full Synthetic Data Cycle
Refer to caption: x2.png

## 3 Experimental Setup

We present our logic expression dataset, which is the foundation for the verification of the language model’s output. Furthermore, we explain the data cycles we analyze in our experiments and describe the used model architecture in detail.

### 3.1 Verification with a Logic Expression Dataset

LLMs usually generate natural language texts and their performance is typically measured by using similarity metrics like the BLEU score Papineni et al. (), ROUGE score Lin () and BERT score Zhang et al. () as well as perplexity Jurafsky and Martin (). However, those metrics rely on measuring similarity of outputs with expected reference data, which can only serve as a proxy for quality of a language model Callison-Burch et al. (); Gehrmann et al. ().
Consequently, we propose using a dataset consisting of logic expressions. Those expressions can be represented as a sequence making them a good fit for language modeling.
In contrast to natural language text, we can easily and systematically evaluate the quality of logic expressions by verifying their correctness.
An expression is defined to be syntactically correct, if it can be parsed without an error. The semantic correctness can be evaluated if an expression is either True or False. If the original dataset only consists of True expressions, then a high-quality trained model should also only generate True expressions.

We build a logic expression as a tree in a recursive way specified in the function GenLogicExpr($d$), where $d$ is the desired depth of a logic expression tree. If the desired tree depth is reached when calling the function, a random Boolean is returned (either True or False). Otherwise, a logic operator (not,and,or) is selected and the function is called again recursively with $d-1$.
To build the entire original dataset we use this recursive function to sample new random logic expression trees with a random depth between $d_{min}$ and $d_{max}$ until we have a dataset with $m$ unique expressions that evaluate in a True Boolean expression. The complexity of the dataset can be easily controlled by adjusting $d_{max}$.
For our experiments we chose a initial dataset size of $m=10,000$ with expressions of minimum depth $d_{min}=1$ and maximum depth $d_{max}=5$.
Algorithm 2 in Appendix A describes the generation of our logic expression dataset in pseudo-code.

The resulting logic expression trees can then be saved as strings, e.g., not ( True and False ).
This allows us to evaluate them in Python using the eval() function to test whether they are correct or trigger an error and whether they evaluate to a True or False Boolean.
Additionally, we can encode those strings to a sequence of tokens (True, False, not, and, or, (, ), <eos>, where <eos> is a stop token and indicates the end of an expression) and use these sequences to train a language model.

### 3.2 Measuring the Diversity of the Model’s Output

A generative model does not only need to generate correct outputs but also a diverse set of outputs.
To measure the diversity of a sample from a LLM we use the Levenshtein diversity Wittenberg et al. () in our experiments. This metric calculates the pairwise Levenshtein distance between each expression in the sample normalized by the number of tokens of the longer expression of each pairwise comparison and averaged over the number of pairwise comparisons. If the average pairwise normalized Levenshtein distance is close to zero the diversity within a sample is low. A value closer to one suggests a stronger diversity within a sample. The Levenshtein distance itself is a metric for measuring the distance between two strings and is defined as the number of edits (deletion, addition, substitution) required to change one string to another Levenshtein et al. (). Normally, this edit distance is calculated on a character basis, however, we calculate it on the tokens from our encoding of the logic expressions to accommodate for the different length each token has in character representation.

### 3.3 Data Cycles

We define a data cycle as the way a dataset $\mathcal{D}_{t}$ is constructed in generation $t$ from the original data $\mathcal{D}_{0}$, potential fresh data $\mathcal{F}_{t}$ from the original distribution, and the generated data from the current and previous generations $\mathcal{S}_{1\dots t}$. This way of constructing a new dataset may vary and different data cycles are possible.
In the image generation domain, previous work suggests that the self-consuming training loop is influenced by the data cycle being used Alemohammad et al. (). Therefore, inspired by their work we conduct our experiments with the following four different data cycles (also depicted in Fig. 2):

Full Synthetic Data Cycle:
In the most extreme case of a self-consuming training loop a new model $\mathcal{M}_{t}$ is only trained on the generated data from the last generation so that $\mathcal{D}_{t-1}=\mathcal{S}_{t-1}$. We call this a full synthetic data cycle.
Normally, new datasets would still contain the original data when they are collected in practice.
However, we can use this data cycle to study the most extreme changes in behavior of models from generation to generation.

Balanced Data Cycle:
We refer to the second data cycle as the balanced data cycle.
In this data cycle, we construct the new dataset $\mathcal{D}_{t}$ from equal parts of all the previous samples $\mathcal{S}_{1\dots t}$ and the original dataset $\mathcal{D}_{0}$ so that every previous generation contributes $m*\frac{1}{t+1}$ logic expressions to the new dataset $\mathcal{D}_{t}$ of size $m$.

Incremental Data Cycle:
The third data cycle we study is called the incremental data cycle. In this data cycle, the new dataset is created by a portion $(1-\lambda)$ of the last dataset $\mathcal{D}_{t-1}$ and a portion $\lambda$ of new sampled data $\mathcal{S}_{t}$ so that $\mathcal{D}_{t}=(1-\lambda)*\mathcal{D}_{t-1}+\lambda*\mathcal{S}_{t}$ while holding the size $m$ of the dataset constant each generation. A $\lambda=1$ would result in a full synthetic data cycle. We chose $\lambda=0.1$ for our initial experiments and later used different values of $\lambda$ to study the influence of this parameter on diversity.

Expanding Data Cycle:
The first three data cycles create a dataset of equal size in each generation. In practice, however, the dataset would grow each generation by adding new generated data to the already existing dataset. Additionally, fresh data samples $\mathcal{F}_{t}$ from the original distribution (e.g. human generated) would also be added at each generation. Consequently, the last data cycle we study is an expanding data cycle. At each generation the new dataset is created so that $\mathcal{D}_{t}=\mathcal{D}_{t-1}+(1-\lambda)*\mathcal{F}_{t}+\lambda*\mathcal
{S}_{t}$, where $\lambda$ is the portion of generated data we add each generation and $(1-\lambda)$ is the portion of fresh real data added at each generation. For this data cycle, we chose $\lambda=0.9$ for our initial experiments and later used different values of $\lambda$ to study the influence of fresh data in an expanding data cycle on diversity.

Figure: (a) Full Synthetic Data Cycle
Refer to caption: x6.png

### 3.4 Model Architecture and Training

We employ a GPT-style LLM using the open-source implementation nanoGPT(^4^44[https://github.com/karpathy/nanoGPT](https://github.com/karpathy/nanoGPT)) (MIT license) in our experiments. The model accepts a context of up to 256 tokens and consists of 6 attention layers with 6 attention heads each and an embedding dimensionality of 384, resulting in roughly $10.6$ million parameters.

During training we use a batch size of 64 and a dropout rate of $0.2$, training for 5000 iterations minimizing cross entropy loss, starting with a learning rate of $10^{-3}$ decaying to $10^{-4}$, to achieve a sufficient fitting of the training data. We split the dataset in 90% training and 10% validation data. During training we calculate the validation error every 250 iterations and use the model parameters with the lowest validation error as our final model $M_{t}$ for each generation $t$ in the self-consuming training loop.

We use the trained model at each generation to sample $m=10,000$ logic expressions. During sampling, we auto-regressively generate new tokens with temperature $0.8$ and feed them back into the model until a stop token <eos> is sampled up to a maximum of 200 tokens per expression. We run the self-consuming training loop for $T=50$ generations in each experiment.

All experiments are performed on a workstation using a NVIDIA Turing architecture graphics card.

## 4 Results

We present our experimental results. First, we describe our results on the performance in terms of correctness of models trained in a self-consuming training loop. Afterwards, we present our results in terms of diversity for those models.

### 4.1 Correctness of Generated Content

We first study the correctness of the expressions within a generated sample $\mathcal{S}_{t}$ from a model $\mathcal{M}_{t}$ at each generation $t$. As described in Sec. 3.1, we consider an expression to be syntactically correct if it can be parsed without error. Since the original dataset $\mathcal{D}_{0}$ only consists of True expressions, we consider a generated expression to be semantically correct if it also evaluates to True. A semantically correct expression is also syntactically correct.

Figure 3 displays the composition of samples $\mathcal{S}_{t}$ generated during a self-consuming training loop. Each subplot presents the results for a different data cycle: a) full synthetic, b) balanced, c) incremental, and d) expanding. Every bar in a subplot displays the composition of $\mathcal{S}_{t}$ generated from model $\mathcal{M}_{t}$ at generation $t$, except for the first bar in each subplot which displays $\mathcal{D}_{0}$. The green portion of a bar indicates the number of syntactically correct expressions that evaluate to True in that sample. The yellow part of a bar represents the number of syntactically correct expressions that evaluate to False, and the red part of a bar displays the number of syntactically incorrect expressions that result in an error when being parsed.

Overall, we see that only very few expressions are syntactically incorrect. This indicates that the models are sufficiently trained and can correctly learn the syntactic rules of a logic expression.
We see a drop of around 20% in the number of semantically correct expressions from the original data $\mathcal{D}_{0}$ to the first sample $\mathcal{S}_{1}$ in every data cycle.
Interestingly, the number of semantically correct expressions increases afterwards over the course of the self-consuming training loop.
The speed at which this number increases depends on the data cycle. We can see the fastest increase for the full synthetic data cycle in which the whole sample consists of True expressions by generation $t=6$. The incremental data cycle takes consistently longer but also nearly reaches this point by generation $t=36$. While the balanced data cycle has an initially steeper increase in semantically correct expressions, it does not reach the point of a completely semantically correct sample by the end of 50 generations but comes very close to it. Lastly, the expanding data cycle shows this trend as well.
Overall, we see that the self-consuming training loop seems to help with generating more semantically correct expressions in our experiments and the rate at which this happens differs by data cycle.

### 4.2 Diversity of Generated Content

Figure: Figure 4: Average pairwise normalized Levenshtein distance over generations of a self-consuming training loop for different data cycles. Line markers added every 5 generations for better display.
Refer to caption: x11.png

Figure: Figure 5: Average pairwise normalized Levenshtein distance over the course of a self-consuming training loop for the incremental data cycle with different portions $\lambda$ of new sampled data. Line markers added every 5 generations for better display.
Refer to caption: x12.png

Figure: Figure 6: Average pairwise normalized Levenshtein distance over the course of a self-consuming training loop for the expanding data cycle for different portions $\lambda$ of generated data and $(1-\lambda)$ of fresh data. Line markers added every 5 generations for better display.
Refer to caption: x13.png

While we can see an increase in correctness over generations in our experiments, contemporary work in the image generation domain suggests, that this comes with a loss of diversity Martínez et al. (); Alemohammad et al. (); Bertrand et al. (). Therefore, in this section we study the diversity within a sample $\mathcal{S}_{t}$ of model $\mathcal{M}_{t}$ for each generation $t$.

Figure 4 displays the average pairwise normalized Levenshtein distance over generations of a self-consuming training loop for different data cycles.
For each data cycle, we observe a decrease in diversity over the course of generations, with the degree of decrease varying depending on the data cycle.
For the full synthetic data cycle we see a steep decline in diversity with a collapse into a single point by generation $t=39$.
The incremental data cycle is initially stable, but decreases in diversity from the 10th generation onwards, with a decrease of 68% in diversity by the end of the 50 generations.
The balanced and expanding data cycle also decrease in diversity, however, this decrease is way slower, with a 30% decrease in diversity for the balanced data cycle and a 22% decrease for the expanding data cycle.
While not all data cycles fully collapse in diversity by generation 50, ultimately, we expect all of them to eventually reach zero diversity if the self-consuming training loop is run for enough generations.

To get further insight into the decline in diversity, we also inspect the number of unique expressions sampled per generation for the full synthetic data cycle. We find that the number of unique expressions is stable at first while the diversity already declines and then rapidly decreases within 5 generations to very few unique expressions
(see Appendix B).

To better understand the influence of the amount of generated data in each generation on the diversity, we also study the incremental data cycle in more detail with different portions $\lambda$ of new sampled data.
Figure 5 displays the average pairwise normalized Levenshtein distance over generations in a self-consuming training loop for the incremental data cycle with different values of $\lambda$.
We observe, that the loss in diversity is stronger for a higher share of generated data added in each generation of the self-consuming training loop. Even with as little as $\lambda=0.25$, we reach the complete loss of diversity within $50$ generations. Only for our experimental run with $\lambda=0.10$ the diversity does not drop to zero, but to a value around $0.17$ after $50$ generations. However, we see that the speed at which diversity decreases does not scale linear with the amount of generated data. Instead we observe a more exponential increase in speed where larger amounts of generated data lead to an even quicker decay in diversity.

Lastly, we study the expanding data cycle in more detail with regard to the proportion between generated data and fresh data added to the training data at each generation.
Figure 6 displays the average pairwise normalized Levenshtein distance over generations in a self-consuming training loop for the expanding data cycle with different portions $\lambda$ of generated data and $(1-\lambda)$ fresh real data.
We observe, that while each configuration declines in diversity, the rate of this decline depends on the proportion between generated and fresh data. The more fresh data is added at each generation, the slower diversity decreases. With no fresh data or only 1% of fresh data the diversity decreases by approximately 36%. If the portion of fresh data is 25% we only observe a decrease in diversity of approximately 11%. This indicates that fresh data cannot stop the self-consuming training loop but enough fresh data can slow down the rate of the decline in diversity.

## 5 Discussion

In our experiments we found that iteratively re-training LLMs from scratch might help with correctness of model output. This is sometimes already done in practice where the output of LLMs is used to improve their performance Huang et al. (); Wang et al. () or generate new training data Yu et al. ().
However, this trend is concerning as we also found that this self-consuming training loop eventually leads to a drastic loss of diversity of a model’s output.
Our results confirm the findings on the self-consuming training loop in the field of image generation Martínez et al. (); Alemohammad et al. (); Shumailov et al. (); Bertrand et al. (), further highlighting the importance of this issue for generative models and LLMs in particular.
These LLMs have an even greater impact on society than image generation models, leading to more LLM-generated data being produced and potentially having a greater impact on society as a whole, as many people already rely on good model performance.

Consequently, as we expect that a self-consuming training loop will harm model performance in the future, researchers and practitioners should carefully choose their data when training their models and test those models sufficiently for diversity. Additionally, the machine learning community at large needs to find ways to deal with this problem as generated text data is already present in new internet scraped datasets.
While our results indicate that fresh real data can slow down the self-consuming training loop, other studies suggest that this is not an option as we will run out of real data eventually and have to rely on generated data for future models Villalobos et al. ().
Additionally, LLM-generated data has already found its way onto the internet, even infecting academic publications Geng and Trotta (); Liang et al. (), and LLM-generated data may soon overtake real data on the internet.

Furthermore, this LLM-generated data is hard to differentiate from real data Kreps et al. (); Sadasivan et al. (); Huschens et al. (), making it difficult to curate datasets scraped from the web. This makes the self-consuming training loop inevitable in the future. Therefore, we advise to further study ways to maintain diversity of generative model outputs like quality diversity methods Pugh et al. (); Fontaine and Nikolaidis ().

## 6 Conclusion

In this paper we studied the behavior of LLMs trained in a self-consuming training loop and found that iteratively training LLMs from scratch with self generated data can initially help with correctness of model outputs. However, this comes at a price, as the diversity of generated data eventually degenerates and collapses into a single point. The rate of this decline in diversity depends on the data cycle creating the training data at each generation. Furthermore, we found that fresh data can slow down this process and preserve diversity of a model output for longer. This should encourage researches to carefully look at their datasets and models to mitigate such a harmful self-consuming training loop in the future.

In future work, we will study how fine-tuning can influence the self-consuming training loop for LLMs. Additionally, we plan to further investigate how the effects of a self-consuming training loop can be slowed down or completely negated.

## 7 Limitations

One major cause for limitations of our work are limited computational resources. First, we ran our experiments with a smaller GPT-style model than for example GPT-4. Even though we expect that larger models with billions of parameters still suffer from the same loss of diversity if trained in a self-consuming training loop, we note that the behavior may vary for those larger models. However, a study for models of those size is not feasible with current compute options nor is it responsible in terms of the expected CO2 emissions Strubell et al. ().

Second, another limitation imposed by our computational resources is that the main results of this work consist of only one run per experimental configuration. With more runs per experiment the results would be more robust, but conducting more runs for our experiments was simply not feasible with our available computing resources. As all our experiments point towards the same direction, however, we do not believe that this limitation is crucial.
Nevertheless, to further mitigate this limitation we performed additional experiments for the full synthetic data cycle with fewer generations over multiple runs and with different initializations in Appendix C. These results show that the trend of a decline in diversity still holds true over multiple runs.

Another limitation is that our work focuses on repeatedly training foundation models from scratch in a self-consuming way. However, in practice those models are normally fine-tuned after the initial training. It is unclear from our work if this fine-tuning process can slow down or negate the effects of a self-consuming training loop when performed with fresh real data. However, we plan to investigate this in future work.

Lastly, one may argue that real text has a way larger vocabulary and more complex structures. Therefore, our results might be slightly different when performed on a real text corpus. However, measuring both quality and diversity of generated text is not trivial and can only achieved by using a proxy Callison-Burch et al. (); Gehrmann et al. (). Consequently, we intentionally performed our study on a logic expression dataset to ensure a controlled experimental setting in which we can measure both quality and diversity of a trained model.

## Appendix A Generation of the Logic Expression Dataset

Algorithm 2 describes the generation of our logic expression dataset in detail.

Figure: Algorithm 2 Generate Logic Expression Dataset

## Appendix B Unique Expressions Generated

Figure: Figure 7: Average pairwise normalized Levenshtein distance (left y-axis) and number of unique expressions in a sample (right y-axis) over generations of a self-consuming training loop for the full synthetic data cycle. Line markers added every 5 generations for better display.
Refer to caption: x14.png

To better understand the decline of diversity, we inspect the number of unique expressions sampled at each generation. We focus on the full synthetic data cycle, as this data cycle fully collapses to a single point within 50 generations.

Figure 7 displays the number of unique expressions (right y-axis) generated at each generation of a self-consuming training loop in comparison to the diversity measured as the average pairwise normalized Levenshtein distance (left y-axis).
We observe that in the beginning the number of unique expressions stays stable while the diversity is already in a steep decline. Only by generation $t=10$ the number of unique individuals starts to decline rapidly and by generation $t=15$ only around 400 unique expressions are sampled while the decline in diversity slows down. In generation $t=21$ the number of unique individuals is below 10 and in generation $t=39$ only a single individual is sampled at each generation. Consequently, the diversity is collapsed to zero.

This shows that it is not enough to only track unique outputs of a model, but that diversity needs to be tracked as well. By the time a decrease in unique results is noticeable, the diversity already decreased significantly.

## Appendix C Impact of Initialization

Figure: (a) True Initialization
Refer to caption: x15.png

Figure: Figure 9: Average pairwise normalized Levenshtein distance over generations of a self-consuming training loop for the full synthetic data cycle with different initializations. Displayed is the mean and standard deviation at each generation for each initialization over 5 runs.
Refer to caption: x19.png

So far we only looked at the self-consuming training loop starting from an initial dataset consisting only of True expressions. To check whether our findings also hold for different initializations of data across multiple runs, we conduct further experiments. Specifically, we analyze the full synthetic data cycle with the original dataset $\mathcal{D}_{0}$ only consisting of (1) True expressions, (2) False expressions, and (3) an equal mixture of both.
The results from Sect. 4 indicate that the effect of the self-consuming training loop for the full synthetic data cycle are already clearly visible in the first 10 generations.
Therefore, we conduct experiments with only 10 generations. With this saved computing time, we conduct 5 independent runs for each initialization.

Figure 8 displays the composition of samples $\mathcal{S}_{t}$ generated during a self-consuming training loop. Each subplot presents the composition for a different initialization for the full synthetic data cycle averaged over 5 independent runs: a) only True expressions, b) only False expressions, and c) half True expressions and half False expressions. Every bar in a subplot displays the composition of $\mathcal{S}_{t}$ generated from model $\mathcal{M}_{t}$ at generation $t$, except for the first bar in each subplot which displays $\mathcal{D}_{0}$. The green portion of a bar indicates the number of syntactically correct expressions that evaluate to True in that sample. The yellow part of a bar represents the number of syntactically correct expressions that evaluate to False, and the red part of a bar displays the number of syntactically incorrect expressions that result in an error when being parsed.

Similar to our results in Sect. 4.1, we observe an initial drop of True expressions by 20% in favor of False expressions for the initialization with only True expressions in the first generation. Afterwards, the number of True expressions in a sample increases again over the course of generations.
For the initialization with only False expressions, we observe a similar effect, only in the opposite direction. The number of False expression first drops by around 20% and then increases again until the whole sample consists of only False expressions.
When initialized with an equal amount of True and False expressions, we can observe that the proportion of both expression types first stays stable and then by generation $t=6$ starts to slightly shift towards True expressions. Upon further inspection of the 5 runs, we find that one run quickly shifts towards only True expressions, one run shifts quickly to only False expression, one run shifts slowly to True expressions, and two runs maintain equal proportions between True and False expressions across the 10 generations.

Figure 9 displays the diversity measured in the average pairwise normalized Levenshtein distance for the three different initializations. Displayed is the mean and standard deviation of five runs over the number of generations. We observe that the diversity declines for all three types of initialization, further confirming our results from Sect. 4.2. The initializations with only True and only False expressions decline at around the same speed. The initialization with equal proportions of True and False expressions declines a bit slower but shows the same effect.

Overall these findings indicate that the effects of the self-consuming training loop apply to different initializations across multiple runs, partially mitigating the limitations mentioned in Sect. 7.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: How do the initial complexity and diversity of the logic expression datasets influence the final conclusions drawn from the experiments? It would be beneficial to explore the sensitivity of the results to variations in these factors.

If the new dataset is synthesized by referring to the original dataset, rather than being directly sampled from LLMs, would the diversity of the final output still exhibit a decline? This question aims to investigate the potential impact of different dataset synthesis methods on the diversity of the generated content.
soundness: 2
strengths: This paper tries to reveal a critical issue in the data construction of LLMs, given the extensive use of WEB data and the challenges in distinguishing LLM-generated content from human-produced content. The authors conduct comprehensive experiments to explore the potential risks associated with directly feeding LLM-generated content into models, highlighting the limitations and risks of the self-consuming training loop.
summary: This paper empirically investigates the self-consuming training loop in GPT-style models. To unambiguously verify the correctness of LLM-generated content, the authors propose to use logic expressions, as opposed to natural language, to construct the training dataset. The authors conducted comprehensive experiments with four distinct data cycles, and the generated content is evaluated in terms of both correctness and diversity. The experimental findings reveal that utilizing the model's own generated data leads to a decline in diversity.
weaknesses: Regarding the sampling methodology and synthetic content: The experimental setup deviates significantly from real-world LLM pretraining scenarios. In practice, the vast majority of pretraining data for open-sourced LLMs originates from human-written sources such as books and web pages. Even when synthetic content is utilized, it is often generated in diverse styles (e.g., textbooks, QA) based on existing human-written web content, rather than being directly sampled from LLMs for training purposes. The authors are encouraged to consider incorporating more realistic data sources and synthetic methods to enhance the practical relevance of their findings.

Concerning the logic expression dataset: While constructing datasets based on logic expressions facilitates the verification of content correctness, it also oversimplifies the underlying problem. The authors should consider the potential impact of the diversity and complexity of logic expressions on the experimental outcomes when designing their experiments. This could involve exploring the use of more complex logic expressions or incorporating elements of natural language to better reflect real-world scenarios.

Regarding the base model: It is recommended that the authors conduct additional pretraining using existing open-sourced LLMs, incorporating model-generated content to further enhance the practical relevance and generalizability of their findings.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. It is recommended that “output” be specified in Algorithms 1 and 2 to enhance the overall readability and comprehensibility.
2. The literature: “AI models collapse when trained on recursively generated data” has given corresponding experimental proofs
and conclusions on the phenomenon of recursive data fine-tuning of language models leading to the collapse of language models.
We hope that the authors will adjust the content of the article so as not to mislead the readers.
3. Figure 3 shows that using complete synthetic data to train the NGL model does not produce error data, while mixing artificial
data with synthetic data produces syntactically incorrect outputs, why is that?
4、 It is suggested to give the formula of the indicator ' Average Normalized Levenshtein Distance ' to make it easier for readers to
understand.
5、 Using logic expressions as training data does have some unique advantages in evaluating the semantic correctness of the
output of a language model, but there are differences between this kind of data and real textual data, and I'm curious how the
semantic correctness and variety of outputs in the article would be reflected on real textual data?
soundness: 3
strengths: 1、 The effects of different data self-recycling methods, different model sizes, and the number of model iterations on language models
are compared in detail.
2、 Levenshtein diversity is used as an evaluation index of data correctness and diversity.
3、 The harm of the phenomenon is analyzed.
summary: In this article, the authors design a series of experiments to explore the impact of self-consumption training loops on
large language models, in which the authors use Levenshtein diversity as an evaluation metric for data correctness and
diversity. Through the experiments, the authors argue that training a large language model using data generated by the
large language model leads to a decrease in the correctness and diversity of expression of the model, and analyze the
corresponding challenges of current large language model training.
weaknesses: 1. The literature: “AI models collapse when trained on recursively generated data” in the recursive data fine-tuning language
model has given the corresponding experimental proof of the phenomenon of the collapse of the language model and the
conclusion. It is not the case, as stated in the introduction, that only the use of recursive language training in the image
domain leads to model collapse. This paper explores the effect of data self-recycling on training in the non-fine-tuning case
from a different perspective, with limited innovation.
2. Levenshtein diversity has long been applied to measure text diversity, and is not considered NOVEL.(Ref:Beijering, Karin,
Charlotte Gooskens, and Wilbert Heeringa. "Predicting intelligibility and perceived linguistic distance by means of the
Levenshtein algorithm." Linguistics in the Netherlands 25, no. 1 (2008): 13-24.)
3. The number of model parameters used in the experiment is low and the type of data used is single, so the relevant
conclusions obtained are not rigorous.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: See weaknesses.
soundness: 2
strengths: * This paper studies an interesting question, which is a valuable contribution to the further development of LLM and a deeper understanding of self-consuming training.
* The paper is well-written and easy to follow.
summary: This paper presents an initial study of the influence of self-consuming style training on the generation quality of LLMs. A logic expression dataset is used to avoid the inaccurate measurement of generated text quality or diversity.  In the evaluation part, four types of self-consuming training are incorporated to comprehensively compare the effect of self-generated data.
weaknesses: * The author missed a series of related works on self-consuming training (self-training) of LLM, and here are some examples [1][2][3].

* The experiment setup is not convincing to me. The generation quality is not equivalent to the correctness of the generated logic expression. Therefore I don't think the evaluation successfully reflects certain properties of self-training of LLMs. 

* Logic expressions merely form a small part of data that LLMs can generate. Experiments with only logical expressions cannot lead to a general conclusion, for instance, self-training in LLM may or may not result in XXX.


[1] ReST-MCTS∗: LLM Self-Training via Process Reward Guided Tree Search

[2] Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models

[3] Reinforced Self-Training (ReST) for Language Modeling

---

confidence: 4
contribution: 4
initial_rating: 10
presentation: 4
questions: My main suggestion is to try this work with a more sophisticated but still machine-evaluatable language, such as SQL. This is a harder problem, but one with probably more diversity, and it might help in making the results more robust to any criticism that the language was too simple. I realize that would be difficult or impossible to do for this submission, but it would make the work stronger.
soundness: 4
strengths: The problem is very clearly framed and good background from the similar problem in image generation is helpful. The author's approach to make a tractable experiment by using logic statements is quite clever, and makes it easy to understand the impacts of "self-consuming" training. The authors present a solid range of experiments, focusing on generations of self-consumption, different degrees and styles of self-consumption, and--in the appendices--model size. The results are stark and convincing, at least in the context of the logic statements they evaluate.
summary: The authors test whether the re-use of already-generated examples from an LLM in further pre-training of the same model result in lower quality output. They measure the output in two ways: correctness and diversity. They use a clever, if somewhat limiting, approach to automatic evaluation: teaching the model to generate logic statements which evaluate to `True`, e.g., by the Python `eval` command, allowing them to evaluate correctness easily. To evaluate diversity they examine the number of unique statements generated as well as their diversity, measured using the Levenshtein Diversity, an averaged, normalized Levenshtein distance between all pairs of generated statements. Within this framework, they show that the re-use of previously generated statements as further training data invariably makes the model more "correct" in the sense that it generates only True statements, but also that the diversity inevitably suffers, in the most extreme cases to the point that the model generates only one output. Diversity begins to fall dramatically before the outputs become truly non-unique.
weaknesses: The authors are aware of some of the limitations of the work--it is possible to claim that a sufficiently large model might not suffer from these problems, but the authors have made a good attempt to address this up to a certain scale, and as they point out the cost (both in dollars and CO2) of such an experiment at large scale would be prohibitive and probably unethical. A more subtle limitation might be that the range of language tested in this paper is necessarily quite small. It would be interesting to extend the experiments to a larger language, perhaps something like SQL which is still easily verified but has a richer vocabulary.

Meta-review: The article investigates the effects of using a large language model's (LLM's) own generated data to further train itself (a "self-consuming training loop").  To accurately assess the generated content, the authors used logic expressions (which can be automatically evaluated for correctness) as training data, rather than natural language. The design makes the evaluation easy and task. They conducted experiments with four different self-consumption training cycles, evaluating the generated output for both correctness and diversity using metrics like the number of unique outputs and Levenshtein distance. The key finding is that while retraining with self-generated data can increase the correctness (in this case, generating valid logic statements), it consistently decreases the diversity of the output,

The reviewers agree it is a critical problem to investigate, and the proposed approach to make a tractable experiment by using logic statements is clever. The conclusion on its own setting is also supported by the experiments results. 

The major concern is its generalizability. The setup in the study is significantly different from the real world LLM pre-training scenarios. First, demonstrating similar effectiveness in a more realistic domain would strengthen the work's relevance, e.g. SQL query generation. Second, the evaluation focuses solely on smaller scale data and models without comparison to established baselines. Given the known performance differences between model scales, it is unclear whether these findings generalize to larger models.

Decision: Reject