How bad is training on synthetic data? A statistical analysis of language model collapse
Authors: Mohamed El Amine Seddik
Suei-Wen Chen
Soufiane Hayou
Pierre Youssef
Merouane Abdelkader DEBBAH
Venue: COLM (2024)
Keywords: model collapse, recursive training, generative models
TL;DR: Study of model collapse through an introduced toy language model.

Abstract: Model collapse, as introduced in (Shumailov et al., 2023), refers to the phenomenon where training models on synthetic data generated from previously trained models leads to a deterioration in performance. This recursive training loop makes the tails of the original distribution disappear, thereby making future-generation models forget about the initial (real) distribution. With the aim of rigorously understanding model collapse in language models, we consider in this paper a statistical model that allows us to characterize the impact of various recursive training scenarios. Specifically, we demonstrate that model collapse cannot be avoided when training solely on synthetic data. However, when mixing both real and synthetic data, we provide an estimate of a maximal amount of synthetic data below which model collapse can eventually be avoided. Our theoretical conclusions are further supported by empirical validations.

Published as a conference paper at COLM 2024
How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse
Mohamed EI Amine Seddik
Suei-Wen Chen NYU Abu Dhabi Abu Dhabi, UAE
Soufiane Hayou Simons Institute Berkeley, USA hayou@berkeley..ed edu
Technology Innovation Institute Abu Dhabi, UAE mohamed. beddik@tii.e.
swc435dnyu.
Pierre Youssef
Merouane Debbah
NYU Abu Dhabi
Khalifa University of Science and Technology Abu Dhabi, UAE merouane dewbahhku..e.o ac.ae
Abu Dhabi, UAE
yp27@nyu.c edu
Abstract
The phenomenon of model collapse, introduced in (Shumailov et al. 2023 refers to the deterioration in performance that occurs when new models trained synthetic data generated from previously trained models. This recursive training loop makes the tails of the original distribution dis- appear, thereby making future-generation models forget about the initial (real) distribution. With the aim of rigorously understanding model col- lapse in language models, we consider this paper a statistical model that allows us to characterize the impact of various recursive training scenarios. Specifically, we demonstrate that model collapse cannot be avoided when training solely on synthetic data. However, when mixing both real and syn- thetic data, we provide an estimate of a maximal amount of synthetic data below which model collapse can eventually be avoided. Our theoretical conclusions are further supported by empirical validations.
1 Introduction
The large-scale adoption of large language models (e-g. ChatGPT (OpenAI 2024)) will inevitably lead to enormous amounts of synthetic (generated) data "polluting" the original human-created web data. Since language models are trained on web data, this raises some concerns about the impact of this synthetic data the next generations of LLMs. One can think of a train-generate loop where models from the current generation generate data that contaminate existing web data, and the next generation models are trained this contaminated data. This loop was studied in several works (Shumailov et al. 2023) Alemohammad et al.) 2023: Briesch et al., 2023) where the authors conclude that synthetic data generally hurts performance as the number of generate-train increases, and that (naturally) the impact on model performance is linked the amount of real data in the training set. A particular phenomenon, coined model collapse (Shumailov et al. 2023), refers to the model's tendency produce limited repetitive outputs making recursive training on such outputs forget about the tails of the original underlying distribution of real data. This was further studied in (Guo et al. 2023) where the authors show that recursive training synthetic data leads to "self-consuming " loop that affects linguistic diversity.
Intuitively, model collapse a result of the distribution shift that occurs when training gen- erative models recursively synthetic data from previous generation models. Shumailov et al. (023) have discussed two main sources of model collapse; 1) statistical approximation finite
error: which is inherently related to the fact that generative models are trained number of samples, and therefore is impossible that the learned model captures all the in- formation about the original distribution. functional approximation error: which results from
1

---

Published as a conference paper at COLM 2024
the fact that the models are insufficiently expressive in real implementations, even if neural networks are known to be universal functional approximators from a theoretical standpoint. The authors provide further theoretical intuition to characterize the effect of these approxi- mation errors, relying on simple mathematical models such as single-dimensional Gaussian distribution.
In this paper, we aim to provide a rigorous theoretical framework to understand the ef- fects of recursive training with synthetic data. In particular, we focus on the statistical approximation error and introduce a simple next-token -peekicitreiiiioon language model to char- acterize model collapse. Our model allows us to gain insights into the behaviour of the self-consuming train-generate loop leading to model collapse. From a theoretical standpoint, we consider two main recursive training scenarios:
Fully Synthetic: Training with data sampled from the previous generation model.
Partially Synthetic: Training with a mixture of data sampled from the previous generation model and original data.
We demonstrate that model collapse always occurs in the first scenario and characterize the rates at which it occurs. Furthermore, in the second scenario, we provide an upper bound on the sample size of generated data below which model collapse can eventually be attenuated. Our results are further confirmed through simulations of general scenarios with the introduced statistical model, as well as with realistic GPT22-ttle language models on real data. Our findings suggest that the amount of generated data should be considerably smaller compared to the original data to avoid model collapse.
With the adoption of generative Large Language and Vision models, the amount of synthetic data on the web is growing at an unprecedented rate - see for example (del Rio-Chanona et al. 2023) where the authors conducted an empirical study of the amount of synthetic data via activity monitoring on Stack Overflow, and (Alemohammad et al. 2023)
Related work:
where the authors showed that a dataset used to train Stable Ditfusion contains synthetic data (Schuhmann et al.) 2022). In fact, practitioners are willingly using synthetic data to train next-generation models (Ben Allal al. 2024 Gunasekar et al. 2023 Chen et al. 2024).
As we mentioned above, several works studied the recursive training loop where next- generation models are trained on synthetic data generated from previous-generatoo models. Shumailov et al. (2023) studied model collapse, a phenomenon that occurs in recursive training where the quality of model outputs tend to degrade by becoming e.g. repetitive. Similar phenomena were studied in (Alemohammad et al. 2023) where the authors call it Model Autophagy Disorder (MAD). Another empirical study by (Briesch et al. (023) studied this same Self-Cons Self-Consuming phenomenon and observed that the degeneracy rate (naturally) depends on the number of fresh data in the training sample. In the same direction, several works showed that incorporating synthetic data in the training can hurt the performance of trained diffusion models (Bohacek & Farid 2023 Martínez et al.) 2023a b).
Only few works have tackled this question from a theoretical perspective. Shumailov et al. (2023) considered a simple recursive Gaussian distribution to provide an intuition as to why model collapse occurs, but no training is considered in that work. In (Fu et al. 2024), authors studied recursive training of diffusion models (generally used to learn distributions over images) and obtained an upper bound on the total variation distance between the distribution of the original data and that of the synthetic data after T generations, Dohmatob al. (2024b) studied the impact of synthetic data on scaling laws and, in simple setting of linear regression, Dohmatob et al. (2024a) studied the behaviour of the test error of different generations and showed that a linear dependency of the degradation on generation number.
In this work, we are interested in characterizing the distribution shift in synthetic data generated with a language model, as the number of generations increases. We consider a linear Softmax classifier for next-token prediction and study the distribution of the learned conditional probabilities as the number of generations increases. The closest work to ours is (Fu et al. 2024) where the authors study the distribution of synthetic data generated by
2

---

Published as a conference paper at COLM 2024
diffusion model instead, whereas the statistical model considering is closer in spirit to the realm of language models.
The remainder of the paper is organized as follows. Section 2 presents our theoretical setup where we introduce our statistical model and the considered recursive training scenarios. Our main theoretical results are presented in Section 3 We further present some experiments in Section 4 to support our findings. Finally, Section concludes the paper.
Theoretical Setup
2.1 Statistical Language Model
We consider a language model of vocabulary size s, context length denoted by l and we further denote by the number of possible contexts which is at most s', We suppose that the language data is generated from some unknown conditional probabilities given the contexts. That is, the probability of the next token being E context j (fi....,h) E [c] is denoted by
[s] = [1)-.))} given some
Py P(Y=
where X and Y denote discrete random variables representing context and the next- token respectively. In practice, we do not have access to the true conditional probabilities Pik but rather (large) corpus sampled according to Pı-- In other words, are given dataset (x))y)) HE[M of M samples of contexts and next-token pairs represented by X E {eI....,ec) and y: E es] where ej's denote the canonical vectors.
Given this dataset, consider approximating the underlying conditional probabilities via the Softmax classifier, which entails minimizing the categorical cross-entropy loss function:
arg min
1 M
Σ yi log
(W xT))).
(1)
W= wu JE
where ((D) Ei (xp();)) is the Softmax function and the functions exp and log are applied
exp(r)
entry-wise. Note that in current state-of-the-art language models, the x are context representations computed via transformer models, whereas, in our setting, we choose to work with one-hot embeddings as representations for tractable theoretical analysis. Solving the above objective (see Appendix 8.1) yields the estimated conditional probability Ộæ of Pjk which expresses as the following empirical mean:
exp(r0 ej)
1
Pjk
Σ yı with C, IEC,
(1
[n]
x = =ei
(2)
Ei=1 exp(10) eq) |c,l
The estimated conditional probabilities Pj are the result of training on the original data. These conditional probabilities can be used to generate new synthetic data, which can be used (with or without additional fresh data from the original dataset) to train the next- generation model. In this paper, we are interested in characterizing the behaviour of 6 in this recursive training loop which we will formally define in the next section. Hereafter, without loss of generality, we consider fixed context with N == |Cyl training samples {(ej, y:); of context-next-token pairs, where y! IR are independent multinomial random variables with one trial and parameter
p (P Ps) := (pjt...., Pı)) E [0, 1|s From we notice that p.r is an estimate of PyR and we further denote
(3)
p (1) (PI...., ,.....,,,) P.) == (f. Ps)) E [0,
(4)
As such, (0) := corresponds to the ground truth distribution whereas p(1) denotes the first-generation model trained on the real data {e,, hic,
3

---

Published as a conference paper at COLM 2024
p (10)
p (20)
p (30)
p (40)
Figure 1: Evolution of p(m) in the Fully Synthetic setting for vocabulary size 3, context
length L 4, total contexts = s = 81 and sample size 1000. The initial distribution p(o) is some random distribution over tokens. The trained conditional distributions converge
towards Dirac measures over generations illustrating total collapse in Definition 1
2.2 Recursive Training
In this section, we introduce the notations for recursive training. At generation in 2 1, sup- pose that we have samples (f),, generated by some past models pl for t E (O...., 1} respectively, As such, similarly to 2,, the model at generation m expresses as
R
p (m)
y1
(5)
n
In other words, p(m) is obtained by training the model on a mixture of real and synthetic (m) data generated from previously trained models. Here, ni stands for the number of
samples used to train the m-th generation model that are generated by model p(f), and (m)) n((m) = nD +... -
is the total number of training samples used to train p(m). Note that we do not generate samples from p p(0) but rather we are given a corpus {yj (0) h: which represent original data.
(im)
Note that by definition all the models p(m) are unbiased, i.e. Ep(") p, which means that they recover the original distribution in case of infinite sample sizes. However, in practical settings the sample sizes are finite, and therefore p p(T)) may deviate from p since its variance
can be large for small values of n; (m) In the remainder of the paper, we present analytical
results to rigorously quantify the impact of sample sizes ni;") on the variance of p(")) and how they affect the learned distribution eventually cause model collapse (Shumailov al., 2023), a degenerative process affecting future generation models by either losing information about the tails of the initial distribution or inducing distribution shifts over generations.
Specifically, we aim to quantify the rate of such deterioration, and to this end, we introduce stricter version of model collapse that we call total collapse and define as follows.
Definition 1 (Total Collapse). We say that total collapse occurs in the recursive training process if p( ((m) converges to some Dirac mass ó; for some token i E [s] as m grows.
Total collapse refers to the case where, under recursive training, the trained model ((") completely loses information about the original distribution over generations, leading to poor linguistic diversity[] This phenomenon is illustrated in Figure 1 where we see convergence towards the vertices of the probability simplex, i.e. Dirac measures, over generations. With this definition of total collapse, we provide a quantitative analysis of two cases of recursive training:
Note that here we define total collapse with respect to a single pre-fixed context. This can be generalized to the event where total collapse occurs for all contexts. However, the theoretical analysis in this case would require more refined treatment of several statistical quantities to obtain uniform bounds over contexts. We leave this for future work.
4

---

Published as a conference paper at COLM 2024
Fully Synthetic: Training with synthetic data from the last model. Each generation p(m) is trained only on data generated by the previous model p(" 1), More precisely, for some fixed n E N we let ni n 1( m-11 1} for all 2 1 and S m. Partially Synthetic: Training with mixture of real and synthetic data from the last model. Each generation p(") trained on mixture of real data and synthetic data generated by the previous model p (("-1)) More precisely, for some fixed n E IN we let no
[m) N, 1 π and n:t)) 0 for all m 2 2 and 0 < < m, and H = N.
Fully Synthetic corresponds to the theoretical setting considered in (Shumailov et al. 2023). However, in that paper, only Gaussian distribution analyzed instead of discrete distri- butions over all possible tokens as language models entail. We point out that this setting is unlikely happen in real-world applications but serves the worst-case scenario.
On the other hand, the Partially Synthetic setting is more realistic for future generation models it corresponds training on mixture of real and synthetic data. We consider this setting to assess whether it is possible to avoid collapse by having a fraction of the original data in the training mixture. We answer this question positively in the next section. Moreover, we show through simulations in Section 4 that conclusions from our theoretical analysis on both Fully Synthetic and Partially Synthetic hold beyond these simple settings, such as training a mixture of all generations even using realistic transformer models.
3 Main Results
To investigate the model collapse phenomenon and the rate at which it occurs, define the following statistical quantities that capture the randomness of p( (mm)
(m))
000 max Pi
(m) On = 1| (m)
(m)
and Sm :- E[Gm)]
iE(s]
These quantities measure how far away p( p(m) is from some Dirac mass (Total Collapse, Definition D Since the maximum value of all three quantities 1, the closer they are to 1,
the closer (()) is to some Dirac mass, and the less diverse p(n) is as a language model, As
such, ( ||o or Onr being equal to 1 is equivalent to p(m) being Dirac mass which reflects total collapse. To quantify the distribution shift incurred by the aforementioned recursive
training scenarios, we further consider the 1-norm between two distributions μ4, E Rs defined by ||и v|l Σ (i) v(i).
We assume that the initial distribution (0) is nontrivial, specifically S < 1 and (p || < 1.
Under this assumption, provide results on the rate of total collapse and further quantify distribution shift under recursive training. All the proofs are presented in Appendix
3.1 Fully Synthetic: Training on synthetic data
We start by describing the recursive training process in which only synthetic data from the last generation model are used. This process can be viewed as a Markov chain on the set of probability measures s-1 n Ns on [s] with denominator where
A 1 == {v E R +
= L, 2 0 for all i}
We point out that the 1-norm is twice the total variation distance || vll which is another commonly used metric for probability distributions and was considered by Fu et (024 for studying model collapse in the case of diffusion models.

---

Published as a conference paper at COLM 2024
is the probability simplex. Since the probability of reaching 8, is positive provided p; > this random walk which starts at pl)) has absorbing states {6; : [s] s.t. p; > 0). As
result, the random walk converges to one of the absorbing states almost surely (Kemeny
et al. 1969), which means that total collapse is bound to happen in the Fully Synthetic setting. setting
To characterize the rate at which total collapse occurs, let us denote by T := inf(t N
[|p |00000000 13 2 the random time at which the model first becomes Dirac mass, and let Pm P (1p(m) 00 = 1) denote the probability that the m-th generation has collapsed. Our first result, presented in Theorem 1 provides the rate of convergence via Sm, Pm and IE(T). Theorem 1 (Control on Total Collapse). Consider the Fully Synthetic setting and let $ := (supp(p) p)1 denote the support size of p, namely := Ki [s] : pi 0}|.
1. The expected sum of squared probabilities Sm is given by
m
Sm 11 1
(1 (1-
(6)
2. The probability pm that lotal collapse has occurred by generation m satisfies
1- So
1- - - 51) S Pm S
1- 1-3 (1 1/n)m..
(7)
3. The generation T at which total collapse happens satisfies
So nn - 1/3
1 1
[E[] S 1 + (1-
So)n(n -1-11111-mt
1).
(8)
In essence, Theorem 1 describes the behavior of ((m) as a function of the model generation m, the sample size n, and the dispersion of the initial distribution Sg. Specifically, we draw the following observations:
Effect of the number of generations m: As m increases, S,n and pm tend to 1 as per 6) and 7 making total collapse increasingly more likely. Note also that this
dependence is exponential in mi making total collapse in this case relatively fast.
Effect of "synthetic" sample size n: The smaller u is, the more likely p p(m) is to have
collapsed for given generation m per the bounds on p'm in 7 and the faster
total collapse is expected to happen as suggested by ( The dependence in this
case is polynomial in n.
Effect of Sp: Larger values of Sy correspond to faster collapse as suggested by (8); namely, starting from an original distribution that is not diverse enough speeds up total collapse. The upper and lower bounds on E [T] are both linear in (1 -S).
We point out that when the number of contexts c is large, the number of samples per context would be fairly small, which suggests that total collapse, given a single context, is expected to happen fairly quickly. The bounds on E[T] state that the expected collapse time is at least of order and at most of order n²2 though experiments (see Figure 2 suggest that the upper bound is not sharp and should be close to (n)) instead. On another note, characterize below in Proposition 7 the limiting distribution as m gets to infinity, which demonstrates the direction of total collapse.
Proposition 1. In the Fully Synthetic case, we have P lim
p (m) -1-
5, for all i i [s].
Proposition I describes the limiting distribution when total collapse occurs in terms of the initial probabilities p; over tokens. Specifically, the resulting Dirac mass o; is likely to be supported some token i with high initial probability pt. This formally supports the description of early and late model collapse in Shumailov et al. (0033: In the early phase of recursive training, the tails of the original distribution disappear because the probability

---

Published as a conference paper at COLM 2024
So 0.01, 581
Sp = 0.1, 5=
Sp =p 0.5, 5 $010
Sg 0.7, 5
log
10
730
833
Token
800
Ceneration
X
200
X
130
200
300
100 200
300
400 0
100
200 300
420
100
200
300
420
Sample size
Figure 2: Fully Synthetic case for different initial distribution p Total collapse time is plotted as function of the initial distribution and the sample size n. (Top) The initial distribution with different values of So and support size The x-axis represents tokens 21, 600} while the y-axis represents the probabilities in log scale. (Bottom) Each cross represents the average total collapse time over 100 runs for a particular sample sizes (10,50) 50, 100, (50,...,,40) The red dashed line depicts the lower bound on IET given by
Pj (n) of outputting unlikely tokens (those j's for which pj is small) will decrease as (m))
converges some Dirac mass 6,. After many generations, all but one p will go to exhibiting late model collapse where all the randomness of the original distribution is lost.
3.2 Partially Synthetic: Handling model collapse with real data
As described in the previous section, total collapse is unavoidable when training solely on synthetic data. In this section, we consider the case in which real data is incorporated at each generation. In this case, pi") can be simply seen a weighted average of (1) and an estimate of pmm-11 with samples. We also assume that p(1) nontrivial, thereby
implying that, on average, ((m) will not be Dirac mass. In what follows, we quantify the variability the data distribution across different generations, as well as the distribution
drift [|)(m) -p(P|| from the fist-generation model. We particularly show that collapse can be avoided if enough real data is injected into the recursive training process.
Theorem 2 (Model Variation). Consider the Partially Synthetic selfing. For > 1 have
1 2x - (1- a
(1- (1-
aB"h
Sm+1
1
(9)
1+ 11+1/NNa
(1+1//N)
N
where N+a and p= a (1+ } x
Theorem provides a control of the variance Sn in the Partially Synthetic setting. Essentially, when n < N, we have Sm AJ 1 + (1 So s So, which is not surprising since the training dominated by real data in this case. However, even
data for each generation model when the number of synthetic data is much larger than the original dataset (i.e. n > N), we have a Re ~ $ and hence S R 1/N + (2 + 1/N) -/N)-1(1 -1-1)(11))2 - 1/N) which approaches So for large N.
To further refine our analysis, we present a result that directly controls the deviation ElIp(") (()) between the conditional distributions from first and m-th generations. This allows us to have a quantitative control over the distribution shift. When n is sufficiently small, we have sharper control over this deviation by exploiting the concentration results
7

---

Published as a conference paper at COLM 2024
from (Mardia et al. 2020). Essentially, this allows us to estimate the maximum number of synthetic samples to ensure that the distribution p(m) stays close to p())
Theorem 3 (Model Deviation). Consider the Partially Synthetic setting and define
Cpn Cise 2F
if
C 1
n # s;
8/2
Gn(s)= =
CiS (2
if
C n
CQ 2;
(10)
2)
if s <
+2
where Co
2: ~ 3.19 and C
2 2.93, and s is the vocabulary size. Then, m 2 2, 23/2
be
E| Bl
2 Gn(s).
We point out that the upper bound on the deviation E|]p p(11 is independent of the generation m. Since the deviation from p0)) to p()) is inevitable and independent of n, we give the result in E [l]p(") pO') from which E E (0) can be estimated by the triangle inequality. Theorem 3 allows us to estimate the maximum number of synthetic
samples that can be used if we want p(m) to stay c-close to p(l) in L norm. For example,
when Con/e + 2 < s, for any e > 0 we can take
V2TNNe
n1 S 2nE -2 "min s 2, log
(11)
6es
in order for - p to be less than e. In other words, to ensure small L deviation should be taken to be logarithmic in the ratio Ne/s, which highlights that the amount of synthetic data should be exponentially smaller compared to real data in order to ensure that p(")) remains close to p()) In Appendix 9 we provide experiments showing the effect of the sample size n and the initial distribution p. In F we show that if we fix the initial distribution and increase the amount of synthetic data the dispersion om stays relatively constant while the distribution drift [|p(m)) -p() increases. In contrast, when we fix and increase the values of So, Um increases but ||('') - (()) actually decreases as depicted in Figure 6 This behavior can be explained by Theorem in Appendix 8 which is more
general than Theorem 3 in the sense that captures the dependence of E p (')) ll on the randomness of p and hence provides a sharper bound on the expected deviation. We provide additional simulations Appendix 9.2 to account more general scenarios.
4 Experiments
To support our findings in realistic setting, conduct experiments with decoder-only generative model for text generation. For this model, we consider the aforementioned Fully Synthetic and Partially Synthetic settings using the model parameters in Appendix 9.1 We consider simple character-level tokenizer using the tiny Shakespeare datase yielding vocabulary of size = 65. We first train a model for 2000 iterations on this dataset
and consider the trained model the ground-truth distribution (0) The next-generation models are trained recursively using the exact same architecture and training parameters.
This setting allows only on the effect of statistical approximation error.
reduce the effect of functional approximation error thereby focusing
The results of these experiments summarized in Figure The top left plot therein depicts the deviation || - (()) which is averaged over 300 random contexts generated by
https:
input. txt
8

---

Published as a conference paper at COLM 2024
synthetic
0.8
synthetic
2.50
0.7
synthetic
2.25
synthetic 20% real 80%
2.00
0.6
Gen
Gen 7
1.75
Gen
Gon 8
0.5
1.50
9
0.4
Gen
Gen
1.25
1
2
3
5
6
7
10
10a
generation
iterations
synthetic 50% real 50%
synthetic 20% real 80%
2.50
Gen 1
Gen
2.50
Gen
Gen 6
2.25
2.25
2.00
Gen 3
Gen
2.00
Gen
Gen 8
Gen 4
Gen
Gen
Gen 9
1.75
3 1.75
Gen
1.50
1.50
1.25
1.25
10"
101
iterations
iterations
Figure 3: Experiments with a GPT2-type generative model. The top left plot depicts the deviation |!'") p())/1 varying the generation model m for synthetic data and mixture of real and synthetic data. The three other plots show the behavior the validation loss over generations. Essentially, training solely synthetic data causes model collapse and affects the usual scaling laws (Dohmatob et al. 2024a).
the ground-truth generative model (0)), From this plot, we clearly see that p - (1) diverges generations in the Fully Synthetic setting, while mixing with real data ensures stability over generations.
The effect of synthetic data is further noticed in the validation loss of the next-generation models where we see an overfitting effect in the Fully Synthetic setting. We point out that this effect might also be associated with the functional approximation error and was rigorously studied by Dohmatob et al. (2024a) in the case of linear regression, where the authors have shown that synthetic data affect the usual scaling laws. We believe that similar conclusions can be obtained with our statistical model by incorporating the functional approximation error, this can achieved for instance by supposing that the context embeddings ej g's are high-dimensions Gaussian vectors instead of canonical vectors, thereby introducing the embedding dimension as a parameter controlling model complexity.
5 Discussions & Conclusion
In this paper, we studied model collapse in language models through simple statistical model. We provided theoretical analysis when training with only synthetic data and when adding real data from the original distribution. Our results demonstrate that model collapse always happens when the model is training solely on synthetic data, whereas controlling deviation from the initial distribution requires careful choice of the amount of synthetic data to inject in the training set. We also provided experiments showing that these findings extend beyond the simple theoretical settings.
Our current results describe only the statistical approximation error since all generation models are unbiased in our theoretical framework. However, we discussed in the previous section, this framework can be extended to account for the functional approximation error by considering high-dimension Gaussian vectors context embeddings instead of canonical vectors. Another possible extension is to study the effect of in-context learning (Wu et al.) 2023) model collapse, which is key feature of transformer-based models.

---

Published as a conference paper at COLM 2024
Despite the simple setting of our current investigation, we believe that it lays the ground- work for better understanding and mitigation of model collapse in language models, thereby opening the way for the development of more general theoretical frameworks to study next-generation language models dynamics.
6 Limitations
Our analysis primarily focuses simplified recursive training setup, which may not fully represent current large language model training practices. The theoretical results are derived for a basic language model, and extending the analysis more complex architectures like transformers remains challenging. Our empirical validations use relatively small-scale experiments due computational constraints, and the generalizability of these findings to large-scale language models requires further investigation. While we explore more realistic settings in the appendix, such as "Most Recent Models" and "Randomly Sampled Data", these still represent idealizations of real-world scenarios. Additionally, our treatment of synthetic data generation does not account for various post-processing techniques that may be employed in practice. Future work should address these limitations by expanding the theoretical framework to more complex model architectures, conducting larger-scale experiments, and considering broader range of synthetic data generation and utilization approaches.
References
Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G. Baraniuk. Self-consuming generative models go mad, 2023.
Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Cosmopedia, 2024. URL https::/huggingfaac.
cosmopedia
Matyas Bohacek and Hany Farid. Nepotistically trained generative-ai models collapse, 2023,
Martin Briesch, Dominik Sobania, and Franz Rothlauf. Large language models suffer from their own output: An analysis of the self-consuming training loop, 2023.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine- tuning converts weak language models to strong language models. arXiv preprint arXi::2401.01335, 2024.
Maria del Rio-Chanona Nadzeya Laurentsyeva, and Johannes Wachs. Are large language models a threat to digital public goods? evidence from activity on stack overflow, 2023.
Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Model collapse demystified: The case of regression, 2024a.
Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, and Julia Kempe. A tale of tails: Model collapse as change of scaling laws, 2024b.
Shi Fu, Sen Zhang, Yingjie Wang, Xinmei Tian, and Dacheng Tao. Towards theoretical understandings of self-consuming generative models, 2024.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, ct al. Textbooks are all you need. ar Xiv preprint axXit:2306.11644 2023.
Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, and Chloé Clavel. The curious decline of linguistic diversity: Training language models synthetic text, 2023.
10

---

Published as a conference paper at COLM 2024
John G Kemeny, J Laurie Snell, et al. Finite markov chains, volume 26. van Nostrand Princeton, NJ, 1969.
Jay Mardia, Jiantao Jiao, Ervin Tánczos, Robert D Nowak, and Tsachy Weissman. Concen- tration inequalities for the empirical distribution of discrete distributions: beyond the method of types. Information and Inference: Journal of the IMA, 94):813- 2020.
Gonzalo Martinez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Marc Juarez, and Rik Sarkar. Combining generative artificial intelligence (ai) and the internet: Heading towards evolution or degradation?, 2023a.
Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Marc Juarez, and Rik Sarkar. Towards understanding the interplay of generative artificial intelligence and the internet, 2023b.
OpenAl. Gpt-4 technical report, 2024.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wight- man, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget, 2023.
Tsachy Weissman, Erik Ordentlich, Gadiel Seroussi, Sergio Verdu, and Marcelo) J Weinberger. Inequalities for the 11 deviation of the empirical distribution. Hewleet--Packee Labs, Tech. Rep, 2003.
Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter L Bartlett. How many pretraining tasks are needed for in-context learning of linear regres- sion? Xiv preprini arXit:2310.08391, 2023.
7 Technical Lemma
For completeness, we include the concentration results from (Mardia al. 2020) which we used to prove Theorem 3 For a probability distribution p on [$] define
Tp == max min (p((() p(A)
(12)
Ag
and notice that Tp S 1
Define the function p: (0,1/2) + R R
2.
log 1-x
p(x) = 1- 2x
x
and extend & by continuity to (1122) = Observe that @ strictly decreasing and 2 < ((x) < co. The following Lemma concerns the concentration of empirical distribution which captures the dependence on the sample size n, dimension as well as the structure of the underlying probability distribution via (TTTp)
Lemma 1. Let be a probability distribution on [s] and P be the associated empirical distribution
obtained from n samples. Then for > we have
np(
P P(Pp
pll E) S exp
Gn (s),
where Co
3 2
C
6e n3/2
and
Cis (Con/s)s/2
if Can + 2 $ 5
Cpn + 2;
Gs(s)
Cal Cise t0 (2 -2)
Co if e
+ 2 < s;
(13)
ifs
Con
4 +2.
11

---

provides an Published as a conference paper at COLM 2024
The upper bound
nQ( )²
exp P( - pll1) S (2 - exp
4
in fact holds for all values of s and (Weissman et al. 2003), but this bound is very poor when s/ is large, which is commonly the case for language models. Lemma |
improvement in the regime of small sample size $ s.
General Results and Proofs
8.1
Derivation of Formula (2)
Proof. The categorical cross-entropy loss reads
1
M
exp(20f xi) E exp(w) x;)
L w.)
Yik log
M
The gradient of the loss is expressed as
a E -w;) xx)
ac
M
dev
Σ Vit
1 +
exp - wi) x;)
where
a
- Li exp((x wx) Tx)x;; if k=
Σ exp((10) ww) xxxx)
awv
=
eexp((wx wi) xx) if k k l.
Therefore,
Oc
1
M
Ljtk exp((xx; wws) x;)
exp exp((w; ww;) Tx;)
V/k
+
Vii
x,
dr
M
+ E ji exp (zo; ww:) x;)
1+ E
Ejst exp(exx) wc) xx)
1
M
exp(w xi)
Ejtt exp(xeepp 20,7 x;)
1/i
y/k
Xi
M
1 exp(ex) w. x;)
ΣE=1 exp( w xi)
1
exp( w x;)
exp W x;)
Vit
Yik
xi
M
1 exp(w0 x;)
EE exp(20 xi)
1
M
( x;)
Vii Σラ exp(ee 1 xi)
Vikk X;
M
1
M
exp ( x;)
Vikk xi
M
- exp(w) w xi)
dc Finally, solving for dwn;
0 yields 2b.
12

---

Published as a conference paper at COLM 2024
8.2 Proofs for the Fully Synthetic Case
Proof of Theorem 1
Write pi (m)
x x(!1)/ /n where x(m))
B(n, Pi
(or 1/) is binomial when
conditioned on m
So we have
E Elp
Elp!m)?
(m)
p (m
1
11 Pi
1 (m-1)
+
1
1
Pi
1)-
11
Σ Elp (m)2
p (n=1))]
(1
(m-1)
+
Pi
11
12
and by the law of total expectation
11
1 (1- n
Sm 11
+
Sm-1 =
(1 (1-S
So)-
(14)
Note that Sm / as > and
S E ( E pi
Sm E Σ
(mr) (m)
pm))
|1p(m)
1000 1llpp
pi
Recall that Pm = P (II(())|a 1000 =1) and T == inf(m E N 11((") 1} 2 1. Since
x=1 Om 2 1/5 and [| loo E (
n 1}, we have
1 Pm (1 pm) < EU Sm E (ml (m) 1o S 1. Pm + (1- (1-11) 1/n) - pm),
from which follows thanks to 14
For k = 1,2,... we have (PT > k) = P( p (k) 10 1) = 1 Px and thus
1- 50
1/n) k < P k) < n(1 So) 1/n)*,
1/5
Ie= PP k). which establishes 8 because [
Proof of Proposition 7 Fix i E [s). For m 2 1 consider the events En, == {T s m) and
Fm = {p:" (m)
1}. Then p!n)) E 10 on Em and Fm C Em- Observe that
Em Um and Fm 7 U lim p ((m)) 6,), m=-000
where (m) 1 by Theorem and in particular P P(Em) > 0 for m large. Thus
Pi lim E[p; (m)) = lim (P(EE) E(p:
(m)
Em + P(E E(p: (m)) E
11:
lim Elpp (m) Em] lim P(pp (m)
P(F n Em P(Em)
Em) lim
lim P(Fm)
lim P((m)) PmUmFm)) m=+00
P(Em)
which proves the assertion.
8.3 Proofs for the Partially Synthetic Case
Proof of Theorem 2
Let v.
(m)
IE
2 pi
(m)
N' = N + and y; (m)
(v ...., Uy8 ). Then
(m)
(m))
Pi
+
1
1
p
N
N
13

---

Published as a conference paper at COLM 2024
and for m 2 2
2
2
N
v. (())
N22
E
iik
(0)
+
Σ
Vix
(m-
+
Σ
Vix
(0) (m-
Vixk
-1k=1
N2 [Npi+ + (N2 N7 + np; + (n²
n))
(m 1)
+
2
N
IB
Vij Vix
(0) (m-1) 1)
N2
Pi
N N n² n (m-1)
2
(0) (m-
#1
N
+
N pf
+
N Vi +
N
E
Vij
Vik
(15)
1m-
is conditionally independent of Vij given pi p: (m-
(0)
For 2 2, since Vix
by conditioning
(0) on Vif and Pi (m- we have
(0) (m-1)) Vi. Vix = EE (0) 11]
F
Vif Pi
and hence for m 2 2,
(mn)
Pi N
N2 N
N
pf
+
12 n
N22
Vi
Im 1) +
N22
2
Σ
N
IF
yij
(0)
Pi
(m
1]
(m) for m 2 3
By the definition of p;
E
Vif
00)
Pi
(M
11
1
E
Vij
10)
N
Vik
(0)
+
1 E
Vi.j
00:
Σ
Vit
22
N
N
k=1
k=1
Pi
+
(N
1)p
11 N
(0)
(m 2)
Vij
Pi
N'
and
E yij 01
(0)
(1- 1
p:
P (N 1)p?
PP
Pi +
p
N
N
which gives
(0) (m 1) Vij Pi
1- 1 m-
(x)
No
p: + (N - 1)p?
11
m-2
(0) (1)
I
+
E
yi, p
1 N'
N
N'
Setting a= == n/N', we have N = (1 - N'N and hence
Pi + (N 1)p?
10) (m 1i
E
Vij Pi
N
(m)
for all m 2 2. Plugging this back to the expression
15
for v
gives
(m)
(1 a1 + 2x)
p; (1
1 N (1 (1-a²) p² + [11
1+
1
1
11-
a
N
N
N
Let f= a [(1 + xt) a R/
Then for 2 1,
11
1
1 Bmh+1
a(2 a)
1 8Th
Pi
N
1
B
B
1)
[1
8m
a² 1 - m
+
1
pł
N
1 - B
1 - B
kP;
(
1
1 +
1+ (1 + 11/N))
N
(1- }}?
1
1+a
N 8
1+ (1+1/N)
14

---

Published as a conference paper at COLM 2024
and summing across i [s] gives the expression for Sm+1-
Note that 0 < B < a < 1, which gives both an upper bound and lower bound on V (m))
In
particular, for m 2
(m)
1 N IN (1 (1+2a)); 2x)p; + (1 (1-N) 1) (1 + a)
V
1+ 11N)
and therefore
1 1+ (1 + 1/N)a (1 + 2a) + 1 (1-N) (1 + Ss
Sm
=1 [1 - x) (1 a1( So)
1 (1 11NNa
where
11 11) (1 a1 SS))
1 a (1 So), So
(1+111/N)
+ 3a
since 0 < 1/N 1/2.
We state more general result which implies Theorem 3 Theorem 4. Let (s) and @ be as in Lemma 1
Then for 2 we have
(llp(++1) - 101 < NV p(5)
11 (s(,),
(16)
where
1
1
21E
max p (1) (A)p
(
A)l) (
(N+*) n
N
2
AC
Proof of Theorem
4 Since all generations share the same original data source, we can write
p (k+1)
N
n
where p H1
Σ y
(K)
(17)
N N
11
N N n
and (v/" (v(")) 1 are i.i.d. multinomial with parameter p(8) and one trial (k 2 1). This gives
plki1) - p1)) N +
p
p())
and applying the triangle inequality yields (It()+)) (1)
n
p(k) 10)
p(k)
(())
||
p(0))
(1))
N
N
+
Taking the expectation and solving the recursion gives for k 2 1
kil
Ellp (++ -p(01 <
n
E
p(i) - 60) ||,
(18)
N + n
From Lemma 1, we have
((p70)-p0)) pu)l 21 p(k)
p
where 74. m. Integrating over t [0, o0), get
E
(
(8) 1 (k)
< Gn(s))
π
"TT((TT))
15

---

Published as a conference paper at COLM 2024
and by Jensen's inequality and the concavity of x + xxxx) -1/2,
E
p - pb0) S n1
Gn)
(
1/2
nol n (77,)) Gn(s).
π
Thus
El Ellp(k*1) R++))
-p 10
π
k Gn (s)
k+1-j
1
(19)
V
+
V (((E)))
It remains to upper upper bound EMT, since q is decreasing. For A C [s] let A denote its complement. Then by 17 for A S [s] we have
N2
11
(A)
(N + n) p (1) (A)p (2) (A9)+ (A9)+ (N + n)2 (l)))((())) (A)
Nn
(A9)) (A) 11 (A) (A) (A9)
(N + n)²
N2
n?
Nn
1 (N + n) P (1) (AA)p(()A)) (N + n)² pkk) (A) +
(N n)2
Let == = maxsc(s) p (a) (A) (8) (At), so the inequality above implies
B[Ar+1
p(), p(ip 5
(N + n) d1 + (N + n1²
N2
n²
E
max ((k) (A) (A9) p(() AC||
+
Nn (N n)²
N2
n²
Nn
(N + n) A1 +
+
4(N + n)²
(N + +n2
and thus
[2 + Nn + n2/4 (N n)²
N 2 N + )² 1 (1 - E(Li)) (
E E(A++1] <
4 4
Observe that for 0 5 < 1 we have min(x, (x, x) < 2x(1 x), E E[TT&] I 21E(Ap) Writing
1 - 2E(1il
(
N
:
2
N +
have IB [[TT]] S 5, so from 19
we see that
KI1 i
IB (++)) p 111
(1)
π Gn((s
Σ
nπ p({)
(s.
(20)
nt((t)
N N+n n
Proof of Theorem 3 Theorem 3 is an immediate consequence of Theorem by replacing @ with its minimum value 2.
Additional Experiments
9.1 Architecture & training parameters for GPT2 experiments
We consider that all generation models have GPT2-type" vanilla architecture which is decoder-only generative model with the configuration and training parameters as summa- rized in Table These parameters were chosen to achieve the best validation loss when training the first-generation model on data produced by ground-truth generative model p()) defined in Section
16

---

Published as a conference paper at COLM 2024
Context length
128
Embedding dimension
256
Number of layers
Number of self-attention heads
4
Vocabulary size Dropout Learning rate
0.2
10 3
Batch size Max iterations
256
2000
Table 1: Architecture and training parameters in the setting of Section 4
9.2 Additional Experiments with the Statistical Model
To further investigate and demonstrate the generality of theoretical findings, we present empirical results on two more scenarios that better represent recursive training in real-world settings, follows:
Most Recent Models: Each generation p(") is trained on synthetic data from the most recent K models for a fixed window size More precisely, for some fixed n E IN
1} we let n!")) / 1 max(0, K) i m - for all 2 When this case degenerates to the Fully Synthetic setting.
Randomly Sampled Dala: Each generation p(m) is trained on a mixture of synthetic
data from possibly all the previous models and the real data. More precisely, the
m-th generation model is trained on = 1i + +...+Mo +...
(m)
" samples with
1 == Σ 18: = =}, (t (t= 0,
m 1),
samples from the t-th generation, where {gilie|m) are independent and uniformly distributed on {0, 10,1, 1} indexing mmevious-generation models. This setting describes the scenario where data generated by all past models are mixed in pool from which the training data for the next generation is collected.
Experiments for these two cases are shown in Figure Column 1 corresponds to Fully Synthetic, columns and 3 for Most Recent Models, and column shows the case of Randomly Sampled Data. As we can see, when the window size K is increased, total collapse is delayed but still eventually happens. Observe that there are multiple visible horizontal yellow lines in the second row for window size This is because p(m) can converge to any of the Dirac masses ò; provided pi 0 by Proposition [ similar phenomenon can be observed in the second and third columns. On the other hand, in column the model does deteriorate but the deviation plateaus very quickly. In particular, the 100 runs produce only one Dirac mass over the span of 500 generations, suggesting that randomly sampling from all the past models is qualitatively different from sampling from the most recent models with
fixed K. We remark that for K > 1, even p(")) is some Dirac mass, p(m-1) could still regain randomness from models before generation m, but with fixed window size all the randomness eventually disappears.
17

---

Published as a conference paper at COLM 2024
Window Size
Window Size
Window Size = 10
Window Size m
Evolution of 9m
Evolution of Om
Evolution of Ors
Evolution de:
120
200
300
400
500
100
200
300
400
502
1000
200
300
400
500
100
200
300
500
Evolution of |p p
Evolution Ip p
Evolution Iotm (
Evolution | km
Ip/mt
Distribution on of
Distribution
Distribution
Distriutiinn
Mean 17
Mean 95
Meen 216
Mean
Count
22
100
200
300
402
500
100
200
300
400
500
100
200
302
100
503
100
200
300
400
500
Generation m
Figure 4: Experiment for Most Recent Models and Randomly Sampled Data. A hundred experiments were run for 500 generations for different window sizes and fixed sample size 10.
In the first two rows, each yellow line represents the evolution of om and || (m)) - p p() [] one experiment respectively, with the red line being the empirical mean across 100 runs. The bottom row plots the histogram of total collapse time T with the red line being the empirical mean. The initial distribution p satisfies 600, 5 = 52 and So 0.1.
additional samples Evolution of Gg
10 additional samples Evolution of o..
100 additional samples Evolution of on.
500 additional samples Evolution of O..
0.20
Empitcal
[ow]
6
0.15
Evolution of [lpim) 1!
Evolution of Ip -p
Evolution of Ip p
Evolution |p
p
02
(pir)
Empltial
0.0
30
Generstion
with different sample sizes A hundred experiments were run for 50 generations for N 100 and different values of Each yellow line represents the evolution of om (top row) or | p (1) 1 (bottom row) one experiment, with the red line being the empirical mean across 100 runs. The blue dashed lines plot the formula for Sm given by (9) The initial distribution satisfies = 600, = 52, and Sg = 0.1.
Figure 5: Partially Synthetic
18

---

Published as a conference paper at COLM 2024
Sp = 0.01, 5= 484
5o 0.1, 5 52
5o = 0.5, 5=
5o 0.7, 5 16
lagí1
Evolution of
Evolution of
Evolution of
Evolution of
0.
Errpator mean
Sy c/oul
Evolution of [ip!n) p!
Evolution of [ip!m) p!l)
Evolution of [ip!m) pl
Evolution of llp
0.15
Errieioa
0.3
Generation
Figure 6: Partially Synthetic case with different initial distributions p. A hundred experi- ments were run for 50 generations for 10, N = 100 and different initial distributions p shown in the top row. Notice that as 5g increases, the deviation [|p(m) - p1D decreases, as suggest by the inequality (19.
19


Reviews:
[{"review_id": "jNYF1ImcI8", "reviewer": "Reviewer_k9FW", "summary": "The paper performs a theoretical analysis of recursive training loops in language model training, i.e., where synthetic data generated by language models contaminates the data used for training future iterations of language models. Prior work has found that this phenomenon leads to \"model collapse,\" where models' outputs become very limited as the amount of synthetic data in the training set increases. This paper builds on one of these works (Shumailov et. al. 2023), particularly exploring the setting of models for next token prediction. They conclude that theoretically, model collapse should consistently occur in the fully synthetic scenario. In the partially synthetic scenario, an upper bound on the proportion of synthetically generated data in the training set is identified to mitigate model collapse. The authors confirm these results using experiments with toy models.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and the recognition that our work is among the first to provide direct theoretical insights into recursively training language models. We would like to bring the following to the reviewer’s attention:\n1. We choose to give a concise presentation in Sect 2.1 due to space constraint. The distinction between $x_l$ and $e_j$ arises from the assumption that contexts are represented by canonical (basis) vectors $e_j$ (i.e. one-hot embeddings in ML language), as explained in Sect 2.1 (“Note that in current …”). Additionally, in Eq (1), $M$ denotes the number of all (context, next-token) pairs in the dataset, while $N=|C_j|$ denotes the number of data points with context $j$ so $N=N_j$ in fact depends on $j$ and $M=\\sum_j N_j$. We suppress the subscript of $N_j$ for brevity. A derivation for $p^{(m)}$ being unbiased will be added to the appendix.\n\n2. We remark that our main results (in particular Theorems 1 and 4) hold context-wise and easily extend to all contexts under certain conditions. For example, if we assume that the number of synthetic samples $n_j$ for each context j is fixed across generations, we directly obtain uniform bounds on the expected collapse time or distribution shift which holds true for all contexts. \nA natural way to relax this assumption is to consider the autoregressive model governed by Eq (5) so that $n_j$ depends on the conditional distributions given other contexts, but this significantly complicates the analysis. Instead, in Sect 4 we perform exactly such an experiment with transformer model. In the top-left corner of Fig 3, we report the average deviation from the original distribution (across 300 randomly sampled contexts) which confirms the dichotomy between whether or not original data is retained for training, in accordance with our theoretical findings.\n\n3. Our contributions beyond Shumailov et al. (2023) consist in the direct quantification of LM deterioration in recursive training. Their treatment on LM is indirect: theoretically, they only analyze Gaussian distributions which do not apply to LMs where output distributions are discrete. In contrast, our setup is more closely related to the actual setting in LM recursive training and we provide direct analysis accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The topic of the paper is very relevant, since the amount of model generated data on the web has drastically increased in recent years. Previous studies have explored this phenomenon empirically, but few have provided theoretical insights, which are important when deciding how to build the next generation of of language models.\n* The derivations seem sound, although there is a chance that I missed an error in one or more of the proofs.\n* There is a comprehensive discussion of related work, giving a nice overview of the recent study of model collapse\n* The authors generally provide good intuition for the mathematical results", "reasons_to_reject": "* The presentation could generally use some work. The paper is dense and hard to follow. For example, in the setup in 2.1, there’s switching between terms, e.g., x_l and e_j which are multiplied in the final linear layer, and N and M for number of training samples. I can’t follow the reasoning for the use of these multiple terms, and they make section 2.1 hard to understand. \n* The theoretical analysis is only for a single context. It’s unclear which conclusions would change when considering the large number of possible contexts. While the authors point this shortcoming out, the absence of such an analysis makes the paper much weaker, in my opinion, since this is the main contribution. \n* Experimental results are only for toy settings, although it would be difficult to perform experiments on realistically-sized language models with realistically-sized datasets\n* The contributions above Shumailov et. al. 2023 are arguably limited", "questions_to_authors": "* When commenting on the issue that it “is impossible that the learned model captures all the information about the original distribution”, is the issue that results from this related to the capacity of the model being trained, or to the data generated from previous models?\n* By canonical vectors, do you mean canonical basis vectors? They’re called one hot embeddings elsewhere, and it may be more clear to stick to one term.\n* After some thought, I can see why all models p^{(m)} are unbiased, but I think it would be best to explain this more explicitly to the reader", "extra_scores": {}}, {"review_id": "eqJXJ6gYof", "reviewer": "Reviewer_3UBU", "summary": "The paper addresses a problem that has emerged as part of the move towards large-scale generative AI content. More specifically, the contribution is primarily of a theoretical nature but this also gets supported by some corresponding experimental work. The research question is to establish what happens to a model that gets augmented with synthetic data (or is entirely based on synthetic) data when recursively being trained on the output of the next generation of training. The take-home message is that fully synthetic data will ultimately lead to what has been coined “model collapse” in prior work. However, augmenting real data with synthetic data is shown to be less prone to the problem, in particular when the proportion of synthetic data is being controlled.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and suggestions. We provide responses to the reviewer's main concerns.\n\n1. To study *the impact (over multiple generations) of using synthetic data in the pretraining of LLMs*, we consider recursive training where each generation is trained on either synthetic data or a combination of synthetic and real data. The Fully Synthetic setting (while it may not correspond to any practical use case) proves the point that information will eventually be lost over generations if we train new LLMs solely on synthetic data. On the contrary, in The Partially Synthetic setting (real+synthetic), degradation in model performance can be controlled if we carefully choose the ratio of real to synhetic data\n\n2. Synthetic data can indeed be generated and used in various ways. E.g, one could *prune* the synthetic dataset to only keep low probability samples (to make them more present in the training data for next gen LLMs). In this paper, we only consider synthetic data obtained directly by sampling the output of an LLM (without post-sampling processing). Other approaches are beyond the scope of this work. We will include this discussion in the revised paper.\n\n3. Our theoretical results consider a simple model for tractability. Extending the theory to transformers is currently challenging given the fact that the transformer architecture is highly complex. However, our experiments demonstrate the generalizability of our findings (derived on the simple model) in the context of transformers. In Fig 3, the Fully Synthetic and Partially Synthetic settings behave qualitatively differently – the former exhibits model collapse while the latter stabilizes in terms of model deviation. Even though the exact quantification of distribution drift is difficult to estimate for transformers, the result of this experiment aligns with our theoretical derivations in Theorems 1 and 4.\n\n4. We respectfully disagree with the reviewer on this point. We do not provide experiments with larger models because it is computationally infeasible given our compute budget. In this regard, we would like to refer the reviewer to the last point of the Review Guidelines, where it is stated that \"**we ask that you take into account that most researchers do not have access to large-scale compute**\".\n\n5. Extended limitations section: we will include the aforementioned points and concerns raised by the reviewers.\n\nWe hope our response clarifies some potential misunderstandings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "There are several key strengths of the paper including: \n\n(1) Topically this is a great fit. \n\n(2) The use of synthetic data to increase the pool of training data has become a very active research area, in particular since the release of GPT-2 a few years ago. This could be done for a number of reasons but often due to the lack of quality training data (and sometimes the cost associated with annotating more). My reading of the paper is that the authors foresee a future where much of the data out there is being generated by LLMs (although I do not think this is made explicit in the paper). \n\n(3) The authors provide a detailed theoretical analysis and back up their findings with (some limited) experimental work.", "reasons_to_reject": "I have a number of concerns, some about the underlying assumptions and others in regards of the experimental work:\n\n(1) The use of synthetic data has been demonstrated to be effective in some settings (and for some NLP tasks) but not for others. However, the usual approach appears to be that of data augmentation where a proportion of synthetic data is being generated (and then filtered before being added to the original dataset). I also understand the appeal of plain synthetic data in circumstances where data privacy might be an issue (e.g. synthetic patient records). However, I wonder how realistic the assumption of continuous recursive training is. Clearly, theoretically it is something worth exploring but I am lacking the actual use case. \n\n(2) In the discussion of related work I would like to hear more about different approaches / paradigms / families of synthetic data generation and then (later) have a discussion about how that choice might affect the findings. \n\n(3) The assumptions made in the theoretical setup (Section 2.1) simplify the concept of language models quite substantially (e.g. not considering transformer architectures). How generalisable are the findings with such simplifying assumptions? \n\n(4) Similarly, the choice of experimental setup in Section 4 is also quite narrow. A fairly tiny corpus is picked on a simple prediction task and GPT-2 is being deployed. All this hints at difficulties of obtaining generalisable insights for practical applications at scale. \n\n(5) There might be some good answers to my concerns and part of this should probably be done as part of a Limitations section. Currently there is no such discussion. \n\n\nSome other issues: \n\n(1) It would be good to include some compelling running examples. The authors reference (Shumailov et al., 2023), and that paper appears to be doing a great job in that respect. \n\n(2) Almost all entries in the bibliography have essential details missing.", "questions_to_authors": "Please see comments above", "extra_scores": {}}, {"review_id": "Gv5EcBqJa7", "reviewer": "Reviewer_Uzr5", "summary": "This paper presents a theoretical analysis on language model collapse from model generated synthetic data, in a recursive training setup. In this setup, the new generation of model is trained on synthetic data generated from the last generation. From theoretical perspective, the authors prove that model generated synthetic data will cause language model collapse when it is the single data source in the recursive training setup. When combining with real training data, the authors give an upper bound of model generated synthetic data to avoid the collapse. Besides, this paper also provides some empirical results and analysis to align with their theoretical anaylsis.\n\nOverall, the conclusions and analysis presented in this paper are interesting and insightful for language model collapse related research. It is an important step to understand the impact of model generated data in LLM training.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and suggestions. We recognize that our main statistical analysis results are based on the more restrictive recursive training setup and that our experiments are conducted on a small model. Due to limited computational resources, we did not conduct experiments on larger language models. In addition, we address the first limitation in App. C.2 (Fig 4) by performing experiments in more realistic settings “Most Recent Models” and “Randomly Sampled Data”. These two settings are meant to model the scenario of collecting text data online which have been contaminated with machine-generated content. In particular, when data generated by the most recent K models are sampled as the training set (the Most Recent Models setting), model collapse is delayed but still eventually happens, qualitatively similar to the Fully Synthetic setting. On the other hand, our theoretical result suggests that the Partially Synthetic setting behaves differently qualitatively. The deviation probability in Theorem 4 provides guidance on how much real data to incorporate to curb distribution shift.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a solid theoretical analysis to reveal the impact of model generated synthetic data in recursive training setup. This paper would be interesting to LLM developers and researchers to get them more insights.", "reasons_to_reject": "Though the analysis is interesting, all the conclusions in this paper have a strong condition: recursive training setup. This condition might not apply to most of the existing LLM effort. Besides, the empirical results in this paper are based on a lightweight setup, while realistic LLM training has much larger scale. It is not clear if the same empirical results can hold on large scale setup.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a statistical model to characterize the impact of recursive training scenarios to understand model collapse in language models. Reviewers all agreed that this work is very timely relevant and presents a solid theoretical analysis, together with experimental work. There are a few concerns about the assumption of continuous recursive training, the very lightweighted empirical results, as well as generalization to more realistic settings. We suggest the authors to follow reviewers’ comments to revise their draft and also add limitation discussion.

[comments from the PCs] Please follow up on the AC request, and also include a limitation section.
Decision: Accept