How bad is training on synthetic data? A statistical analysis of language model collapse

Authors: mohamed  el
amine  seddik
suei-wen  chen
soufiane  hayou
pierre  youssef
merouane  debbah
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-18 09:51:42

Abstract: Model collapse, as introduced in (Shumailov et al., 2023), refers to the phenomenon where training models on synthetic data generated from previously trained models leads to a deterioration in performance. This recursive training loop makes the tails of the original distribution disappear, thereby making future-generation models forget about the initial (real) distribution. With the aim of rigorously understanding model collapse in language models, we consider in this paper a statistical model that allows us to characterize the impact of various recursive training scenarios. Specifically, we demonstrate that model collapse cannot be avoided when training solely on synthetic data. However, when mixing both real and synthetic data, we provide an estimate of a maximal amount of synthetic data below which model collapse can eventually be avoided. Our theoretical conclusions are further supported by empirical validations.

Main paper: 
 introduction: 
 The large-scale adoption of large language models (e.g. ChatGPT (OpenAI, 2024) ) will inevitably lead to enormous amounts of synthetic (generated) data "polluting" the original human-created web data. Since language models are trained on web data, this raises some concerns about the impact of this synthetic data on the next generations of LLMs. One can think of a train-generate loop where models from the current generation generate data that contaminate existing web data, and the next generation models are trained on this contaminated data. This loop was studied in several works (Shumailov et al., 2023; Alemohammad et al., 2023; Briesch et al., 2023) where the authors conclude that synthetic data generally hurts performance as the number of generate-train increases, and that (naturally) the impact on model performance is linked to the amount of real data in the training set. A particular phenomenon, coined model collapse (Shumailov et al., 2023) , refers to the model's tendency to produce limited or repetitive outputs making recursive training on such outputs forget about the tails of the original underlying distribution of real data. This was further studied in (Guo et al., 2023) where the authors show that recursive training on synthetic data leads to a "self-consuming" loop that affects linguistic diversity.Intuitively, model collapse is a result of the distribution shift that occurs when training generative models recursively on synthetic data from previous generation models. Shumailov et al. (2023) have discussed two main sources of model collapse; 1) statistical approximation error: which is inherently related to the fact that generative models are trained on a finite number of samples, and therefore it is impossible that the learned model captures all the information about the original distribution. 2) functional approximation error: which results from the fact that the models are insufficiently expressive in real implementations, even if neural networks are known to be universal functional approximators from a theoretical standpoint. The authors provide further theoretical intuition to characterize the effect of these approximation errors, relying on simple mathematical models such as single-dimensional Gaussian distribution.In this paper, we aim to provide a rigorous theoretical framework to understand the effects of recursive training with synthetic data. In particular, we focus on the statistical approximation error and introduce a simple next-token-prediction language model to characterize model collapse. Our model allows us to gain insights into the behaviour of the self-consuming train-generate loop leading to model collapse. From a theoretical standpoint, we consider two main recursive training scenarios:• Fully Synthetic: Training with data sampled from the previous generation model.• Partially Synthetic: Training with a mixture of data sampled from the previous generation model and original data.We demonstrate that model collapse always occurs in the first scenario and characterize the rates at which it occurs. Furthermore, in the second scenario, we provide an upper bound on the sample size of generated data below which model collapse can eventually be attenuated. Our results are further confirmed through simulations of general scenarios with the introduced statistical model, as well as with realistic GPT2-style language models on real data. Our findings suggest that the amount of generated data should be considerably smaller compared to the original data to avoid model collapse.With the adoption of generative Large Language and Vision models, the amount of synthetic data on the web is growing at an unprecedented rate -see for example (del Rio-Chanona et al., 2023) where the authors conducted an empirical study of the amount of synthetic data via activity monitoring on Stack Overflow, and (Alemohammad et al., 2023) where the authors showed that a dataset used to train Stable Diffusion contains synthetic data (Schuhmann et al., 2022) . In fact, practitioners are willingly using synthetic data to train next-generation models (Ben Allal et al., 2024; Gunasekar et al., 2023; Chen et al., 2024) .As we mentioned above, several works studied the recursive training loop where nextgeneration models are trained on synthetic data generated from previous-generation models. Shumailov et al. (2023) studied model collapse, a phenomenon that occurs in recursive training where the quality of model outputs tend to degrade by becoming e.g. repetitive.Similar phenomena were studied in (Alemohammad et al., 2023) where the authors call it Model Autophagy Disorder (MAD). Another empirical study by (Briesch et al., 2023) studied this same Self-Consuming phenomenon and observed that the degeneracy rate (naturally) depends on the number of fresh data in the training sample. In the same direction, several works showed that incorporating synthetic data in the training can hurt the performance of trained diffusion models (Bohacek & Farid, 2023; Martínez et al., 2023a; b) .Only a few works have tackled this question from a theoretical perspective. Shumailov et al. (2023) considered a simple recursive Gaussian distribution to provide an intuition as to why model collapse occurs, but no training is considered in that work. In (Fu et al., 2024) , authors studied recursive training of diffusion models (generally used to learn distributions over images) and obtained an upper bound on the total variation distance between the distribution of the original data and that of the synthetic data after T generations. Dohmatob et al. (2024b) studied the impact of synthetic data on scaling laws and, in a simple setting of linear regression, Dohmatob et al. (2024a) studied the behaviour of the test error of different generations and showed that a linear dependency of the degradation on generation number.In this work, we are interested in characterizing the distribution shift in synthetic data generated with a language model, as the number of generations increases. We consider a linear Softmax classifier for next-token prediction and study the distribution of the learned conditional probabilities as the number of generations increases. The closest work to ours is (Fu et al., 2024) where the authors study the distribution of synthetic data generated by a diffusion model instead, whereas the statistical model we are considering is closer in spirit to the realm of language models.The remainder of the paper is organized as follows. Section 2 presents our theoretical setup where we introduce our statistical model and the considered recursive training scenarios.Our main theoretical results are presented in Section 3. We further present some experiments in Section 4 to support our findings. Finally, Section 5 concludes the paper. 
statistical language model: 
 We consider a language model of vocabulary size s, context length denoted by ℓ and we further denote by c the number of possible contexts which is at most s ℓ . We suppose that the language data is generated from some unknown conditional probabilities given the contexts. That is, the probability of the next token being k ∈ [s] := {1, ..., s} given some context j = (j 1 , . . . , j ℓ ) ∈ [c] is denoted byp jk := P{Y = k | X = j},where X and Y denote discrete random variables representing a context and the nexttoken respectively. In practice, we do not have access to the true conditional probabilities p jk but rather a (large) corpus sampled according to p jk . In other words, we are given a dataset {(x l , y l )} l∈[M] of M samples of contexts and next-token pairs represented by x l ∈ {e 1 , . . . , e c } and y l ∈ {e 1 , . . . , e s } where e i 's denote the canonical vectors.Given this dataset, we consider approximating the underlying conditional probabilities via the Softmax classifier, which entails minimizing the categorical cross-entropy loss function:EQUATIONwhereσ(v) = exp(v) ∑ s k=1 exp(v k )is the Softmax function and the functions exp and log are applied entry-wise. Note that in current state-of-the-art language models, the x l 's are context representations computed via transformer models, whereas, in our setting, we choose to work with one-hot embeddings as representations for tractable theoretical analysis. Solving the above objective (see Appendix 8.1) yields the estimated conditional probability pjk of p jk which expresses as the following empirical mean:EQUATION)The estimated conditional probabilities pjk are the result of training on the original data. These conditional probabilities can be used to generate new synthetic data, which can be used (with or without additional fresh data from the original dataset) to train the nextgeneration model. In this paper, we are interested in characterizing the behaviour of pjk in this recursive training loop which we will formally define in the next section. Hereafter, without loss of generality, we consider a fixed context j with N := |C j | training samples {(e j , y l )} l of context-next-token pairs, where y l ∈ R s are independent multinomial random variables with one trial and parameter p = (p 1 , . . . , p s ) := (p j1 , . . . , p js ) ∈ [0, 1] s .(3)From (2), we notice that pjk is an estimate of p jk and we further denoteEQUATION)As such, p (0) := p corresponds to the ground-truth distribution whereas p (1) denotes the first-generation model trained on the real data {(e j , y l )} l∈C j .p (0) p (10) p (20) p (30) p (40) Figure 1 : Evolution of p (m) in the Fully Synthetic setting for vocabulary size s = 3, context length ℓ = 4, total contexts c = s ℓ = 81 and sample size n = 1000. The initial distribution p (0) is some random distribution over tokens. The trained conditional distributions converge towards Dirac measures over generations illustrating total collapse in Definition 1.In this section, we introduce the notations for recursive training. At generation m ≥ 1, suppose that we have samples {y(t)l } l generated by some past models p (t) for t ∈ {0, . . . , m -1} respectively. As such, similarly to (2), the model at generation m expresses asp (m) := 1 n (m) m-1 ∑ t=0 n (m) t ∑ l=1 y (t) l .(5)In other words, p (m) is obtained by training the model on a mixture of real and synthetic data generated from previously trained models. Here, n (m) tstands for the number of samples used to train the m-th generation model that are generated by model p (t) , andn (m) = n (m) 0 + • • • + n (m)m-1 is the total number of training samples used to train p (m) . Note that we do not generate samples from p = p (0) but rather we are given a corpus {y (0) l } l which represent original data.Note that by definition all the models p (m) are unbiased, i.e. Ep (m) = p, which means that they recover the original distribution in case of infinite sample sizes. However, in practical settings the sample sizes are finite, and therefore p (m) may deviate from p since its variance can be large for small values of n (m) t . In the remainder of the paper, we present analytical results to rigorously quantify the impact of sample sizes n (m) t on the variance of p (m) and how they affect the learned distribution to eventually cause model collapse (Shumailov et al., 2023) , a degenerative process affecting future generation models by either losing information about the tails of the initial distribution or inducing distribution shifts over generations. Specifically, we aim to quantify the rate of such deterioration, and to this end, we introduce a stricter version of model collapse that we call total collapse and define as follows.Definition 1 (Total Collapse). We say that total collapse occurs in the recursive training process if p (m) converges to some Dirac mass δ i for some token i ∈ [s] as m grows.Total collapse refers to the case where, under recursive training, the trained model p (m) completely loses information about the original distribution p over generations, leading to poor linguistic diversity.1 This phenomenon is illustrated in Figure 1 where we see convergence towards the vertices of the probability simplex, i.e. Dirac measures, over generations. With this definition of total collapse, we provide a quantitative analysis of two cases of recursive training:• Fully Synthetic: Training with synthetic data from the last model. Each generation p (m) is trained only on data generated by the previous model p (m-1) . More precisely, for some fixed n ∈ N we let n(m) t = n • 1{t = m -1} for all m ≥ 1 and 0 ≤ t < m.• Partially Synthetic: Training with a mixture of real and synthetic data from the last model. Each generation p (m) is trained on a mixture of real data and synthetic data generated by the previous model p (m-1) . More precisely, for some fixed n ∈ N we let nEQUATIONm-1 = n and n (m) t= 0 for all m ≥ 2 and 0 < t < m, and n(1)0 = N.Fully Synthetic corresponds to the theoretical setting considered in (Shumailov et al., 2023) . However, in that paper, only Gaussian distribution was analyzed instead of discrete distributions over all possible tokens as language models entail. We point out that this setting is unlikely to happen in real-world applications but serves as the worst-case scenario.On the other hand, the Partially Synthetic setting is more realistic for future generation models as it corresponds to training on a mixture of real and synthetic data. We consider this setting to assess whether it is possible to avoid collapse by having a fraction of the original data in the training mixture. We answer this question positively in the next section. Moreover, we show through simulations in Section 4 that conclusions from our theoretical analysis on both Fully Synthetic and Partially Synthetic hold beyond these simple settings, such as training on a mixture of all generations or even using realistic transformer models. 
main results: 
 To investigate the model collapse phenomenon and the rate at which it occurs, we define the following statistical quantities that capture the randomness of p (m) :∥p (m) ∥ ∞ := max i∈[s] p (m) i , σ m := ∥p (m) ∥ 2 2 = s ∑ i=1 p (m) i 2 and S m := E[σ m ].These quantities measure how far away p (m) is from some Dirac mass (Total Collapse, Definition 1). Since the maximum value of all three quantities is 1, the closer they are to 1, the closer p (m) is to some Dirac mass, and the less diverse p (m) is as a language model. As such, ∥p (m) ∥ ∞ or σ m being equal to 1 is equivalent to p (m) being a Dirac mass which reflects total collapse. To quantify the distribution shift incurred by the aforementioned recursive training scenarios, we further consider the 1-norm2 between two distributions µ, ν ∈ R s defined by ∥µ -ν∥ 1 := ∑ s i=1 |µ(i)ν(i)|. We assume that the initial distribution p (0) is nontrivial, specifically S 0 < 1 and ∥p (0) ∥ ∞ < 1. Under this assumption, we provide results on the rate of total collapse and further quantify distribution shift under recursive training. All the proofs are presented in Appendix 8.We start by describing the recursive training process in which only synthetic data from the last generation model are used. This process can be viewed as a Markov chain on the set of probability measures ∆ s-1 ∩ 1 n N s on [s] with denominator n, where∆ s-1 := {v ∈ R s : v 1 + v 2 + • • • + v s = 1, v i ≥ 0 for all i}is the probability simplex. Since the probability of reaching δ i is positive provided p i > 0, this random walk which starts at p (0) has absorbing states {δ i : i ∈ [s] s.t. p i > 0}. As a result, the random walk converges to one of the absorbing states almost surely (Kemeny et al., 1969) , which means that total collapse is bound to happen in the Fully Synthetic setting.To characterize the rate at which total collapse occurs, let us denote by T := inf{t ∈ N : 1. The expected sum of squared probabilities S m is given by∥p (t) ∥ ∞ = 1} ≥EQUATION)2. The probability ρ m that total collapse has occurred by generation m satisfiesEQUATION)3. The generation T at which total collapse happens satisfiesEQUATIONIn essence, Theorem 1 describes the behavior of p (m) as a function of the model generation m, the sample size n, and the dispersion of the initial distribution S 0 . Specifically, we draw the following observations:• Effect of the number of generations m: As m increases, S m and ρ m tend to 1 as per ( 6) and ( 7), making total collapse increasingly more likely. Note also that this dependence is exponential in m, making total collapse in this case relatively fast.• Effect of "synthetic" sample size n: The smaller n is, the more likely p (m) is to have collapsed for a given generation m as per the bounds on ρ m in (7), and the faster total collapse is expected to happen as suggested by ( 8). The dependence in this case is polynomial in n. • Effect of S 0 : Larger values of S 0 correspond to faster collapse as suggested by (8); namely, starting from an original distribution that is not diverse enough speeds up total collapse. The upper and lower bounds on E[T] are both linear in (1 -S 0 ).We point out that when the number of contexts c is large, the number of samples n per context would be fairly small, which suggests that total collapse, given a single context, is expected to happen fairly quickly. The bounds on E[T] state that the expected collapse time is at least of order n and at most of order n 2 , though experiments (see Figure 2 ) suggest that the upper bound is not sharp and should be close to O(n) instead. On another note, we characterize below in Proposition 1 the limiting distribution as m gets to infinity, which demonstrates the direction of total collapse.Proposition 1. In the Fully Synthetic case, we have P limm→∞ p (m) = δ i = p i for all i ∈ [s].Proposition 1 describes the limiting distribution when total collapse occurs in terms of the initial probabilities p i over tokens. Specifically, the resulting Dirac mass δ i is likely to be supported on some token i with high initial probability p i . This formally supports the description of early and late model collapse in Shumailov et al. (2023) : In the early phase of recursive training, the tails of the original distribution disappear because the probability of outputting unlikely tokens j (those j's for which p j is small) will decrease as p (m) converges to some Dirac mass δ i . After many generations, all but one p (m) jwill go to 0, exhibiting late model collapse where all the randomness of the original distribution is lost.As described in the previous section, total collapse is unavoidable when training solely on synthetic data. In this section, we consider the case in which real data is incorporated at each generation. In this case, p (m) can be simply seen as a weighted average of p (1) and an estimate of p (m-1) with n samples. We also assume that p (1) is nontrivial, thereby implying that, on average, p (m) will not be a Dirac mass. In what follows, we quantify the variability in the data distribution across different generations, as well as the distribution drift ∥p (m)p (1) ∥ 1 from the first-generation model. We particularly show that collapse can be avoided if enough real data is injected into the recursive training process.Theorem 2 (Model Variation). Consider the Partially Synthetic setting. For m ≥ 1 we haveEQUATION)where α := n N+n andβ := α (1 + 1 N )α -1 N .Theorem 2 provides a control of the variance S m in the Partially Synthetic setting. Essentially, when n ≪ N, we have S m ≈ 1 N + (1 -1 N )S 0 ≈ S 0 , which is not surprising since the training data for each generation model are dominated by real data in this case. However, even when the number of synthetic data is much larger than the original dataset (i.e. n ≫ N), we have α ≈ 1 ≈ β and hence S m+1 ≈ 1/N + (2 + 1/N) -1 (1 -1/N)(2 -1/N)S 0 , which approaches S 0 for large N.To further refine our analysis, we present a result that directly controls the deviation E∥p (m)p (1) ∥ 1 between the conditional distributions from first and m-th generations. This allows us to have a quantitative control over the distribution shift. When n is sufficiently small, we have a sharper control over this deviation by exploiting the concentration results Published as a conference paper at COLM 2024 from (Mardia et al., 2020) . Essentially, this allows us to estimate the maximum number of synthetic samples n to ensure that the distribution p (m) stays close to p (1) .Theorem 3 (Model Deviation). Consider the Partially Synthetic setting and defineEQUATION)where C 0 = e3 2π ≈ 3.19 and C 1 = 6e π 3/2 ≈ 2.93, and s is the vocabulary size. Then, for m ≥ 2,E∥p (m) -p (1) ∥ 1 < 1 N πn 2 G n (s).We point out that the upper bound on the deviation E∥p (m)p (1) ∥ 1 is independent of the generation m. Since the deviation from p (0) to p (1) is inevitable and independent of n, we give the result in E∥p (m)p (1) ∥ 1 , from which E∥p (m)p (0) ∥ 1 can be estimated by the triangle inequality. Theorem 3 allows us to estimate the maximum number of synthetic samples n that can be used if we want p (m) to stay ϵ-close to p (1) in L 1 norm. For example, when C 0 n/e + 2 ≤ s, for any ϵ > 0 we can takeEQUATIONin order for E∥p (m) p (1) ∥ 1 to be less than ϵ. In other words, to ensure small L 1 deviation n should be taken to be logarithmic in the ratio Nϵ/s, which highlights that the amount of synthetic data should be exponentially smaller compared to real data in order to ensure that p (m) remains close to p (1) . In Appendix 9, we provide experiments showing the effect of the sample size n and the initial distribution p. In Figure 5 , we show that if we fix the initial distribution and increase the amount of synthetic data n, the dispersion σ m stays relatively constant while the distribution drift ∥p (m)p (1) ∥ 1 increases. In contrast, when we fix n and increase the values of S 0 , σ m increases but ∥p (m)p (1) ∥ 1 actually decreases as depicted in Figure 6 . This behavior can be explained by Theorem 4 in Appendix 8, which is more general than Theorem 3 in the sense that it captures the dependence of E∥p (m)p (1) ∥ 1 on the randomness of p and hence provides a sharper bound on the expected deviation. We provide additional simulations in Appendix 9.2 to account more general scenarios. 
experiments: 
 To support our findings in a realistic setting, we conduct experiments with a decoder-only generative model for text generation. For this model, we consider the aforementioned Fully Synthetic and Partially Synthetic settings using the model parameters in Appendix 9.1. We consider a simple character-level tokenizer using the tiny Shakespeare dataset 3 yielding a vocabulary of size s = 65. We first train a model for 2000 iterations on this dataset and consider the trained model as the ground-truth distribution p (0) . The next-generation models are trained recursively using the exact same architecture and training parameters. This setting allows us to reduce the effect of functional approximation error thereby focusing only on the effect of statistical approximation error.The results of these experiments are summarized in Figure 3 . The top left plot therein depicts the deviation ∥p (m)p (1) ∥ 1 which is averaged over 300 random contexts generated by (Dohmatob et al., 2024a) .the ground-truth generative model p (0) . From this plot, we clearly see that ∥p (m)p (1) ∥ 1 diverges over generations in the Fully Synthetic setting, while mixing with real data ensures stability over generations.The effect of synthetic data is further noticed in the validation loss of the next-generation models where we see an overfitting effect in the Fully Synthetic setting. We point out that this effect might also be associated with the functional approximation error and was rigorously studied by Dohmatob et al. (2024a) in the case of linear regression, where the authors have shown that synthetic data affect the usual scaling laws. We believe that similar conclusions can be obtained with our statistical model by incorporating the functional approximation error, this can be achieved for instance by supposing that the context embeddings e i 's are high-dimensional Gaussian vectors instead of canonical vectors, thereby introducing the embedding dimension as a parameter controlling model complexity. 
discussions & conclusion: 
 In this paper, we studied model collapse in language models through a simple statistical model. We provided theoretical analysis when training with only synthetic data and when adding real data from the original distribution. Our results demonstrate that model collapse always happens when the model is training solely on synthetic data, whereas controlling deviation from the initial distribution requires careful choice of the amount of synthetic data to inject in the training set. We also provided experiments showing that these findings extend beyond the simple theoretical settings.Our current results describe only the statistical approximation error since all generation models are unbiased in our theoretical framework. However, as we discussed in the previous section, this framework can be extended to account for the functional approximation error by considering high-dimension Gaussian vectors as context embeddings instead of canonical vectors. Another possible extension is to study the effect of in-context learning (Wu et al., 2023) on model collapse, which is a key feature of transformer-based models.Despite the simple setting of our current investigation, we believe that it lays the groundwork for better understanding and mitigation of model collapse in language models, thereby opening the way for the development of more general theoretical frameworks to study next-generation language models dynamics. 
limitations: 
 Our analysis primarily focuses on a simplified recursive training setup, which may not fully represent current large language model training practices. The theoretical results are derived for a basic language model, and extending the analysis to more complex architectures like transformers remains challenging. Our empirical validations use relatively small-scale experiments due to computational constraints, and the generalizability of these findings to large-scale language models requires further investigation. While we explore more realistic settings in the appendix, such as "Most Recent Models" and "Randomly Sampled Data", these still represent idealizations of real-world scenarios. Additionally, our treatment of synthetic data generation does not account for various post-processing techniques that may be employed in practice. Future work should address these limitations by expanding the theoretical framework to more complex model architectures, conducting larger-scale experiments, and considering a broader range of synthetic data generation and utilization approaches. 
technical lemma: 
 For completeness, we include the concentration results from (Mardia et al., 2020) which we used to prove Theorem 3. For a probability distribution p on [s] defineEQUATIONand notice that π p ≤ 1 2 . Define the function φ: [0, 1/2) → R by φ(x) := 1 1 -2x log 1 -xx and extend φ by continuity to φ(1/2) := 2. Observe that φ is strictly decreasing and 2 ≤ φ(x) < ∞. The following Lemma concerns the concentration of empirical distribution which captures the dependence on the sample size n, dimension s as well as the structure of the underlying probability distribution via φ(π p ). Lemma 1. Let p be a probability distribution on [s] and p be the associated empirical distribution obtained from n samples. Then for ϵ > 0 we haveEQUATION)The upper boundP(∥ p -p∥ 1 ) ≤ (2 s -2) exp - nφ(π p )ϵ 2 4in fact holds for all values of s and n (Weissman et al., 2003) , but this bound is very poor when s/n is large, which is commonly the case for language models. Lemma 1 provides an improvement in the regime of small sample size n ≲ s. 
derivation of formula (2): 
 Proof. The categorical cross-entropy loss reads asL(w 1 , . . . , w s ) = - 1 M M ∑ i=1 s ∑ k=1 y ik log exp(w ⊤ k x i ) ∑ K j=1 exp(w ⊤ j x i ).The gradient of the loss is expressed as∂L ∂w k = 1 M M ∑ i=1 s ∑ ℓ=1 y iℓ ∂ ∂w k ∑ j̸ =ℓ exp((w j -w ℓ ) ⊤ x i ) 1 + ∑ j̸ =ℓ exp((w j -w ℓ ) ⊤ x i ) ,where∂ ∂w k ∑ j̸ =ℓ exp((w j -w ℓ ) ⊤ x i ) = -∑ j̸ =k exp((w j -w k ) ⊤ x i )x i if k = ℓ, exp((w k -w ℓ ) ⊤ x i ) if k ̸ = ℓ.Therefore,∂L ∂w k = 1 M M ∑ i=1 y ik -∑ j̸ =k exp((w j -w k ) ⊤ x i ) 1 + ∑ j̸ =k exp((w j -w k ) ⊤ x i ) + ∑ ℓ̸ =k y iℓ exp((w k -w ℓ ) ⊤ x i ) 1 + ∑ j̸ =ℓ exp((w j -w ℓ ) ⊤ x i ) x i = 1 M M ∑ i=1 ∑ ℓ̸ =k y iℓ exp(w ⊤ k x i ) ∑ s j=1 exp(w ⊤ j x i ) -y ik ∑ j̸ =k exp(w ⊤ j x i ) ∑ s j=1 exp(w ⊤ j x i ) x i = 1 M M ∑ i=1 ∑ ℓ̸ =k y iℓ exp(w ⊤ k x i ) ∑ s j=1 exp(w ⊤ j x i ) -y ik 1 - exp(w ⊤ k x i ) ∑ s j=1 exp(w ⊤ j x i ) x i = 1 M M ∑ i=1            s ∑ ℓ=1 y iℓ =1 exp(w ⊤ k x i ) ∑ s j=1 exp(w ⊤ j x i ) -y ik            x i = 1 M M ∑ i=1 exp(w ⊤ k x i ) ∑ s j=1 exp(w ⊤ j x i ) -y ik x iFinally, solving for ∂L ∂w k = 0 yields (2).Proof of Theorem 1. Write p(m) i = X (m) i /n where X (m) i ∼ B(n, p (m-1) i) is binomial when conditioned on p (m-1) . So we haveE[p (m) i 2 | p (m-1) ] = 1 n p (m-1) i + 1 - 1 n p (m-1) i 2 s ∑ i=1 E[p (m) i 2 | p (m-1) ] = 1 n + 1 - 1 n s ∑ i=1 p (m-1) i 2and by the law of total expectationEQUATION)Note that S m ↗ 1 as m → ∞ andS m = E ∑ i p (m) i p (m) i ≤ E ∑ i p (m) i ∥p (m) ∥ ∞ = E∥p (m) ∥ ∞ .Recall that ρ m := P ∥p (m) ∥ ∞ = 1 andT := inf{m ∈ N : ∥p (m) ∥ ∞ = 1} ≥ 1. Since σ m ≥ 1/ s and ∥p (m) ∥ ∞ ∈ { 1 n , . . . , n-1 n , 1}, we have 1 • ρ m + 1 s (1 -ρ m ) ≤ Eσ m = S m ≤ E∥p (m) ∥ ∞ ≤ 1 • ρ m + (1 -1/n)(1 -ρ m ), from which (7) follows thanks to (14).For k = 1, 2, . . . we have P(T > k) = P(∥p (k) ∥ ∞ < 1) = 1 -ρ k and thus 1 -S 0 1 -1/ s (1 -1/n) k ≤ P(T > k) ≤ n(1 -S 0 )(1 -1/n) k , which establishes (8) because E[T] = ∑ ∞ k=0 P(T > k).Proof of Proposition 1. Fix i ∈ [s]. For m ≥ 1 consider the events E m := {T ≤ m} andF m := {p (m) i = 1}. Then p (m) i ∈ {0, 1} on E m and F m ⊆ E m . Observe that E m ↗ ∪ m E m and F m ↗ ∪ m F m = { lim m→∞ p (m) = δ i },where P(∪ m E m ) = 1 by Theorem 1 and in particular P(E m ) > 0 for m large. Thusp i = lim m→∞ E[p (m) i ] = lim m→∞ P(E m )E[p (m) i | E m ] + P(E c m )E[p (m) i | E c m ] = lim m→∞ E[p (m) i | E m ] = lim m→∞ P(p (m) i = 1 | E m ) = lim m→∞ P(F m ∩ E m ) P(E m ) = lim m→∞ P(F m ) P(E m ) = lim m→∞ P(F m ) = P(∪ m F m ),which proves the assertion.Proof of Theorem 2. Let νEQUATION)For m ≥ 2, since y(m-1) i,k is conditionally independent of y (0) i,j given p (m-1) i, by conditioning on yi,j and p (m-1) iwe haveE y (0) i,j y (m-1) i,k = E y (0) i,j p (m-1) i and hence for m ≥ 2, ν (m) i = p i N ′ + N 2 -N N ′2 p 2 i + n 2 -n N ′2 ν (m-1) i + 2n N ′2 N ∑ j=1 E y (0) i,j p (m-1) i.By the definition of pEQUATION.Setting α := n/N ′ , we have N = (1α)N ′ and henceE y (0) i,j p (m-1) i = p i + (N -1)p 2 i Nfor all m ≥ 2. Plugging this back to the expression (15) for ν(m) i gives ν (m) i = (1 -α)(1 + 2α) N p i + 1 - 1 N (1 -α 2 )p 2 i + α 1 + 1 N α - 1 N ν (m-1) i . Let β := α 1 + 1 N α -1 N . Then for m ≥ 1, ν (m+1) i = 1 N 1 -β m+1 1 -β -α(2 -α) 1 -β m 1 -β p i + 1 - 1 N 1 -β m+1 1 -β -α 2 1 -β m 1 -β p 2 i = 1 N p i 1 + (1 + 1/N)α 1 + 2α -1 - 1 N αβ m + (1 -1 N )p 2 i 1 + (1 + 1/N)α 1 + α - 1 N αβ mand summing across i ∈ [s] gives the expression for S m+1 .Note that 0 < β < α < 1, which gives both an upper bound and a lower bound on ν(m) i . In particular, for m ≥ 2 ν (m) i < 1 1 + (1 + 1/N)α 1 N (1 + 2α)p i + 1 - 1 N (1 + α)p 2 iand thereforeS m < 1 1 + (1 + 1/N)α 1 N (1 + 2α) + 1 - 1 N (1 + α)S 0 = 1 - 1 -1 N (1 + α)(1 -S 0 ) 1 + (1 + 1/N)α ,whereγ := 1 -1 N (1 + α)(1 -S 0 ) 1 + (1 + 1/N)α ∈ 1 + α 2 + 3α (1 -S 0 ), 1 -S 0 since 0 ≤ 1/N ≤ 1/2.We state a more general result which implies Theorem 3. Theorem 4. Let G n (s) and φ be as in Lemma 1. Then for k ≥ 1 we haveEQUATIONwhereζ = 1 2 - 1 2 -2E max A⊆[s] p (1) (A)p (1) ([s] \ A) N N + n 2 .Proof of Theorem 4. Since all generations share the same original data source, we can writeEQUATIONi } i=1,...,n are i.i.d. multinomial with parameter p (k) and one trial (k ≥ 1). This gives 1) , and applying the triangle inequality yieldsEQUATION∥p (k+1) -p (1) ∥ 1 = n N + n p (k) -p (1) 1 ≤ n N + n p (k) -p (k) 1 + n N + n p (k) -p (1) 1 .Taking the expectation and solving the recursion gives for k ≥ 1EQUATION)From Lemma 1, we haveP p (k) -p (k) 1 ≥ t p (k) ≤ G n (s)e -nφ(π k )t 2 /4where π k := π p (k) . Integrating over t ∈ [0, ∞), we getE p (k) -p (k) 1 p (k) ≤ G n (s) π nφ(π k ) ,and by Jensen's inequality and the concavity of x → φ(x) -1/2 , E p (k)p (k) 1≤ π n G n (s)E φ(π k ) -1/2 ≤ π nφ(Eπ k ) G n (s).ThusEQUATION)It remains to upper upper bound Eπ j since φ is decreasing. For A ⊆ [s] let A c denote its complement. Then by ( 17), for A ⊆ [s] we havep (k+1) (A)p (k+1) (A c ) = N 2 (N + n) 2 p (1) (A)p (1) (A c ) + n 2 (N + n) 2 p (k) (A) p (k) (A c ) + Nn (N + n) 2 p (1) (A c ) p (k) (A) + p (1) (A) p (k) (A c ) ≤ N 2 (N + n) 2 p (1) (A)p (1) (A c ) + n 2 (N + n) 2 p (k) (A) p (k) (A c ) + Nn (N + n) 2 .Let λ k := max A⊆[s] p (k) (A)p (k) (A c ), so the inequality above impliesE[λ k+1 | p (1) , p (k) ] ≤ N 2 (N + n) 2 λ 1 + n 2 (N + n) 2 E max A⊆[s] p (k) (A) p (k) (A c ) p (k) + Nn (N + n) 2 ≤ N 2 (N + n) 2 λ 1 + n 2 4(N + n) 2 + Nn (N + n) 2and thusE[λ k+1 ] ≤ N 2 E[λ 1 ] + Nn + n 2 /4 (N + n) 2 = 1 4 - 1 4 -E[λ 1 ] N N + n 2 .Observe that for 0 ≤ x ≤ 1 we have min(x, 1x) ≤ 2x(1x), soE[π k ] ≤ 2E[λ k ]. Writing ζ := 1 2 - 1 2 -2E[λ 1 ] N N + n 2we have E[π k ] ≤ ζ, so from (19) we see thatEQUATION)Proof of Theorem 3. Theorem 3 is an immediate consequence of Theorem 4 by replacing φ with its minimum value 2. 
architecture & training parameters for gpt2 experiments: 
 We consider that all generation models have GPT2-type4 vanilla architecture which is a decoder-only generative model with the configuration and training parameters as summarized in Table 1 . These parameters were chosen to achieve the best validation loss when training the first-generation model on data produced by ground-truth generative model p (0) as defined in Section 4.  
 
 Appendix: Note that here we define total collapse with respect to a single pre-fixed context. This can be generalized to the event where total collapse occurs for all contexts. However, the theoretical analysis in this case would require more refined treatment of several statistical quantities to obtain uniform bounds over contexts. We leave this for future work.We point out that the 1-norm is twice the total variation distance ∥µ -ν∥ TV , which is another commonly used metric for probability distributions and was considered byFu et al. (2024) for studying model collapse in the case of diffusion models.https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/ input.txthttps://github.com/karpathy/nanoGPT 


Decision: This paper presents a statistical model to characterize the impact of recursive training scenarios to understand model collapse in language models. Reviewers all agreed that this work is very timely relevant and presents a solid theoretical analysis, together with experimental work. There are a few concerns about the assumption of continuous recursive training, the very lightweighted empirical results, as well as generalization to more realistic settings. We suggest the authors to follow reviewers’ comments to revise their draft and also add limitation discussion.

[comments from the PCs] Please follow up on the AC request, and also include a limitation section.

Reviews: review: paper_summary: paper_summary: The paper performs a theoretical analysis of recursive training loops in language model training, i.e., where synthetic data generated by language models contaminates the data used for training future iterations of language models. Prior work has found that this phenomenon leads to "model collapse," where models' outputs become very limited as the amount of synthetic data in the training set increases. This paper builds on one of these works (Shumailov et. al. 2023), particularly exploring the setting of models for next token prediction. They conclude that theoretically, model collapse should consistently occur in the fully synthetic scenario. In the partially synthetic scenario, an upper bound on the proportion of synthetically generated data in the training set is identified to mitigate model collapse. The authors confirm these results using experiments with toy models.
strength_weakness: strength_weakness: * The presentation could generally use some work. The paper is dense and hard to follow. For example, in the setup in 2.1, there’s switching between terms, e.g., x_l and e_j which are multiplied in the final linear layer, and N and M for number of training samples. I can’t follow the reasoning for the use of these multiple terms, and they make section 2.1 hard to understand. 
* The theoretical analysis is only for a single context. It’s unclear which conclusions would change when considering the large number of possible contexts. While the authors point this shortcoming out, the absence of such an analysis makes the paper much weaker, in my opinion, since this is the main contribution. 
* Experimental results are only for toy settings, although it would be difficult to perform experiments on realistically-sized language models with realistically-sized datasets
* The contributions above Shumailov et. al. 2023 are arguably limited
questions: questions: * When commenting on the issue that it “is impossible that the learned model captures all the information about the original distribution”, is the issue that results from this related to the capacity of the model being trained, or to the data generated from previous models?
* By canonical vectors, do you mean canonical basis vectors? They’re called one hot embeddings elsewhere, and it may be more clear to stick to one term.
* After some thought, I can see why all models p^{(m)} are unbiased, but I think it would be best to explain this more explicitly to the reader
paper_summary: paper_summary: The paper addresses a problem that has emerged as part of the move towards large-scale generative AI content. More specifically, the contribution is primarily of a theoretical nature but this also gets supported by some corresponding experimental work. The research question is to establish what happens to a model that gets augmented with synthetic data (or is entirely based on synthetic) data when recursively being trained on the output of the next generation of training. The take-home message is that fully synthetic data will ultimately lead to what has been coined “model collapse” in prior work. However, augmenting real data with synthetic data is shown to be less prone to the problem, in particular when the proportion of synthetic data is being controlled.
strength_weakness: strength_weakness: I have a number of concerns, some about the underlying assumptions and others in regards of the experimental work:

(1) The use of synthetic data has been demonstrated to be effective in some settings (and for some NLP tasks) but not for others. However, the usual approach appears to be that of data augmentation where a proportion of synthetic data is being generated (and then filtered before being added to the original dataset). I also understand the appeal of plain synthetic data in circumstances where data privacy might be an issue (e.g. synthetic patient records). However, I wonder how realistic the assumption of continuous recursive training is. Clearly, theoretically it is something worth exploring but I am lacking the actual use case. 

(2) In the discussion of related work I would like to hear more about different approaches / paradigms / families of synthetic data generation and then (later) have a discussion about how that choice might affect the findings. 

(3) The assumptions made in the theoretical setup (Section 2.1) simplify the concept of language models quite substantially (e.g. not considering transformer architectures). How generalisable are the findings with such simplifying assumptions? 

(4) Similarly, the choice of experimental setup in Section 4 is also quite narrow. A fairly tiny corpus is picked on a simple prediction task and GPT-2 is being deployed. All this hints at difficulties of obtaining generalisable insights for practical applications at scale. 

(5) There might be some good answers to my concerns and part of this should probably be done as part of a Limitations section. Currently there is no such discussion. 


Some other issues: 

(1) It would be good to include some compelling running examples. The authors reference (Shumailov et al., 2023), and that paper appears to be doing a great job in that respect. 

(2) Almost all entries in the bibliography have essential details missing.
questions: questions: Please see comments above
paper_summary: paper_summary: This paper presents a theoretical analysis on language model collapse from model generated synthetic data, in a recursive training setup. In this setup, the new generation of model is trained on synthetic data generated from the last generation. From theoretical perspective, the authors prove that model generated synthetic data will cause language model collapse when it is the single data source in the recursive training setup. When combining with real training data, the authors give an upper bound of model generated synthetic data to avoid the collapse. Besides, this paper also provides some empirical results and analysis to align with their theoretical anaylsis.

Overall, the conclusions and analysis presented in this paper are interesting and insightful for language model collapse related research. It is an important step to understand the impact of model generated data in LLM training.
strength_weakness: strength_weakness: Though the analysis is interesting, all the conclusions in this paper have a strong condition: recursive training setup. This condition might not apply to most of the existing LLM effort. Besides, the empirical results in this paper are based on a lightweight setup, while realistic LLM training has much larger scale. It is not clear if the same empirical results can hold on large scale setup.

Comments: title: Response
Response to rebuttal
Update
comment: Thank you for the clarifications. While some of my concerns are assuaged, my overall perception of the work remains the same. I will therefore keep my score the same.
I thank the authors for their response to my concerns. The rebuttal does contain partial answers to some of my concerns and I assume that the authors will update the paper accordingly if invited to submit a revision. 

My two concerns regarding (a) tiny sample size and (b) lack of realistic use case remain as before. I also note that these two points have been raised in the other reviews as well. Nevertheless I will raise my overall score slightly as I do not think that there is a systematic flaw in the work. It simply is not strong enough in my view to warrant a clear accept.
Given that the discussion deadline is in two days, we would appreciate it if the reviewer could provide feedback on our rebuttal, particularly noting if there is anything unclear.