Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
Authors: Matthias Gerstgrasser
Rylan Schaeffer
Apratim Dey
Rafael Rafailov
Tomasz Korbak
Henry Sleight
Rajashree Agrawal
John Hughes
Dhruv Bhandarkar Pai
Andrey Gromov
Dan Roberts
Diyi Yang
David L. Donoho
Sanmi Koyejo
Venue: COLM (2024)
Keywords: model collapse, curse of recursion, generative models, model-data feedback loops
TL;DR: Accumulating Data Prevents Model Collapse

Abstract: The proliferation of generative models, combined with pretraining on web-scale data, raises a timely question: what happens when future models are trained on model-generated data? Recent investigations answered that such model-data feedback loops cause performance to progressively degrades with each model-data iteration until fitted models become useless, a phenomenon termed model collapse. However, those studies largely assumed that new data replace old data over time, where a more realistic assumption is that data accumulate over time. In this paper, we ask: what effect does accumulating data have on model collapse? 
We first empirically study this question by pretraining sequences of language models on text corpora. After confirming that replacing the original real data by each generation's synthetic data does indeed tend towards model collapse, we demonstrate that accumulating synthetic data with real data avoids model collapse; these results hold across a range of sizes, architectures, and hyperparameters. We obtain similar results for other deep generative models: diffusion models for molecule conformation generation and variational autoencoders for image generation. To understand why accumulating data can avoid model collapse, we use an analytically tractable framework introduced by prior work in which a sequence of linear models are fit to previous models' outputs. Previous work used this framework to show that if data are replaced, the test error increases with the number of model-fitting iterations; we extend this argument to prove that if data instead accumulate, the test error has a finite upper bound independent of the number of iterations, meaning model collapse is avoided. 
Our work provides consistent empirical and theoretical evidence that data accumulation avoids model collapse.

Published as a conference paper at COLM 2024
Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
Matthias Gerstgrasser"
Rylan Schaeffer" Stanford University
Apratim Dey" Stanford University apd 995 oord.ord.d.oo edu
Harvard & Stanford University matthiaséseas. harvard. edu
Rafael Rafailov* Stanford University rafailov@stanford.eddu
Henry Sleight*
John Hughes* +
Tomasz Korbakt
Constellation
Constellation
Constellation
Rajashree Agrawal* Constellation
Dhruv Pai Stanford
Andrey Gromov UMD College Park
Daniel A. Roberts MIT & Sequoia Capital
Diyi Yang Stanford University diyiy@stanford.c
David Donoho Stanford University
Sanmi Koyejo Stanford University sanmi@stanf edu
donoho@stanforde edu
Abstract
The proliferation of generative models, combined with pretraining on web- scale data, raises a timely question: what happens when future models trained on model-generated data? Recent investigations answered that such model-data feedback loops cause performance to progressively degrades with each model-data iteration until fitted models become useless, a phe- nomenon termed model collapse. However, those studies largely assumed that new data replace old data over time, where a more realistic assumption is that data accumulate over time. this paper, we ask: what effect does accumulating data have on model collapse? We first empirically study this question by pretraining sequences of language models on text corpora. After confirming that replacing the original real data by each generation's 's synthetic data does indeed tend towards model collapse, we demonstrate that accumulating synthetic data with real data avoids model collapse; these results hold across a range of sizes, architectures, and hyperparameters. We obtain similar results for other deep generative models: diffusion models for molecule conformation generation and variational autoencoders for image generation. To understand why accumulating data can avoid model collapse, we use an analytically tractable framework introduced by prior work which a sequence of linear models are fit to previous models' out- puts. Previous work used this framework to show that if data are replaced, the test error increases with the number of model-fitting iterations; we extend this argument to prove that if data instead accumulate, the test error has a finite upper bound independent of the number of iterations, meaning model collapse is avoided. Our work provides consistent empirical and theoretical evidence that data accumulation avoids model collapse.
Introduction
The advent of large-scale generative models such as GPT-4 (Achiam et al., 2023), DALL-E (Ramesh et al., 2022) and Stable Diffusion (Rombach et al., 2022) has revolutionized the
DDnnotes equal authorship. *Dnotes equal contribution.
1

---

Published as a conference paper at COLM 2024
Replace Data
Accumulate Data
Modell
Model
Training
Thrts
Model
Unta
Midell
Data
Real
Model
Data 2
Madell
Daca
Data
Generated
Modal-
Data
Fil
Datoo
Mara
Sample
That Tome
Test Lote
Mudal-Filing Heroati
ModallFFiting Laration
Figure 1: Two Settings to Study Model Collapse. Model collapse a phenomenon where sequences of generative models trained on their own outputs progressively degrade until the latest model becomes useless. Left: Many prior works studied model collapse where data are replaced with each model-fitiig iteration. Right: We study model collapse where data accumulate with each iteration and demonstrate accumulating data avoids model collapse.
field of artificial intelligence. These models, trained on vast web-scale datasets, exhibit remarkable capabilities in generating text, images, and other media (Brown et al., 2020; Saharia et al., 2022). However, as these models become more widely used, an increasing amount of generated data populates the web. This raises a critical question: what are the consequences of pretraining generative models on datasets containing their own outputs?
Recent studies have investigated this question, revealing that training generative models on their own outputs can cause the performance of such models to progressively degrade with each model- model-fitting iteration, eventually rendering newer models useless (Hataya et al., 2023; Martínez et al., 2023a; Shumailov et al., 2023; Alemohammad et al., 2023; Martínez et al., 2023b; Bertrand et al., 2023; Briesch et al., 2023; Dohmatob et al., 2024a;b) (see Appendix A for review and discussion of prior work). This phenomenon was consequently labeled model collapse. Model collapse warns that democratizing access to generative models runs the risk of polluting the very data necessary to train future iterations of generative models.
To better understand this phenomenon many prior works have considered a setup that assumes each model's generated data replaces previous data. In theory, this leads to very natural comparisons across generations as the total number of training points for each model remains fixed. In practice, subsequent generations of deep generative models are often trained with increasing data over time e.g., 1.4 trillion tokens for Llama 1 (Touvron et al., 2023a), 2 trillion for Llama (Touvron al., 2023b), 15 trillion for Llama 3 in which both human- and machine- machine-generated generated data are accumulating in data collected from the internet. It was noted in some of those works Hataya et al. (2023); Martínez et al. (2023a); Alemohammad et al. (2023); Bertrand et al. (2023); Dohmatob et al. (2024b) that model collapse can be either slowed down or negated by mixing real data in with generated data.
To that end, in this work we study the effect of accumulating data rather than replacing data. Our data-accumulating setting is, in some sense, maximally pessimistic: it considers hypothetical future where synthetic data are uncontrollably dumped on the internet to be vacuumed up for training the next iteration of generative models. Nevertheless, we find that model collapse is avoided when accumulating data.
model collapse,
2

---

Published as a conference paper at COLM 2024
Language Models Pretrained on TinyStories
Model
GPT-2 (9M)
Replace
Accumulate
lamm-2 (12M)
2.6
Llama-2 (42M)
Cros Entrogy (Test)
Llama22 (126M)
2.4
2.2
2.0
1.8
1.6
2
Model-Fitting
Model Fitting
Figure 2: Data Accumulation Avoids Model Collapse in Language Modeling. Sequences of causal transformer-based language models are pretrained on TinyStories (Eldan & Li, 2023). Cross-entropy validation loss increases when replacing data (left), but not when accumulating data (right). Synthetic data was sampled with temperature = 1.0.
We begin by studying model collapse experimentally with deep generative models trained realistic data: transformers causal language modeling (Sec. 2.1), diffusion models on molecular conformation (Sec. 2.2) and variational autoencoders on images (Sec. 2.3). After confirming that replacing data at every iteration indeed causes test error to increase with the number of iterations, we empirically find that accumulating synthetic data with real data avoids model collapse for all models and for all data modalities test. To understand why replacing data and accumulating data have different consequences for model collapse, we turn to an analytically tractable framework of sequence of linear models, each trained on synthetic outputs generated from the previous-iteration' 's fitted linear model (Mobahi et al., 2020; Dohmatob et al., 2024a). Within this framework, Dohmatob al. (2024a) demonstrated that if data are replaced with each model-fitting iteration, the test error increases linearly with the number of iterations n. We extend Dohmatob et al. (2024a)'s analysis to prove that if data instead accumulate, then the test error has finite and (to us, surprisingly) well-controllee upper bound independent of the number of model-fitting iterations.
Altogether, work suggests that data accumulation may be robust to model collapse and emphasizes the importance of considering accumulating data and other real-world data dynamics in the analysis of model collapse in generative models trained web-scale data.
2 Accumulating Data Avoids Model Collapse in Deep Generative Models
We first investigate model collapse experimentally in deep generative models. Here, and for the remainder of this manuscript, the term model collapse refers to notably worsening error over increasing iterations of the model-data loop, while avoiding model collapse refers instead to bounded error over such iterations. We use three diverse experimental setups of causal transformers, diffusion models, and variational autoencoders trained on real text, molecular conformation, and image datasets, respectively. We find that replacing data yields model collapse for all models and all datasets, whereas accumulating data avoids model collapse.
2.1 Transformer-Based Causal Language Modeling
Experiments We first train causal transformers (Vaswani et al., 2017) on text data. Specif- ically, we pretrain 9M parameter GPT-2 (Radford et al., 2019) and 12M, 42M and 125M parameter Llama2 (Touvron et al., 2023b) language models for single epoch on TinyS- tories (Eldan & Li, 2023), a 470M token GPT-3.5/ GPT-3.5/4-eenrrate 4-generated dataset of short stories at kindergarten reading level. For each model-fitting iteration dataset of the same size as Siorie from the previous iteration's 's language model and then either replace concatenate the previous dataset with the newly generated dataset. In
2, we sample new
3

---

Published as a conference paper at COLM 2024
languge Thyfooees
(A30)
02
0.0
0.5
3.5
0.2
0.4 0.5
1.5 30 3.2 3.0 3.5 3.5
Figure 3: Data Accumulation Avoids Model Collapse in Language Modeling. Learning curves for individual model-fitting iterations when repeatedly replacing data (top), and when accumulating data (bottom).. Note: Epochs correspond to more gradient steps for accumulate than replace because the number of training data grows for accumulate.
each model-fitting iteration, we then pretrain a newly initialized model on the replaced or concatenated dataset from the previous iteration. We experiment with sampling the new datasets using temperatures 0.3 or 1.0. We chose this combination of architectures, scales, dataset, and sampling because the setup necessitates pretraining sequences of language models computationally costly endeavor - but we also wish to study realistic conditions where generative models are high-performing and generative diverse outputs. Because small language models (below 10M parameters) pretrained on TinyStories were shown to be able to generate athheret--llbit-simpe of architectures, scales, dataset and temperature hopefully strikes a good balance between being representative, being diverse and being computationally feasible.
English sentences (Eldan & Li, 2023), this choice
Results We found that for all architectures, parameter counts, and sampling temperatures, as the number of model-fitting iterations increased, replacing data led to an increase in test cross entropy (Fig. 2 top) whereas accumulating data led to equal-or-lower test cross entropy (Fig. 2 bottom). Lower temperature (0.3) led faster increase in test error than higher temperature (1.0) (Appendix Fig. 13), but the trend was consistent for both temperatures turee. Appendix Table shows samples of generated texts for GPT2 (9M) and Llama2 (125M) models at model-fitting iterations 5 when both accumulating and replacing data, as well as iteration 10 (GPT2, replacing only). We run extensive ablations for potential confounds; see Appendix C for additional experiments, figures and discussion.
2.2 Diffusion Models on Molecular Conformation Data
Experiments We next train sequences of diffusion models on molecular conformation data. Specifically, we train GeoDiff (Xu et al., 2022), a geometric diffusion model for molecular conformation generation, on the GEOM-Drugs (Axelrod Gomez-Bombarelli, 2022) dataset. We down-sample the training split of GEOM-Drugs to 40, 000 molecular conformations, which we use as our initial training set, and perform 50 diffusion steps for each prediction. For the loss, use the standard loss used by GifDiff: weighted variational lower bound to the conditional likelihood; for more details, see Xu et (2022).
Results Over 8 model-fitting iterations, we find test loss increases when replacing data, matching our language model experiments, and test loss remains relatively constant when accumulating data (Fig. 4). Unlike with language models, we found that when replac- ing data, performance worsens significantly in the first model-fitting iteration trained on synthetic data and does not degrade further substantially in subsequent iterations.
4

---

Published as a conference paper at COLM 2024
Diffusion Models For Molecule Generation
Replace
Accumulate
300
Loss (Test)
200
100
2
4
6
8
0
2
6
Model-Fitting Iteration
Model-Fitting Iteration
Figure 4: Data Accumulation Avoids Model Collapse in Geometric Diffusion Modeling. GeoDiff, a diffusion-based molecular conformation generation model, is trained on a subset of Drugs data containing molecular structures found in drugs. Test loss degrades when replacing data (left) but not when accumulating data (right).
Variational
Autoencoders For Image Data
Replace
Accumulate
0.030
0.028
Loss (Test)
0.026
0.024
0.022
0.020
2
6
0
2
Model-Fitting Iteration
Model-Fitting Iteration
Figure 5: Data Accumulation Avoids Model Collapse in Variational Autoencoders for Image Generation. Sequences of variational autoencoders (VAEs) are trained on CelebA Test loss degrades when replacing data (left) but not when accumulating data (right).
2.3 Variational Autoencoders on Image Data
Experiments We lastly train sequences of variational autoencoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014) on CelebA (Liu et al., 2015), a dataset of 200k images of human faces split between train and test sets, chosen balance between being realistic dataset with many samples, color images and resolution, and computational feasibility of training multiple iterations of models on accumulating data. The loss is the standard VAE loss: reconstruction error plus the KL divergence between the encoder's 's output Gaussian and the isotropic Gaussian prior. See Appendix D for more experimental details.
Results We find that replacing data at each iteration again exhibits model collapse: the test error rises swiftly with each additional iteration, and each iteration yields lower quality and less diverse generated faces until all model generations represent single mode as shown in the left panel of Figure 6. In contrast, accumulating data at each iteration significantly slows

---

Published as a conference paper at COLM 2024
Figure 6: Sampled Images from Left: Replacing data with data generated by the previous iteration's 's newly trained VAE yields lower quality and eventually leads to complete mode collapse. Middle: Accumulating data with data generated by the previous iteration's 's newly trained VAE preserves the quality and diversity of generated data across iterations. Right: Baseline samples after 100 training epochs on the dataset.
model collapse: the test error increases significantly slower with each additional iteration. While the diversity of generations does go down as compared in the middle and right panel of Fig. 6, still represents major axes of variation in the dataset, such as gender, but no longer seems to generate other details, along more minor axis of the data manifold, such as glasses and accessories. We discuss further analysis of VAE reconstructions in Appendix D. Interestingly, unlike language modeling, the test error of accumulating data does increase with the number of iterations (albeit much more slowly than with replacing data).
3 Accumulating Data Avoids Model Collapse in Linear Models
To gain mathematical understanding and intuition, we employ an analytical framework introduced in prior work (Mobahi et al., 2020; Dohmatob et al., 2024a) to understand the difference between data accumulation and data replacement. We will show that it predicts the same types of test error behaviors for these two data-use strategies that were measured empirically. The framework considers sequence of linear models that are fit to the synthetic data sampled from the linear generative model model based on the previously fit linear models. Within this framework, Dohmatob et al. (2024a) showed that if data are replaced across model-fitting iterations, then the test squared error increases linearly" with the number of iterations n. Here, we extend Dohmatob et al. (2024a)'s ' argument to show that data instead accumulate across model-fitting iterations, then the test squared error is upper bounded by a relatively small constant, meaning model collapse is avoided2.
The content in this section relies heavily on the framework and pioneering contributions of Dohmatob et al. (2024a). Our contribution is to study different way to use synthetic data in training, namely accumulate, which better align with real-world settings. We show that our empirical results could have been anticipated on theoretical grounds, by applying the same analysis framework as in Dohmatob et al. (2024a) to this specific training dataset pattern.
An approach 'halfway' between the 'replace' and 'aaccumultt'' approaches would replace the previous dataset with pure synthetic dataset size iT at the i-th iteration. Analyzing this goes mostly in parallel, except the 1 1/22 mentioned in running text now becomes 1/i. Consequently, the MSE scaling becomes MSE x (log(n)); the 'halfway' approach with pure synthetic data, but more of it, again has test error growing unboundedly with iterations. Thanks to Elvis Dohmatob, Yunzhen Feng and Julia Kempe for communicating this observation. See Appendix E for an extended discussion.
In this theoretical section, we identify the term model collapse with the situation where test error diverges to infinity (at any rate) as iterations progress. Other authors may employ similar terminology while identifying it with different properties of test error. For example, Alemohammad et al. (2023) use the term MAD to refer to the situation where the distance between the distribution of the original data and that of the subsequent generative models grow farther apart, without necessarily diverging-
6

---

Published as a conference paper at COLM 2024
3.1 Notation and Preliminaries
We adapt notations from Dohmatob et al. (2024a). Define the distribution pw on Rd R given by (x, PI
Original Data Distribution.
iff
(Input)
2),
(Noise) ~ (O, independent of x, (Label) y x w +e.
The positive integer d is the input-dimension, the matrix Σ E Rdxd is the true covariance structure of the input the vector w" is the true linear relationship used to generate the original data and the scalar is the level noise. We start at iteration n = 1 with T initial independent data points (xi, yi) each following ,I, that is, Vi = = + for each i= 1,, We form the design matrix X E RTxd with x .... Xy as rows. We also form vectors Y and with i-th coordinate y1 and E; respectively. In what follows, assume that X has full column rank, i.., T 2 d, X X is invertible and the model underparameterized.
Synthetic Data Generation Process. We generate synthetic data from the following se- quence of distributions:
PE 2²2
02 7...
Ps
where E is the number of iterations. The scheme is outlined as follows.
For 1:
Accumulating Covariates Features: X1 def X
Accumulating Targets: Y def 91 def Xwo Eg, where Ej def N(0,027))
Fit linear model: かò = X
Sample synthetic data for the next iteration: 82 def Xt + E2, where Ez ~ o2
For 2 2:
Accumulating Dovariates/Featuree X [ E Rdxxn
Accumulating Targets: Y,T = [YT 1 PT E RIxnT
Fit linear model: @ú, def X
Sample synthetic data for the next iteration: Yx41 Xtd, + Ex41, where E,41 (0,
Here, for a matrix A with full column rank, At = (A AA is the Moore-Penrose pseudo-inverse of A. The noise terms E,, Eg...., the sovariates/featuree Since X has full column rank, so does X, for every 1.
,, Fn are independent of each other and of
Test Error. We are interested in the dynamics of the test error Etest (te))(w,)) of this sequence of linear model w), 20.,... Note that evaluation of the model is done the true distribution PI,"" , even though the model is trained on the accumulated synthetic data. For any linear estimator ie computed from the training data, we measure test error in the standard way:
Etest (w) def E (x/ee
ytest yttest2 )2]
0² E[[[w w0' 1
(1)
where the expectation is taken over the training data and (Xtest, ylest) Pe an independent of the training data.
7

---

Published as a conference paper at COLM 2024
Regge Regibrization: Data Dimesioon:
Nakie (+))
Num. Sampho per Itealon (?) 100
Nam. Sampls por Itasikn (T) 316
Mar, Simples Reusion (T) 2030
0.21
1.2
10.3
100.0
(1m0)
Test Evver Type
Nammoical
Thearciall
200
320
Mis4Fiinnn PHOTOR
Mode Frisig Te-30oo
Figure 7: Accumulating Data Avoids Model Collapse in Linear Regression. We consider sequences of linear models recurrently fit to generated targets by previous iterations of models. Top: If each linear model is fit to the generated targets of only the preceding linear model, i.e., data are replaced, then the test error grows linearly with the number of iterations n. Bottom: If each linear model instead fit to the generate targets of all the preceding linear models, i.e., data accumulate, then the test error has a finite upper bound independent of the number of iterations. This suggests that data accumulation might be a robust solution for mitigating model collapse. For log test error and higher iterations, see Appendix Fig. 16.
3.2 Precise Test Error Characterization Under Accumulating Data
Our goal is to establish an analytic formula for the test error of the nth model in the data accumulation setting. We begin by characterizing the relationship between the fitted linear parameters W and the true parameters w". We remind the reader that we assume that X has full column rank, i.e., X X is invertible. Proofs are deferred to App. B.
Theorem 1. In the data accumulation setting, Vn 1, the fitted linear parameters w; can be expressed as:
a
E
tÚ 2 (X xX
T
Σ
where, recall, w" is the true parameter, the original design matrix, and E, is the extra noise added at the i'th iteration.
Theorem For an n-fold synthetic data generation process with T 2 d + samples per ileration and isotropic features (I def L4), the test error for the ridgeless linear predictor wú learned on the accumulated data up iteration is given by:
02d
2²d
π22 6
AAccum (2x)
(3)
T- 1
T
where, recall, o² is the noise variance of the fake data generation process, d is the input dimension, and T is the number of samples (i.e., data points) added per iteration.
How does test error with accumulating data compare against test error with replacing data? Under otherwise identical assumptions, Dohmatob et al. (2024a) proved in the data-
8

---

Published as a conference paper at COLM 2024
replacing setting that the test error is given by
o2d
Replace
(z0n)
(4)
lest
- d -
When data are replaced, the test error grows linearly with the number of iterations и (Fig 7 top). In contrast, when data accumulate, Theorem 2 shows the test error is upper bounded regardless of the number of iterations n:
o²d
π
FAccum (10m) S T-d X test
(5)
6
This striking difference can be intuitively explained: the data replacement setting, because previous data were discarded, the model is more strongly affected by the new noise that each iteration of generated data introduces, and adds that to the effects experienced iterations. But in the data accumulation setting, because iteration contributes fraction 1/i to the training dataset, the additional noise from the ith iteration of synthetic data has its effect on the model MSE shrunken proportional to 1// (due to squared error). The summability of 1/2 prevents the test error from growing indefinitely. This suggests that accumulating generated data with real data can indeed avoid model collapse.
earlier
3.3 Numerical Confirmation of Analytical Results
To confirm the analytical results, we numerically simulate the setup. The numerics almost perfectly matched the analytics (Fig. 7): when data are replaced, the test error grows with the number of iterations n, with the prefactor set by the noise-to-signal ratio d'd/(T - d 1), but when data accumulate, the test error rapidly plateaus with the prefactor similarly set. For log test error and higher model fitte-liiiigg iterations, see Appendix Fig. 16.
Discussion
This work explored the phenomenon of model collapse, important concern AI- generated content permeates the internet and future training datasets. Prior work has shown that training on model outputs can lead to degraded performance (Martínez al., 2023a;b; Shumailov et al., 2023; Alemohammad et al., 2023; Hataya et al., 2023; Bertrand et al., 2023; Briesch al., 2023; Dohmatob et al., 2024a;b), implying that future model training faces a difficult challenge of ensuring strict data hygiene. For significantly more thorough discussion of related work, please see Appendix A.
Our findings show that if data accumulates, model collapse no longer occurs. We show this experimentally on causal transformers for language modeling, diffusion models for molecule generation, and variational aufo-encoders on image generation as well theoretically for linear regression. Together, these results strongly suggest that the "curse of recursion" may not be as dire as had been portrayed provided we do not delete real data masse.
Looking to the future, many questions worth investigating remain. For instance, in future work we would like to explore different data generation and accumulation regimes, such as (1) additional "real" data being introduced in each model-fitting iteration and (2) different schedules of how much synthetic data is generated at each iteration and (3) human faman-filltring of generated data, e-g,, as done in RLHE
Lastly, it is worth noting that "model collapse" ways by various researchers; so care is required in comparing claims across articles. In reviewing the literature, error blowup (as here); (1) modal collapse - collapse to one (or a few) modes; (2) collapse to uniformity; and (3) amplification of artifacts introduced by models fit previous synthetic data. Future work should map out what factors cause which to occur and what preventative strategies are effective at addressing each.
term of art - has been used in various
identified at least four related phenomena: (0) unbounded test
Forr notational simplicity, we assume that Dohmatob al. (2024a)'s To def T and og def
d.

---

Published as a conference paper at COLM 2024
5
Acknowledgements
The content of this paper does not necessarily reflect the position or the policy of any of the funding agencies/ erencies/entities. entities. No endorsement should be inferred. M.G. acknowledges support through grant from the Cooperative AI Foundation. R.S. acknowledges sup- port from Stanford Data Science and from OpenAT's Superalignment Fast Grant Research Fellowship. A.G. acknowledges support from the NSF CAREER grant DMR-2045188 the Sloan Foundation, and by the Laboratory for Physical Sciences through the Condensed Matter Theory Center. D.R. acknowledges support from the National Science Foundation under Cooperative Agreement PHHY-209786 (the NSF Al Institute for Artificial Intelligence and Fundamental Interactions) and appreciates both the sanction and support of Sequoia Capital. S.K. is partially supported by NSF III 2046795, IIS 1909577, CCF 1934986, NIH 1R01MM111226-014 NIFA award 2020-67021-32799 the Alfred P. Sloan Foundation, and Google Inc.
10

---

Published as a conference paper at COLM 2024
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ar preprint ar Xf::23303.00 2023.
Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G Baraniuk. Self-consuming generative models go mad. ar arx preprint ar xrXiv:2307.0188 2023.
Simon Axelrod and Rafael Gomez-Bombarelli. Geom, energy- anngya-n molecular con- formations for property prediction and molecular generation. Scientific Data, 9(1):185, 2022.
Quentin Bertrand, Avishek Joey Bose, Alexandre Duplessis, Marco Jiralerspong, and Gau- thier Gidel. On the stability of iterative retraining of generative models on their own data. Xrv preprint arXio:2310000429, 2023.
Martin Briesch, Dominik Sobania, and Franz Rothlauf. Large language models suffer from their own output: An analysis of the self-consuming training loop, arXiv preprint ar arXiv:2311.16822 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Adoances in neural information processing systems, 33:1877-1901, 2020.
Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7:331-368, 2007.
Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in Neural Information Processing Systems, 34:10131-10143, 2021.
Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Model collapse demystified: The case of regression. arXiv preprint arXi::2402.07712 2024a.
Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, and Julia Kempe. A tale of tails: Model collapse as a change of scaling laws. arXiv preprint 4xXi::2402.07443, 2024b.
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759, 2023.
Ryuichiro Hataya, Han Bao, and Hiromi Arai. Will large-scale generative models corrupt future datasets? In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 20555-20565, 2023.
Ayush Jain, Andrea Montanari, and Eren Sasoglu. Scaling laws for learning with real and surrogate data. Xr preprint ar Xrin:22002.03376 2024.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. aiv preprint arXiv:1312.6111,, 2013.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611-626, 2023.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Matteo Marchi, Stefano Soatto, Pratik Chaudhari, and Paulo Tabuada. Heat death of generative models in closed-loop learning, 2024.
11

---

Published as a conference paper at COLM 2024
Gonzalo Martinez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Marc Juarez, and Rik Sarkar. Combining generative artificial intelligence (ai) and the internet: Heading towards evolution or degradation? arXiv preprint ar (XX::23333001555 2023a.
Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Marc Juarez, and Rik Sarkar. Towards understanding the interplay of generative artificial intelligence and the internet. arXiv preprint arXio:2306.0613 2023b.
Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regular- ization in hilbert space. Advances in Neural Information Processing Systems, 33:3351-3361, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):,, 2022.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, Pp. 1278-1286. PMLR, 2014.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEÉE/CVF conference computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Ad- vances in neural information processing systems, 35:36479-36499 2022.
Mohamed El Amine Seddik, Suei-Wen Chen, Soufiane Hayou, Pierre Youssef, and Merouane Debbah. How bad is training on synthetic data? a statistical analysis of language model collapse, 2024.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXie:2305.17493 2023.
James B Simon, Madeline Dickens, and Michael Deweese. Neural tangent kernel eigenvalues accurately predict generalization. 2021.
Rohan Taori and Tatsunori Hashimoto. Data feedback loops: Model-driven amplification of dataset biases. In International Conference on Machine Learning, pp. 33883-33920. PMLR, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar arXi9:2302.13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ar preprint arXiv:2307.0920 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances neural informa- fion processing systems, 30, 2017.
Alexander Wei, Wei Hu, and Jacob Steinhardt. More than toy: Random matrix models predict how real-world neural representations generalize. In International Conference on Machine Learning, pp. 23549-23588 PMLR, 2022.
12

---

Published as a conference paper at COLM 2024
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-artaature language processing. ariio preprint arXiv:1910.03371,, 2019.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint aXXiv:2203.02923, 2022.
13

---

Published as a conference paper at COLM 2024
A Summarization and Discussion of Prior and Related Work
Prior Empirical Work A growing body of recent work has investigated the phenomenon of iteratively training models on data generated by previous models, e.g., Hataya et al. (2023); Martinez et al. (2023a); Shumailov et al. (2023); Alemohammad et al. (2023); Martínez et al. (2023b); Bertrand et al. (2023); Briesch et al. (2023); Dohmatob et al. (2024a;b) and (in different context) Taori & Hashimoto (2023). Hataya et al. (2023) and Martínez et al. (2023b) conducted experiments replacing real training data with generated data at each iteration, assuming that the dataset size remains fixed over time. They found that this iterative retraining procedure can lead to model degradation if the proportion of synthetic data becomes too high. Similarly, Shumailov et al. (2023) ran experiments with Gaussian mixture models, VAEs, and language models in which the total number of samples per iteration was held constant, and the samples always originated with the previous model rather than aggregating over time. Building on this work, Alemohammad et al. (2023) considered three treatments of data: fully replacing real data with synthetic data, augmenting a fixed real dataset with additional synthetic data, and mixing new real data with synthetic data at each iteration. In almost all of their experiments, they drew fixed size dataset from the most recent model at each iteration, without accumulating data. Bertrand et al. (2023) also assumed that dataset size and mixing proportions are constant over time in their theoretical stability analysis and empirical validation.
Prior Theoretical Work Over the last few years, there has been significant research effort contributing to our theoretical understanding of model behavior when synthetic data are integrated into training. The most closely related works to ours are Dohmatob et al. (2024a) and Dohmatob et al. (2024b); of course, the inspiration for the linear regression model studied in this paper directly comes from Dohmatob et al. (2024a). Dohmatob et al. (2024a) performs an in-depth analysis of high dimensional linear and ridge regression when the training data used per iteration are generated from the previous iteration's fitted model. They are able to conclude that the test error grows linearly with the iteration count in their setup, as well as derive more interesting and more nuanced results using random matrix theory. They also discuss how to mitigate model collapse through optimal regularization both when the training data are noise-free free and noisy versions of the previous model's synthetic outputs. A related noise-free setup was studied by Mobahi et al. (2020) in the case of oel-dditillattion. Although Mobahi et al. (2020) considers a more general setup with ridge regression as special case, they use noiseless predictions from the previous model as the training data for the next model, and show that eventually, the predictions shrink to zero. Through this, they highlight that self-distillation induces regularization in the function space, which initially is beneficial for reducing over-fitting, but eventually over- regularization causes underfitting and hence performance decay. Dohmatob et al. (2024b) go beyond the linear model to study model collapse - they study the tails of LLM outputs vs. real data and provide scaling laws that clearly identify regimes of model degradation when synthetic data misses lails present in real data. They identify an interesting phase transition in the test error scaling law depending on the size of the real dataset size in comparison to (a functional of) the chopped-off tail, and conclude that enough real data is able to mitigate model collapse. All these works consider the scenario where the amount of training data available per iteration is fixed (and does not grow with the iteration count), and it is certainly possible with larger amount of synthetic data (from prediction by the previous model), several of these scalings would improve significantly. For example, in Equation (12) of Dohmatob et al. (2024b), one obtains the linear scaling (with iteration count) of test error simply because the amount of synthetic data generated per iteration the same. If one generated synthetic data with size proportional to the iteration count, then at iteration n the scaling would, instead of be like n² </(1- -c) for c < 1. When one does not increase the dataset size, Dohmatob et al. (2024b) points out that increasing the proportion of real data would help one to avoid model collapse altogether. However, even if one did increase the amount of synthetic data with iteration count, Theorem 3.2 coupled with Corollary 3.3 in Dohmatob et al. (2024b) would tell us that the amount of real data was all that mattered the amount of real data is large, we overcome model collapse. If one only had synthetic data (and no real data), no matter how large, it would be impossible to regain the original
14

---

Published as a conference paper at COLM 2024
real-data scaling laws. The scenario study is highly inspired by these pioneering works, but still, in our view, different. We consider the case when we keep augmenting synthetic data (generated by the previous model trained on all the previous data so far) as iterations progress, much akin to how - in our view - the internet evolves. We observe that we can avoid model collapse in this setting. The analysis of previous models in our case is more involved, since the data used for training at iteration и is not homogeneous - different models from the past impart different statistical aspects to different parts of the training data. We also note related augmentation model studied by Jain et al. (2024) they perform risk minimization augmenting real data with synthetic data available from a potentially different independent source. One of their messages is that augmentation of (even) pure noise can be looked upon as adding a ridge penalty and hence, in certain cases, can improve test error. Their setup, however, is different from ours, since the synthetic data in their setup is not obtained by a learning algorithm employed on the real data, and the process is not iterative. However, morally, each iteration of ours involves risk minimization on data statistically composed of an equal mixture of data generated from the previous models, and hence each iteration of ours can be mapped to the general framework developed in Jain et al. (2024), although the dependencies among the various models trained in our setup introduce theoretical complications that do not seem to be too easily addressed by the theory developed in Jain et al. (2024). Shortly after v1 of our manuscript was uploaded to ArXiv, two other manuscripts appeared, dealing with the theoretical aspects in a setting similar to ours. Theorem 1 of Marchi et al. (2024) obtains the same square summability scaling of the variance as us. Seddik et al. (2024) studies collapse in language models in both purely synthetic and partly synthetic regimes and obtains deviation bounds as model iterations progress.
Considering Accumulating Data The two papers we found that partially considered accu- mulating data are Martinez et al. (2023a) and Alemohammad et al. (2023). Alemohammad et al. (2023) did so in one-half of one experiment: StyleGAN2 trained on FliqrFaces 128 128 (App. Fig. 8). The authors concluded that accumulating data does not avoid model collapse, but merely slows it down. However, we believe that closer examination of their results (App. Fig. 8) reveals that accumulating data causes the test error to plateau to relatively low error with increasing numbers of model-fitting iterations. This result would support our conclusion that accumulating data avoids model collapse and does not merely delay it. The results from Martínez et al. (2023a) are harder to evaluate; model collapse only seems to occur when the amount of synthetic data added per model-fitting iteration is 2x the total amount of accumulated data, and the subsequent work by the authors switched from
0.7
0.4
20
Precision
0.3
0.65
Recall
FID
1.5
0.2
10
0.6
0.1
2
Generations
Generations
Generations
#HHO-SyyoGANN2, À =
Fully synthetic loop
Synthetic augmentation loop
Figure 8: Clarification of Data Accumulation in Alemohammad et al. (2023). Figure from Alemohammad et al. (2023) (above) shows that linearly accumulating data ("Synthetic augmentation loop") causes poor behavior to plateau with the number of model-fitting iterations. Alemohammad et al. (2023) write, "Our experiments [...] support our main conclusion [that] fixed real training data only delays the inevitable degradation of the quality or diversity of the generative models over generations." " We believe is that our evidence and their evidence is more consistent with the conclusion that accumulating data avoids model collapse and does not merely delay it.
15

---

Published as a conference paper at COLM 2024
accumulating data to replacing data (Martinez et al., 2023b). We think understanding what conditions and why these discrepancies exist is an interesting future direction.
Avoiding Model Collapse Several papers present methods for avoiding or slowing model collapse. Bertrand et al. (2023) shows in the replacing data setting that model collapse will not occur if the initial generative models approximate the data distribution well enough and the proportion of real data is sufficiently large with respect to the synthetic data. Dohmatob et al. (2024b) similarly demonstrates that the replacing data setting, carefully selecting real data to mix with synthetic data can avoid model collapse. Other solutions may also be possible in various models and under various assumptions. To our knowledge, no paper has claimed an "optimal" strategy to avoid model collapse, and neither has ours.
16

---

Published as a conference paper at COLM 2024
B Proofs of Mathematical Results
We point out a lemma useful to prove Theorem 2.
Lemma 3. Let T and d be positive integers with T 2 d + 2, and let X E Rxxd be random matrix with iid. rows from N (0, Σ) with Σ positive definite. Then, X has full rank a.s. Moreover, it holds that:
1
Ex((x x)
27
(6)
T
d 1
Proof. See Dohmatob al. (2024a).
Assuming Lemma 3 and Theorem 1, we present the proof of Theorem 2.
Proof of Theorem 2. From Theorem 1, we have:
Σ E
in w0 + (X X
where wt is the true parameter, X is the original data matrix, and E, are the noise terms at each iteration, with E, ~ NN, o2 The test error is given by:
Etest (bn)) = [ == [[2]
(8)
where the expectation taken over all random quantities involved.
Substituting th into the test error expression and using the fact that def
la, we get:
Exest (0n) =
E Σ
x)XTx)=2 2x
Σ
1
E
Σ
o² 12
xx x)-2x]]
tr(
(x(((xx)))
Using Lemma 3, we have:
d
Ex (r((xTx)
(9)
T -
Therefore, the test error for ridgeless regression with isotropic features in the data accumu- lation setting is:
0² d
odd
π²
Etest (wn)) =
Σ
12
T
d
1-1
T
d
6
as E0 12 72/6.
Finally, we prove Theorem
Proof of Theorem We prove this theorem by induction.
Base case: For ı = 1, we have:
i = x = (x x) (Xw" + +E, = =" + +( X) -1xTE,
which satisfies the lemma.
17

---

Published as a conference paper at COLM 2024
Inductive step: Assume that for some n 2 1, we have:
m T
w,==" w0 + (X -
Now, consider 0x+1;
Φx+1 X Vn+1
= (8 xx+1) -1 -1x++ Yn+1
121 x TTyy Yi
n n+1 (X) T x)
Recalling that 9j:
Xwo* + El, i=
8,
[x0 + E,, 2 i < n+1
Substituting this back into the expression for wn+1:
0 n+1
1
X x)-
x "Xx"" + +EI) +
n n
Σ x (X10;- + Ei)
n
1-2
x Xw"" + +x E + n+1
n1 n 1 (XT x) 1
Σ
(x T Xto;; + x Fi)
n+ (X x)-
x xxo" +x T E + Σ
(x
Xto; + x Eiti))
i=1
" XT xw" + Σ x Xtd; + n+1
(X x)-1
Σ x Ei
n
Now, using the induction hypothesis:
+ +
r0.+1
(X Txx)) x)-
x xT Xw"
Ex x w² +
(X
x)
11 ++1
1 (X (XTx))
(1 11x+)xX 1)x Xw"
Σ x
x) x
Σタ2 Fy E
n1 #1
1
11
E, + Σ x E
= w² + n 1
(X -1
Σ x
j=
= w² 1 (X x)
1xT
ΣE E.
Now, we need to simplify the term Σ" E 7 + + E, We can do this by counting the number of times each E; appears in the double sum: Ei appears n times in the double sum and once in the single sum, so its coefficient is "+1 Ez appears n - times in the double sum and once in the single sum, so its coefficient is 4. This continues along till we reach Eg, which appears once in the double sum and once in the single sum, so its coefficient is 2. E,41 appears only once in the single sum, so its coefficient is - Therefore,
E Σ
u+1 n + 2
E
E, = (n2 +
i
i
18

---

Published as a conference paper at COLM 2024
Substituting this back into the expression for wn+1;
2 n 1 ~"
1
X
x)
1x
(n+1)
(n
1=1
E
-+1 1
n
1
E
~ + (X
i
Therefore, by mathematical induction, the lemma holds for all и 2 1.
A Note on Extensions to Ridge Regression and Kernel Methods. To reiterate comment made previously by Dohmatob et al. (2024a), although we present our results in the context of ordinary linear regression in R our analysis can be readily extended to ridge regression and the kernel setting (Caponnetto & De Vito, 2007; Simon et al., 2021; Cui et al., 2021; Wei et al., 2022). We focus here a simple useful model for studying model collapse.
19

---

Published as a conference paper at COLM 2024
C
Additional Details and Ablations on Language Model Experiments
Implementation Details
Model training was implemented using Huggingface Transformers (Wolf et al., 2019). Dataset generation was implemented using vllm (Kwon et al., 2023).
Model
Iteration (A)
Sample Generation
Llama2 (125M)
Once upon time, there was little girl named Lily. She loved to look up at the night sky and dream about flying. One night, she saw a shooting star and wished she could fly like a bird. Suddenly, she saw beautiful light in the sky. It was so bright and everything looked bright. She felt happy and excited. Suddenly, she heard someone calling for help. was her mom and she had arrived home from school. Lily ran to her mommy and said, "Mommy, 1 saw a shooting star! I dreamed about flying like bird." Her mommy smiled and said, "Wow, that sounds wonderful!" Once upon time, there touch things, especially cookies, and cookies. One day, her family had a special surprise for her. It was a big box full of cookies! Lily was so happy that started to laugh. A little mouse was swimming around in the pond and playing with special toys. They spent all day playing with their toys, scaring and focusing toys with flowers. Once upon a time, there was a little fish named Nemo. Nemo loved to play with his friends in the ocean. One day, Nemo's 's friend, little crab named Nemo, came to Nemo's house with new toy. Timmy wanted to play with the big red boat, but Nemo would not want to share. Nemo knew Nemo was right. He decided to share the boat with Nemo. Nemo agreed and they both played with the big, red boat.
5 (R)
little girl named Lily. She loved to
GPT2 (9M)
5 (A)
5 (R)
Once upon time, there was fish named Spike. Remy had big mushroom that he loved to eat every day. One day, Remy found bone in the grass and picked it up. But then, he saw a big bird on the ground. Or he was not scared anymore. The bird flew up to Fin and accidentally dropped the bone in the grass. Pinky would not hear that the bird was not the worm either. The bird did not like Remy. Once upon a time, there was a little girl named Lily. She loved to play with her toys and run around outside in the snow. One day, she saw another tank. She thought it's important to be careful and ask her mom if she could go outside outside. But then, her mom asked her to clean it up, but Lily didn't want stay inside. She asked mom, "Where are you playing outside with me?" Her mom replied, "We found a big snowman and it might be dangerous, but it's too hot and cozy."
10 (R)
Table 1: Data Accumulation Avoids Model Collapse in Language Modeling. Both 125M- parameter Llama2 as well as 9M GPT-2 models show decreasing quality when replacing data (R), but maintain high-quality text generations when accumulating data (A).
Additional Plots
In addition to Figure 3 in the main text, Figures 9-12 show learning curves in larger print, with x-axes showing either epochs or gradient steps, and with axes shown in linear-linear or log-log scale, respectively.
Ablations
In addition to the experiments shown in the main paper, we conducted several ablation studies.
To offer quick summary, when accumulating data, subsequent model iterations are trained on larger datasets than when replacing data. To control for this, we also perform experiments
20

---

Published as a conference paper at COLM 2024
Iteration
Language Models Pretrained on TinyStories
1.0
Replace
Accumulate
5.0
3.0
4.0
5.0
Cross Entropy (Test)
(v16) Z-149
3.5
2.5
2.0
1.5
5.0
Entropy (Test)
4.0
Camme (126M)
2.5
1.5
1.5
Cross Entropy (Test)
Ulmaa22 (42M)
3.0
2.0
1.5
4.5
Cross Entroyy (Test)
3.5
3.0
2.0
Epoch
Epoch
Figure 9: Data Accumulation Avoids Model Collapse in Language Modeling. Learning curves for individual model-fitting iterations when repeatedly replacing data (left), and when accumulating data (right). Note: Epochs correspond to more gradient steps for accumulate than replace because the number of training data grows for accumulate.
21

---

Published as a conference paper at COLM 2024
Iteration
Language Models Pretrained on TinyStories
Replace
Accumulate
3.0
4.0
1x 10"
5.0
Cross Entropy (Test)
3xx
(W6) 22222
Cross Entropy (Test)
Ulmma (126M)
100
2 10"
4 100
Cross Entropy (Test)
Llama22 (42M)
3 1000
2
Cross Entropy (Test)
Llama-2 (12M)
1000
x 10"
10:4
10
108
10
10 Epoch
10
Epoch
Figure 10: Data Accumulation Avoids Model Collapse in Language Modeling. Learning curves for individual model-fitting iterations when repeatedly replacing data (left), and when accumulating data (right), in log-log scale. Note: Epochs correspond to more gradient steps for accumulate than replace because the number of training data grows for accumulate.
22

---

Published as a conference paper at COLM 2024
Iteration
Language Models Pretrained TinyStories
1.0
Replace
Accumulate
2.0
5.0
3.0
4.0
5.0
Entropy (Test)
4.0
(v96) 2-149
1.5
1.5
Cross Entropy (Test)
Llama-2 (12614
3.0
2.0
1.5
1.5
Cross Entropy (Test)
Uamu (42M)
3.5
3.0
2.0
5.0
Cross Entropy (Test)
3.5
2.0
100000
200000
300000
100000
200000
300000
Gradient Step
Grodient Step
Figure 11: Data Accumulation Avoids Model Collapse in Language Modeling. Learning curves for individual model-fitting iterations when repeatedly replacing data (left), and when accumulating data (right).
23

---

Published as a conference paper at COLM 2024
Iteration
Language Models Pretrained TinyStories
1.0
Replace
Accumulate
2.0
3.0
4.0
1x
5.0
Cross Entropy (Test)
Uva 2:140
3
10"
4xx 100
Cross Entropy (Tes)
Lima-22 (126M)
3. 100
2 100
4xx.000 109
Cross Entropy (Tess)
Liama-22 (42M)
3 3.x.10
100
Cross Entropy (Test)
almms- (12M)
3.x.U 1000
x 10P
10
10a
10a
10'
10
10P
Gradient Step
Gradient Step
Figure 12: Data Accumulation Avoids Model Collapse in Language Modeling. Learning curves for individual model-fitting iterations when repeatedly replacing data (left), and when accumulating data (right), in log-log scale.
24

---

Published as a conference paper at COLM 2024
in which data is replaced, but the size of the (fully synthetic) dataset is grown to match the training set size the accumulation regime. We find that model performance still degrades (albeit at lower rate). This is shown in Appendix C, Table 2, right-most column. Second, possible concern could be that degrading performance when replacing data could be due to low model performance in iteration 1 (and thus the quality of the first synthetic dataset). We control for this by varying the amount of training performed in iteration 1 only and find that this has no significant impact. Lastly, we find that our results are also consistent across varying dataset sizes and training epochs.
Controlling for dataset size. One possible concern is that when accumulating data, the train dataset size will grow at each model-fittigg iteration, meaning subsequent models will be trained on more aggregate data than their counterparts in the replacement regime. To experiments controlling for this. In this "replace-multilee" regime,
control for this, we create a fully synthetic dataset at the end of each model-fitting iteration, but grow the size of this dataset to match that of the accumulated data in the accumulation regime. Table rightmost column shows that in this regime, evaluation loss still increases over model-fitting iterations.
Generation temperature. Most of our language model experiments were run with sam- pling temperature 1.0 during generation of new datasets. To ensure that this choice is not critical, we also run one experiment with temperature 0.3, and that this shows similar results (with even larger increases in validation loss in the replacement regime than temperature 1.0), as shown in Table 2, row 2, and Figure 13.
Dataset size and training epochs. We similarly vary the size of the initial (and subsequent) training datasets and number of training epochs, and see that this has no qualitative effect on the results (Table 2, rows 3 & 4 show training on 1/5th of the TinyStories dataset for & 3 epochs, respectively).
Model quality after first model-fitting iteration. Finally, we control specifically for model (and thus synthetic dataset) quality after the first iteration, to rule out an undue influence of a "bad" first synthetic dataset on subsequent training. Figure 14 shows performance in subsequent iterations for different amounts of training in the first iteration, showing no qualitative differences.
Model
t=1
t=4 (acc)
t=4 (repl)
t=10 (repl)
t=4 (*)
GITT-2 (9M)
1.81
1.78 1.78 (-0.04)
2.31 (+0.49)
2.79 (+0.97)
2.11 (+0.30)
Llama-2 (12M)
2.06
1.98 (-0.07)
2.58 (+0.53)
2.89 (+0.83)
n/a
Llama-2 (42M)
1.86
1.79 (-0.07)
2.34 (+0.47)
n/a
n/a
Llama-2 (126M)
1.71
1.64 (-0.06)
2.15 (+0.44)
n/a
n/a
GPTT2 (9M) (temp=0.3)+
1.82
1.75 (-0.06)
5.82 (+4.00)
9.85 (+8.04)
n/a 2.91 (+0.35)
GPT-2 (9M) (small dataset)**
2.56
2.28 (-0.28)
3.21 (+0.65)
3.72 (+1.16) n/a
ibid (+ 3 epochs)*
1.99
1.87
(-0.12)
2.62 (+0.63)
n/a
Table 2: Evaluation cross-entropy loss for different models at model-fitting iterations 1, and 10 for replacement and accumulation regimes. (*) indicates replacement regime with growing dataset size to ablate for total train set size. indicates experiments run using an earlier version of our code with minor numerical differences.
25

---

Published as a conference paper at COLM 2024
GPT-2 (9M) Pretrained on TinyStories
Iteasion
10
Tempeoture 03
Tempeoture
4.0
5.0
Cooss Lntropy (1oo)
Cross Litroy (Fes)
Figure 13: Accumulating data shows stable behavior across different generation tempera- tures for GPT-2 (9M) model, while replacing data does not.
GPT-2 (9M) Pretrained on TinyStories
1.0
Esoclo (3x. horatoe)
Epects lteration
horation)
2.0
Cioa Eroopp (Te4)
0.0
Clos Entrer (1t83)
0.5
3.0
6.5
3.0
3.8
38
0.0
0.2
0.4
0.6
1.0
0.2
0.4
0.6
1.0
9.2
0.6
0.0
0.5
Epock
Egech
Epoch
Figure 14: Model quality after the first model-fitting iteration does not qualitatively change behavior in subsequent iterations. Columns show differing training amount (as measure by epochs) in first iteration.
26

---

Published as a conference paper at COLM 2024
D Additional Details on VAE Experiments
Experiment Details. As pre-processing, we crop and down-sample the images to 64x64 pixels. We use standard convolutional architecture for the VAE model consisting of convolutional layers with 32, 64, 128, 256, and 512 channels, respectively, and similar convolutional decoder structure. The latent space is 128-dimensional isotropic Gaussian, represented by 2 MLP layers. Each data iteration consists of 100 training epochs, after which we generate 163K new training images by sampling latents from the Gaussian prior and the passing them through the generator model.
Analysis of Reconstructions. Figure 15 shows reconstructions after replacing (left) and accumulating (center) data, compared to baseline (right). Analyzing the reconstruction of test set images also reveals interesting findings the model trained only on data from the prior iteration has indeed collapsed and cannot represent any other classes besides the single mode it generates. Interestingly, the model trained on aggregated data still maintains it's capabilities and generates accurate reconstructions, including smaller details such as glasses and hats. We hypothesize that this model maintains it's generative capabilities, but these details become a more minor sub-manifold in the latent space, which is realigned with the newly-generatee data, hence why they appear less often in the generated images, which use samples from the prior.
Figure 15: Data Accumulation Maintains Model Capabilities. Image reconstructions from the test set. Left: Training on prior iterations collapses the model's capability, and subsequently, it can only represent a single mode. Middle: training on aggregated data preserves model capabilities and leads to little to no degradation in the reconstructed images. Right: Baseline reconstructions after 100 training epochs on the dataset.
27

---

Published as a conference paper at COLM 2024
E Linear Regression: Replacing Data with Increasing Sample Size
In the framework of Mobahi et al. (2020) and Dohmatob et al. (2024a), we consider sequences of linear models fit the previous model's synthetic outputs. Within this framework, Dohmatob et al. (2024a) proved that if data are replaced with each model fitting iteration and the training data cardinality remains constant, then the test squared error scales linearly with the number of model fitting iterations n:
02d - - X n
Replace test
(00n)
(10)
In this work, we lightly adapt the argument of Dohmatob et al. (2024a) to study the effects if data accumulate with each model fitting iteration. We We specifically considered the case where the training data cardinality increases by constant T with each model-fitting iteration i.e. the ith model is fit using T x i data, where T data are "real" and then each subsequently fit model contributes its own T synthetic data the accumulating data. In this setting, the test squared error is upper bounded independent of the number of iterations.
old
o²d
x π
Accumulltee (#n) = T-d-1 X Fest
I T - d
(11)
-
6
k=1
In the main text, we focus on the replace and accumulate data settings because prior work focused replacing data and we wished to study how accumulating data affects model collapse. However, much richer landscape of outcomes is possible. For instance, and as pointed out in personal correspondence with Dohmatob et al. (2024a), what we term the "Replace-Multiplee" setting, in which one fits the i-th linear model using i data sampled from the (i 1)-th linear model. Replace-Multiple is useful baseline for Accumulate because matches the amount of training data at each model fitting iteration. Under Replace-Multiple, the test squared error grows logarithmically:
consider
Replace-Multippl Multiple (w.)) Eestt
odd
R x
Odd
T-d-1 x log(n)
(12)
T-d
k=
Replace-Multiple has the drawback of not matching the total amount of compute of Accu- mulate since each iteration of Replace-Multijp Multiple draws i samples from the most recent model, whereas Accumulate draws T samples from the most recent model. Other baselines are also possible, but we leave these to future work. We focus on accumulating data as we feel real and synthetic data are likely to accumulate in the real world as time progresses.
28

---

Published as a conference paper at COLM 2024
F Additional Linear Regression Numerical Results
Ridge
Data Dimersion:
Nake (+)
Num.. Samples Itaation (?)
Samples teration (?)
har Samp hostie 1000
1.5
10.3
Texl Eroo Type (avemiical
107
Thwarcioll
Sqand
32"4
162
Sqarrs Enee
160
Ene
02
32-4
100
233
150
Finiig
m1gg bwories
Mo54FFirie Fitting
Figure 16: Accumulating data across iterations avoids model collapse in linear regression. We consider sequences of linear models recurrently to generated targets by previous iterations of models. Replace (Top): If each linear model is fit to the generated targets of only the preceding linear model i.e. data are replaced, then the test squared error grows linearly with the number of model-fitting iterations iterations Replace-Multiple (Middle): each linear model is fit to of data Accumulate), then the test squared error grows logarithmically with the number of model-fitting iterations; see Appendix for more details. Accumulate (Bottom): If each linear model is instead fit the generate targets of all the preceding linear models i.e. data accumulate, then the test squared error has a finite upper bound, independent of the number of iterations. This suggests that data accumulation might be robust solution for mitigating model collapse. This figure is similar to Figure 7 but displaying log test squared error and more model-fiting iterations for additional clarity.
samples from the (i 1) model (i.e. the same amount
29


Reviews:
[{"review_id": "6x7FYHNOij", "reviewer": "Reviewer_Djpz", "summary": "The paper focuses on the study of model collapse in deep generative models. It analyzes how the performance of these models deteriorates when they are trained on their own outputs. Two specific scenarios are examined: one where the data is replaced in each model iteration, and the other where data is accumulated over iterations. The authors offer a theoretical analysis on this in the context of linear regression. The paper finds that data accumulation is effective in preventing model collapse, in contrast to data replacement which leads to increased test error. Afterwards, authors conduct empirical experiments on various models such as language models on texts, diffusion models on small molecules, and variational autoencoders on images to verify the practical applicability of their the hypothesis.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Djpz, \n\nThank you for your review and useful comments. Overall, we want to stress that the main message of our paper is to show that accumulating vs replacing data leads to a stark difference in model collapse (or lack thereof), a point overlooked in prior literature. We believe, as you also state, that we thoroughly demonstrate this. To your specific points:\n\n> I feel like the major weakness is the limited scope of theoretical analysis to linear regression [...] the conclusions may not immediately generalize to more complex models.\n\nWe consider the main contribution of our paper to be the experiments, which are more extensive and diverse than prior work in the model collapse literature. The theory was intended as the simplest analytically-tractable example to explain mathematically why accumulating data makes a difference. In our revised manuscript, we swap the order of experiments and theory to emphasize the experiments.\n\n> the authors could expand the theoretical framework to include non-linear models.\n\nOur results hold for nonlinear kernel regressors; we would be happy to state such results in the appendix. However, deriving results for other nonlinear models like logistic regression is non-trivial, as we know of no closed-form solution. In general, theory in this direction is very hard. We also feel that this would detract from the manuscript’s main message, which we believe is an important and highly timely one, and one for which we provide ample evidence.\n\n> Plus, authors can validate the practical applicability of their claims on larger model settings,\n\nWe want to clarify that we pre-train these models from scratch, at each iteration, with growing dataset sizes. Thus, to evaluate a sequence of models with just 5 model-fitting iterations, we need to train for the equivalent of 20 full pre-training runs (1+2+3+4+5 for accumulate, 1+1+1+1+1 for replace). The experiments in the paper altogether constitute hundreds of full pre-training runs. Doing this on billion-parameter models and web-scale data would require compute matching OpenAI, Google or Meta, which sadly we do not have.\n\nBecause of this, we carefully chose the tinystories dataset precisely because it was specifically designed to allow insightful experiments with smaller models. In particular, it was designed to be well-fit by 10-30M parameter models, and we evaluate up to an order of magnitude larger than that.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper offers a solid contribution to the understanding of model collapse in generative models, which is critical for the continuous improvement of model reliability and performance. The comparative analysis of data replacement vs accumulation presents a clear and well-supported argument for the benefits of the latter strategy (data accumulation). The empirical evidence covering multiple deep generative models and datasets validate of the conclusions, demonstrating that this approach has practical applicability.", "reasons_to_reject": "I feel like the major weakness is the limited scope of theoretical analysis to linear regression. This means that the conclusions may not immediately generalize to more complex models. To better connect the theoretical analysis and application in modern deep generative models, the authors could expand the theoretical framework to include non-linear models. For example, given that the authors trying to validate their hypothesis on language models on discrete sequence data, which can be seen as a sequence of classification problems, authors can consider to extend their theoretical conclusion to logistic regression?\n\nAlso, why data accumulation prevents model collapse can be further explored and articulated. Further theoretical insights or explanatory hypotheses would enrich the reader's understanding of the underlying mechanisms. Plus, authors can validate the practical applicability of their claims on larger model settings, for instance a Llama 3B?", "questions_to_authors": "see weaknesses", "extra_scores": {}}, {"review_id": "OF67qnx3bh", "reviewer": "Reviewer_Vd4L", "summary": "This paper investigates the phenomenon of model collapse, where training generative models on their own outputs leads to progressive degradation in performance. The authors compare two settings: 1) replacing the training data with new synthetic data at each model-fitting iteration, and 2) accumulating synthetic data with real data at each iteration. They prove theoretically for linear models that accumulating data bounds the test error independent of the number of iterations, while replacing data causes test error to grow linearly with iterations. The authors then empirically demonstrate consistent results for deep generative models including causal transformers on text, diffusion models on molecules, and variational autoencoders on images. They find that across architectures and datasets, accumulating data prevents the model collapse that occurs when replacing data.\n\nOn a first glance, the paper is clearly written, with theoretical results derived mathematically and then validated numerically.Upon further reflection, the theoretical formulation, along with the experiment setup guided by the theory, seems to be a bit distant from the real world setting especially language modeling. In fact, I think how it should be formulated inevitably is deeply integrated with the nature of each modality (e.g., what is the ground truth of the modality).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for your in-depth review and the interesting and on-point discussions you raise.\n\n> Lack of discuss on data quality [...] It is essential to engage with the leading edge of synthetic data in each domain to advance our understanding.\n\n> There is growing evidence suggesting that language models are capable of generating synthetic data of superior quality compared to real data [1], which can be more beneficial for certain downstream tasks.\n\nWe agree that research on using synthetic data to improve models or accelerate learning is an exciting new topic.\n\nHowever, we see this as a step beyond our paper. Our paper is not asking the best way to use synthetic data. Our paper is asking whether future generative models will become useless in a pessimistic future where synthetic data is dumped without restraint on the internet. Our conclusion is no. Our point is not that random synthetic data is good for models, but that such data is not especially harmful so long as data accumulates. We believe this is an important and timely point to make, considering model collapse has recently been picked up by mainstream media as a catastrophic problem that threatens the future of generative AI.\n\nWe add to our Discussion that identifying how to best use synthetic data is an exciting research frontier and cite your recommended paper “Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling”.\n\n> Impact of data size in comparisons\n\nAppendix C includes this ablation (“Controlling for dataset size”) and our updated manuscript will have more experiments. For hyperparameters, is there any specific experiment you would like to see?\n\n> Test loss on real data might not be the best metric to compare\n\nTest loss on real data is an extremely standard metric for evaluating models. While today’s language models indeed have significant post-processing applied, such methods rely on the base models having low pretraining loss. Applying DPO or some other preference optimization method to a high-perplexity LM will not work.\n\n> I would suggest the author measure the loss on a held-out dataset\n\nTo be clear, we do measure the loss on a held-out split of the real data.\n\n> maybe loss on synthetic data is a better metric to compare\n\nMeasuring the model’s loss using model-generated synthetic data opens the door to undesirable minima. For instance, if the model always outputs the same token, its cross entropy loss on the generated data will be 0, but its outputs are useless.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper formulates the timely and critical issue of model collapse in generative models, providing both theoretical and empirical insights.\n- The theoretical analysis for linear models is rigorous and clearly presented. The key result that accumulating data bounds test error is novel.\n- The authors conducted extensive experiments on transformer, diffusion, and VAE models on text, molecule, and image datasets. This breadth makes the results widely interesting and applicable.", "reasons_to_reject": "- **Test loss on real data might not be the best metric to compare, and thus does not faithfully characterize the real world scenario**: The primary metric used to assess model performance in this framework is the test loss on real data. However, in the context of language model pre-training, reducing test loss might not always correlate with overall model effectiveness. There is growing evidence suggesting that language models are capable of generating synthetic data of superior quality compared to real data [1], which can be more beneficial for certain downstream tasks. This phenomenon challenges the assumption that minimizing test loss on real data accurately reflects model quality. In this case, I think the mode collapse on the \"replacing data\" part should not happen because maybe loss on synthetic data is a better metric to compare. Their specific setup (small language models, and small data), simply hasn't reached the inflection point, which makes the current results a bit misleading and not generalizable. I would suggest the author measure the loss on a held-out dataset, or work in a more frontier synthetic data setting, where the generated data could be of a higher quality. \n- **Lack of discuss on data quality**: The paper lacks a discussion on the quality of synthetic data across different domains. Synthetic data quality can possibly alter our perceptions of what constitutes ground truth in each modality. For language modeling, synthetic data might represent \"ground truth\" better as the data quality increases, since in practice we train LMs to perform cognitive reasoning tasks, not simply language itself. This argument might still hold for image generation, but may not for molecule modeling. It is essential to engage with the leading edge of synthetic data in each domain to advance our understanding.\n- **Impact of data size in comparisons**: The study attempts to compare the effects of accumulating versus replacing data, but it does not sufficiently account for the influence of data size on test loss. In practical scenarios, test loss is often dependent on the amount of data (e.g., LM) due to factors such as learning rate schedules and optimization algorithms (e.g., Adam). This oversight suggests that we need a more nuanced analysis that considers how increasing data size impacts learning outcomes beyond the simplistic linear model framework employed.\n\n[1] Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling", "questions_to_authors": "- **Optimal Use of Synthetic Data**: Figure 4 suggests that incorporating synthetic data can reduce test loss on real data, yet it raises an important question about the diminishing returns of synthetic data. Specifically, it would be insightful to determine the point at which further iterations of adding synthetic data no longer enhance model performance (if model performance can be properly measured). This inflection point is observed earlier in domains like molecule generation and image generation compared to language modeling. \n- **Typo**: reference in 2.3 is missing: Code in App. ??", "extra_scores": {}}, {"review_id": "hPNkKXDa2s", "reviewer": "Reviewer_7uQd", "summary": "This paper investigates the phenomenon of model collapse, where the performance progressively degrades when they are trained on their own generated outputs over multiple iterations. The key contribution os this paper is showing that accumulating the generated synthetic data with the original real data across iterations, instead of replacing data, can prevent model collapse. They authors provide both theoretically and empirically analysis.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 7uQd for their review and interesting discussion. Our responses are below:\n\n> The exploration on larger model is desired [...] such as those with billion-parameter scales\n\nWe agree larger models would be better but billion-parameter models are computationally unaffordable for us. We pre-train from scratch, repeatedly, on growing data. 5 model-fitting iterations is equivalent to 20 full pre-training runs; the experiments in the paper together are well into the hundreds of full pre-training runs. To evaluate 7B parameter models on a 2T token starting dataset, we would need to train for 40T tokens for a single 5-iteration experiment – more than twice as many as Llama3! And this is not even taking into account generating the synthetic tokens. We cannot afford this.\n\nBecause of this, we carefully choose the TinyStories dataset, as this has been designed specifically to allow for meaningful experimentation with smaller models.\n\nThat said, if you have a specific suggestion of a model size and dataset that are both achievable with reasonable compute budgets, and would provide insight that our current experiments do not, we would be very happy to discuss this!\n\n\n> The paper mainly relies on linear models for its theoretical analysis [...] this simplification might not fully capture the complexities of real-world scenarios.\n\nWe agree that linear models may fail to capture real-world scenarios, which is precisely why we complemented the theory with three medium-to-large-scale realistic experiments on deep generative models across 3 data modalities.\n\nOur main results are these extensive experiments. We will swap the order of experiments & theory to emphasize this.\n\n> This paper lacks of discussion and comparison with other popular methods in this area\n\nCould the reviewer please clarify what “other popular methods in the area” means? Our work is scientifically studying a particular system, i.e., sequences of generative models trained on outputs from their predecessors. \n\n> The discussion about the related work is missing.\n\nWe have written a 2 page Related Work and will add this to the Appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. They paper invistigates a critical and timely issue in the field of AI, i.e., model collapse in generative models. And it mainly focuses on data accumulation.\n2. The paper proveds a theoretical foundation to understand the model collapse when data is replaced accross iterations and how accumulating data can prevent this phenomenon.\n3. The authors have conducted extensive experiments across different types of models and data modalities to show the effectiveness.", "reasons_to_reject": "1. The exploration on larger model is desired. This paper disscussed some models (linear models, transformers, diffusion models, and VAEs), but it does not extend its exploration to large models, such as those with billion-parameter scales, which is the main-stream nowadays.\n2. The paper mainly relies on linear models for its theoretical analysis. While useful, this simplification might not fully capture the complexities of real-world scenarios. \n3. The discussion about the related work is missing.\n4. This paper lacks of discussion and comparison with other popular methods in this area.", "questions_to_authors": "Please refer to the negative comments. No further questions.", "extra_scores": {}}, {"review_id": "bUTa6OCLRh", "reviewer": "Reviewer_TmgS", "summary": "Post rebuttal:\n\nI increased the rating from 6 to 7 because of the clarification from the authors addressed my concerns to a certain extent.\n\n---\n\nInitial review:\n\nThis work investigates the problem of model collapse, which usually happen when the model is trained continually with its own output data. Unlike previous research, in which only newly generated data is added, this study assumes that both real data and synthetic data accumulate with evolving.\n\nThe authors conduct empirical analysis in a simple setting with linear model, followed by extended experiments in Transformers, Diffusion model and VAE, while the training data in multiple modalities is selected accordingly.\n\nBoth theoretical and empirical analysis support the conclusion that simply adding real data in multi rounds of training can solve the model collapse problem.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer TmgS,\n\nThank you for your review.\n\n> is it possible to accumulate the real data as well? In nature the real data will be generated by humans over time. The experiments will provide more insights.\n\nWe agree this is an interesting question and a more realistic modeling assumption. For the current paper, we intentionally wanted to present a very pessimistic scenario of accumulating data, to show that even under pessimistic conditions, simply keeping previous data avoids model collapse.\n\nIn practical terms, for experiments accumulating real data does present a challenge, as you can’t create new real data in an experiment, so one would have to start out with just a fraction of the dataset in the initial iteration.\n\n> A clarification (as well as a detailed section of related work) from the authors will be appreciated.\n\nWe will update the manuscript with an in-depth 2-page Related Work section\n\nTo your specific question, our setting is different from the synthetic augmentation loop in Alemohammad et al! In their work, they primarily consider the case where a constant amount of synthetic data is replaced in each iteration, in addition to a constant amount of real data. That is, their setting is closer to what we consider the “replacement” setting, and they do conclude that this does not avoid model collapse. They do briefly look at accumulating data as well, but only in one part of one experiment, and they conclude that this merely slows model collapse down.\n\nWe will include a new appendix section closely examining Figure 7 of Alemohammed et al. We believe that a closer examination suggests that test error plateaus (and this would likely have been more apparent in subsequent iterations), consistent with our results and in contrast with their conclusion that “fixed real training data only delays the inevitable degradation of the quality or diversity of the generative models over generations.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem of *model collapse* is quite important in many fields, as the synthetic data is becoming a considerable part of training dataset for foundational models.\n2. This study provides a clear analysis of this problem, by providing experiments in widely used model architectures, with multiple modalities, including language, images and protein.\n3. Surprisingly, all experiments support the assumption of the authors, that simply adding the original real data into the newly generated data can solve the problem of *model collapse*. This insight is straight-forward, clear, and easy-to-implement, thus potentially impactful to the community.", "reasons_to_reject": "1. A minor concern: is it possible to accumulate the real data as well? In nature the real data will be generated by humans over time. The experiments will provide more insights.\n2. A major concern: the statement of previous study [1] in this paper seems inaccurate, the \"synthetic augmentation loop\" (section 2 in [1]) defined as training model with **fixed set of real data and synthetic data from previous iteration** seems to be an exact same setting as that in this paper. Hence, the novelty and contribution of this paper highly depends on the difference with previous study. A clarification (as well as a detailed section of related work) from the authors will be appreciated.\n\nI would increase the rating if I mis-understood above issues.\n\n[1] Alemohammad et al. Self-Consuming Generative Models Go MAD. ICLR 2024", "questions_to_authors": "Stylistic error: broken reference in Section 2.3", "extra_scores": {}}]

Author rebuttal:

Meta-review: I think this is a great paper. The problem, model collapse, is interesting. It provides interesting empirical work with interesting theoretical work on a toy case. I disagree with reviewer Djpz that the theoretical analysis is too limited -- it's hard to analyze beyond the linear case. I thought the authors did a tremendous job in engaging with the reviewer feedback. Apologies for the shorter meta-review -- it is simply due to the fact that the reviewers did a great job on this one.
Decision: Accept