Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World

Authors: Joshua Kazdan, Rylan Schaeffer, Apratim Dey, Matthias Gerstgrasser, Rafael Rafailov, David L. Donoho, Sanmi Koyejo
Primary area: generative models
Keywords: model collapse, model-data feedback loops, synthetic data, sampling bias, deep generative models, model misbehavior

Abstract:
The increasing presence of AI-generated content on the internet raises a critical question: What happens when generative machine learning models are pretrained on web-scale datasets containing data created by earlier models?
Some authors prophesy _model collapse_ under a '_replace_' scenario: a sequence of models, the first trained with real data and each later one trained _only on_ synthetic data from its preceding model. In this scenario, models successively degrade. Others see collapse as avoidable; in an '_accumulate_' scenario, a sequence of models is trained, but each training uses all real and synthetic data generated so far. In this work, we deepen and extend the study of these contrasting scenarios. First, collapse versus avoidance of collapse is studied
by comparing the replace and accumulate scenarios on each of three prominent generative modeling settings; we find the same contrast emerges in all three settings. Second, we study a compromise scenario; the available data remains the same as in the _accumulate_ scenario -- but unlike _accumulate_ and like _replace_, each model is trained using a fixed compute budget; we demonstrate that model test loss on real data is larger than in the _accumulate_ scenario, but apparently plateaus, unlike the divergence seen with _replace_.
Third, we study the relative importance of cardinality and proportion of real data for avoiding model collapse. Surprisingly, we find a non-trivial interaction between real and synthetic data, where the value of synthetic data for reducing test loss depends on the absolute quantity of real data. Our insights are particularly important when forecasting whether future frontier generative models will collapse or thrive, and our results open avenues for empirically and mathematically studying the context-dependent value of synthetic data.

Decision: reject
Reviewer scores: 8, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1730604639946

thread_type:
official_review

<thread_object>
creator: reviewer_9sjl

<content>
summary:
This paper addresses the risks of training generative models on datasets increasingly dominated by AI-generated (synthetic) data, studying "model collapse"—where model performance deteriorates as synthetic data accumulates. Through theoretical and empirical analyses, the authors explore when and how this collapse can be mitigated. They test three data-handling approaches (replace, accumulate, and accumulate-subsample) across various generative model settings. Findings suggest that model collapse is avoidable if synthetic data is accumulated alongside real data, and highlight how different ratios of real to synthetic data influence model performance.

strengths:
1. The studied problem of model collapse under increasing AI-generatd data is prompt and interesting.

2. The theories and empirical studies round up the good work.

3. This paper offers actionable guidance on how to balance real and synthetic data in training.

weaknesses:
1. Some of the theoretical proofs hinge on idealized assumptions (e.g., independence and Gaussian distribution in certain proofs). These may not fully apply in practical, real-world datasets.

2. The "accumulate-subsample" paradigm offers a practical perspective on fixed compute budgets but is tested under simplified conditions. Additional real-world constraints (e.g., dynamic memory handling) are not considered in depth.

3. While language models are likely the most relevant generative model for readers, the theoretical analysis does not extend to this model type and is instead focused on Gaussian and kernel density estimation models. Additionally, the size of the language model used in this study is quite small. While contemporary models like Llama range from 7B to 70B parameters, the paper’s experiments are limited to Gemma2 with only 2B parameters.

questions:
1. The study's empirical analysis is limited to a 2B parameter language model. Given the prevalence of larger models like Llama 3 and Mistral, do the authors plan to extend their experiments to these more widely used models to assess the generalizability of their findings?

2. This work primarily explores synthetic data quantity, but the quality of synthetic data is also a significant factor. Would lower-quality synthetic data exacerbate collapse, and could filtering or curating synthetic data mitigate this effect?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730651878993

thread_type:
official_review

<thread_object>
creator: reviewer_bken

<content>
summary:
The paper investigates the impact of iteratively training a generative model on a combination of real and synthetic data generated from a previously trained model. This is an important area of research to explore and understand, as exclusively training models on synthetic data will eventually fail, as highlighted by prior work. 
The authors conduct experiments to validate previous findings, offering new evidence to clarify cases in which model collapse can be avoided. They also explore additional experimental settings in which they control for the ratio of synthetic/real data and the absolute quantity of real data to gain further insight into the limits of training with synthetic data.

strengths:
1. Motivation. The research question is clearly defined and contextually relevant. Exploring the impact of training with synthetically generated data is important for advancing the field of machine learning.

2. Experiment section. The experiments are thorough, relevant, and insightful. The authors verify that the hypotheses they are testing hold across a wide range of models and data generation processes. They present interesting experiments that control for different parameters, offering clear insights into the scope of the problem. The authors clearly list the questions they aim to answer and provide solid experiments to support their conclusions, along with clear and detailed explanations of their experimental setup.

3. Writing and structure. The paper is well written and follows a logical and clear structure.

weaknesses:
1. Contribution. The majority of the experimental section (4/9 pages) validates the findings of previous work (Gerstgrasser et al., 2024). The experiments verify that retraining with a mix of synthetic and real data can avoid model collapse in three additional settings. Although it is valuable to reproduce and verify previous findings, this should not constitute the main contribution of a novel work. Emphasizing and clarifying the unique contributions and findings of this paper would strengthen its impact.

2. Presentation. Overall, the presentation of the work could be improved in several ways. First, a clearer discussion of related work to better position this paper's contributions is needed. In the introduction, various papers on model collapse are listed with the comment that they have differing methodologies and conclusions. What are the differing methodologies and conclusions? A brief summary of these differences and a clearer statement of this work's position relative to others is required. This should be easy to achieve as the paper is one page short of the limit, so I am not sure what justifies the almost nonexistent discussion of the listed previous work. Second, the presentation of figures could be improved (e.g., readability of axis labels, font size, relative scaling of the plots). Some figures use notation like 1.2×10^0 instead of simply 1.2, and there is a lot of wasted empty space in the figures. Lastly, there are several errors in the references, as several published works are cited as arXiv references. This gives the impression of an unpolished work.

Minor

1. Some of the writing is needlessly sensationalized. For example, describing "that model collapse is caused by deleting past data en masse and avoided by instead accumulating real and synthetic data" as a "provocative pair of claims" feels excessive. It’s clear that using real data works and that purely generated data does not, so it is not surprising that a middle ground exists. Especially since this finding was already made by previous work. 

2. The use of tweets and LinkedIn posts as references gives the paper an unserious tone, though this may be a matter of personal preference.

questions:
1. This paper presents iterative settings in which performance either suffers from model collapse (replace) or avoids it (accumulate and accumulate+subsample). Have the authors explored ways to identify the breaking point between these two modes? For example, would the results hold if, instead of adding $n$ samples at each iteration, we add an increasing amount of synthetic data (say, $2n$ each time)?

2. Another question is about the relative quality of the synthetic data. Presumably, there is a connection between how well the model learns the task and whether its generated data can effectively train a second iteration. Do the authors have comments on this?


3. Regarding the observations in the blob experiment, the authors note a tendency: "Interestingly, for specific pairs of datasets and numbers of samples per iteration, training on real data while accumulating synthetic data can yield lower loss on real test data than training on real data alone." For these specific settings where the true distribution is known (I believe that it is the case?), it would be valuable to report the NLL value of the true distribution as a reference point.

4. Have the authors considered extending the result from Theorem 1 to the accumulate-subsample setup? Or relating this result to their experimental results? This could strengthen the contribution of this work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696903256

thread_type:
official_review

<thread_object>
creator: reviewer_xzu3

<content>
summary:
this work study the model collapse with the sythetic data.

strengths:
1. The research topic is very interested. It tries to study the model collapse issue when the world have more and more sythetic data due to generative model.

2. The paper contain a lot of experiemts to valdiate their claim.

3. The finding in this work is interesting. they find a non-trivial interaction between real and synthetic data, where the value of synthetic
data for reducing model test loss depends on the absolute quantity of real data.

weaknesses:
1. the paper starts with MULTIVARIATE GAUSSIAN MODELING, can the findings still hold for other model, like VAE, diffusion model, or Transformer. Somehow,  I feel the paper make too strong claim since there are many generative model.

2. the reprodubility is unclear at this time.

3. the math proof in Appendix is mainly for Guassina model, it is unclear how it can generalize to more generative models.

questions:
1. the paper starts with MULTIVARIATE GAUSSIAN MODELING, can the findings still hold for other model, like VAE, diffusion model, or Transformer. Somehow,  I feel the paper make too strong claim since there are many generative model.

2. the reprodubility is unclear at this time.

3. the math proof in Appendix is mainly for Guassina model, it is unclear how it can generalize to more generative models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730767696313

thread_type:
official_review

<thread_object>
creator: reviewer_mnch

<content>
summary:
Update: I appreciate the authors efforts to address my concerns, and thus would like to raise my score. Ideally, I would change it to a 7 b/c I feel it does deserve acceptance but not spotlight/oral. 

This paper studies how training on synthetically generated data in the model-data feedback loop. In particular the authors claim to focus on a more nuanced setting than previous papers on the topic by considering settings where (i) parts of the real-data and synthetic data are used together (instead of discarding the real-data entirely in favor of synthetic), (ii) only a fixed budget is available for updating the model, and (iii) consider how important the proportion vs. cardinality is when updating the model. The authors demonstrate that keeping some of the real data can help prevent collapse, that the right amount of synthetic + real data can improve performance, and in some cases, no amount of synthetic data is outperform real data. The authors provide some proofs to demonstrate their claims too.

strengths:
- *Important Problem Setting.* The problem of model collapse is a pressing problem and I found the authors investigation into the more nuanced issues of the model-data feedback loop compelling: that likely parts of synthetic and real data will be aggreagated, the budget assumptions etc. 

- *Thorough Experiments.* The authors conduct well-designed experiments on a variety of models and settings, including Gaussian models, kernel density estimators, linear regression and language models on a variety of datasets. While some of these models and datasets are fairly simple, they nonetheless help suport the author's claims. 

- *Theoretical Insights.*  In addition to the empirical insights, the authors are able to proud some theoretical guarantees. This helps complement the empirical results with simpler models.

weaknesses:
- * Novelty.* The novelty of this work is fairly limited since it builds quite a bit on existing work. It starts by addressing some of the claims from Gertgrasser et al, and then uses the model/settings in Shumailov et al. The settings the authors tried out (not replacing data en-masse. etc) are important but do seem a bit incremental conceptually.

- *More Real-settings.* The authors motivate their work a fair bit from the perspective of language models. I would like to have seen more experiments on this setting under a couple of different model strengths and synthetic data quality.

questions:
- Can the authors please clarify their novelty a bit more?
- Can the authors please discuss how they expect their results to hold when training significantly larger models, or changing the quality of the synthetic data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732653619367

thread_type:
public_comment

<thread_object>
creator: ~julia_kempe1

<content>
title:
Public comment by 9 members of the model collapse community

comment:
1/2

Dear Reviewers and AC,

In this public comment, we would like to respectfully point out that this paper fails to adequately acknowledge all perspectives (and large amounts of related works) on model collapse and, by extension, on mitigating it. As such, their definition and approach to model collapse offers at best a partial view. We feel it seriously misrepresents other works, which should not happen when the purported goal is to unify fractured literature (see point 1 below). 

Claims that warrant a nuanced viewpoint are stated unequivocally: for instance (line 21): *".. previous work claimed that model collapse is caused by replacing all past data with fresh synthetic data at each model-fitting iteration and that collapse is avoided by instead accumulating data across model-fitting iterations"* hardly summarizes the status of the literature, which contanis a wealth of explanations and mechanisms.

Moreover, we want to point out omission of prior work that already studied parts of "new contributions" of this paper, namely the benefits of synthetic data when real data is scarce (see point 3 below).
We also highly comment reviewer *BKEn* for noting that the contribution of the presented work is essentially additional validation of the findings of previous work (Gerstgrasser et al., 2024). Where novel theory would have been welcome (subsampling from a mixture of real and synthesized data to compare apples to apples in terms of dataset size), it seems lacking (point 2 below). In detail.

1. **Clarification and Definition of Model Collapse.** We appreciate the effort in addressing and attempting to unify the fractured literature on the perils and promises of synthetic data. However, to achieve this goal effectively, the paper must clearly define "model collapse" and elucidate what avoiding model collapse means. The definition of model collapse varies across prior works:
- Shumailov et al. (2024) define model collapse as the phenomenon where “a degenerative process affecting generations of learned generative models, in which the data they generate end up polluting the training set of the next generation.” They continue to define that model collapse is not just that the final model is catastrophically broken, it is also gradual loss of representativeness and improbable events.They called them *stages* in the paper. 
- Model collapse is not only about finiteness of synthetic or real data, it is really the interaction of finiteness and overregularization akin to Mobahi et al. (https://arxiv.org/abs/2002.05715), which can happen in kernel density estimation, as shown in Shumailov et al. (Nature, 2024). 
- Dohmatob et al. (2024b) identify model collapse as the alteration of scaling law curves, where training on synthetic data breaks the scaling law compared with the scaling law of training on real data. Continuous retraining leads to a n-fold divergence. Avoiding model collapse in this context involves achieving a scaling law curve comparable to that of real data.

Gerstgrasser et al. (2024) and this paper conceptualize model collapse as a scenario where continuous retraining on synthetic data results in diverging performance, with avoiding collapse equated to maintaining a non-diverging performance. These definitions are all different and none seems more valid than any other. As a consequence there is a body of works taking different viewpoint on what "avoiding model collapse" means. Let us just list:

- Zhang, Jinghui, et al. "Regurgitative training: The value of real data in training large language models." arXiv preprint arXiv:2407.12835 (2024). "All three strategies can improve the performance of regurgitative training to some extent but are not always able to fully close the gap from training with real data."
- Alemohammad, Sina, et al. "Self-improving diffusion models with synthetic data." arXiv preprint arXiv:2408.16333 (2024) also terms preventing MaDness (collapse) as whether we could improve beyond the generator of the syntehtic data.
- Peterson, Andrew J. "AI and the Problem of Knowledge Collapse." arXiv preprint arXiv:2404.03502 (2024) also calls the prevention of collapse to be preserve an accurate vision of the truth for the community.
- Feng, Yunzhen, et al. "Beyond Model Collapse: Scaling Up with Synthesized Data Requires Reinforcement." arXiv preprint arXiv:2406.07515 (2024) also defines avoiding model collapse as attaining performance beyond the generator that synthesize data.
- Wang, Ze, et al. "Bias Amplification: Language Models as Increasingly Biased Media." arXiv preprint arXiv:2410.15234 (2024) treats mixing and accumulation as mitigation methods.

(cont. in 2/2)
</content>
</thread_object>

</thread>
<thread>
creation date:
1733163264822

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to the AC and Reviewers

comment:
Dear AC and Reviewers,

Thank you for reading our paper and offering helpful suggestions.  Because the discussion period is ending, and we have not received responses or updated scores from two reviewers, we would like to summarize the main concerns of the reviewers and how we addressed them.  

__Larger models__: All reviewers questioned how our results would change under different model sizes and/or strengths, since larger models would likely improve synthetic data quality.  To address this concern, we ran additional SFT experiments with Gemma-2-9B and Gemma-2-27B (Figure 3).  These experiments suggest that our results hold regardless of model scale, but larger models collapse more slowly under the replace paradigm.

__Reproducibility__: Reviewer xzu3 states “the reprodubility is unclear at this time.”  We have provided 5 sets of experiments run across 3 seeds with error bars.  The results are all consistent with each other.  Moreover, for the final version, we have agreed to release the Github for the project, which provides access to all of our models, sweep settings, and runs.  In our opinion, objections to the paper on the grounds of reproducibility are unfounded.

__Novelty__: Two reviewers (MNCh, BKEn) raised concerns about the novelty of our work.  Originally, Gerstgrasser et. al. postulated that accumulating data avoids collapse.  However, their theoretical and experimental settings were not extensive enough to support the universality of the claim.  We provided theoretical evidence in 2 new settings and experimental evidence in 3 new settings to support the generality of Gerstgrasser’s claim.  Note that previous model collapse works studied some of the same modelling settings, but they lower-bounded the proportion of real data (Bertrand et. al. (2024), Dohmatob et. al. (2024b)), whereas we allow it to go to 0.  We also replicated Gerstgrasser’s pretraining experiment with more training iterations and provided evidence that in low data settings, an optimal amount of synthetic data can improve model  SFT performance.  Finally, we explored a new compute-limited paradigm that we call “accumulate-subsample.”  Accumulate-subsample represents a middle-ground between replace and accumulate.  The extensive new theoretical and experimental results that we have provided justify our work’s novelty.

__Proof Assumptions__: Two reviewers (9sJL, xzu3) object to our proof settings, noting that theorems about Gaussian estimation and KDEs do not generalize to LLMs.  Currently, the field does not know how to analytically probe general LLMs, and using Gaussians and KDEs to gain intuition about language models has a strong precedent in Shumailov et. al. (2023).  Shumailov explored these settings under the replace paradigm to provide evidence of collapse, so we chose to use the same settings under the accumulate paradigm to show that collapse does not occur.  As a result, our lack of analytic proofs for deep language models should not be viewed as a shortcoming of our paper. 

__Related Works__: Reviewer BKEn commented that we were missing a related works section.  We have added one, and will revise it in accordance with his suggestions in our final version.

__Conclusion__:
Our paper has significantly improved through our efforts to address the reviewers’ concerns.  We hope that the remaining reviewers will respond and acknowledge these improvements.  If they do not respond, __we implore the AC to consider that our revisions have obviated most of the objections posed to our paper.__

Sincerely,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies model collapse, an important shortcoming of generative models when they are trained on synthetic data. The major concern that reviewers have, including public reviewers from the model collapse community, is that the paper is of limited novelty and doesn't fully contextualize itself in the broader model collapse literature. The reviewers judged this paper fails to properly discuss key papers making similar contributions. There has been a long exchange between the authors and the reviewers. My recommendation for rejection is based on reading the paper and on the authors' failure to address all the reviewers' concerns when it comes to previous work.