Strong Model Collapse

Authors: Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, Julia Kempe
Primary area: learning theory
Keywords: Model Collapse, Regression, High dimensional asymptotics, Synthetic Data, Scaling Laws

Abstract:
Within the scaling laws paradigm, which underpins the training of large neural networks like ChatGPT and Llama, we consider a supervised regression setting and establish a strong form of the model collapse phenomenon, a critical performance degradation due to synthetic data in the training corpus. Our results show that even the smallest fraction of synthetic data (e.g., as little as 1 per 1000) can still lead to model collapse: larger and larger training sets do not enhance performance.  We further investigate whether increasing model size, an approach aligned with current trends in training large language models, exacerbates or mitigates model collapse. In a simplified regime where neural networks are approximated via random projections of tunable size, we both theoretically and empirically show that larger models can amplify model collapse. Interestingly, our theory also indicates that, beyond the interpolation threshold (which can be extremely high for very large datasets), larger models may mitigate the collapse, although they do not entirely prevent it. Our theoretical findings are empirically verified through experiments on language models and neural networks for images.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1729697970602

thread_type:
official_review

<thread_object>
creator: reviewer_g7ed

<content>
summary:
This paper studies the model collapse phenomenon in a supervised regression setting. The authors study whether model collapse is inevitable or whether proper mixing strategies can mitigate it. They also study the connection between model size and robustness to model collapse.

*Setup and tools*: The theoretical analysis focuses on linear models and random projection models (simplification of neural networks). The authors derive their proofs using linear pencils and operator-valued free probability theory (OVFPT).

*Contributions*: The authors provide two main theoretical results: 1) they show that even the smallest fraction of synthetic data can lead to model collapses where increasing the model's size does not improve the performance. This is dubbed "strong model collapse". 2) They study the impact of the model's size on the severity of the collapse and show that bigger models are more prone to model collapse when the distribution shift between synthetic and real data is important. They also show that an interpolation threshold exists (model size = sample size) beyond which larger models are more robust to collapse. This is akin to a double descent phenomenon. The theoretical results are experimentally verified on synthetic data and the authors also conduct experiments with neural networks on MNIST and GPT2-small on BabyStories data that corroborate their results.

strengths:
- The problem tackled by the authors is of great interest for the training of large-scale generative models.
- I find the derivation of the proof elegant and the usage of the OVFPT novel in this setting.
- The derived results are insightful and well-analyzed by the authors, along with the empirical validation on synthetic data that helps to convey their implications to real-world scenarios.
- The authors conduct experiments on real data with (small) neural networks and LLMs that corroborates their theoretical analysis.
- The paper is well-written and logically structured, making it easy to follow the authors' arguments and methodology.

weaknesses:
1) Could the authors detail more the key steps/ideas of the proofs in the main paper (paragraph **Proving Theorem 1**)? I believe such tools could be beneficial to the reader, even in other applications.
2) Could the authors discuss the estimation of the quality of synthetic data (parameter $c^2$) on the real applications with MNIST and BabiStories? How would one assess it when training a large-scale model like an LLM  or a VLM?
3) Could the authors discuss in more length the "cost" of approximating neural networks as random projections? In particular, what is lost in terms of generality, does it approximate any architecture (e.g., can LLMs be studied in the NTK or infinite-width limit?)

questions:
See weaknesses.

I list the potential typos found in the text below. *"a-->b"* means I think a should be replaced by b.

*Section 2.2*
- Paragraph **Random Projection Models** (l224 & l225 ): it seems that dimensions of $S, x, v$ do not match. According to previous notations, $x \in \mathbb{R}^d$ implies $S \in \mathbb{R}^{m \times d}$ (instead of $d \times m$ as it is currently (l224)) and $v \in \mathbb{R}^m$ (instead of $v \in \mathbb{R}^k$ as it is currently (l225)).

*Section 3.1*
- Paragraph **Example: The Isotropic Case** (l299): *"the synthetic data $P_1$"* --> it should be *"the synthetic data $P_2$"*

*Section 3.2*
- Paragraph **Are Larger Models More Prone...** (l398 & l401): *"represented by crosses"* --> it should be *"represented by stars"* (l398). I am not sure but according to Figure 1 *"for both high-quality and low-quality (squares)"* --> it should be *"for both very high-quality and high-quality (squares and diamonds)"* (l401)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730557858263

thread_type:
official_review

<thread_object>
creator: reviewer_5qx9

<content>
summary:
This paper considers the questions: is model collapse inevitable when training with synthetic data? Are larger models more prone to collapse than smaller ones? The paper concludes that model collapse is inevitable if training with some non-zero about of synthetic data. Larger models may suffer more from model collapse in the low quality data setting. The paper contains theoretical analysis for linear models and random projection models, and empirical results for GPT-2 models trained on the BabiStories dataset.

strengths:
- The paper shows model collapse in theoretical settings of the linear and random projection models
- The paper shows nice alignment between the experiments and theoretical analysis (eg Fig 3)

weaknesses:
I'm a bit unclear on the takeaway from the experimental results with GPT2. Specifically, the paper seems to make mixed claims for the case of large models beyond the interpolation threshold; it is said that large models may mitigate the collapse beyond the interpolation threshold and that large models tend to amplify collapse beyond the interpolation threshold in the experimental results.

questions:
- For the empirical results, it is said that the curves are expected to plateau in Fig 5. Can this be shown empirically by further scaling?
- Can the double descent phenomenon be showed more clearly in the experimental results?
- Separately, I would be interested in more analysis specifically in the high quality data setting, ie what can be said about the amount of model collapse as the synthetic distribution approaches the real distribution?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730929002047

thread_type:
official_review

<thread_object>
creator: reviewer_wc8a

<content>
summary:
This paper studies model collapse, which relates the performance degradation of ML models due to synthetic training data. For linear regression and random projections model, the authors establish scaling laws that characterize this degradation, as a function of the proportion of the synthetic data as well as the quality. Their findings suggest that larger models become more susceptible to this degradation. Experiments pertaining to training GPT-2 with synthetic data are used to demonstrate validity of their results beyond the limited theoretical models.

strengths:
- The paper is very well written; the problem statement and the contributions are clearly established. 
- The theoretical results seem sound and intriguing, and I appreciate the authors efforts to present simplified examples to help the reader understand the implications better.
- The findings seem to transfer to practical learning settings fairly well.

weaknesses:
- Model Assumptions: I am unsure of the modelling of synthetic data and how well it translates into practice. Specifically, the authors model synthetic data using a label shift, assuming that the data (X) marginal remains the same. However, it seems unrealistic for autoregressive training (a key experiment in the paper), where the input tokens for next token generation come from the synthetic distribution. 
- Experimental Details: The theoretical results establish a strong dependence on the quality of synthetic data. However, the experiments with real data (MNIST/GPT-2) do not provide quantitative metrics to measure the degradation in the synthetic data source (either accuracy of MNIST classifier or perplexity/goodness scores of the trained GPT-2 generator), which makes it hard to ascertain which paradigm (in fig.1) the experiments align best with or what level of degradation in practice results in the observed trend. 
- Minor Issues
    - Typos: Line 225 (v \in R^{m}), line 299 (synthetic data P2), line 398 (represented by stars)
    - Suggestions for Clarity:
        - Akin to theorem 1, is it possible to present a simplified version of theorem 2 for the general audience? As it is, definition 2 and theorem 2 are hard to digest just on their own.
        - Line 481, before stating the result, can the authors explain in words the process of iterative mixing proposed in Ferbach et al., 2024? It would make the manuscript more self-contained.  
    - Missing Citation: Line 424, for the MNIST dataset, please include the citation for "Gradient Based Learning Applied to Document
 Recognition", LeCun et al, 1998 
    - Visualization: For fig.1, please consider changing the y-axes test error to the same scale. Right now, it is hard to compare the error values or the slope in subplot 1 to those in 2 and 3.

questions:
- Model Assumptions: Regarding the label shift assumption, what are the authors thoughts on how the results would change if the domain shift were to be accounted for as well? 
- Experimental Setup: 
    - I understand that it might not be computationally feasible for GPT-2 in the given timeframe, but could the authors provide an ablation on the MNIST experiment with regards to the quality of synthetic data? That is, models trained on labels taken from increasingly worse synthetic classifier and the corresponding trends. 
    - Minor: Regarding the GPT-2 experiment, if BabiStories is itself synthetic, why not use TinyStories as real data and BabiStories as the synthetic dataset for training the models, as opposed to creating a new GPT generator?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732048547577

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank all the reviewers for their overall positive reception of our work. We are also grateful for their thoughtful and helpful feedback.

In this rebuttal, we carefully address all the questions and comments of the reviewers. We have also run additional experiments to address specific requests from the reviewers, to help clarify some of our results.

Our rebuttal comes alongside an updated version of the manuscript, that includes new figures and text changes (in blue).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732992526699

thread_type:
public_comment

<thread_object>
creator: ~rylan_schaeffer2

<content>
title:
Scientific Claims Require Discussion & Citation of Intentionally-Omitted Contradictory Prior Work

comment:
**TLDR: The authors' decision to not cite our directly relevant published work - of which they are well aware through significant interactions - while using our methodology and reaching contradictory conclusions without explanation, warrants careful examination.**

“Strong Model Collapse” reaches conclusions that appear to be in tension with recent and relevant work, particularly [Gerstgrasser et al. 2024's "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data," published at COLM 2024](https://openreview.net/forum?id=5B2K4LRgmz#discussion). However, while the authors of “Strong Model Collapse” are well aware of “Is Model Collapse Inevitable?” and even use its methodology, the authors do not cite the work.

Note: I am a co-first author of “Is Model Collapse Inevitable?”

**Scientific Contradictions with Prior Work:**  The scientific claims of “Strong Model Collapse” merit further discussion in light of existing work. Specifically, “Strong Model Collapse” concludes that model collapse "generally persists even when mixing real and synthetic data, as long as the fraction of training data which is synthetic does not vanish” and that "model collapse cannot generally be mitigated by simple adjustments such as data weighting unless these strategies asymptotically remove all but a vanishing proportion of synthetic data from the training process.” However, our work and other existing literature (including other manuscripts by the authors of “Strong Model Collapse”) demonstrate that model collapse can be robustly avoided, even when the fraction of real data vanishes - an apparent contradiction that warrants careful examination.

**Uncited Reuse of Experimental Methodology:** The experimental setting of pretraining sequences of GPT-2 causal transformers on TinyStories was first used in “Is Model Collapse Inevitable” and is now re-used in “Strong Model Collapse” without citation. While perhaps conceptually straightforward, finding the correct experimental combination of models, data and optimization that faithfully emulates model collapse in the face of high costs from pretraining sequences of language models was non-trivial. “Is Model Collapse Inevitable” also contributed results in the linear regression setting, the same setting considered in Section 2 of “Strong Model Collapse,” that similarly go uncited.

**The Authors Acted Intentionally:** While missing a citation is in general an innocuous and easily corrected oversight, in this particular case, the authors of “Strong Model Collapse” had direct engagement with “Is Model Collapse Inevitable” beginning in April 2024, including (i) providing feedback that was incorporated into the April 29th 2024 Arxiv update and (ii) publicly tweeting about our manuscript over summer.

**Conclusion:** We respectfully ask the reviewers and the AC to ask the authors of “Strong Model Collapse” to address:

1. The relationship between this work and existing literature on avoiding model collapse, and

2. How the seemingly contradictory conclusions arise from different modeling assumptions, and which assumptions best reflect real-world scenarios.

Evaluating which assumptions are most relevant to real-world scenarios is essential for assessing the practical implications of this work.

Note: I have omitted additional relevant evidence to preserve anonymity of the authors. If the AC wishes to see said evidence, I can provide it.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733197473546

thread_type:
public_comment

<thread_object>
creator: ~rylan_schaeffer2

<content>
title:
Scientific Claims Require Discussion & Citation of Intentionally-Omitted Contradictory Prior Work (Part 2)

comment:
Dear Reviewers & Area Chairs,

I would like to again emphasize that, based on the response by the authors of Strong Model Collapse below, the authors are:

1. clearly aware of [prior published work](https://openreview.net/forum?id=5B2K4LRgmz#discussion) that directly contradicts their narrative and their scientific claims, while
2. using the methodology introduced by said prior work that they simultaneously insult and do not credit.

The authors' response demonstrates that the authors intentionally chose to not discuss these complexities, despite clearly having much to say on the topic. Rather, the authors are instead relying on a bad-faith interpretation of the ICLR reviewing guidelines to justify their intentional omission of a (reasonably) well-cited published work that demonstrates the fraction of real data can vanish without exhibiting "a critical degradation" of performance.

While the authors are well within their rights to challenge prior work, **suppressing contradictory evidence is antithetical to the scientific process and muddies the research community's understanding of the (potential) harms of model collapse.**

To make two additional comments: 

1. The ICLR guidelines are meant to help authors avoid being penalized for being unaware of contemporaneous work. The authors of Strong Model Collapse are well aware of our paper since April 2nd 2024, and are even credited for their feedback in its footnotes twice. The ICLR guidelines are not meant to excuse intentionally not citing work that the authors are familiar with.

2. To be clear, I am not claiming credit for TinyStories (and we appropriately cite it). I am claiming credit for demonstrating that GPT2+TinyStories is a viable and insightful setting for studying model collapse, which is a non-trivial extension and (to the best of my knowledge) was novel. None of the 10 citations the authors cite below contain the string "model collapse".
</content>
</thread_object>

</thread>
<thread>
creation date:
1744494345846

thread_type:
official_comment

<thread_object>
creator: program_chairs

<content>
comment:
This statement clarifies the review process for #4448 ("Strong Model Collapse").

The reviewers were unanimously positive about the paper.  The AC recommended reject on the basis of a public comment stating a missing citation to a COLM 2024 paper.  Per ICLR Reviewer Guidelines, the COLM 2024 paper is within the grace period of not requiring a citation (published after July 1st, 2024).  

The authors acknowledge knowing about an earlier version of the paper that was on arXiv, but chose not to cite that paper.  Per ICLR Reviewer Guidelines, authors are encouraged to cite papers that are not peer-reviewed, but it’s ultimately a judgement call (by the reviewers). 

In private discussions (on OpenReview but not viewable to the public), the reviewers ultimately decided that missing this citation cannot be the sole basis for rejection.  The AC overruled the reviewers and recommended reject.  The Programs Chairs received a complaint and asked an internal committee as well as the Senior AC to review the case.  The unanimous decision is to agree with the reviewers, thus accepting the paper. Any paper that achieves an average score above a threshold is automatically considered for spotlight.

To be absolutely clear, ICLR’s decision-making is completely within this scope.  ICLR is not making any conclusions about any allegations of plagiarism or other misconduct at other venues.  Any implications by any party suggesting otherwise is false.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper received unanimously positive reviews from double blind peer review. During the discussion period, a public comment was posted about the lack of citation and discussion for a related paper, first published at COLM 2024 in October 2024. While ICLR's policy exempts papers published after July 1, 2024 from citation requirements, the authors acknowledged being aware of an early conception and development of the COLM paper, but had decided not to cite it.

There was significant discussion among the authors, area chair, and reviewers for whether the authors are required to cite the paper. The program chairs consequently convened a panel of prior ICLR program chairs who weighed whether the COLM paper needed to be cited. The panel voted unanimously, given the evidence, that the authors are not required to cite the COLM paper. Since the main reason for rejection was the missing citation and otherwise received positive reviews, the paper is hence accepted.

The original meta-review for the paper is below.

-----------------

(I) The paper discusses the very relevant topic of model collapse in the presence of synthetic data generated by LLMs in the training datasets. The authors hypothesize that even a small fraction of synthetic data influences the performance of pre-trained LLMs. All reviewers agree on the quality of the manuscript in terms of being well-written and having practical findings. Furthermore, the reviewers appreciated the soundness of the theoretical results. I agree with the judgment of the reviewers.

(II) However, a comment by a community member indicated that the authors were aware of a closely related prior publication and did not cite it. Based on the author's own words (https://openreview.net/forum?id=et5l9qPUhm&noteId=k34IfU2Hvs), they deliberately decided to not cite the prior work :

"... we still don't think Gerstgrasser et al. (2024) is scientifically sound (for example, in terms of the technical contributions and most importantly, the misleading/inaccurate claims of solving/mitigating model collapse, as explained above), and therefore didn't feel the need to cite."

Based on established practices of good scientific conduct, closely related prior work should be cited and appropriately discussed. The AC concludes that no author should willingly refuse to cite a closely related prior work published at a peer-reviewed conference, even if they might disagree with the arguments, quality, or methodology of a prior work. Instead, the authors should cite the prior work and criticize it.

(III) During the discussions between the AC and the paper reviewers, an argument was raised that the ICLR policy permits not citing papers published after 1 July. In this concrete case, the committed prior work was published at COLM 2024 after 1 July.

 The applicable policy (https://iclr.cc/Conferences/2025/ReviewerGuide) states that:

"We consider papers contemporaneous if they are published within the last four months. That means, since our full paper deadline is October 1, if a paper was published (i.e., at a peer-reviewed venue) on or after July 1, 2024, authors are not required to compare their work to that paper. Authors are encouraged to cite and discuss all relevant papers, but they may be excused for not knowing about papers not published in peer-reviewed conference proceedings or journals, which include papers exclusively available on arXiv. Reviewers are encouraged to use their good judgment and, if in doubt, discuss with their area chair. “

However, the policy focuses on good faith ommissions of the prior work due to not knowing such papers:

“… Authors are encouraged to cite and discuss all relevant papers, but they may be excused for NOT KNOWING about papers not published in peer-reviewed  …, which includes papers exclusively available on arXiv.”.

However, the case of this submission is different, because the authors admitted to being aware of the work but deliberately decided to not cite it. As a result, I do not see this case as excusable under the ICLR regulation, as the latter implies a good faith omission of prior work.

(IV) As a result, I recommend rejecting the paper.