Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification

Authors: Yunzhen Feng, Elvis Dohmatob, Pu Yang, Francois Charton, Julia Kempe
Primary area: foundation or frontier models, including LLMs
Keywords: Learning with Synthetic Data, Data Curation, Avoiding Model Collapse

Abstract:
Large Language Models (LLM) are increasingly trained on data generated by other LLMs, either because generated text and images become part of the pre-training corpus, or because synthetized data is used as a replacement for expensive human-annotation. This raises concerns about *model collapse*, a drop in model performance when their training sets include generated data. Considering that it is easier for both humans and machines to tell between good and bad examples than to generate high-quality samples, we investigate the use of verification on synthesized data to prevent model collapse. We provide a theoretical characterization using Gaussian mixtures, linear classifiers, and linear verifiers to  derive conditions with measurable proxies to assess whether the verifier can effectively select synthesized data that leads to optimal performance. We experiment with two practical tasks -- computing matrix eigenvalues with transformers and news summarization with LLMs -- which both exhibit model collapse when trained on generated data, and show that verifiers, even imperfect ones, can indeed be harnessed to prevent model collapse and that our proposed proxy measure strongly correlates with performance.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730144547476

thread_type:
official_review

<thread_object>
creator: reviewer_rfsv

<content>
summary:
The authors study whether filtering synthetic data (collected from a generator) using verifiers can prevent model collapse. In a binary classification setting, they relate downstream model performance to two quantities: the generator's error $p$ and a quantity related to the verifier's accuracy $p_*$. When the generator's error surpasses the threshold related to the verifier's accuracy ($p > p_*$), model collapse occurs. They then conduct three empirical experiments (binary classification w/ linear models, eigenvalue prediction w/ transformers, and news summarization with Llama-2) in which they show that downstream performance correlates with the verifier performance $p_*$, which they can estimate using some labeled training data.

strengths:
This paper was well-written, flowing through a toy but well-executed theory section (Section 4), and then a set of three nice experiments (Sections 5-6). The grounding in actual experiments on transformers was great; in particular, the news summarization experiment for tasks without exact match notions of correctness improved the generality of this work. I appreciated that the appendices were very thorough, and I was able to find answers to most of my questions by looking through these. 

I'm not confident in my familiarity with related work, but I believe that the setting studied here (and the style of analysis, beginning with a toy model and ending with empirical work) is novel and helpful to the synthetic data community.

weaknesses:
*Novelty of results.* From a practitioner's point of view, the conclusion from these results is simple: when it is possible to evaluate a verifier on labeled data, picking a more accurate verifier gives better results on that same task. This seems to be obvious / empirically quite clear in existing work (e.g. Appendix A.1), and I'm not sure if the specific definition of $p_*$ offers gains over just picking by existing accuracy metrics.

questions:
- How might this generalize to language modeling / task-agnostic pretraining, where there is not necessarily a sense of "correctness" for verification?

- Lines 390-395 point out that when using a weak verifier, there's a dependence on $n'$, the amount of synthetic data to train on. This seems practically quite important to establish. Can the authors comment more on how to "take into account the quantity of data selected" (line 395)?

Misc suggestions for a final version:

- I would suggest moving Appendix C into the main text. If necessary for space, I would move Section 3 to the appendix, since these results are reasonably well-known from the test-time compute literature.

- https://arxiv.org/pdf/2209.03942 is a related reference

- Line 365 typo: "and the ground truth $w_*$. 5Having the verified"
- Line 12 typo: "other LLM[s]"
</content>
</thread_object>

</thread>
<thread>
creation date:
1730647651055

thread_type:
official_review

<thread_object>
creator: reviewer_rwot

<content>
summary:
The paper presents a verification method to prevent model collapse that happens when models are trained on synthetic data. The proposed proxy $p_{*}$ offers a practical way to estimate the potential performance of models trained on verified synthesized data. Based on it, the synthetic data can be selected to avoid degrading the performance of the model that is trained on mixed real and synthetic data. Tested across three different tasks, the method shows its effectiveness and practical applicability.

strengths:
1. The paper provides a theoretical framework for understanding the role of verification in preventing model collapse.
2. The paper is well-written and easy to follow. 
3. The method has been demonstrated effective via empirical experiments in different domains.

weaknesses:
1. Model collapse not only happens when trained on synthetic data but also occurs when synthetic data is involved in the training set iteratively. This paper fails to address model collapse from this dimension. 
2. As stated in the paper, Llama 3 performs worse than Llama 2 when as a verifier. Therefore, an advanced model is not necessary to be a good verifier. There could be more discussion on the reason behind it and how to choose verification models. 
3. This paper primarily focuses on accuracy for analysis, yet it would be insightful to consider cases where accuracy alone is insufficient for assessing model performance, such as alignment in large language models (LLMs), and explore how the synthetic data selection strategy could be applied in such complex scenarios.

questions:
Figure 5 shows when more selected synthetic data are trained on, both the performance of Oracle and $p_{*}$ become worse, why does this happen? Does it mean synthetic data can still potentially make the model collapse?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730647993128

thread_type:
official_review

<thread_object>
creator: reviewer_mxv4

<content>
summary:
This paper studies a phenomenon known as "model collapse," in which LLMs that are trained on synthetic (LLM-generated) data suffer drops in performance compared to LLMs trained on real data. The authors propose a verification technique for selecting synthetic data, and analyze cases in which this process can improve over collapsed LLMs trained on randomly selected synthetic data. The theoretical analysis is done using a simplified setting of a Gaussian mixture and linear classifiers, and the results are verified empirically in the same setting. Finally, the authors present empirical results using a pretraining setting for eigenvalue prediction and news summarization.

strengths:
This paper covers an important topic, and establishes some of the fundamentals for effectively training on synthetic data -- this is likely going to be an important area of study as we continue to scale foundation models. 
- The authors perform a theoretical analysis that characterizes how high-quality the synthetic data needs to be in order for it to be a useful stand-in for real data, i.e., in order to avoid model collapse. 
- The authors present practical results for fine-tuning Llama-2 on a news summarization task as well as a pretraining task in which the transformer is trained to predict matrix eigenvalues. For the news summarization task, the authors find that Llama-3 is a worse verifier than Llama-2, which is a peculiar finding. 
- For the real data experiments, the authors confirm that the model collapse phenomena actually occurs, and that their verification procedure actually improves over the model collapse baseline.

weaknesses:
- The theoretical analysis could be improved with some form of a finite-sample result. The current result only applies in the infinite sample limit in a simplified setting, although the result is validated empirically (using finitely-many samples, of course). 
- The writing in Section 3 can be improved substantially. The purpose of this section and its contribution to the overall narrative are currently unclear, which can be mostly fixed by leading with a crisp motivational statement. More generally, I'm still not sure that I understand the takeaways of this section -- why is it that the model "lacks the inherent capability to autonomously select the best predictions," when using beam search? Is this a statement about decoding, or about the model? Earlier, in the intro, the authors say that models "cannot intrinsically identify the best solution using perplexity alone," yet beam search *is* essentially based on maximizing the likelihood of outputs. 
- 4.1 in the theoretical analysis says that the $y'$ variables "[have] been generated by an AI model," but it's unclear what this statement means mathematically. This should be more formal, particularly since it's part of the theoretical analysis. 
- Assumption 4.1 introduces "bits" $q_1, ..., q_N$. I assume these represent whether or not a sample has been pruned, however this is not defined until after it's already mentioned, and it is up to the reader to infer what these bits represent. 
- The authors compare to "selection with weak supervision," but there is little elaboration about what this actually means. Weak supervision is an overloaded term and refers to many things -- one reading of this could be that the authors refer to programmatic weak supervision, which could make sense because programmatic weak supervision in some sense generates synthetic training data and verifies it via abstain votes.

questions:
There are a few minor writing inconsistencies in the introduction (and more generally, I feel that the writing of the intro could be improved): 
- "We validate our theoretical findings in three empirical settings" and "We conduct two large-scale experiments to test our theoretical insights :" are somewhat at odds. And if they aren't, then the writing for this part should be clarified. 
- Ranking is only mentioned once, at the very end of the intro. It is unclear how ranking fits into the verification process -- data selection does not necessitate ranking, although ranking is indeed a way to perform data selection. What is meant by this? 
- Is verification really all we need? Some form of "verification is all you need" is mentioned throughout the paper, but it seems like there is a long list of things that we need. 
- While I imagine that this would be computationally expensive, it would be useful for the authors to include a scaling law analysis under synthetic data, and for that matter, an example of pretraining on synthetic general data. The proposed technique indeed improves over the collapsed baseline, but it does not necessarily approach the performance of training on real data. There are many questions related to scale that I don't think this paper addresses, but I am not going to factor this into my evaluation, as this seems like obtaining an answer to this would be quite expensive. In any case, I would be interested to hear any speculative thoughts that the authors might have about this.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699484105

thread_type:
official_review

<thread_object>
creator: reviewer_cepm

<content>
summary:
The work investigates synthetic data in LLM training and concerns about model collapse (catastrophic drop in model performance) when LLMs are trained with model-generated labels. Specifically, the work investigates model collapse as a decision problem and utilizes a framework where a verifier (pruner in the manuscript) vets synthetic examples in an attempt to limit the risk of model collapse. Experiments have demonstrated that this verification-based approach can prevent collapse even when the verifiers are imperfect and are not oracles.

strengths:
- Paper is very well written and easy to understand
- Both synthetic data and model collapse are very important topics in the literature and are of great interest to the audience
- Theoretical derivation is complete and appears to be correct
- Very thorough quantitative experiments with strong results

weaknesses:
- In terms of LLMs, qualitative examples showing model collapse in some of the tasks may benefit the delivery further

questions:
- I am interested in whether the proposed methods can be applied to other modalities (e.g., VLMs, Vision-Language-Action Models in robotics) - many problems face more challenges in data annotation cost and also resort to synthetic data.
- Many LLMs are trained with synthetic data specifically on preferences (e.g., RLHF, DPO). I wonder if parallels can be drawn between the verifiers and a reward model and if model collapse is particularly severe with Vicuna-style finetuning.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732096546630

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response: How to do data verification and finite sample (Part 1)

comment:
We thank all the reviewers for their insightful and high-quality feedback. We are glad all reviewers appreciated our theoretical analysis on how verifiers, even imperfect ones, can improve synthetic data, as well as the insights derived from our work—specifically the proxy $p_*$— and found they are well-supported by a comprehensive set of experiments with increasing complexity. We are happy that many reviewers also praised our writing structure and found the observation that a better model is not necessarily a better verifier to be a peculiar finding.

In the following general response, we address some common questions raised in the reviews. A new version incorporating reviewer suggestions has been uploaded, with changes marked in blue.

1. **How can we effectively approach data verification for pretraining when accuracy is not available as a metric, alongside analyzing scaling for pretraining on synthetic general data?** Reviewer rwoT highlighted the challenge of '*cases where accuracy alone is insufficient for assessing model performance*'; Reviewer rFSV inquired about how the method '*generalizes to language modeling or task-agnostic pretraining, where there is not necessarily a sense of "correctness" for verification*'; and reviewer mXV4 raised the question of conducting '*a scaling law analysis under synthetic data, and for that matter, an example of pretraining on synthetic general data*'; Additionally, Reviewer CePm touched on a related issue, suggesting the extension of verification methods to other modalities.

These are excellent questions and remarks following up on our work. Indeed, the challenging aspect for general settings lies in effectively building and leveraging verifiers to select high-quality data. First of all, verifiers can encompass a combination of metrics and criteria tailored to specific domains, often extending beyond accuracy. Any correlated feedback signals can be used to guide data selection, including the concept of "programmatic weak supervision" mentioned by Reviewer mXV4. In the context of physical synthetic data, verifiers could include differential equations designed to model the setting; in other scenarios, existing rule-based knowledge can be leveraged. Individual verifiers for each domain can be combined into a mixture of verifiers. To generalize these methods, verifiers could be augmented with LLM acting as processors. Rules, prompts, or even search strategies (retrieval) could be integrated as in-context information, similar to approaches that enhance generated outputs [1, 2]. 

The most general form of verification could involve leveraging a pretrained general model itself. For instance, the training corpus for Llama-3 was preselected (verified) by Llama-2. This process employs a broader notion of 'correctness', evaluating whether each data point meets the quality threshold necessary for use as pretraining material. Pretrained models acquire some general rules for identifying high-quality data, which can be assessed through metrics like perplexity and other interpretability measures. However, optimizing the use of pretrained models and understanding the extent to which these metrics correlate with data quality remain open areas for further investigation. Recent advancements in enhancing LLMs to act as judges [3,4] underscore their potential for general and alignment tasks, even without explicit metrics. This approach aligns with the intuition presented in our paper that classification tasks are simpler than generation tasks. Furthermore, fine-tuning models specifically for judgment tasks could enhance their effectiveness and utility.

Recent advances in data selection help provide valuable insights. A common approach involves using a high-quality dataset to train a model that generalizes and identifies high-quality data within a larger set [5]. The selected subset can then be used to train an improved model, forming an iterative data engine that can incorporate external information at each stage. Several statistical methods have also been proposed [6, 7], which focus on aligning data with the general distribution in a well-represented feature space without relying on accuracy metrics. For synthetic data specifically, the controllability of the generation process may allow us to tailor these methods in tandem with data generation, resulting in a well-verified dataset.

With these methods, we could test whether the optimal combination achieves scaling performance surpassing that of real data. This would examine whether we could avoid model collapse in the scaling laws, as suggested in [8].
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper raises an interesting concern that the use of synthetic/LLM-generated data in training causes "model collapse" -- a drop in model performance. Specifically, the authors investigate model collapse as a decision problem with a framework where a verifier vets synthetic examples in an attempt to limit the risk of model collapse. They study whether simply filtering synthetic data (collected from a generator) using verifiers can prevent model collapse. The theoretical analysis is done using a simplified setting of a Gaussian mixture and linear classifiers, and the results are verified empirically in the same setting. Finally, the authors present empirical results using a pretraining setting for eigenvalue prediction and news summarization.

I side with the reviewers who all rated towards acceptance.