Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification
Authors: Yunzhen Feng
Elvis Dohmatob
Pu Yang
Francois Charton
Julia Kempe
Venue: ICLR 2025 Poster (2025)
Keywords: Learning with Synthetic Data
Data Curation
Avoiding Model Collapse
Primary area: foundation or frontier models, including LLMs
TL;DR: We theoretically and empirically demonstrate that leveraging verifications to select synthesized data can prevent model collapse.

Abstract: Large Language Models (LLM) are increasingly trained on data generated by other LLMs, either because generated text and images become part of the pre-training corpus, or because synthetized data is used as a replacement for expensive human-annotation. This raises concerns about *model collapse*, a drop in model performance when their training sets include generated data. Considering that it is easier for both humans and machines to tell between good and bad examples than to generate high-quality samples, we investigate the use of verification on synthesized data to prevent model collapse. We provide a theoretical characterization using Gaussian mixtures, linear classifiers, and linear verifiers to  derive conditions with measurable proxies to assess whether the verifier can effectively select synthesized data that leads to optimal performance. We experiment with two practical tasks -- computing matrix eigenvalues with transformers and news summarization with LLMs -- which both exhibit model collapse when trained on generated data, and show that verifiers, even imperfect ones, can indeed be harnessed to prevent model collapse and that our proposed proxy measure strongly correlates with performance.

Published as a conference paper at ICLR 2025
BEYOND MODEL COLLAPSE: SCALING UP WITH SYN- THESIZED DATA REQUIRES VERIFICATION
Yunzhen Feng 1.2. Elvis Dohmatob 13,4,+ Pu Yanga Francois Charton' Julia Kempe
Meta FAIR "New York University *Concordia University
"Mila *Peking University
"Equal Contribution. Part of the work was done during an internship at Meta.
Correspondence to y ff2311 e@nynnnneeee .dd
ABSTRACT
Large Language Models (LLM) are increasingly trained on data generated by other LLMs, either because generated text and images become part of the pre-training corpus, or because synthetized data is used as a replacement for expensive human- annotation. This raises concerns about model collapse, drop in model performance when their training sets include generated data. Considering that is easier for both humans and machines to tell between good and bad examples than to generate high-quality samples, we investigate the use of verification on synthesized data to prevent model collapse. We provide a theoretical characterization using Gaussian mixtures, linear classifiers, and linear verifiers to derive conditions with measurable proxies to assess whether the verifier can effectively select synthesized data that leads to optimal performance. We experiment with practical tasks computing matrix eigenvalues with transformers and news summarization with LLMs which both exhibit model collapse when trained on generated data, and show that verifiers, even imperfect ones, can indeed be harnessed to prevent model collapse and that our proposed proxy measure strongly correlates with performance.
INTRODUCTION
As generative models for language, images, and video continue to achieve human- level performance (Touvron et 2023; Achiam et al., 2023; Ramesh et al., 2021; Rombach et al., 2022; OpenAl, 2024), they are increasingly used to synthesize data across diverse domains, including coding (Haluptzok al., 2022) and mathematics (Trinh et al., 2024). With this rise in Al-generated data, there is growing interest in its potential to replace expensive human annotators.
This gradual replacement of human-writter corpora by machine-generated tokens gives rise to number of concerns, notably the risk of "model collapse" (Shumailov et al., 2023; 2024), where iterated training synthesized data brings drop in model performance, and, ultimately, "dumber models". The phenomenon was observed empirically in (Hataya et al., 2023; Martínez et al., 2023a;b; Bohacek & Farid, 2023; Briesch al., 2023; Guo al., 2023; Taori & Hashimoto, 2023) and theoretically analyzed in (Alemohammad et al., 2023; Bertrand et al.. 2023; Dohmatob et al., 2024c:a). In particular, Dohmatob et al. (2024c;a) show that model collapse is an effective change of scaling law: as data becomes more synthetic, larger training sets do not enhance performance. Current thinking around model collapse recommends that synthetic data to avoided for model training lest the system deteriorate completely.
This raises a critical question: are synthesized data so fundamentally flawed that they must be identified and discarded using detection techniques (Zhang et al., 2024b), corrected through various refinement methods (Gillman al., 2024) or used as negative feedback (Alemohammad al., 2024)? In this work, argue that synthesized data contain abundant valuable information that can be effectively leveraged. We advocate for a paradigm verifier-based selection of synthesized data and provide criteria for when this process is successful even with imperfect verifiers.
Throughout this paper, we focus on a scenario where synthesized data serve as the labels (answers) to questions common framework in tasks such question answering, code generation, and mathematical reasoning. To begin, we explore the information contained in synthesized data through controlled experiment: training transformers linear algebra task. This setting allows to

---

Published as a conference paper at ICLR 2025
Verteer
GranndTu
Grane Trus
Par harbrmanco
Model Veriiiation
Paze
Gese sttor
Generator
Sood
Whhul Melfeadio
(Model Collapee)
Number of Syrtheszed Dats
Orignal Date
Seleciec Systheeswd Data
(a) The generator is capable of producing high- quality data, and verifiers can extract this data through selection to prevent model collapse, which occurs in the absence of such selection.
(b) In theory, consider Gaussian mixture model with a linear generator and linear pruner, The pruner improves synthesized data through se- lection, resulting in improved performance.
Figure 1: Illustrative figures for our proposal (a) and for the theoretical and simulation settings (b).
both understand the solutions and easily synthesize data, placing it into a long established corpus of works that study interesting phenomena in a controlled setting to advance our understanding of the underlying mechanisms in larger models in the wild. For example, see (Power et al., 2022; Garg et al., 2022; Charton, 2024; Dohmatob et al., 2024c). In this context, we find that the accuracy of the most accurate synthesized solution among the top candidates is three times higher than that of the solution selected by the model itself based on perplexity - its own measure of quality. This finding provides positive answer to question: the model is indeed capable of generating high-quality solutions. However, it cannot intrinsically identify the best solution using perplexity alone. Thus, effectively scaling with synthesized data requires robust selection and verification process.
We proceed to provide a theoretical characterization using Gaussian mixtures and linear classifiers, employing an external verifier to select the generated data. In the limit of infinite synthesized data, analyze the conditions required for the generator and verifier to enable the model, trained on selected synthesized data, to achieve Bayes-optimal results. Notably, there is sharp phase transition from zero accuracy due to errors in the synthesized data to optimal accuracy. Our theory confirms that there abundant information within the synthesized data, and that proper, not necessarily perfect, selection can overcome model collapse. We also identify measurable proxy function that characterizes the ability of the verifier to select synthesized data that results in good models when retraining on it.
We further conduct large-scale experiments to test our theoretical insights: (1) the above mentioned transformers on a linear algebra task and (2) news summarization using the LLM Llama-2 (Touvron et al., 2023). In both experiments, we apply various verifiers for selection and compute our theoretical proxy function. Our results show that relying solely on generated data leads to poorer performance than that of the generator itself, even with increased data volume, indicating model collapse. However, even with noisy and imperfect selection methods, synthesized data can be improved to yield models that outperform the original generator. Our proxy function strongly correlates with final performance across all cases. In particular it elucidates the counter-intuitive fact that a stronger model is not demonstrate by showing inferior performance when selecting
automatically better selector with Llama-3 compared to self-selection with Llama-2 for -lame-2-generated text.
We summarize our contributions follows:
We provide a theoretical analysis in the high-dimensional limit with infinite data proving that synthesized data with appropriate selection lead optimal performance, and provide
characterization of necessary conditions via a measurable proxy function (Section 4),
We validate our theoretical findings in three empirical settings of increasing departure from our theoretical assumptions:
Simulation results with linear classifiers, tested in a finite-data regime (Section 5),
Linear algebra tasks with transformers trained in generative way (Section 6).
News summarization using Llama-2 (Section 6.3).
RELATED WORK
We here only list works of direct relevance to ours, with an extensive reference list in Appendix A.
Model Collapse. With the advancement of generative models, synthesized data generated by these models has become increasingly prevalent online, mixing irreversibly into our training corpora. Recent studies have highlighted the potential for dramatic deterioration in downstream models,
2

---

Published as a conference paper at ICLR 2025
phenomenon known as "model collapse" (Shumailov al., 2023). Empirical studies have demon- strated this issue in various settings (Hataya et al., 2023; Martínez et al.. 2023a;b; Bohacek & Farid, 2023; Briesch et al., 2023; Dohmatob et al., 2024b; Yang et al., 2025; Synthetic data can also amplify biases through feedback loops (Taori & Hashimoto, 2023). Syn- thesized datasets have been shown to reduce diversity (Padmakumar & He, 2024; Guo et al., 2023) and cause distributional distortions (LeBrun et al., 2021). Theoretical analysis also examines the effects of iterative training on self-generated data (Alemohammad et al., 2023; Bertrand et al., 2023; Dohmatob et al., 2024a; Seddik et al., 2024). Notably, Dohmatob et al. (2024c) warn that model collapse signifies a break in customary neural scaling laws (Kaplan et al., 2020; Hoffmann al., 2022), where increasing synthesized data volume does not enhance performance as effectively as scaling with human-geneen -enerrtde data. As a result, recent works have focused on avoiding or correcting synthetic data to prevent model collapse. Gillman et al. (2024) propose using a correction function informed by expert knowledge to modify the synthesized data. Alemohammad ct al. (2024) leverage model trained synthetic data as negative guidance for diffusion models. Zhang et al. (2024b) employ the confidence score and an Al detection classifier to discard synthesized data. In contrast, we propose leveraging the synthesized data through selection techniques.
et al., 2025).
Benefits of Synthesized Data. Synthetic data holds great potential, as it is much easier and cheaper to scale compared to human-labeled data. Numerous empirical studies have demonstrated the benefits of synthesized data across a wide range of settings. Common practices include cases where the downstream task slightly differs from that of the data-generating model, where the generating model is significantly stronger than the consuming one, or when better prompt engineering and external information are utilized. In Appendix A. we provide taxonomy that outlines when and how synthesized data can be advantageous. Data selection is already employed in some works, particularly in code generation and mathematics, where natural verifiers such as compilers, solutions, or heuristic verifiers exist. For instance, Haluptzok et al. (2022) generate synthesized code and filter out incorrect samples. Ulmer et al. (2024) use conversational metrics to filter synthetic dialogue data. Trinh et al. (2024) utilize symbolic deduction engine to verify correct solutions for Olympiad geometry problems. Setlur et al. (2024) apply a final answer verifier to distinguish between good and bad synthetic data. Although verifiers are used in these cases, their effects performance have not been systematically explored, especially in terms of how different types of verifiers influence outcomes.
Data Selection. Data selection is a preprocessing technique typically applied original datasets with human labels, as discussed in various related works (see Appendix A.3). In contrast, our work focuses on data selection for synthesized data, which does not represent noisy version of the ground truth but rather a skewed distribution. Our aim is not to propose new selection methods but to demonstrate how selection can prevent model collapse and analyze the conditions under which it effective.
WARMUP
We first focus on transformer models on a mathematical task, which offers an interpretable setting to understand the generation quality since we have clear metric for error and computable ground truth, as well as the ability to generate as much (original) data as desirable.
Setting. We follow Charton (2022), who showed that transformers (Vaswani et al., 2017) can learn to predict the eigenvalues of 5 real symmetric matrices with independent and identically distributed entries, rounded three significant digits. All training, test, and synthe- sized data are generated by sampling matrices with independent entries from 2/[-10, 10). A prediction is considered correct the relative error in the L norm is below a certain tolerance T. The synthesized data generator is trained on limited sample of 200,000 examples with Adam for 65 epochs. Details on the tokenizer and optimization can be found in Appendix B.
Table I: Generator accuracy for different beam sizes.
Left: the most accurate solution in the beam is evaluated. Right: the solution with lowest perplexity is evaluated.
Verify all beams
Verify the best beam
Tolcrance r
2%
1%
0.5%
2%
1%
0.5%
Beam 50
90.4
60.4
22.9
65.9
19.2
2.4
Beam 25
88.0
53.2
16.8
66.1
19.3
2.4
Beam 10
83.7
43.1
10.5
66.2
19.5
2.5
Beam 5 Greedy
79.3
34.9
7.1
66.5
19.7
2.4
66.9
20.2
2.4
66.9
20.2
2.4
We aim to evaluate the quality of the generator's outputs and determine how much useful information can be extracted from it. To achieve this, employ beam search, inference technique that
3

---

Published as a conference paper at ICLR 2025
maintains multiple candidate sequences (referred to "beams") during the generation process to explore a larger portion of the solution space. Beam search tracks the top l candidate sequences based on their cumulative probability scores, expanding each by considering all possible next tokens. When k = 1, beam search is equivalent to greedy decoding. The algorithm continues this expansion until the full generation process complete, allowing capabilities and better understand its limitations and strengths.
assess the upper bound of the generator's
In Table 1, we report the test accuracy of generated predictions using beam search. We consider two settings: evaluating only the best beam identified by the model (right) to understand the model's actual predictions, and another evaluating all & candidates in the beams, with the best one contributing to the accuracy (left) to assess the model's 's full potential.
For self-selection, increasing the number of beams does not lead to any improvement in accuracy, despite significantly higher inference costs, However, in the left setting, moving from greedy decoding (beam 1) to beam 50 improves accuracy from 20.2% to 60.4%, with r 1%. This indicates that the model has considerable potential to generate improved solutions, but it lacks the inherent capability autonomously select the best predictions. Among the vast amount of synthesized data, there may be high-quality data, but effective selection is necessary.
THEORETICAL INSIGHTS
In this section, we aim to theoretically explore training and verification using synthesized data, and characterize the conditions under which verification enhances performance. To streamline our analysis, we focus exclusively on training with synthesized data, since the inclusion of additional real data is always advantageous. We consider learning with family of high-dimensional data distributions. Within this context, the verification process is implemented applied to synthesized data. Crucially, we will not assume that the pruning strategy has access to the ground truth, as such an assumption would overly restrictive for practical applications; rather, we will formulate theory for general pruners, which could for instance be like another trained model. full exposition of our general theory is in Appendix E; for case of exposition, we specialize here to Gaussian mixtures.
a pruning strategy
4.1 SETTING
Data Distribution. We will consider distributions P over IR x {0, 1] with certain high dimensional concentration properties of a general form (Condition E.1). special case are binary Gaussian mixtures: features have conditional distribution given by y ~ N (Ny, E), where By (2y 1)H,
for some l E R&d and Σ is desinive-defiiite matrix with E ||>]|| = || tr E = 1. For further ease of exposition we will only consider balanced distributions:
P(y = 1) = Ply 0) 1/2, for (a, P.
Synthesized Data. Let D2 ((x1,)))... .))...... NN } } be a dataset of N i.i.d. pairs from the true distribution P and let D'y [11 V))...,) (NN, )r } be the synthesized data generated from the same distribution, but where label y/ (instead yı) has been generated by an Al model. Further, we shall denote by p E j0, 1), the probability that the label y/ of a synthesized example (æ, v/) is different from the true label yı, i.e
p P(v( # y.).
(I)
Note that does not depend on the example index i, due to the i.i.d. assumption. We make no further assumptions on the synthesize data.
Downstream Model and Pruning. We will model our data selection (whether with or without feedback) via pruning strategy q = (....... .9N) where 4 is a bit which indicates whether the ith training example from D'y has survived pruning. For the downstream models we consider the family:
F x. w) 9 ox w) E (0, 1),
(<) 1
parametrized by a vector of weights w E Rd and sigmoid non-linearity a. Let w be obtained via logistic regression fitted D'y with ridge regularization parameter À > 0. Thus, is the unique
4

---

Published as a conference paper at ICLR 2025
minimizer of the following objective function:
L(w)= = N Σ (l(((aa) + 2 1001?
where l the binary cross-entropy. The corresponding downstream classifier is IN = söx where the notation fur refers to the linear classifier induced by a weights vector E IR i. fm (sign(æ* T w) + 1)/2. Test Accuracy. The test accuracy of the downstream model fN is defined by aCc(SN) = (x) (aayes (z)), for random test point (z, v) P, where /Byyes == Elyl 2] is the Bayes-optimal classifier. particular, note that acc( 100% by construction. The quantity acc( (cc(fN) will be the main object of our analysis, and we will be interested in how it depends on errors the generator P and the choice of pruning strategy q, the infinite-sample limit N +
4.2 PRUNING STRATEGY
We consider a wide class of parametrized pruning strategies y, which we term Verification-Pruningy (see Appendix E). They satisfy the following reasonable property:
Assumption 4.1 (Independent Selection). The bits .1......9....................... 9N E {0, 1} are independent. Thus, in particular, whether any training example e; := (zs, vo) E D'y survives pruning or not is independent of what happens the other examples Ejp--
In its most general form, our verification perffication-pruning family (refer to Appendix E for details) is described by four parameters (do: 1 thoi, vioo), defined as follows:
Φ (g = y' = k k), dkke = P(q = 1 1 = E, y kt)
(3)
Assumption 4.1 implies that for any class labels k, l E {0. the random variables (Zas))ie[)) are i.i.d. with Bernoulli(pze) distribution, with P&* = (1 - p2 and Pyε = pike/ k * e, with E [0, 1) as in equation For simplicity of exposition, we will focus on symmetric pruning strategies where Φ1 on eb and 4/o1 = 1110 = 사b. Such pruning strategies are therefore described by a pair of parameters (Ø. v)
defined by 24kt- 11/ k, 3/2 l,
(4)
Let us provide two more notable examples of (symmetric) pruning strategies.
No Pruning. The case (Φ) = (1, 1) corresponds to no pruning, i.e using the entire training dataset.
Oracle Pruning. The case (db, 1/- 01. The pruning strategy only keeps indices corresponding to examples in the dataset which have correct label (all corrupted labels are discarded).
Supervised Pruning, We shall also consider supervised pruning strategies g defined by
= (x] Uprane) 0],
(5)
for some Wprune E Rdd This pruning strategy filters out all examples on which there is disagreement on the assigned label. Appendix E we show how we can map this to (ø, )))-runingg
4.3 PERFORMANCE OF MODELS TRAINED WITH PRUNING: INSIGHTS FROM INFINITE-SAMPLA REGIME
The following is our main theoretical result (see Theorem E.3 for full statement). It characterizes test accuracy ace(fN) the downstream model on pruned data as a function of p (the label disagreement) and the parameters (ó, v)) of the pruner, in the theoretical limit of infinite training data (N + xc). Theorem 4.2 (Simplified version of Theorem E.3). Let Assumption define the breakdown point p. E (0, 1) by p+ == 1/(1 + v//6). For the family of data distributions obeying Condition E.1 (including the Gaussian mixture), for downstream model IN trained on data from a generator with error rate p. pruned with an verification-type strategy with parameters (ò, v) in the limit N + co it holds a.s that: (i) pf < pa then aco(fN) = 100%. (ii) If > p- then ace( [N) = 0%. The pruner is overwhelmed by so many inaccuracies in the synthesized data, and the downstream model learns the exact opposite of the true class labels.
be order. Fix p. &. vb and
5

---

Published as a conference paper at ICLR 2025
1.0
0 IT
Thus, there is sharp phase- transiiionn around the corruption level p. = 1/(1 + 0/6): as p is in- creased past level P+. the downstream model fN
acc(fi))
0.35
0.5
0.4
abruptly switches from being perfectly accurate, to perfectly inaccurate! The proof (see Appendix E.7 for sketch) explicitly computes empirical test accu- racy in terms of Nie- := E'= Zike, which follow a binomial distribution, bounding the gap the popu- lation accuracy, and using concentration of measure type techniques. Note that the sharp transition due to the anfinit--sample regime, where we can avoid samtle corrections. p. could be used as a mea- surable proxy in the experiment.
0.45
0.5
0.4
0.6
0.8
1.0
Agen/t
Figure 2: Empirical confirmation of Theorem 4.2. Comparing the breakdown points of differ- ent generators and pruners of different strengths. Synthesized data is generated from linear model Wgen with classification error rate p = 9gen / [0, 1j. The data is pruned with another lincar model Wprrue which has classification error Oprune T. Broken lines correspond to the prediction of The- orem 4.2, while solid points correspond exper- iments. Notice sharp phase transitions where the model suddenly switches from perfect accuracy to worse worse-than- chancchance, as the theorem predicts.
See Figure (and Figure in Appendix E) for an empirical illustration of the theorem.
Remark 4.3. Note that the 100% accuracy achiev- able Theorem 4.2 is idealized, and is expected to only hold in the infinite sample regime (with possibly large but fixed input dimension).
4.4 SOME CONSEQUENCES OF THEOREM 4.2
We now present some illustrious applications of Theorem E.3. These examples are empirically confirmed in Figure 2 (see also Figure in Appendix E).
No Pruning. Here, we have / á = 1 and so the downstream model achieves 100% accuracy for all values of corruption parameter p up the breakdown point p. 1/2 predicted by Theorem E.3
Oracle Pruning. For this scenario, /// = and so Theorem E.3 predicts that the downstream model JN achieves 100% accuracy for all values of corruption parameter p up to the breakdown point P. = 1. This is perhaps not so surprising in hindsight. The point is that even for moderately large values of v/0, the breakdown point p. still be quite close to 1.
Supervised Pruning. Consider isotropic Gaussian mixture data with means = and a pruning strategy as in Eq. (5). The parameters (Ø, v) only depend on the angles 0 Ogens 0 Oranee 0 E [0, r]:
O Ogen <(Wgen: H),, Oyrane == (WW p),
(6)
0 = L(Wprune: Wgen).
This is because, the Pke's defined in equation 4 now correspond to orthant probabilities for a trivariate normal distribution, with correlation coefficients given by these angles (see also Figure 2). In the Appendix E.6, we provide the calculation from the angles to v and Φ. In practice we can directly measure the proxy P., which encapsulates all the aforementioned correlations.
Decoupling the Generator and Verifier. Although the generator and verifier are coupled together in supervised pruning, there are some intuitions that help us decouple them: (I) a better generator always improves performance, (2) when the verifier is poor, such in cases of no pruning random pruning, we have a low breakdown point and require a good generator to achieve good performance, and (3) when the verifier is sufficiently good, close to oracle, the breakdown point is high, and any non-degenerate generator is sufficient.
SIMULATIONS ON SYNTHESIZED DATA
The theoretical results are based on the best-case scenario having unlimited access to synthesized data and operating in the high-dimensional limit. In this framework, the impact of generator and verifier on performance reflected in binary outcomes: 100% or 0%. In this section, we present simulation results from finite regimes to illustrate the practical implications of our theoretical framework. Specifically, we explore how the generator and verifier influence performance, and how performance scales with increasing, yet finite, volume of synthesized data.

---

Published as a conference paper at ICLR 2025
Oracle Data
Oracle Selection
Weak Supervision Oprsse /024
Weak Opnun - 1/12
Random Selection
Training
Number Training Data: No
Number of Training
164
124
Reatio Dree
Reaatun Srree
to
to
Number of Synthesized Data n'
Number of Synthesized Data n
Number of Synthesized Data n
Strakegy
Strategy
Strakegy
Figure 3: Simulations with Gaussian mixtures. (Top row) Relative error (accuracy relative to optimal accuracy) scaling a function number of selected data, n', used to train the model. T 0.15, N1 10". The Bayes optimal classifier achieves approximately 94% accuracy this distribution. (Bottom row) p. values for all settings.
5.1 SETTING
Following the theoretical setting, we consider the same Gaussian mixture and linear models for the generator and selector, Let w. fixed unit vector in R The distribution Porig is
zly Nyyrw, Ia/d), for E [-1,+1].
Here, J is positive scalar that controls the overlap.
Synthesized Data Generation and Verification. We sample No data points from the distribution Pang form the original dataset, which is used train a linear model, d.,,, employing ordinary least squares. We then generate a large synthesized dataset using the trained model wiN, with sigmoid function, which undergoes selection by various verifiers into sets Deen, In our study, these verifiers are linear models parameterized by various W where Orrune denotes the angle between the pruner and the ground truth w.. Having the verified synthesized data, then randomly select n'' data from No to train and evaluate the final models.
5.2 LESSONS LEARNED
In Figure 3, conduct several simulations varying No. 0prune, and n'. A larger No indicates more effectively trained generator, utilizing a greater amount of original data. Oprune determines the oracle verifier. verifier's quality, with Oprane corresponding the of the ground truth We further explore scenarios where pprune 24 synthesized samples used train the final model. We plot the scaling curves n' approaches infinity, a scenario analyzed by our theory. The "random" line represents the process of randomly selecting n' data points from the synthesized dataset without any verification. The "clean" line indicates training of the model using n' data points from the original distribution. We make the following observations:
# and = The variable represents the number of
12-
Oracle Verification Matches Training with Oracle Labels. The oracle verifier achieves the best performance, matching training with clean data across all settings and attaining Bayes optimal accuracy. This validates the theory that synthesized data has the full information which can be extracted with verification.
The Effectiveness Weak Verifier Depends on n'. In practical scenarios, obtaining oracle-level verification is often challenging, weaker forms of verification are typically employed. While weak verifier generally leads to poorer performance, consistent with the decaying threshold points predicted by theory, there is surprising "sweet spot" at certain data size (for the curves with Oprane - Training around 10,000 data outperforms training with more data. Therefore, the choice of verifier should also take into account the quantity of data selected.
P. strongly correlated with the final performance. Across all settings, the relative values of p. consistently align with the models' performance. The simulations provide practical validation of the theoretical results and are consistent with the interpretation presented in Section 4.4. Oracle supervision yields optimal performance, and, in general, improving both the generator and verifier leads to enhanced results.
7

---

Published as a conference paper at ICLR 2025
Table 2: Implementation our three experiments: progressively explore our insights through three real-world experiments. First, we conduct simulations in a finite-data regime, where all settings align with theoretical expectations. Next, we examine transformers trained on generation tasks, evaluated using a 0-1 metric. Finally, we analyze large language models with general metrics.
Settings
Data
Task
Model
Verifiers
Metric
Simulation
Gaussian mixture
classification generation generation
linear model transformer
linear models (noisy) verifiers
classification accuracy accuracy w. tolerance
Arithmetic
synthesized data
Summarization
XLSUM dataset
Llama
Llamas
similarity w. Rouge-1
6 EXPERIMENTS
In both the theoretical results and the simulations, examine classification scenario where oracle supervision can select "100% correct" synthesized data. However, outside of these controlled settings, synthesized data may only approximate the ground truth, meaning the selected data might be closer to the truth rather than exactly correct. In this section, we explore verification for more general tasks using generative models through two experiments: (1) training transformer to predict the eigenvalues of a matrix, and (2) fine-tuning Llama-2-7B on a news summarization task. Table provides an overview of how gradually relax the setting from theoretical empirical. For the purposes of our discussion here, we refer to model collapse whenever the performance of the model trained on synthetic data is worse than that of the original generator.
6.1 TRANSFORMER FOR ARITHMETIC TASKS
Recall the setting described in Section 3, where a transformer is trained in a generative manner using 200K samples of matrices and their corresponding eigenvalues. This setting allows for the synthesis of unlimited data by generating random matrices and using the generator to predict their eigenvalues.
We first introduce the oracle verifier that measures the rel- ative L' distance between the model's predictions and the correct solutions. We use greedy decoding to generate a prediction for each matrix, and only matrices with predic- tions within a tolerance of T 1% (as determined by the oracle verifier) are retained. Additionally, we introduce set of noisy verifiers, where data with predicted solutions exceeding the 1% tolerance are still included with a prob- ability of Puoise Since we are evaluating accuracy in a binary (0/1) manner for each data point, we can directly compute &b and á in Equation 3, with y' = replaced by being within 1% relative L tolerance. The noisy verifier corresponds to setting ø = and vi = Pnoise. Conse- quently, according our theory, the proxy given by p. = 1/ + Pnoise). For perfect oracle verifier, Pnoise = O and p. 1; for random selection without verification, 0.5.
Generator
Accuracy with 1% telerance
data size: 1M 50
data size: 10M
45
40-
30
23
20
0.5
0.6
Figure Transformers computing eigen- values. Correlation between accuracy with 1% tolerance, p.. and the number of syn- thesized data. Model collapse is observed without verification (p. = 0.5), while higher values of p. result in improved performance. Results are averaged over 5 seeds.
Pnoise = 1 and
We generate various verified synthesized datasets, with data sizes ranging from million 5 million, and up to 10 million samples. Using these datasets, the transformer is trained from scratch and evaluated with greedy decoding. We report how different verifiers, specifically different proxy values Ps, and varying dataset sizes affect the final performance in Figure 4. We observe the following:
Model Collapse is Observed. When using synthesized data without any verification, at p. = 0.5. even with significantly larger dataset - 10 million samples, or 50 times the size of the generator's original training set - the trained model performs worse than the generator itself (represented by the dashed line). This outcome indicates the occurrence of model collapse.
Ps as a Reliable Proxy for Final Performance. When leveraging verification, we observe a consistent increase in performance. With the oracle verifier (p. 1), the accuracy nearly doubles. This observation aligns with practices in code generation and mathematics (Haluptzok et al., 2022;
This is equivalent to introducing random noise in the selection of both correct and incorrect data.
8

---

Published as a conference paper at ICLR 2025
Trinh al., 2024), where natural verifier, such precomputed solutions, unit tests, compiler, available. Across all dataset sizes, P. shows a strong correlation with the final performance. Furthermore, using 10 million samples approximates the effect of having infinite synthetic data; the turning point on the green curve, which surpasses the generator's performance, occurs around 0.65 in Figure 4. This value is close to the generator's error rate (i.e., minus the accuracy, represented by the dashed line) and reflects the phase transition predicted by our theory.
6.2 GENERALIZATION OF THE PROXY
In the theory and the task of predicting eigenvalues, the evaluation is two-way classification problem. For synthetic y' and ground truth y. the indicator which measures the similarity, is 0/1 variable. We can rewrite p == 1 - E[s] and define @ = Pg problems, we may instead have distance measure between y and y We extend the definition of p. under the assumption that s is normalized to lie in [0, 1]. and redefine & and d/ as follows:
s = 1). s = Pi = 1 = 0). For general
E.ggs/(L1 pp), J E. s)1/p.
When s takes values in {0, 1}. these definitions are consistent with those in Equation (3). With this extension, now move consider general problems in NLP.
6.3 LLMS FOR NEWS SUMMARIZATION
We now turn to one of the most standard tasks in NLP: news summarization. We utilize the English summarization subset of the XLSUM dataset (Hasan et al., 2021). the largest publicly available summarization dataset, consisting of 307,000 training samples and 11,500 test samples. Each sample in this dataset contains a news article x paired with a professionally annotated summary y. Unlike in the previous cases, we no longer have infinite (x. y) pairs due to the finite number of news articles available. Therefore, we fine-tune Llama-2-77 model (Touvron et al., 2023) on only 12.5% of the training set to serve as the generator. The synthesized data (news summaries) generated using the articles of the entire training set with greedy decoding. This approach reflects real-world conditions where synthetic data generation can significantly outpace human annotation. this setting, we use the Rouge-1 score (Lin, 2004) to evaluate the generated summary y'. Rouge- 1 assesses the quality of the generated summary by measuring the overlap of individual words between the generated and human-writted summaries y.
We consider three selection strategies: (1) Selection with Oracle: We calculate the Rouge score between the generated summary and the ground truth summary, keeping the data with the highest scores; (2) Selection with Llama-3: We leverage a fine-tuned I.lama3 model with higher Rouge score than the generator and keep the data with the lowest perplexity; (3) Self-Selection: We the generator to keep the data with the lowest perplexity. We apply three selection rates: 12.5% (when the selected synthesized data is the same size as the original data used train the generator), 25%, and 50%. For each combination, we collect the selected synthesized data to finetune the Llama-2 model. We present scaling law curves that illustrate how the Rouge- score improves with increasing amount of selected synthesized data used for training. Throughout the experiments, all finetuning was performed with full parameter training. Details are provided in Appendix D.
The results and p. are shown in Figure 5. We observe the following:
Model Collapse Observed. The Random Selection curve represent training with synthesized data directly without verification. In Figure 5 Left, using the same amount of synthesized data results in worse performance compared to using the original data, indicating model collapse (comparing "Random Selection' with 'Generator).. Only with more data, the Random Selection lines improve and nearly match the performance of the generator.
Selection by Oracle Performs Best. The dataset curated with oracle selection surpasses the perfor- mance of the generator in all settings. Oracle selection with 12.5% of the data selected even surpasses the model trained with 100% of the training set and original labels. The oracle verifier consistently has the best p. compared with the other methods.
p. Strongly Correlates with Performance. Surprisingly, self-selection leads to better performance than the generator, while Llama-3 verification results in performance similar to random selection. These findings seem counterintuitive, given that Llama-3 actually achieves better Rouge-1 score.

---

Published as a conference paper at ICLR 2025
Oracle Selection
Uarme-3 Selection
Self Selection
Random Selection
Gemeratoo etorr
Full Origin
Select 12.5%
Select 25.0%
Select 50.0%
0.34
0.34
0.24
0.80
0:11
0.31
ROUCE-1
ROUGE-1
ROUGE-1
0.30
0.30
0.35
1.20
6.29
0.2
1Pt
20'
Number of Synthesized Data
Number of Synthesized Data
Number of Synthesized Data
06
0.
Oracle
Llame-3
Self
Random
Oracle
Llame-J
Self
Random
Oracle Llama-3
Self
Random
Figure 5: News summarization with LLMs. (Top Row) The three figures from left to right represents models
trained 12.5%. 25%, 50% selected data individually. Each figure includes four curves illustrating different training scenarios: (1) selection with oracle, (2) selection with Llama-3 as a weak supervision, (3) self-selection, and (4) random selection. Additionally, two horizontal lines are included for comparison: one representing the generator model and the other representing model trained with 100% data with original labels. (Bottom row) Computed values p. for the verifiers for corresponding proportions of selected data. Training with data selected by a verifier with higher p. achieves better performance.
However, they consistent with the measured p.. Intuitively, self-selection tends to favor easy- to-learn samples, resulting in good performance with fewer data points. Although Llama-3 can generate better summaries, the synthesized data are produced by Llama-2, which is less correlated with Llama-3. As discussed in Equation 6, p. depends on three correlations and Llama-3 is not necessarily a better verifier in this context. This comparison highlights the challenges of selecting an appropriate verifier when fixed oracle, like those used in code or math tasks, is not available. In such cases, our proposed p. can serve as a valuable proxy before any training is conducted. These observations resonate with the recent work (Fang al., 2024) where it was also empirically observed that more accurate models might not always lead to better data filters.
7 DISCUSSION AND LIMITATIONS
In this paper, we consider a novel problem related to synthesized data: how to prevent model collapse through data selection. We propose to leverage a verifier to improve the synthesized data. We emphasize it is crucial to focus not only on the quality of the generator but also on having a high-quality verifier to select the data. We theoretically show that verification is all you need for synthesized data and identify a proxy function for performance after data selection. Through three solid experiments, we demonstrate that decent selector can prevent model collapse and our proxy function is reliable measure. Our work is of significant theoretical and practical importance in the era of large models with increasing use of generated data.
Can a model improve itself? In our paper, we present results showing that a math transformer using beam search does not improve test accuracy, while Llama-2, through self-selection of generated data, can yield model that performs better than the original generator. In the first experiment, the model selects a better y (output) for each x (input) (label selection). However, at the distribution level, the selected y not better. In the second experiment, the model selects better (æ, pairs from a larger pool of news articles (z) than available in the generator's training set. This introduces new information through the and results in shift in the distribution of æ. Due to this distribution shift, the trained model can outperform the original generator even with self-selection.
One limitation of this study is that only considered data selection Besides data selection, data curation also includes methods such as data augmentation, data regenera- tion, and weighting. The exploration of general data curation methods avoid model collapse is left for future work.
means of data curation.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
YF and JK acknowledge support through NSF NRT training grant award 1922658. Part of this work was done while JK and YF were hosted by the Centre Sciences de Donnees (CSD) at the École Normale Supérieure (ENS) in 2023/24, and JK and YF would like thank CSD and ENS for their hospitality. YF and PY would like to thank Yanzhu Guo, Di He, Zhenyu He for discussions and suggestions. This work supported in part through the NYU IT High Performance Computing resources, services, and staff expertise.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.08874, 2023.
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. arXiv preprint 47X25:2402.16627 2024.
Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G. Baraniuk. Self-consuming generative models go mad. arXiv preprint arxiv:2307.01850 2023,
Sina Alemohammad, Ahmed Imtiaz Humayun, Shruti Agarwal, John Collomosse, and Richard Baraniuk. Self-improving diffusion models with syntheric data, 2024. URL http:://t//// //axiv.. org/abs/ 2408. 16333.
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self- distillation in deep learning. The Eleventh International Conference on Learning Representations, 2022.
Reyhane Askari-Hemmat, Mohammad Pezeshki, Elvis Dohmatob, Florian Bordes, Pietro Astolfi, Melissa Hall, Jakob Verbeek, Michal Drozdzal, and Adriana Romero-Soriano. Improving the scaling laws of synthetic data with deliberate practice. arXiv preprint arXiv:2502.15588, 2025.
Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. Synthetic data from diffusion models improves imagenet classification. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
Quentin Bertrand, Avishek Joey Bose, Alexandre Duplessis, Marco Jiralerspong, and Gauthier Gidel. On the stability of iterative retraining of generative models on their own data, arXiv preprint arxiv:2310.0042 2023.
Matyas Bohacek and Hany Farid. Nepotistically trained generative-ai models collapse, 2023.
Martin Briesch, Dominik Sohania, and Franz Rothlauf. Large language models suffer from their own output: An analysis of the self-consuming training loop, 2023.
Max F Burg, Florian Wenzel, Dominik Zietlow, Max Horn, Osama Makansi, Francesco Locatello, and Chris Russell. Image retrieval outperforms diffusion models on data augmentation. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
François Charton. Linear algebra with transformers. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL / ew, hwt.:/ettooooaee forum? f 7FXX.
François Charton. Learning the greatest common divisor: explaining transformer predictions. In
The Twelfth International Conference on Learning Representations, 2024. URL https://
openreview. seeenreview.nee/forrre wwt./fouumt/ d=cmcDG5NPK..
11

---

Published as a conference paper at ICLR 2025
Rudrajit Das and Sujay Sanghavi. Understanding self-distillation in the presence of label noise. In International Conference on Machine Learning. pp. 7102-7140. PMLR, 2023.
Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Model collapse demystified: The case of regression. Advances in Neural Information Processing Systems, 37:46979 47013, 2024a.
Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, and Julia Kempe. Strong model collapse. arXiv preprint ar X1x::2400.00840, 2024b.
Elvis Dohmatob, Yunzhen Feng, Pu Yang. François Charton, and Julia Kempe. tale of tails: Model collapse as a change of scaling laws. In Forty-first International Conference on Machine Learning. 2024c. URL https::tp/..... openrevieweeeeeeeeeeeeeeee net/ www.oooiddoowwaaro.o.ooooooo 7shW.
Bin Dong, Jikai Hou, Yiping Lu, and Zhihua Zhang. Distillation ~~ early stopping? harvesting dark knowledge utilizing anisotropic information retrieval for overparameterized neural network. arXiv preprint #rXiv:1910.01255 2019.
Lisa Dunlap, Alyssa Umino, Han Zhang, Jiezhi Yang. Joseph E Gonzalez, and Trevor Darrell. Diversify your vision datasets with automatic diffusion- based augmentation. Advances in Neural Information Processing Systems, 36, 2023.
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07755 2023.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T Toshev, and Vaishaal Shankar. Data filtering networks. In The Twelfth International Conference on Learning Represen- tations, 2024.
Yunzhen Feng, Ariel Kwiatkowski, Kunhao Zheng, Julia Kempe, and Yaqi Duan. Pilaf: Optimal human preference sampling for reward modeling. arXiv preprint arXiv:2502.04270, 2025.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International conference on machine learning, pp. 1607-1616. PMLR, 2018
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
Nate Gillman, Michael Freeman, Daksh Aggarwal, HSU Chia-Hong, Calvin Luo, Yonglong Tian, and Chen Sun. Self-correcting self-consuming loops for generative model training. In Forty-first International Conference Machine Learning, 2024.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11 11644, 2023.
Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, and Chloé Clavel. The curious decline of linguistic diversity: Training language models synthetic text, 2023.
Patrick Haluptzok, Matthew Bowers, and Adam Tauman Kalai. Language models can teach them- selves to program better. In The Eleventh International Conference on Learning Representations, 2022.
Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguis- tics: ACL-IJCNLP 2021, pp. 4693-4703, Online, August 2021. Association for Computational Linguistics. URL ht https:/ /aa//aaaanntt aclanthology org/ 2021.0021222 findings- acl. 413.
Ryuichiro Hataya, Han Bao, and Hiromi Arai. Will large-scale generative models corrupt future datasets? In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 20555-20565, October 2023.
12

---

Published as a conference paper at ICLR 2025
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and XI- AOJUAN QL IS SYNTHETIC DATA FROM GENERATIVE MODELS READY FOR IMAGE RECOGNITION? In The Eleventh International Conference Learning Representations, 2023.
URL https::/p//... / //oppneeeeeeeeeeeeeeeeeeeeeeeeee net / forum? id=nUmCc7.5KFF
Reyhane Askari Hemmat, Mohammad Pezeshki, Florian Bordes, Michal Drozdzal, and Adriana Romero-Soriano. Teedback-guide data synthesis for imbalanced classification. arXiv preprini arXiv:2310.00158, 2023.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.025: 0253.,251,.22 2015.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoe, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022.
Sham M Kakade, Karthik Sridharan, and Ambuj Tewari. On the complexity of linear prediction: Risk bounds, margin bounds, and regularization. Advances in neural information processing systems, 21, 2008.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.088 003611, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6900 69800, 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Germain Kolossov, Andrea Montanari, and Pulkit Tandon. Towards statistical theory of data selec- tion under weak supervision. In The Twelfth International Conference on Learning Representations, 2024. URL https: / http///ppp/eeeeeeeeeee/eeeeeeeee net / forum? i d- CNg@n 6p.
Benjamin LeBrun, Alessandro Sordoni, and Timothy J J Donnell.. Evaluating distributional distortion in neural language modeling. In International Conference Learning Representations, 2021.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinay Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.002677 2023.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International conference on machine learning. pp. 12888-12908 PMLR, 2022.
Chin-Yer Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Marc Juarez, and Rik Sarkar. Combining generative artificial intelligence (ai) and the internet: Heading towards evolution or degradation? arXiv preprint arxiv: 2303.0/255, 2023a.
Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Marc Juarez, and Rik Sarkar. Towards understanding the interplay of generative artificial intelligence and the internet. arXiv preprint arxiv: 2306.00 06000 2023b.
David McAllester. Simplified pac-bayesian margin bounds. In Learning Theory and Kernel Machines. Springer Berlin Heidelberg, 2003.
Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 33:3351-3361, 2020.
13

---

Published as a conference paper at ICLR 2025
OpenAL. Video generation models world simulators. https::/epenai.ccm// viceo--eneeatioo--ooooo -- -0t--s00011--mmmmaaoar..aa/a//aa 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Vishakh Padmakumar and He He. Does writing with language models reduce content diversity? In International Conference on Learning Representations (ICLR), 2024.
Baolin Peng. Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, arXiv preprini arXiv:2304.03277, 2023.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arX11:2201.02117. 2022.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 8821-8831. PMLR, 18 24 Jul 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Mohamed El Amine Seddik, Suei-Wen Chen, Soufiane Hayou, Pierre Youssef, and Merouane Debbah. How bad is training on synthetic data? a statistical analysis of language model collapse. arXiv preprint arXiv:24040059% 2024.
Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. RI on incorrect synthetic data scales the efficiency of Ilm math reasoning by eight-fold. arXiv preprint arXiv:2406. 145.3222...553,, 14532, 2024.
Shai Shaley-Shwartz and Shai Ben-David. Understanding Machine Learning From Theory to Algorithms. Cambridge University Press, 2014.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Ander- son. The curse of recursion: Training generated data makes models forget. arXiv preprint arxiv:2305. /7493, 2023.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. Ai models collapse when trained on recursively generated data. Nature, 631, 2024.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 15:19523-19536, 2022.
Rohan Taori and Tatsunori Hashimoto. Data feedback loops: Model-driven amplification of dataset biases. In International Conference on Machine Learning, pp. 33883-33920, PMLR, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.099 09880998888... 2023.
Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476 482, 2024.
Dennis Ulmer, Elman Mansimov, Kaixiang Lin, Justin Sun, Xibin Gao, and Yi Zhang. Bootstrapping Ilm-based task-oriented dialogue agents via self-talk. arXiv preprint 24XI::2401.000000 2024.
Soobin Um. Suhyeon Lee, and Jong Chul Ye. Don't play favorites: Minority guidance for diffusion models. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttpss///peeeeeeeeeeee net forum? di (d
14

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances in Neural Information Processing Systems, pp. 6000-6010, 2017.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484-135000 2023.
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need. arXiv preprini urXiv:23/2.02/11 2023.
Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. RLCD: Reinforcement learning from contrastive distillation for LM alignment. In The Twelfth International Confer- ence on Learning Representations, 2024. URL httpss////ppeeeeee.rrr... net wttp://wwwee/foummiicicc forum?ide v3xXtxwKi6.
Pu Yang, Yunzhen Feng, Ziyuan Chen, Yuhang Wu, and Zhuoyuan Li. Spend wisely: Maximizing post-training gains in iterative synthetic data boostrapping. arXiv preprint arXiv:2501.1890 2025.
Weizhe Yuan, Richard Yuanzhe Pang. Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXi::2401.10020 2024.
Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets L.M finetuning: The effect of data, model and finetuming method. In The Twelfth International Conference on Learning Representations, 2024a. URL https: opwnrevieweeeeeeeeeeeeeee net forum? iidiiddCHHN SHCnKDeTws.
Jinghui Zhang, Dandan Qiao, Mochen Yang, and Qiang Wei. Regurgitative training: The value of real data in training large language models. arXiv preprini 2.X.::2407.12835, 2024b.
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreter: Integrating code generation with execution and refinement, arXiv preprint 20t.:2202..464555 2024.
15

---

Published as a conference paper at ICLR 2025
A MORE WORKS ON SYNTHESIZED DATA
A.1 TAXONOMY FOR SYNTHESIZED DATA
Contrary to the phenomenon of model collapse, synthesized data has been shown to improve per- formance in numerous empirical studies. We now provide a taxonomy outlining when and how synthesized data is beneficial. Specifically, we identify four key components: prompt engineering knowledge from advanced models distributional shift and targeted curation E. and external verifiers Most empirical studies can be categorized based on one or more of these components. We use and to denote the components each reference leverages.
Code and Math. Haluptzok et al. (2022) generate synthesized data for codes and use verifier to filter and show that the model can "self-improve" with its own synthesized data. Gunasekar al. (2023) filter high-quality data from the web and prompt GPT-3.5 with a specially curated prompt set covering both quality and diversity. Wei et al. (2023) leverage a diverse and large set of open-source code snippets to curate code instructions as prompts with good coverage and high quality. Zheng et al. (2024); Trinh et al. (2024) leverage symbolic deduction engine as verifier to test the correctness of each branch for solving Olympic geometry problems.
Alignment. During standard fine-tunings, synthesized data is often generated by stronger model like GPT-4 Peng et al. (2023)
Wang et al. (2023) use a good set of prompts and inputs with heuristic verifier to filter out low-quality ones and maintain high diversity. Bai et al. (2022)
use the model itself to critique whether its own generation is harmful, given already harmful prompts with gold standards from humans. For alignment with reinforcement learning, (Ouyang et al., 2022)
use humans as verifiers to compare synthesized data generated by the current model with good set of prompts. Some papers propose reinforcement learning with Al feedback (RLAIF) that leverages another LLM as the verifier to match human verification. The verifier is stronger model, instruct-tuned Palm2 L, while the network being trained is the Palm2 XS. However, ( (Yang et al., 2024) later found that using better prompts harmful or harmless responses can surpass RLAIF. Yuan et (2024) accieve surprising results with iterative fine-tuning and generating good prompts with in-context learning. Recently, (Feng et al., 2025) discuss how to optimally sample synthetic data for alignment.
(Lee et al., 2023)
that direct
Knowledge distillation. Most papers in the knowledge distillation area involve using a better model to distill for general performance or specific tasks, with and involved from case to case. One where GPT-4 is prompted to generate stories
example is the tiny story cases (Eldan & Li, 2023) for four-year-olds that are used to train GPT-Neo with good performance.
Image Domain. (Kirillov et al., 2023) and (Li et al., 2022) luse a distributional shift from high- quality to low-quality data to label and curate a vast amount of unlabeled data. Specifically, (Li et al., 2022) also trains verifier to filters high-quality data. Um et al. (2024) A specifically curate minority groups with a diffusion model to enhance performance. He et al. (2023); Dunlap et al. (2023) generate synthesized data that aids in classification tasks by tailoring the synthesized employ guided curation (with supervision) to curate data from diffusion models. Burg al. (2023) find that while synthesized data from a diffusion model helps improve downstream tasks, such as classification, using the pre-training data of the diffusion model alone gives even stronger performance.
data to match the model's learning objectives. Azizi et al. (2023); Hemmat et al. (2023)
A.2 KNOWLEDGE DISTILLATION WITH SOFT LABELS
Related to synthesized data, there is a long history of using synthesized labels in image classifica- tions. In the domains of self-distillation and knowledge distillation (Hinton et al., 2015; Furlanello et al., 201 8), data with soft labels generated from the teacher model can significantly improve the performance of the student model. These soft labels convey additional insights-referree to as 'dark knowledge' thht have been theoretically linked to specific advantageous adaptations. These include implicit biases that mitigate overfitting (Mobahi et al., 2020), mimicry of early stopping (Dong et al., 2019) for improved optimization under label noise (Das & Sanghavi, 2023), and adjustments to accommodate specific data structures (Allen-Zhh & Li, 2022). We only consider synthesized data with fixed labels in the current practice of L.MSs and diffusion models.
16

---

Published as a conference paper at ICLR 2025
A.3 DATA SELECTION
Comprehensive surveys on data selection for language models can be found in Albalak et al. (2024), along with theoretical studies on selection in high-dimensional settings (Sorscher al., 2022; Kolossov et al., 2024). Specifically, Kolossov et al. (2024) also explore the use of surrogate models for producing labels during selection, followed by curation of the original labels. In our study, selection is applied synthesized data where original labels are not available, resulting in distinct phenomena compared to these approaches on original data.
B PREDICTING THE EIGENVALUES
We leverage the code base provided by Charton (2022) facebookresearch/ under the license CC BY-NC 4.0.
https://github.co
Input and Tokenization. Transformers trained to predict the eigenvalues of 5 x 5 symmetric real matrices. Model inputs are sequences of 25 real entries, rounded to three significant digits, and tokenized as triplets of signs( + or -), mantissas (from 0 to 999) and power of ten exponents (from E-100 to E100). For instance, the 2 x 2 matrix,
2.3 0.6035
0.6035
-3.141
will be encoded as the sequence of 12 tokens: + 23 E-1 + 604 E-3 604 E-3 - 314 E-2. Model outputs are vectors of real eigenvalues, rounded to three significant digits, and tokenized as before, triplets of sign, mantissa and exponent (the P1000 encoding from Charton (2022)).
Model and Optimization. We train sequence-to -oweencceet--suunnneeee transformers (Vaswani et al., 2017), with 4 layers in the encoder, and one in the decoder, 512 dimensions and attention heads, to minimize a cross-entropy loss, using the Adam optimizer (Kingma & Ba, 2014) 4), with fixed learning rate of 5010, after an initial linear warm-up phase over the first 10,000 optimization steps. The model trained for 400 epochs before overfitting.
Evalution. Model accuracies are measured on a held-out test set of examples not seen at train- ing. Model predictions are evaluated by decoding the output sequence numbers (P1, P2: P3, P1. Ps), and assessing that prediction p(P1: P2, P3. P1. Ps) of eigenvalues v(U1, U2 U3, V4, U5) is correct if the relative error in L norms is below some tolerance т, L.e. if
vector of 5 real
U3 Pal <
vel.
i=1
We tolerances 7 of 5, 2, and 0.5%.
B.1 FINETUNING MODELS WITH SYNTHESIZED DATA
In all previous experiments, the data generated from the generator (using beam or reject sampling) were used to train new model. In this section, we consider using data generated from the generator to finetune models pre-trained on a small sample of ground truth data. consider four cases:
Fine-tuning the generator (Model A).
Fine-tuning model pre-trained to the same accuracy as the generator (62%, Model B).
Fine-tuning model pre-trained to higher accuracy (93%, Model C).
Fine-tuning model pre-trained to low accuracy (4%. Model D).
Table compares accuracy of the four fine-tuning cases to that of a model trained from scratch. Fine-tuning only achiueves better performance when the pre-teained model achieved higher accuracy than model A. In all other cases, fine-tuning brings no improvement. Note that fine-tuning model on its own generated data achieves the worst result, clear case of model collapse.
17

---

Published as a conference paper at ICLR 2025
Model A (66%)
Model (62%)
Model (93%)
Model (4%)
From scratch
Rejection
61.8
72.9
82.1
66.3
72.1
Beam
74.1
82.6
87.3
78.3
84.0
Beam
72.7
81.3
86.8
76.8
80.4
Beam 25
71.3
79.8
84,4
73.3
79.9
Beam 10
67.5
75.1
83.5
68.0
73.9
Beam 5
64.9
70.8
80.1
65.6
69.1
Beam 61.6
62.1
75.6
55.8
60.5
Table 3: Performance of models fine-tuned on 1M examples generated by the generator. 7 = 2%
B.2 COMPUTATIONAL RESOURCES
We leverage V100 GPU with 32GB of memory for all experiments involving linear algebra. The training time ranges from to days, depending on the data size and the number of epochs.
C GENERALIZATION OF THE PROXY
Recall that in our theoretical setting, we define all variables in a two-way classification problem. with y the ground truth label and y' the predicted synthesized label. We introduce indicator variable 8, defined as:
1 yf 0
if ff
We can rewrite all formulas in terms of s:
p:= E(s].
and
Φ 1 1), P= 0).
When moving beyond the classification setting, is longer binary 0/1 variable. In more general cases, represents a measure of similarity or distance between the original target and the synthesized target y'. Without loss of generality, we assume is normalized to lie within the range [0, 1]. For instance, in the news summarization experiment, s corresponds to the ROUGE-1 score between the ground truth text y and the synthesized text y', and equals if and only if y 3//
With this generalized measure we can extend the previous definitions of Φ and & as follows:
E, [gs]
E. (q((1 - s)]
P
When takes values in {0, 1), these definitions are consistent with those in Equation
D NEWS SUMMARIZATION
We leverage the XLSUM dataset (Hasan al., 2021) https: http://ggggggggaa.g.e//// co/ Catasets/csebuett
x1 sum under the license CC-BY-NC-SA 4.0.
Data preprocessing. For each data in both training and test dataset, it consists of news report and summarization, denoted as (news, summarization). We write each data in the following form:
Article: news. A summary of the article: summarization.
Fine-tuning and generating details. Throughout all phases of evaluation and generation, employ greedy decoding. Given that news summarization is low-entropy task, greedy decoding is chosen to ensure quality generation. Consistent with common practice, fine-tuning is limited to single epoch. Through out the experiments, all the finetuning with full parameter tuning to better capture the scaling law suggested in Zhang al. (2024a).
18

---

Published as a conference paper at ICLR 2025
Implementation details. We leverage the official implementation in Huggingface for training, under the license Apache 2.0. Specifically, for training the generator, we start our training with the pre-trained Llama-2, and set the learning rate to 5e-5, the learning rate scheduler as 'cosine', the number of epochs to 1, the total batch size to 32, the block size to 1024 and the others to the default value. For generating the synthesized data, each news in the training set. For training based on the selected synthesized data, we also start our training with the pre-trained Llama-2, and set the learning rate to 2c-5, the learning rate scheduler as 'constant' and the others to the same. For evaluation, we first use greedy strategy to generate summarization for each news in the test set, and then calculate the Rouge-1 score between the generated summarization and the corresponding ground truth, and finally report the average of the Rouge-1 scores of all test data. When calculating the perplexity, we only calculate the perplexity for the generated summary. When fine-tuning the Llama-3 model, we use the full XISSUM dataset to achieve good performance. The resulting model achieves Rouge-1 score of 34.5.
greedy strategy generate a summarization for
Computational Resources. All experiments were conducted using a dedicated computational cluster equipped with 4 NVIDIA A800 GPUs, each with 80 GB of memory. Our training and inference processes performed on the cluster.
Estimated Time Training the whole dataset for an epoch takes about 6 hours. Generating the whole dataset takes about day. During evaluation, need to first generate and calculate the rouge score, which takes around 40 minutes for one checkpoint.
Quantitative Results. We show two additional qualitative examples from the testset of the news summarization task to compare how the model performs with each selection strategy on the same news as follow. Notice how the generation without selection collapses: it heavily relies on clichéd format that lacks specific information. While this approach might work for simple news by piquing curiosity, it fails convey the depth or significance of the current news. The model trained on generated samples collapses to these formats. The Llama-3 selection does not address this issue, aligning with our results that Llama-3 selection performs poorly despite using better model. In contrast, generations with self-selection and oracle selection produce summaries that are specific, informative, and directly address the key event, significantly improving the quality of the output.
Quantitative examples in news summarization
Currently just the US dollar, the euro, the yen and the pound are in the group. The IMF said the yuan "met all existing criteria" and should become part of the basket in October 2016. IMF chief Christine Lagarde said it was "an important milestone in the integration of the Chinese economy into the global financial system". " She added it was also a recognition of the progress that the Chinese authorities have made in the past years in reforming China's monetary and financial systems. The yuan will now make up part of the IMF's Special Drawing Rights (SDR)- - an asset created by the IMF which serves almost used for transactions between central banks and the IMF, and is used to decide the currency mix that countries like Greece, for example, receive when the IMF provides financial aid. The last change made to the basket was in 2000, when the euro replaced the German mark and the franc. China is the world's second largest economy behind the US and asked for the yuan to become reserve currency last year. Analysis: Andrew Walker, BBC World Service economics correspondent More than anything this move a symbol a powerful one of China's 'ss metcoric rise, from poverty to pillar of the global economy. Until now only the currencies of the four lcading developed economies were used in the basket that determines the value of the SDR. So for China to join this very small club is quite a statement of how the world economy has changed. There could also be some real benefits to China. If the currency's elevated status leads to more being held by central banks businesses and businesses, it would be helpful for the government's finances. A state's 's monopoly over the issue of currency indirectly gives it extra source of revenue. But the main point is that the decision is new way of conveying the increasingly apparent message that China has central role in driving the performance of the world economy. Vaswani: The Chinese yuan is going global What's next? Some analysts have suggested that by 2030 the yuan will become one of
Article:
currency. is
Pttpps: thub. htttes////gthhbb...ommggnggnnngo httee//traraffrre///e/e/te http:/mmmeeee/bbeeo mai in/exappees//s pytorch/1 aaagaagggemoeeeoo ing/run_clm..mmmmmmm
19

---

Published as a conference paper at ICLR 2025
the top three major international currencies, together with the dollar and the euro. Concerns about Beijing keeping the yuan artificially low to help exporters is one reason the currency has previously failed to meet the criteria for reserve currencies set out by the IMF. However, Chinese officials have a made concerted effort to build support for the yuan's 's inclusion, and a recent IMF staff report endorsed such a move. Initially, the currency's inclusion would be largely symbolic gesture, some analysts have said. They have also said the yuan's 's continuing inclusion in the basket would depend on whether China progresses with its financial reforms. Generation of the model trained with oracle selection: China's 's yuan has been added to the International Monetary Fund's s basket of reserve currencies, the IMF has announced. Generation of the model trained with Llama-3 selection: 2015 was good year for China's currency.
Generation of the model trained with Self-selection: The International Monetary Fund (IMF) has approved the inclusion of the Chinese yuan in its basket of reserve currencies. Generation of the model trained without selection: 2015 will be big year for the Chinese yuan.
Ground truth: The International Monetary Fund (IMF) has announced that China's currency, the yuan, will join the fund's basket of reserve currencies.
Quantitative examples 2 in news summarization
His brother Mahinda Rajapaksa is expected to be installed as prime minister, having held the role as caretaker since November. The brothers' party, Sri Lanka People's ' Front, has secured two-thirds' "super majority" of seats needed carry out its promised constitutional changes. The party won 145 of the 225 seats, plus five more seats from its allies. Mahinda Rajapaksa earlier tweeted that Indian Prime Minister Narendra Modi had called to congratulate him. The controversial Rajapaksa family has dominated Sri Lankan politics for two decades. Mahinda Rajapaksa was previously president, from 2005 to 2015. The opposition of former prime minister Ranil Wickremesinghe has been trounced, having lost all but one of the 106 seats it held in the outgoing parliament. The main opposition party is now new group established by the son of Ranasinghe Premadasa, former president who was assassinated in 1993. Sri Lanka has been one of the few nations to hold an election despite the coronavirus pandemic. The vote had already been postponed twice because of the virus. The country has had relatively few confirmed infections and deaths of coronavirus with total of 2,839 cases and 11 deaths. Another victory for controversial brothers By Anbarasan Ethirajan, BBC South Asia Analyst Once again it's a huge victory for the Rajapaksa brothers. Just nine months after his impressive win in the presidential election, Gotabaya Rajapaksa has led his Sri Lanka People's 's Front to a two-thirds majority. He is hugely popular among the Sinhala majority for crushing the Tamil Tiger separatist rebels in 2009 when he was defence secretary. Many in the country also credit his administration for bringing stability and successfully containing the coronavirus outbreak. Gotabaya Rajapaksa is accused of human rights abuses during the civil war and also of targeting those who dissented. He always dismissed such accusations but they have not gone away. A surge in Sinhala nationalism in the run-up the election has also worried Sri Lanka's 's minority communities. Muslim leaders say their community is still reeling from the vilification that followed the devastating Easter Sunday suicide attacks by Islamist militants last year, which killed more than 260 people. With their dominant majority, the Rajapaksas could attempt to change the constitution, increase the powers of the president so reversing the work of the previous government to introduce more checks and balances. Activists, already alarmed by the diminishing space for dissent and criticism, fear such an eventuality could lead to ever greater authoritarianimm.
Article:
Generation of the model trained with oracle selection: 70-ycar-old Gotabaya Rajapaksa has won landslide victory in Sri Lanka's parliamentary election.
Generation of the model trained with LJama-3 selection: 72-year-old Gotabaya Rajapaksa has won Sri Lanka's presidential election, securing two-thirds majority in parliament. Generation of the model trained with Self-selection: 70-year-old Gotabaya Rajapaksa has won landslide victory in Sri Lanka's parliamentary election.
Generation of the model trained without selection: 145 seats needed for a two-thirds majority 106 seats won by the main opposition party 49 seats won by the ruling party 13 seats
20

---

Published as a conference paper at ICLR 2025
won by the Tamil National Alliance 5 seats by the Janatha Vimukthi Peramuna seat won by the United National Party seat won by the Jathika Hela Urumaya 1 seat won by the Civic Force
Ground truth: Sri Lankan President Gotabaya Rajapaksa has declared victory in the country's parliamentary election.
E A GENERAL THEORY OF PRUNING WITH VERIFICATION
In Section 4 have presented special case of our general theory, which we describe here in more generality and detail. While some of our exposition here overlaps with Section 4, we prefer to leave complete text that provides a stand-alone overview.
E.I DATA DISTRIBUTION
Consider probability distribution P over R. {0,1) with the following high-dimensional property Condition E.1. Given N N(d) iid. samples (æ1, U1),.... (IN, YN from P with N N(d), the following hold estimates w.p - o(1) uniformly on all i, E [N], in the limit d + 00
| 2 = 1,
x. x 0 yA ys if yj
b,
where b < < are constants. For simplicity of presentation of our results, We will further assume that b aa
The above structural condition is inspired by assumption in Das & Sanghavi (2023). For simplicity of exposition, we will only consider balanced distributions, meaning that
P(y = 1) = Ply = 0) = 1/2, for (I,y) P.
Gaussian Mixture Example. As a first example, in the case of Gaussian mixtures where the features have conditional distribution given by
(1y, 2),
(7)
(8)
where My (2y 1)H,, for some μ E Idd and positive-definite matrix with E ||=||2 = 2 1, we may take
a = || b aa.
(9)
Condition E.1 then holds thanks to concentration, with Ndd) = ((d)
E.2 TRAINING DATA, DATA PRUNING, AND DOWNSTREAM MODEL
Let DN = [[111 ((((1,9))
(ZN. } be dataset of N i..d. pairs from the true distribution and (IN, } version of the dataset (also i...d) with labels y/ instead of y:- For example, this could be labels generated by an AI trying reproduce real-world data. D' the data on which the downstream model is trained.
let D'x =
We will consider family of models given by
P(y = 1 x, w) = = (xx (xxTx) w) E (0,
parametrized by a vector weights w E Rdd Here, ar is the sigmoid function defined by
a(x) I+e-2
(10)
21

---

Published as a conference paper at ICLR 2025
For the loss function, use binary cross-entropy (BCE), defined by
{(U y) = -y log D - (1 11-38 log(1 - 0)- Let WN be obtained via logistic regression fitted on D'y with ridge regularization parameter Thus, te is the unique? minimizer of the following objective function:
(11)
L(w) = N Σ (0(((( (1) w), v.) + 10ol3
Here di is bit which indicates whether the ith training example has survived pruning. The numbers q = (....... ..x) is called praming strategy. The corresponding downstream classifier is fN f@x where the notation Sw refers to the linear classifier induced by weights vector E IRd i.e
11, if x: w 0, (0, otherwise.
(x)
(12)
The test accuracy of the downstream model JN is defined by
ace( acc(fx) == P( (x) [B)) for random test point (r, y) ~ P. where Baaee (2) Ely|æ 2] is the Bayes-optimal classifier. In particular, note that acol Bogee) 100% by construction.
This quantity will be the main object of our analysis, and we will be interested in how it depends the corruption level p and the choice pruning strategy in the infinite-sample limit N -> oc.
For later reference, we also define an empirical version, namely the accuracy of fN evaluated on the clean dataset DN namely
acel = |M (i M (x,) = ys
(13)
where M = (i [N] q = 1} collects the indices of training samples which survive pruning by q-
E.3 A CLASS OF PARAMETRIZED PRUNING STRATEGIES
Given hyper-parameters do, Φ1, tho,, vho [0. 1J. we consider broad class of parametrized pruning strategies with the following property. For any class labels k, l E {0, 13, the random variables [[ defined by Zike 1/3/ k, v l.q. 1] are ii.d. with Bernoulli distribution Bern(pke), where
Pkt = P(q: = L, L, = E, k)
P(g: = 1/4 = l, = k))((1/ = y: k)P(y; = k)
(14)
[bb(1= 2 if k = l.
(2uep/2,
else.
and the numbers p. de and vrze are defined by
= P((/ * y1). dx = Pp( = 1 vo k; yı = k), the = P( = 11 E,y; = k). (15)
Consequently, if N&s the number of training examples that have true label k, fake label and survive pruning, then
Ng
Zike
(16)
which is has binomial distribution Bin(N, P.r). As mentioned in the main text, for simplicity of exposition we considered the following simplifying assumption:
Φ 6u Φ. doo1 1/100 v.
(17)
Such pruning strategy will be referred to as a verification-pruning strategy with parameter (ó. v)). Remark E.2. Note that the parametrization (ø, v)) and (poo, P11) describe the same verification- pruning strategy via the following bijective transformation.
Poo P11 =1 p)6/2, Po1 P10 p1//2.
(18)
Unieity is due strong convexity of objective function.
22

---

Published as a conference paper at ICLR 2025
E.4 EXAMPLES
Let us present some notable examples of verification-pruning strategies.
No Selection. The case (Ø, vp) (1, corresponds selection, i.e the entire training dataset is used.
Oracle Selection. The case ()) (1, 0). The selection strategy only keeps indices correspond-
ing to examples in the dataset which have correct label (all corrupted labels discarded).
Supervised (Margin-Based) Selection. Let Uprune E R. and consider the pruning strategy defined by
qi (2) Wprune) 0].
This pruning strategy simplify filters out all examples on which it disagrees on the assigned label.
E.5
PERFORMANCE BOUNDS FOR MODELS TRAINED WITH VERIFICATION SELECTION
100
1.0
Pue P.
0.9
80
Accuracy
w/P
0.7
0.21
0.42
0.63
20
0.84
0.5
1.0
0.2
1.0
0.4
0.8
Corruption level
wvg
Figure 6: Empirical Confirmation of Theorem E.3. Comparing the breakdown points of different models. Here, the task classifying a Gaussian mixture, with infinite training samples from datasets generated from model with classification eπrоr rate p (x-axis). Notice the sharp phrase- transitions where the model suddenly switches from perfect accuracy to worse-than-chance a phenomenon predicted by Theorem E.3. Left. Performance of verification-pruning strategies with different values of the hyper-paranen parameters (ø, 4). Recall that the case /// = corresponds to no pruning, while //0 with sup(p aco(fx) 2 90%] (computed empirically), against the analytic estimate p, (t) given in Theorem E.3 (for = 0.1). Again, the results in excellent agreement with the predictions of the theorem.
corresponds oracle selection. Right. Comparing Pup. approximated
The following is one of our main results (proved in Appendix F).
Theorem F.3. Suppose Condition is in order. Fix Φs 1,, 1 E (0, 1) and define p p+(1) E (0, by
1-1
1
(T = 1- - t+ (1+
P 1 (1 t)w/$
(19)
P <. P.. PT (t), then the limit - oo holds w&p 1 - (() that the aco(JN) = 100% for downstream
model IN trained on data from a generator with error rate pruned with a verification-pruning
strategy with parameters (Ø.
On the other hand, p > pt. then in the limit N - 0o holds w.p o(1) that the acc(FN) = 0% for downstream model fx.
Thus, there is sharp phase-transition around the corruption level p. b/0): as p is increased past level p., the downstream model fr abruptly switches from being perfectly accurate, perfectly inaccurate!
23

---

Published as a conference paper at ICLR 2025
See Figure 6 for empirical illustration of the theorem.
The thresholds p. (t) appearing in the above theorem are proxies for the so-called breakdown points Pup 2 Pdown defined by
Pup inf p [0, 1l acc(fN) 3 0% in the limit N - o0)
(20)
Pdown sup {D E [0, 1] aco( fN) " 100% in the limit N + 00
(21)
Theorem E.3 implies Pdown 2 Px (t) and Pup S (t) for all E (0, 1). Consequently,
Corollary E.4. Under the hypotheses of Theorem E.3, it holds that Pup Pdown-
E.6 SOME CONSEQUENCES OF THEOREM E.3
We now present some illustrious applications Theorem E.3. These examples are empirically confirmed in Figure 6.
No Selection. Here, we have v/6 = and so the downstream model achieves 100% accuracy for all values of corruption parameter p up the proxy breakdown point predicted by Theorem E.3 is then p. = 1/2 - t/22.
For this scenario, w/0 and so Theorem E.3 predicts that the downstream model NN achieves 100% accuracy for all values of corruption parameter p up to the breakdown point P. 1. This is perhaps not so surprising in hindsight. The point is that even for moderately large values of // the proxy breakdown point p. given in equation 19 can still be quite close to 1.
Oracle Selection.
Self-supervised (Margin-Baseed Selection. Consider Gaussian mixture data with means =y and consider margin-based pruning strategy in Equation (5). It is clear that ø and v only depend on all the 3 angles between the set of vectors (w., Wgen, Wprune }, with w. = p..
Supervised Selection. Consider isotropic Gaussian mixture data with means -,,, and pruning strategy as in Eq. (5). The parameters (Ø, v) only depend on the angles ggen, Oprune, E [0, a] given by
8 gen == (WWgen: H),, Oprane = < (Wpronestt)
(22)
0 (Wppunne Wgen).
Conditioned y 1 and using B||2||2 || 12 N(4, (1- 11 Ia). We can rewrite x as:
tr
(and ||l| 1), get
1p 7
(23)
where 77 L,), For simplicity of calculation, let further assume here that [|WUgen 12
/uprune| 1 =1
Φ1 = P(x Wprune > 0 1 = 1,, = =1)
P(r Wprune > 0,y 1,x Wgen 0)
IP(y 1, 'geenn > 0)
P( Wprune
> [ull22((1-1 1112)) cos ppnnne 1) Wpen > -11lel12(1111 || cos 0gen)
(p(1) Wggen > 11] (1)) cos Ogen)
In the second step, the definition of conditional probability and equation 23 from line 2 to line 3. The random variables 77 Wgen and r) Wprune are jointly Gaussian with:
gege
N (0,
cos
Wprune
cos 0
24

---

Published as a conference paper at ICLR 2025
Let & be the CDF of the standard normal distribution and Ø2 the CDF of the bivariate normal distribution, where (2(2, yi P) is defined as:
(2 yi = P(X S x, S
for (X,Y) ~ (o.00
(o. (o.(o
)))
Denote 61 = -llll||2d(1 || cos penne and Co = - -11a112d(11
|| [ cos Ogn We have:
1 1 (()) x(22) + P22(C1, C2 cos
1 (c22)
All the distributions are symmetric, and we have do = Φ1 Φ. In the same spirit, vb = 1110 = 1'01.
with
+ (2 c2;
(c22
E.7 SKETCH OF PROOF OF THEOREM E.3
The proof is based on the following representation (refer to Proposition F.2) the accuracy of the downstream classifier I' evaluated on the the clean training dataset DN namely
fx) Nu11x<122 + Noo- + N1o1 + No1 1012221/2
(24)
N11 + Noo Noo + No1
for some random some random variables A, B, C, D E (0, 1) which depend on the Nue's 's defined in equation 16.
Remark E.5. We only compute the accuracy (N) of the downstream model fN evaluated on the clean training dataset Dx. By classical results in learning theory (McAllester, 2003; Shalev-Shwartz & Ben-David, 2014; Kakade et ul., 2008), we know that the gap to the population version (test accuracy) ace(f shrinks to at rate O(1/VN), and since the claim in Theorem E.3 is made only in the limit N -> OC, we are good.
Next, in Proposition F.3 and Proposition F.4, necessary and sufficient conditions are established to ensure A, D 1/2 and B, C > 1/2, and therefore (ce( ([N) 100%. These conditions are given explicitly in terms of the Nur's. Finally, in Proposition F.5, concentration of measure is used to control the N&t's ss and present the aforementioned conditions terms of the P 's defined in equation 14, and therefore in terms of é, and z0 alone, giving condition equation 19.
F PROOF OF THEOREM E.3
Our analysis based on non-trivial extensions of arguments by Das and Sanghavi (2023). Viz,
We allow for selection mechanism (aforementioned work does study selection, just self- distillation), and
We use careful asymptotic analysis to avoid solving certain complicated fixed-point equations defining the weights vector Ty of the downstream model fN-
F.1 PRELIMINARY COMPUTATIONS
For later use, given a pruning strategy y, define the following objects
I == {j E [N] yj = k}.
(25)
F = [ee E [n]
1/ = =
(26)
M = E [N] qs = =1}
(27)
Nu: = Σ
qi = 14
(28)
R 1 O.
(29)
25

---

Published as a conference paper at ICLR 2025
Thus, N&t is the number of training examples that have true label k fake label e, and survive pruning. The following result will be crucial in the sequel.
Proposition F.1. We have the representation の = EieM 0426 where
A 4
B, fi
a
(30)
C, b
-D, Io
and A, D,C,D solve the following system of equations
yA = a(-(aNn1A-ou aN + bNn C - bN - RA),
BB = a daNN11 aN1. 6No1 bNoo RB),
(31)
7C = d- b. aNo1 - D) - RC).
yD a (bNN1 bNo aNo1 aNoo RD).
Proof. The following result is inspired by Das & Sanghavi (2023) and the proof is similar. Observe that KKT conditions (L(w)) give Σx a0 v()x; y = 0, i.
N w
a0 with = y 0t
0: = o(o)), = x w.
(32)
7
One then computes
N 2, w Σ
kaaaa T xj = q:an + { (((s q:a;) + bt, aat - q;a;) + bs, if E lo.
if h.
02
(33)
as + + Rq bs + at + Rgjai,
i E I i E lo.
where and 2 0 given by
s
Σ yjajs SE4
t:= Σ gj0g.
(34)
We deduce that for any E M,
(as + bt + Rgios);
if E 7 0 1 i n i To 11
(as + bt + Ryaa;); (bs at + Rq:as),
ya, -/00(v)))
(35)
(bs + at + Rqaas)
i I
Due to monotonicity of a, we deduce the existence of A, B, C. D 2 0 such that
A
i 4 4
B,
if I I
(36)
C,
if 4
DD
if 4n
1 -A, 0 v/ 704
YB,
if
(37)
1-7C,
N.M
DD
if
Furthermore, these scalars must verify
7 (as bt + RA) a(-(as bt) RA),
BB = a (as + bl. - RB),
(38)
CC abbs at + RC) a(- (bbbs al) RC),
D abbs + at RDD).
26

---

Published as a conference paper at ICLR 2025
Finally, observe that,
8 = N1A NIO t= No1 No
(39)
from which we get
as + bt =a(Ni1 A No b(No) No
=aN11A A - aN + bNo1 C - bNooD,
bs at b(N111 Nio + a(No1 C NooD)
= IN A- - 6Nio. + aNNiC C D.
Plugging this into equation 38 gives equation 31.
F.2 ANALYTIC FORMULA FOR ACCURACY EVALUATED CLEAN TRAINING DATA
One computes the accuracy acc( fF) of the downstream model evaluated on the clean training dataset
D
(Ce(SN) 1 ( M y1 11 OR y: - 11/2])-
M
We rewrite this follows
|M| xvv(fx)
= Ii M y/4 = 1 > 1/2OR y1 = 0A <1/2
ki 1 NM 9: 1/2) lÍi Lo MM < 1/2H
E 1/211/2 + Σ
1/1//2
iEhiM
iETonM
(40)
-Min M M
M
+ +1404 M (o M
NNu1aAci/2 Noo1,,Dc/22 + NN No1 A,
On the other hand, it is clear that the size of the mask is |M| = Ex,e Nke. Putting things together gives the following result which shall be crucial in the sequel.
Proposition F.2. For any Φ, v0 [0. 1]. there is solution (A, J,B,C,K. D) of the system of equations equation 31 such that
aoc(PN)= N111x<1/2 + +No +Noo1T<1/2 + Nool T>112 + No11
(41)
N1 + Noo + No0 + No1
where Ā = 7A, B= BB =C, and D DD usual.
Thus, attain 100% accuracy, it suffices have A, < 1/2 and B,C > 1/2. The proof of Theorem E.3 will be all about establishing sufficient conditions which ensure these inequalities.
F.3 SUFFICIENT CONDITIONS FOR PERFECT ACCURACY
Note that since 7 = NA with fixed and N + co, we have y - oo and system of equations equation 31 simplify to
B o((NN1(NNN A aN + bN C - bNoo))/1),
A = o(-((N aN bNo, - bNoo = 1 - B,
(42)
D [ 7 6N + T - aNoo /77)
C (0-(NN,,, A + a.o. a:Noo D)/r)
D)
where AA, BB C -C. D 7 usual, and we have used the elementary property
that (-2)) = 1 o(2). Eliminating A and C, the above equations further collapse to
= o((aN11(1 B) - a.N10 + bN,11 (1 D) - bNoo D))7),
= =o((NN1 + 6No1 a(N11 + No 6(No1 + Noo ) D)/7),
(43)
D a((NNin - B) bNiiB + aNo1 D) (NooD)/7)
=((NNii +NNo1 6(Nio + Nio) a(No Noo)
*These simplifications made possible hy the Mean Value Theorem.
27

---

Published as a conference paper at ICLR 2025
Two special cases are tractable.
The Symmetric Case: b = -0.
We have 1) = B, and thus the equations become
D A, - B, D 1- B3
(44)
((((NN1 Noo) - a(N1 + Nio + No1 + Noo) B))7))
& 1/2, then we must have 2 (N11 + /Noo)/((11 + Nio + No1 + Noo), which is impossible if we impose
Noo + No1 < N + Noo.
(45)
ie the number of bad indices which survive is smaller than the number of good indices which survive pruning. Thus, under the previous condition, we must have C B 1/2 and A D 1-B 1/2.
By symmetry of the preceeding argument we know that the condition is also necessary. We deduce the following result.
Proposition F.3. Suppose b = Then, for any solution (A, B, D) of the system of equations equation 31, the inequalities
C=B> 1/2,
== ==11//2
(46)
hold if and only iff N10 No1 < N11 + Noo-
Skewed Case: b = O. Here, we have
B ca((N11 (N1 + B)/o))
(47)
D (a([No - (No1 + D)o))
B & 1/2, then 2 Nu/(N1 + N10). which is impossible if we impose
N10 < N11,
(48)
ie the number of examples with label 1, which incorrectly labelled 0 the dataset, which survive pruning is less than the number of examples with true label 1, which are correctly labelled and survive pruning. We deduce that B > 1/2 under the above condition.
Similarly, if D 1/2, then D S NN1o + Noo). which is impossible we impose
No1 < Noo:
(49)
ie the number of with true label 0 but incorrectly labelled as 1 in the dataset, which survive pruning is less than the number of examples with true label 1, which are correctly labelled and survive pruning. We obtain the following result.
Proposition F.4. Suppose b = 0. Then, for any solution (A, D, of equation 31, we have
C,B > 1/2 iff N10 < N1,
(50)
D. < 1/2 iff No < Noo-
(51)
F.4 CONCENTRATION
We shall now derive conditions which are sufficient ensure the hypothesis in Propositions F.3 and F4,, namely Nut < NEk for all k, (0.1) with l + f. Recall that for any k e {0. 1). the counter Nue random with binomial distribution Bin(N, Pkc). Now, by basic binomial concentration, we know that if p, & E [0. 1) and ø E (0, 1], then for any fixed E (0, 1), it holds w.p 1 - o(1) that
Nue S (1 + N)N
if # e, if l.
(52)
Nue 2 (1 t)Neke.
In particular, w.p 1 o(1), it holds that
Nue S (1 + tke
(53)
NoL 2 (1 t)NPkk-
(54)
Comparing the above inequalities, deduce the following result.
28

---

Published as a conference paper at ICLR 2025
Proposition F.5. If the following condition holds
P01 + P10
1-t
2t
1 with =
(55)
+ P11
1 1+1 +t"
then wap 1 o(1) it holds that
N10 No1 < N11 Noo-
(56)
F.5 PROOF OF THIBOREM E.3
Follows directly from putting together Propositions F.2, F.3, F.4, and F.5, and then solving the inequality
Poi + P10
2pn
pd
+ t
+ P11
2(1 - 6
(1 - p)Φ
for p.
29


Reviews:
[{"review_id": "NoYHQXvuiJ", "reviewer": "Reviewer_CePm", "summary": "The work investigates synthetic data in LLM training and concerns about model collapse (catastrophic drop in model performance) when LLMs are trained with model-generated labels. Specifically, the work investigates model collapse as a decision problem and utilizes a framework where a verifier (pruner in the manuscript) vets synthetic examples in an attempt to limit the risk of model collapse. Experiments have demonstrated that this verification-based approach can prevent collapse even when the verifiers are imperfect and are not oracles.", "questions": "- I am interested in whether the proposed methods can be applied to other modalities (e.g., VLMs, Vision-Language-Action Models in robotics) - many problems face more challenges in data annotation cost and also resort to synthetic data.\n- Many LLMs are trained with synthetic data specifically on preferences (e.g., RLHF, DPO). I wonder if parallels can be drawn between the verifiers and a reward model and if model collapse is particularly severe with Vicuna-style finetuning.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Paper is very well written and easy to understand\n- Both synthetic data and model collapse are very important topics in the literature and are of great interest to the audience\n- Theoretical derivation is complete and appears to be correct\n- Very thorough quantitative experiments with strong results", "weaknesses": "- In terms of LLMs, qualitative examples showing model collapse in some of the tasks may benefit the delivery further", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rcNqTC6TTV", "reviewer": "Reviewer_mXV4", "summary": "This paper studies a phenomenon known as \"model collapse,\" in which LLMs that are trained on synthetic (LLM-generated) data suffer drops in performance compared to LLMs trained on real data. The authors propose a verification technique for selecting synthetic data, and analyze cases in which this process can improve over collapsed LLMs trained on randomly selected synthetic data. The theoretical analysis is done using a simplified setting of a Gaussian mixture and linear classifiers, and the results are verified empirically in the same setting. Finally, the authors present empirical results using a pretraining setting for eigenvalue prediction and news summarization.", "questions": "There are a few minor writing inconsistencies in the introduction (and more generally, I feel that the writing of the intro could be improved): \n- \"We validate our theoretical findings in three empirical settings\" and \"We conduct two large-scale experiments to test our theoretical insights :\" are somewhat at odds. And if they aren't, then the writing for this part should be clarified. \n- Ranking is only mentioned once, at the very end of the intro. It is unclear how ranking fits into the verification process -- data selection does not necessitate ranking, although ranking is indeed a way to perform data selection. What is meant by this? \n- Is verification really all we need? Some form of \"verification is all you need\" is mentioned throughout the paper, but it seems like there is a long list of things that we need. \n- While I imagine that this would be computationally expensive, it would be useful for the authors to include a scaling law analysis under synthetic data, and for that matter, an example of pretraining on synthetic general data. The proposed technique indeed improves over the collapsed baseline, but it does not necessarily approach the performance of training on real data. There are many questions related to scale that I don't think this paper addresses, but I am not going to factor this into my evaluation, as this seems like obtaining an answer to this would be quite expensive. In any case, I would be interested to hear any speculative thoughts that the authors might have about this.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper covers an important topic, and establishes some of the fundamentals for effectively training on synthetic data -- this is likely going to be an important area of study as we continue to scale foundation models. \n- The authors perform a theoretical analysis that characterizes how high-quality the synthetic data needs to be in order for it to be a useful stand-in for real data, i.e., in order to avoid model collapse. \n- The authors present practical results for fine-tuning Llama-2 on a news summarization task as well as a pretraining task in which the transformer is trained to predict matrix eigenvalues. For the news summarization task, the authors find that Llama-3 is a worse verifier than Llama-2, which is a peculiar finding. \n- For the real data experiments, the authors confirm that the model collapse phenomena actually occurs, and that their verification procedure actually improves over the model collapse baseline.", "weaknesses": "- The theoretical analysis could be improved with some form of a finite-sample result. The current result only applies in the infinite sample limit in a simplified setting, although the result is validated empirically (using finitely-many samples, of course). \n- The writing in Section 3 can be improved substantially. The purpose of this section and its contribution to the overall narrative are currently unclear, which can be mostly fixed by leading with a crisp motivational statement. More generally, I'm still not sure that I understand the takeaways of this section -- why is it that the model \"lacks the inherent capability to autonomously select the best predictions,\" when using beam search? Is this a statement about decoding, or about the model? Earlier, in the intro, the authors say that models \"cannot intrinsically identify the best solution using perplexity alone,\" yet beam search *is* essentially based on maximizing the likelihood of outputs. \n- 4.1 in the theoretical analysis says that the $y'$ variables \"[have] been generated by an AI model,\" but it's unclear what this statement means mathematically. This should be more formal, particularly since it's part of the theoretical analysis. \n- Assumption 4.1 introduces \"bits\" $q_1, ..., q_N$. I assume these represent whether or not a sample has been pruned, however this is not defined until after it's already mentioned, and it is up to the reader to infer what these bits represent. \n- The authors compare to \"selection with weak supervision,\" but there is little elaboration about what this actually means. Weak supervision is an overloaded term and refers to many things -- one reading of this could be that the authors refer to programmatic weak supervision, which could make sense because programmatic weak supervision in some sense generates synthetic training data and verifies it via abstain votes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Anvj0y39yD", "reviewer": "Reviewer_rwoT", "summary": "The paper presents a verification method to prevent model collapse that happens when models are trained on synthetic data. The proposed proxy $p_{*}$ offers a practical way to estimate the potential performance of models trained on verified synthesized data. Based on it, the synthetic data can be selected to avoid degrading the performance of the model that is trained on mixed real and synthetic data. Tested across three different tasks, the method shows its effectiveness and practical applicability.", "questions": "Figure 5 shows when more selected synthetic data are trained on, both the performance of Oracle and $p_{*}$ become worse, why does this happen? Does it mean synthetic data can still potentially make the model collapse?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a theoretical framework for understanding the role of verification in preventing model collapse.\n2. The paper is well-written and easy to follow. \n3. The method has been demonstrated effective via empirical experiments in different domains.", "weaknesses": "1. Model collapse not only happens when trained on synthetic data but also occurs when synthetic data is involved in the training set iteratively. This paper fails to address model collapse from this dimension. \n2. As stated in the paper, Llama 3 performs worse than Llama 2 when as a verifier. Therefore, an advanced model is not necessary to be a good verifier. There could be more discussion on the reason behind it and how to choose verification models. \n3. This paper primarily focuses on accuracy for analysis, yet it would be insightful to consider cases where accuracy alone is insufficient for assessing model performance, such as alignment in large language models (LLMs), and explore how the synthetic data selection strategy could be applied in such complex scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FZ5A3J8ua6", "reviewer": "Reviewer_rFSV", "summary": "The authors study whether filtering synthetic data (collected from a generator) using verifiers can prevent model collapse. In a binary classification setting, they relate downstream model performance to two quantities: the generator's error $p$ and a quantity related to the verifier's accuracy $p_*$. When the generator's error surpasses the threshold related to the verifier's accuracy ($p > p_*$), model collapse occurs. They then conduct three empirical experiments (binary classification w/ linear models, eigenvalue prediction w/ transformers, and news summarization with Llama-2) in which they show that downstream performance correlates with the verifier performance $p_*$, which they can estimate using some labeled training data.", "questions": "- How might this generalize to language modeling / task-agnostic pretraining, where there is not necessarily a sense of \"correctness\" for verification?\n\n- Lines 390-395 point out that when using a weak verifier, there's a dependence on $n'$, the amount of synthetic data to train on. This seems practically quite important to establish. Can the authors comment more on how to \"take into account the quantity of data selected\" (line 395)?\n\nMisc suggestions for a final version:\n\n- I would suggest moving Appendix C into the main text. If necessary for space, I would move Section 3 to the appendix, since these results are reasonably well-known from the test-time compute literature.\n\n- https://arxiv.org/pdf/2209.03942 is a related reference\n\n- Line 365 typo: \"and the ground truth $w_*$. 5Having the verified\"\n- Line 12 typo: \"other LLM[s]\"", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "This paper was well-written, flowing through a toy but well-executed theory section (Section 4), and then a set of three nice experiments (Sections 5-6). The grounding in actual experiments on transformers was great; in particular, the news summarization experiment for tasks without exact match notions of correctness improved the generality of this work. I appreciated that the appendices were very thorough, and I was able to find answers to most of my questions by looking through these. \n\nI'm not confident in my familiarity with related work, but I believe that the setting studied here (and the style of analysis, beginning with a toy model and ending with empirical work) is novel and helpful to the synthetic data community.", "weaknesses": "*Novelty of results.* From a practitioner's point of view, the conclusion from these results is simple: when it is possible to evaluate a verifier on labeled data, picking a more accurate verifier gives better results on that same task. This seems to be obvious / empirically quite clear in existing work (e.g. Appendix A.1), and I'm not sure if the specific definition of $p_*$ offers gains over just picking by existing accuracy metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)