Escaping Collapse: The Strength of Weak Data for Large Language Model Training
Authors: Kareem Amin
Sara Babakniya
Alex Bie
Weiwei Kong
Umar Syed
Sergei Vassilvitskii
Venue: NeurIPS 2025 poster ()
Keywords: Large-language models
synthetic data
Primary area: theory
TL;DR: We show that highly accurate LLMs can be learned from training sets consisting entirely of synthetic data and weakly curated data.

Abstract: Synthetically-generated data plays an increasingly larger role in training large language models. However, while synthetic data has been found to be useful, studies have also shown that without proper curation it can cause LLM performance to plateau, or even "collapse", after many training iterations. In this paper, we formalize this question and develop a theoretical framework to investigate how much curation is needed in order to ensure that LLM performance continually improves. Our analysis is inspired by boosting, a classic machine learning technique that leverages a very weak learning algorithm to produce an arbitrarily good classifier. The approach we analyze subsumes many recently proposed methods for training LLMs on synthetic data, and thus our analysis sheds light on why they are successful, and also suggests opportunities for future improvement. We present experiments that validate our theory, and show that dynamically focusing labeling resources on the most challenging examples --- in much the same way that boosting focuses the efforts of the weak learner --- leads to improved performance.

Escaping Collapse:
The Strength of Weak Data for Large Language Model Training
Kareem Amin
Sara Babakniya
Alex Bie
Weiwei Kong
Umar Syed
Sergei Vassilvitskii
Google Research
Abstract
seathetically-generatee data plays an increasingly larger role training large language models. However, while synthetic data has been found be useful, studies have also shown that without proper curation it can cause LLM performance plateau, or "collapse", after many training iterations. In this paper, we formalize this question and develop a theoretical framework to investigate how much curation needed in order ensure that LLM performance continually improves. Our analysis is inspired by boosting, a classic machine learning technique that leverages very weak learning algorithm to produce an arbitrarily good classifier. The approach we analyze subsumes many recently proposed methods for training LLMs on synthetic data, and thus our analysis sheds light on why they are successful, and also suggests opportunities for future improvement. We present experiments that validate our theory, and show that dynamically focusing labeling resources on the most challenging examples in much the same way that boosting focuses the efforts of the weak learner leads to improved performance.
Introduction
Large Language Models (LLMs) represent the frontier of artificial intelligence, and are trained on vast amounts of human-generated data. However, much of the high-quality publicly available data on Internet has been exhausted, and limits on generating new tokens threaten to slow progress on L.LM training.
As consequence, synthetically-geeeraae datasets are playing an important role in the training of LLMs. Synthetic data have been shown to improve the performance of real large models on range of tasks [BKK + 22 ZWMG22 GPS+ 23 SCA [2].. On the other hand, the circuitous nature of training new LLMs on data generated by previous generations of LLMss has caused concerns of
model collapse SSZ 24 ACRL+24)
What makes synthetic data beneficial or harmful? The answer depends on the precise elements of the synthetic data recipe, and of main contributions is theoretical framework that unifies existing elements of synthetic data approaches, facilitating reasoning about when they might succeed or fail.
Authors ordered alphabetically. Author contributions are listed at the end of the paper. Correspondence to Ikamin, babakniya, alexbie, weiveikong, usyed, sergeiv] (rrgeivvoogoo @google com.
I Work done while the author was student at University of Southern California.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Basic learning theory and empirical studies suggest that a necessary condition for avoiding model collapse is that synthetic data is curated in some way to inject signal that is exogenous to the system that produced the original data. This can come in many forms: identification of high-quality subsets of synthetic data, human rewrites of poor responses, separate model rating the responses, etc. A key question is how much curation is sufficient to not only avoid collapse, but also to converge to an optimal LLM? Our answer, which we will make precise, is the minimum amount.
Specifically, we analyze simple procedure for improving an LLM, in which we iteratively (I) generate synthetic responses from the model; (2) obtain additional responses from an exogenous source; and (3) train the next generation of the model with both types of responses. This procedure
captures previous successful approaches for training LLMs on synthetic data [ZWMG22 GPS $PS$$23 23
SC], and so our analysis provides an explanation for why they work. More broadly, it models the ad hoc processes employed by model developers. We show that if at least a B > fraction of the non-synthetic responses (i.e., the ones produced by external signal) are correct, then the iterative procedure converges to an optimal L.LM (i.e., one that returns a correct response to each prompt). See Theorem for the precise statement and exact convergence rate.
Connection to Boosting. At a high level, our analysis shows how to use synthetic data to focus curation on regions of the prompt space where the models perform poorly. In this way, this approach resembles AdaBoost, a classic machine learning algorithm that iteratively focuses weak learning algorithm on training examples where previous weak hypotheses performed poorly. Unlike boosting, however, our assumptions on the data and the learning method are inverted. Instead of weak we assume access to powerful I LLMs that can perfectly model an input distribution, which we call strong learners. However, we also assume access to only weak information about the distribution we wish to model (specifically, that 3 > 0), i.e. weak data. This is in contrast to traditional boosting where the algorithm has access to strong data, i.e., independent and identically distributed (i.i.d.) examples from some target distribution.
learner,
This connection between the theory of boosting and learning from synthetic data has been largely unexamined in the existing literature. Our analysis also suggests practical ways to improve current algorithms for learning from synthetic data. In our experiments, we show that scarce curation resources are better utilized by focusing their efforts on producing responses to the most challenging prompts in the training set.
2 Related work
Training models on human- uuman-generated data only has limitations such as scalability, biases, errors, and potential privacy considerations IKPS+23 SCA+ 224, GAK23 [4)+ [24]+22] highlights a challenge: as LLMs scale, the demand for high-quality data increases, yet access to such data becomes more restricted due copyright and privacy constraints. Given these challenges, integrating synthetic data into training pipelines is essential but comes with risks.
Model Collapse. Several studies highlight critical concern regarding the use of synthetic data in training LLMs, known as model collapse. This phenomenon is caused by improper use of synthetic data in training the model, which can cause performance degradation
even complete failure of the model [SKAK25). [ACRL*24 SSZ 24 HBA23 GSD + 24) have empirically studied model collapse in various settings, demonstrating the detrimental effects of iterative training only synthetic data and highlighting how this process can severely degrade model performance.
DFK24 DFY+24 BBD*24 DD24 SCH+24] study model collapse theoretically. Their results
show that recursively retraining only on synthetic data causes performance degradation in differ- models. However, combining synthetic and labeled training data [BBD + 24. DD24 SCH+24 KSD+24| FBBG24] can mitigate this performance degradation. In contrast to our work, they do not demonstrate continuous improvement toward an optimal model.
Recently, STK24 and [FDY+24) 24) provide theoretical explanations for model collapse under restricted models, including Gaussian mixture models and linear classifiers. [FDY+22 shows that the presence of a verifier to select more desired synthetic data can improve the performance in non-recursive settings. [FSH+24] extends this work to include both noisy labels and features and provide theoretical and empirical results on the impacts of combining synthetic and real data. Our results do not assume specific learning class, instead relying on a black-box strong learning assumption.

---

Self Improving LLMs Self-evolving or self-inppoving LLMs 2TLC*24 is new research di- rection that leverages the model itself generate or guide the creation of high-quality data [WKM 23, H3 2F)+ which can then be used for fine-tuning (YPC+24 CDY+24
or RLHF [PMM - 24), enabling continuous improvement with minimum or no external intervention.
STaR [WMG222 presents a bootstrapping mechanism to enhance the reasoning capabilities of LLMs by iteratively asking the model generate step-by-step "chain-of-thougle questions, filtering out incorrect answers, fine-tuning the original model on all correct rationales, and repeating the process. ReST [GPS 23l proposes a combination of self-generated data and offline reinforcement learning. The method operates iteratively in two primary phases: "Grow" phase, where for each input (context), the LLM generates multiple outputs to expand the training dataset, and an "Improve" phase, which involves ranking and filtering this augmented dataset using learned reward model trained on human preferences. ReSTEM EM [SCA+24] is a modified version of ReST with two main differences; they do not augment the generated data with numan-generater data, and in the "Improve" step instead of fine-tuning the model in the previous iteration, they fine-tune the base model. All of the above methods can be modeled in our framework, and thus we provide better theoretical understanding about why and when such methods can work.
rationales for
theoretical approach understand self- improving algorithms. SZE 25 introduces new metric to analyze how different components contribute self-improvement formally. In parallel, (YFC+25) 25] studies how to optimally allocate fixed computational budget across iterations of synthetic data generation and fine-tuning, showing that exponential growth policies outperform constant or linear ones in both theory and practice.
Recent works SZE++2
DDE+ 2F
take
Preliminary Notation
Datasets. Let X' be the set of all possible prompts, and let y be the set of all possible responses, which we also call labels. An element of X' 3 is labeled prompt. A subset of X is prompt set, and a subset of X x ) a dataset.
For any prompt set P. let P(x) denote the number of times prompt appears in P, and for any dataset D. let D(x, 3) denote the number of times labeled prompt (z, y) appears in D. Typically we have P(z)= E {0, 1} and D(x, y) E {0, 1}. However, we also allow datasets to contain multiple copies of the same element, where the multiplicity, or weight, of an element can be any non- negative real number, i.e., D(z, y) E TR. We write (r. E D if and only if D(r, v) > 0 and [D] = zzy D(x, y)- Datasets with general weights are formed by using the weighted union operation: If Do and D1 are datasets, and Ap, A1 > then D Xo W D2 is the dataset defined by D(x, v) = No Do(a, v) + X1 Di v).
For any dataset D let D(yl ar) DIx, 3)/ D(x, v/) be the fraction of labeled prompts in D with prompt that have response y. Define D(y(z) 0 E, D(z, v)-
A large language model, or LLM, is function that maps each prompt in X' to distribution on V. We will denote LLMs by the symbol and let g(x) denote the distribution over labels y of g when evaluated on prompt x.
Let X' x + (0, 1} be the quality function, where q(., y) = 1 indicates that y a good response to prompt x.
4 Problem Setting
We consider setting where a sequence of LLMs 91, .2.... are learned on a sequence of datasets D1, Dz..... Given a prompt set P, our high-level goal is to produce an LLM that generates high quality responses for every prompt in P. We illustrate this meta-algorithm in Setting [
Unlike classical learning, where the learner has access to samples from the target distribution, we assume that the learner only has access to labeled examples constructed by data generation procedure that we control, denoted by the function GenerateData. Data generation might make use of synthetic data, produced by the previous generation's 's LLM gt 9t-1, and exogenous (i.e., synthetttt signals.
3

---

In order to formalize our goal, we make precise the capabilities of learner, the capabilities of GenerateData, and our notion of quality.
Setting 1 Data Generation Problem
Given: Prompt set P. number of iterations T.
1: 9o =
2: for = 1, 7' do
3:
D GenerateData (t
4: D. D A D
5: 9: learner(D()
6: end for
7: Output gr.
4.1 Strong Learning
We first introduce the concept of a strong learner. Definition (Strong Learner). For any LLM g let g(ylar) be the probability that the distribution g(x:) assigns to response y. The function learner takes as input a dataset D C X' x y and outputs an LLM g such that (3)(=)) (yy(z) for all (I, y) D.
The procedure learner trains an LLM that matches the conditional probability of each response given a prompt the input dataset. That is, we assume that the model class has the capacity to match this distribution exactly, and the learning procedure can find the model parameters that perfectly fit the data. This assumption is motivated by the fact that deep neural networks instantiate all modern LLMs and are both theoretically capable of approximating arbitrary functions [MP99] and frequently observed to fit their training inputs 21B
While LLMs are powerful, the largest models contain billions parameters and are extremely expensive to train. Thus, training 7 state-of-the art models from scratch is prohibitively expensive. In contrast, given model 9t-1 1 trained on D it is significantly less expensive train model ge on D D, for some choice D. In other words, our setting models continued training, where the training mixture for the next LLM is constructed by augmenting the existing data mixture with new examples.
4.2 Data Generation
Creating data for the next generation of an LLM might involve making of synthetic data produced by the current generation of the LLM. To avoid model collapse, some degree of data curation happens in practice. This curation may make use of an exogenous signal previously unknown to our training algorithm. Curation may also take the form of evaluating the quality of existing synthetic data. We discuss each of these capabilities in greater detail.
Synthetic Data. Given an LLM and a prompt x, we can generate a synthetic response for x by sampling from distribution ( Overall, we assume that synthetic data generation is relatively inexpensive, and permit data generation procedures that make calls to previously-trained LLMs.
Noisy Filter. We assume that we can noisily partition dataset into labeled and unlabeled prompts based on the quality function.
Definition 2 (y-noisy Filter). Let E [0, 1]. The function filter,(() takes input dataset D X x y and outputs (S+, P-) = filter,(()) where the following holds with independent probability for each (z, E D:
If q(x, u) 1 then (a, v) S+ with probability at least 7, and otherwise & E P
If qf v) 0 then x E P
In other words, 7 a lower bound on the recall of filter, for recognizing high-quality responses. For many applications, recognizing that synthetic response is high-quality for given prompt is significantly easier than generating the response from scratch. For instance, if the dataset contains arithmetic coding problems, it is relatively easy to programmatically verify a correct answer.

---

Weak Labeler. Key to our work is the notion of a weak labeler, a function, that given any set of prompts produces responses with average quality bounded away from zero.
To formally define it, we use an auxiliary function ap P - V. which generates labels for all prompts in set P.
Definition 3 (ß-weak Labeler). Let B [0, 1.. The function labeler, takes input a prompt set P C X, and uses an auxiliary function ap - y to label every prompt in P. Formally,
labelers(P) = {(z, v) x P. ap(x)) C X y
We say that the labeler is 3-weak if fraction of these labels are high-quality, i.e., for any input prompt set P.
((x,y) v) labeler ((x, v) 11 2 3.
PP
In our setting, each iteration of data generation allowed to make one call to the weak labeler. The role of the labeler is to create new responses set of prompts. We are not prescriptive about how the labeler is implemented, only that provides some 3 fraction of high-quality responses. The labeler does not need to indicate which prompts have been correctly labeled, nor does it need to correctly label a representative portion of its input. For example, the labeler is allowed to only correctly label the "easiest" prompts that it receives input. We think of these responses being produced by an exogenous process, such as consulting with a human directly, having a human correct or critique LLM responses, any other framework for generating responses that are not purely synthetic.
4.3 Objective
Given these capabilities - the ability to synthesize data, assess synthetic data quality, and weakly label new data the goal our algorithm is to construct datasets DI,..., Dr so that
lim Pig-:Paa--ort [9(2, y) 1] = T-000
(1)
where x. P denotes that is chosen uniformly at random from P. and y gT (x) denotes that y is chosen from distribution (z). In other words, as the number of algorithm iterations grows large, the final LLM output by the algorithm returns a correct response to almost every prompt in P. Note that this objective is similar to the objective of classical boosting. Rather than use weak learners to construct good hypothesis. we ask whether strong learners and weak data can be used construct model that provides high-quality results on all prompts.
5
Algorithm
We present an algorithm for learning an LLM from mixture of synthetically generated and weakly labeled data that uses the capabilities introduced in Section
The aforementioned algorithm generates synthetic responses from the last generation of LLM. Synthetic data generation is given multiple opportunities to produce good responses, which are noisily recognized by filter, Prompts that are consistently paired with low-quality responses are passed into labeler which provides minimal amount of signal. A mixture of good synthetically labeled data and 9-weak-labelld data is then incorporated into the training mixture. To state this procedure formally, it will be convenient to introduce the generate subroutine, which issues multiple calls to an LLM per prompt to produce dataset of synthetically labeled prompts.
Definition 4 (Generation). The function generate takes input prompt set P CYY LLM g and positive integer k, and is defined
generate(P:k.g) ((I, vit P g()))
Algorithm 2 formalizes our procedure for data generation, where generation, filtering, and weak- labeling are applied in sequence on each generation of LLM. Whether the data that is being added to the mixture consists of mostly 3-weakly labeled data (D, in Algorithm 2 or y-littered synthetic data (St in Algorithm 2 is parameterized by 0.
5

---

Algorithm 2 Boosting-style algorithm for LLM training
Given: Prompt set P. repeat parameter k, weakly labeled prompt weight a, high-quality fraction B. filter recall % number of iterations T.
1: 90 = 1 and Do- =
Initial LLM and initial training set
2: for 1, .............................. do
3: S; = generate(P:
Generate l synthetic responses per prompt. D Noisily partition high-quality synthetic data from low-quality prompts.
4:
(St PT) - filter,((s)
5:
Dj labelerg (PT) a
>> Weakly label low-quality prompts b Set weight weakly labeled prompts
6:
A
Di
7:
D. Di-1 ADD,
Update training mixture
8:
9: = learner
> Use learner to update LLM
9: end for
6 Main result
Theorem S is our main theoretical result. and states that the final LLM gr output by Algorithm 2 satisfies the convergence requirement Eq. ( TheoremBaan alsoo quantifies the rate of convergence.
Theorem 5. Let E (0, 1). Suppose that in Algorithm
2 we have 0, 8 (0, 1), 7 (0,1)
log (2/()
2a
T
+1
B
Be
and k 2 lo T'ogT+ + log (17/ With probability at least - 1/T over the randomness of the algorithm, the final LLM gr output by the algorithm satisfies
Pr Pee--Pa--rr y) = =1] 2
Note that by setting a = e in Algorithm the iteration complexity becomes T = 0(log(1/e)/B)
Proof sketch. The key step in the proof is showing that, with probability - 1/T, in each iteration I we have Pr Pry--e--(z) [q(x,
y) ((2) = 1] 2 3 for all but (1 - -) fraction of the prompts x E P. Since the algorithm draws l: = 2(1 synthetic responses to each prompt from 9:-1, one of those responses is likely to be correct. As result, correctly labeled prompts
continually added to the training data (via the synthetic dataset S;),, and the quality of the training data steadily improves, causing the performance of the LLMs learned from that training data to approach the optimal performance.
Even when t is large. it is non-trivial to show that Pry-gg-i()) (1)]9(2)))) 1] > B for nearly all prompts x E P. While this fact follows from our assumption about the weak labeler, does not follow straightforwardly. The weak labeler ensures that the average response quality given set of prompts is at least B, but we need a guarantee about response quality that holds uniformly for almost all prompts. Our approach is to first show that P,- (the set of prompts with low-quality responses) shrinks exponentially with f, and then observe that the total weight assigned to these prompts in the training data is fixed at 0 (a free parameter of our algorithm). Consequently, once prompt is assigned a high-quality response by the weak labeler, the weight of that response overwhelms the weight of all previous low-quality responses in the training data. So when the learner fits an LLM this training data, the LLM assigns non-trivial probability mass to the high-quality response; we are able to bound this probability from below by 8.
6.1 Relationship to Boosting
Boosting is a meta-learning algorithm for combining weak hypotheses into highly accurate ensemble classifiers [SF13). While the most common version of boosting is AdaBoost [FS97], we will present a slightly simpler version that still contains all of the essential ideas.
In each iteration of boosting, a training set of binary-labeled examples is given as input to a weak learner. Each training example is associated with a non-negative weight, and the weights sum The weak learner returns hypothesis that achieves weighted error at most 3 on the training set,

---

GSMBK test performance
CSMBK train performance
0.8
0.4
0.7
0.3-
Taio areetry
0.5
0.9
0.2
0.0
0.0
Szund
Aiterorlty
Gobd
Rooeiing
Goldd
Brosting
Sooeiing
Bcosting fneueing
Figure 1: We plot test and train performance of our Algorithm variants on GSM8K, across rounds. We report the mean and np. (* ddof=1) for seeds. For train accuracy plots, we plot both train accuracy @1 (solid) and train accuracy @8 (stacked), Boosting results displayed here use weak data (A).
where 3 E (0, }) is the edge over the trivial hypothesis that randomly guesses each label. The weight on each training example that is correctly labeled by the hypothesis is decreased by a factor exp(#). and the weight on each training example that incorrectly labeled by the hypothesis is increased by the same factor, where f 1 log
2 1-29- Essentially, the weights are adjusted to concentrate on difficult examples. The weights are renormalized to sum I, and the process repeats. After T = (log((//)))) (1/e)/ (8) iterations, majority vote among all the hypotheses achieves unweighted error most E on the training set.
1+23
Comparing Algorithm to the description of boosting given above reveals many similarities. In each iteration of Algorithm 2 prompts are given as input to a weak labeler that has quality E (0, 1), where / is the edge over the trivial labeler that assigns an incorrect response to every prompt. The weight on each prompt that is correctly labeled by the previous iteration's LLM weight on each prompt that is incorrectly labeled by the previous iteration's LLM is increased by least a factor exp(O), where 0 = log I-3 (this fact emerges from our analysis, which proves that size of the set of prompts given to the weak labeler shrinks by a factor at least
to zero, and the
each iteration; Lemma (Tbb in the Appendix). As in boosting, the weights are adjusted to concentrate difficult examples. After T (1lg(1//)/ iterations, an LLM learned from all of the training data achieves error most the overall prompt set (see Theorem 5p.
7 Experiments
Viewing Algorithm 2 meta-algorithm. conduct experiments with specific instantiations using Gemma 2B on math problem solving [CKB+21 GSM8K] and Python coding [AON 21, MBPP] tasks We select these tasks because measures of response quality here consistent and easily verifiable.
7.1 Instantiations of Algorithm 2
Do nothing. Responses produced the current iteration of the model directly used as training data for the next iteration. This corresponds to setting a = 0. omitting the generate operation, and using a thss-th filter line 4 of Algorithm 2 This tracks the setting explored in the "model
collapse" literature (ARLL 24 SSZ 24, GSD 24]
Filter only. Only correct responses in the current iteration used for training in the next iteration. This corresponds to a = 0 and 7 = 1 in Algorithm This reproduces the STaR/ReST approaches SCA + 22
for learning from synthetic data (WWM222 GPS 23
*Gemma models are made available under Google's Gemma Terms of Use. GSM8K and MBPP are made available under the MIT License.
7

---

MBPP test performance
HBPP train performance
0.31
3.0
0.30
T1t 1965- TMP (11882
0.8
0.25
0.20
0.23
0.10
0.02
0.00
0.0
Round
Gord
Scorting
Gold
Bcosting
Sceeting
Bcosting tocuenng
Figure 2: We plot test and train performance of our Algorithm variants on MBPP, across rounds. We report the mean and np. std (*, ddof=1) for seeds. For train pass rate plots, we plot both train pass@I (solid) and train pass @32 (stacked). Boosting results displayed here use weak data (A).
Boosting. The full algorithm of the present paper. In addition to the synthetic data produced by Filter only, we mix in weak data from the labeler. This corresponds to a and B > 0 in Algorithm 2 We use 1/3 in all experiments.
Boosting, w/o focusing. We ablate out focusing on hard examples. To be precise: rather than giving the labeler the prompts we got wrong, P,.. we draw a random set of questions of size IP
We also report two baselines that do not involve iteratively training on model-generated data. PT: the pre-trained model; and Gold SFT: the model after one round of fine-tuning on the human-written responses in the dataset. Note that Gold SFT is the only setup that makes use of written responses, rather than just for answer verification.
7.2 Experimental Details
In all experiments, a round of fine-tuning entails training all parameters of the model for 330 (GSM8K) or 30 (MBPP) steps at batch size 64 (with the exception of Gold SFT where we report the checkpoint with best validation accuracy) We train with standard sequence cross-entropy loss. Training examples are (input, target) pairs, where input is the problem preceded by a 3-shot prompt (see Appendix fooo prompt templates); and target is model response Cluman-writter response for Gold SFT),
Modeling the weak data. We instantiate labeler as a Gemma 2 2B PT model with a fixed total query budget, which is distributed uniformly over all problems it receives. For given problem, sample responses from the model equal to that problem's 's allotted queries. We consider two setups to simulate weak data provided by the weak labeler. Weak data (A): for each question, return all correct responses if there are any. If there are none, we return a random incorrect response. Weaker data (B): we pool together the correct responses to all questions. We add this collection an equal number of incorrect responses, drawn randomly from all incorrect responses to all questions.
We remark that the fixed total query budget setup offers mechanism for satisfying the weak data assumption: the labeler can maintain constant accuracy when targeting increasingly granular (and more difficult) slices of the input distribution by focusing their resources. We see that this is indeed the case experimentally, and plot accuracies in Figure 3 Moreover, a fixed query budget is a natural analogue to the fixed person-hours/monseee hours/monee/coomptte budgets behind labelling effort.
Departure from the theory. In our experiments, we make one main modification from Algorithm 2. Rather than accumulating data and retraining the model each iteration (Algorithm 2 line 9), we instead accumulate updates. That is, we fine-tune on newly introduced data in each iteration, initializing from the checkpoint produced by the prior iteration. We do this for efficiency reasons.

---

GSM8K
MBPP
train
test
train
test
Sctup
Rounds
@
1
@8
greedy
@32
greedy
2112,00
.630 007
.222.o0 003 243
.235 $035,008
.703.om oor
.275.002
0
orig. report
.302*
Gold SFT
.392,020 .220
.755 009
.379,008 .799
.880 039
.987 .099
.237 .018
Do nothing
.136,012
.391.026
.129.000 .22
.238,011
.386.01
.210,024
Filter only
5
.553,
.762 029
.393.000
.632.
.702.
.329. 229.
Boosting (A)
.589.
.849.mm ocs
.456.010
.681 101.
.844
.329.00 .004
5
ww/o focusing
.550
.787 .00
.432,01
.6000 o10
.747
.336.00
Boosting (B)
.565,
.820.um .00
.443
.647 2112
.832
.326,ave
w// focusing
.509.c00
.767 ou7
.430,112
.54.
.698 .110
.327 2207
Table Comparison of 3-shot train and test accuracy @ rates on GSM8K and MBPP for Gemma 2 2B checkpoints produced by various setups. We report the mean and np std ddof=1) for seeds. To report train accuracy @ k, we sample k solutions to each problem at temperature 0.7 and mark it correct if any of k solutions is correct. For test accuracy, we employ greedy sampling. (*): Row 2 cites the figure from the Gemma 2 report [GRP*24] which does not report sampling temperature.
7.3 GSM8K Results
Table I (first half) summarizes our results on GSM8K. We have 7000 training problems, use & = for generate, and allocate the same total query budget of 56,000 to the labeler each round. In we present model responses to selected problems over the course of training.
Appendix C
Baselines validate our experimental setup. Results in the PT and Gold SFT demonstrate that: (1) our evaluation setup is in the ballpark of what is reported the original Gemma report; and (2) our fine-tuning setup indeed can yield significant improvement when the training data is human-written solutions.
Model collapse with no curation. In the Do nothing row, we recover the result from the model collapse literature that iterative fine-tuning without curation does not improve the model and leads to degraded quality.
Comparison between curation variants. Indeed, the present algorithm demonstrates improvements over the ReST-like variant that uses filtering only. The differences are most evident in training accuracy, which is strongly predicted by the theory. Indeed, this is in spite of the fact that opposed to filtering only, boosting introduces incorrect answers to the training data. Furthermore although our theory does not address generalization, we observe that boosting results in improved test accuracy. Finally, the performance of boosting without focusing is quite close random selection is a strong baseline but focusing still leads improvements, especially in terms of training accuracy.
7.4
MBPP Results
Table T (second half) summarizes our results on MBPP. We have 374 training problems, use k = 32 for generate, and allocate the same total query budget of 11,968 to the weak labeler in each round.
Similar results to GSMSK for train pass rate. In terms train pass@ k. we observe similar results to GSM8K experiments, that generally: Boosting > Boosting w/o focusing > Filter only > Do nothing. On weaker data (B), Filtering beats Boosting w/o focusing in terms of pass@1 1.
No clear winner for test pass rate. While all iterative approaches outperform Gold SFT in terms of test pass rate, they all recover similar test performance despite differences training accuracy. Notably, Boosting w/o focusing beats Boosting, and Filter Only outperforms Boosting with weaker data (B). One explanation is the limited amount of training data (384 examples) which prevents generalization; note that Gold SFT does not recover PT test pass rate.

---

8 Conclusion & Future Work
We have shown that under mild assumptions a modicum of curation applied to synthetic data not only avoids model collapse, but leads to arbitrarily high accuracy results. Our analysis is through the lens of boosting and, mirroring that paradigm, we define notions of strong learners and weak data to reach the theoretical conclusions. In taking this view, we provide theoretical explanations for many of the synthetic data methods used in practice.
Many interesting questions remain. An immediate avenue is further relaxing the assumptions (e.g.. having nearly strong learners that only approximately match the conditional distribution) and deriving corresponding convergence rates. A broader goal is using these insights for the burgeoning field of data selection, where we must explicitly model similarities between different examples as part of the analysis.
Author contributions
Sergei conceived the idea to analyze synthetic data training through the lens of boosting.
Sara B and Alex B reviewed related work.
Everyone developed the modeling framework.
Kareem A, Weiwei K, Umar and Sergei V proved the main result.
Alex B designed and ran the experiments.
Everyone contributed to writing the paper and framing its contributions.
References
[ACRL*24] Sina Alemohammad, Josue Casco- Masco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaci, Daniel LeJeune, Ali Siahkoohi, and Richard Baraniuk. Self-consuming generative models go MAD. In The Twelfth International Conference on Learning Representations, 2024.
[AON+21] Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732. 2021.
[BBD*24] Quentin Bertrand. Joey Bose, Alexandre Duplessis, Marco Jiralerspong, and Gauthier Gidel. On the stability iterative retraining of generative models on their own data. In The Twelfth International Conference Learning Representations, 2024.
[BKK 22] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemí Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer. Scott Johnston, Shauna Kravec, Sheer El Showk. Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Saaffiedd-odds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: harmlessness from AI feedback. CoRR, abs/2212.0807 2022.
[CDY+24]
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Confer- ence on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 662 66642 PMLR, 21-27 Jul 2024.
10

---

[CKB*21] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/21 14168, 2021.
[DD24] Apratim Dey and David Donoho. Universality the / pathway avoiding model collapse. arXiv preprint arXiv:2410.228/2 2024.
[DDE*24] Rudrajit Das, Inderjit S Dhillon, Alessandro Epasto, Adel Javanmard, Jieming Mao, Vahab Mirrokni, Sujay Sanghavi, and Peilin Zhong. Retraining with predicted hard labels provably increases model accuracy. arXiv preprint arXiv:2406. /206, 2024.
[DFK24] Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Model collapse demystified: The case of regression. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[4] Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, and Julia Kempe. A tale of tails: Model collapse as a change of scaling laws. Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 11165-11197. PMLR, 21-27 Jul 2024.
[FBBG24] Damien Ferbach, Quentin Bertrand, Avishek Joey Bose, and Gauthier Gidel. Self- consuming generative models with curated data provably optimize human preferences. arXiv preprint arXiv:2407.0944 09499, 2024.
[4] Yunzhen Feng, Elvis Dohmatob, Pu Yang, Francois Charton, Julia Kempe, and FAIR Meta. Beyond model collapse: Scaling up with syn-thesized data requires verification. arXiv preprint arXiv:2406.075/1 2024.
[FS97] Yoav Freund and Robert E Schapire. A Decision-theorett generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 55(1):119-139, 1997.
[FSH+24] Aymane El Firdoussi, Mohamed El Amine Seddik, Soufiane Hayou, Reda Alami, Ahmed Alzubaidi, and Hakim Hacid. Maximizing the potential of synthetic data: Insights from random matrix theory. arXiv preprint 24X0v:2410.08994, 2024.
[GAK23] Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120030):e2305016120, 2023.
(GFA+24) Nate Gillman, Michael Freeman, Daksh Aggarwal, Chia-Hong Hsu, Calvin Luo, Yong- long Tian, and Chen Sun. Self-correcting self-consuming loops for generative model training. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st In- lernational Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 15646-15677. PMLR, 21-27 Jul 2024.
[G] Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. Reinforced self self-training (rest) for language modeling, 2023.
[GRP*24] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanezyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben
11

---

Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry. Chris Welty, Christopher A. Choquette- Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Plucinska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia. Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matr Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang. Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao. Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej. Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold. Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Fecles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at practical size, 2024.
[GSD*24] Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Tomasz Korbak, Henry Sleight, Rajashree Agrawal, John Hughes. Dhruv Bhandarkar Pai, Andrey Gromov, Dan Roberts. Diyi Yang, David L. Donoho, and Sanmi Koyejo. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. In First Conference on Language Modeling, 2024.
[HBA23] Ryuichiro Hataya, Han Bao, and Hiromi Arai. Will large-scale generative models corrupt future datasets? In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 90555-20565. October 2023.
[HGH+23] Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1051-1068, Singapore, December 2023. Association for Computational Linguistics.
[KPS+23] Alexey Kurakin, Natalia Ponomareva, Umar Syed, Liam MacDermed, and Andreas Terzis. Harnessing large-language models to generate private synthetic text. arXiv preprint vrXiv:2306.0168: 2023.
[KSD+24] Joshua Kazdan, Rylan Schaeffer, Apratim Dey, Matthias Gerstgrasser, Rafael Rafailov, David L. Donoho, and Sanmi Koyejo. Collapse or thrive? perils and promises of synthetic data in a self-generating world, arXiv preprint a41i 16713, 2024.
L2] Shayne Longpre, Robert Mahari. Ariel N. Lee, Campbell S. Lund, Hamidah Oderin- wale, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole Hunter, Kevin Klyman, Christopher Klamm, Hailey Schoelkopf, Nikhil Singh, Manuel Cherep, Ahmad Mustafa Anis, An Dinh, Caroline Shamiso Chitongo, Da Yin, Damien Sileo, Deividas Mataciunas, Diganta Misra, Emad A. Alghamdi, Enrico Shippole,
12

---

Jianguo Zhang, Joanna Materzynska, Kun Qian, Kushagra Tiwary, Lester James Val- idad Miranda, Manan Dey, Minnie Liang. Mohammed Hamdy, Niklas Muennighoff, Seonghyeon Ye, Seungone Kim, Shrestha Mohanty, Vipul Gupta, Vivek Sharma, Vu Minh Chien, Xuhui Zhou, Yizhi LI, Caiming Xiong, Luis Villa, Stella Biderman, Hanlin Li, Daphne Ippolito. Sara Hooker, Jad Kabbara, and Alex Pentland. Consent in crisis: The rapid decline of the AI data commons. In The Tiirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[4W Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. On LLMs-driven synthetic data generation, curation, and evaluation: A survey, In Lun- Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 1 11065-11082, Bangkok, Thailand, August 2024, Association for Computational Linguistics.
[MP99] Vitaly Maiorov and Allan Pinkus. Lower bounds for approximation by mlp neural networks. Neurocomputing, 25(1-3):81-91 1999.
[PMM*24] 24] Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. West-of-n: Synthetic preference generation for improved reward modeling. arXiv preprint arXiv:2401.12088 2024.
[SCA Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jachoon Lee, Kelvin Xu, Aaron T. Parisi, Abhishek Kumar, Alexander A. Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Fathy Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. Beyond human data: Scaling self-training for problem-solving with language models. Trans. Mach. Learn. Res., 2024, 2024.
[S] Mohamed El Amine Seddik, Suei-Wen Chen, Soufiane Hayou, Pierre Youssef, and Merouane Abdelkader DEBBAH. How bad is training on synthetic data? a statistical analysis of language model collapse. In First Conference Language Modeling, 2024.
[SF13] Robert E Schapire and Yoav Freund. Boosting: Foundations and algorithms. Kybernetes,
42(1 42(1):164-166, 2013.
[SKAK25] Rylan Schaeffer, Joshua Kazdan, Alvan Caleb Arulandu, and Sanmi Koyejo. Position: Model collapse does not mean what you think. arXiv preprint arXiv:2503.03/50, 2025.
[4] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson. and Yarin Gal. AI models collapse when trained on recursively generated data. Nat., 63 (8122))))755577 2024.
[STK24] Ananda Theertha Suresh, Andrew Thangaraj, and Aditya Nanda Kishore Khandavally. Rate of model collapse in recursive training. arXiv preprint arXiv:2412. 17446, 2024.
[SZE*25] Yuda Song, Hanlin Zhang, Carson Eisenach, Sham M. Kakade, Dean Foster, and Udaya Ghai. Mind the gap: Examining the self- ielf-improvement capabilities of large language models. In The Thirteenth International Conference on Learning Representations, 2025.
[TLC*24] Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, and Jingren Zhou A survey on self-evolution of large language models. arXiv preprint arXiv:2404. /4387, 2024.
[WKM*23 Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Pupers), pages 13484-13508, Toronto, Canada, July 2023. Association for Computational Linguistics.
13

---

[5FC Pu Yang. Yunzhen Feng, Ziyuan Chen, Yuhang Wu, and Zhuoyuan Li. Spend wisely: Maximizing post-training gains in iterative synthetic data boostrapping. arXiv preprint arXiv:2501.18662. 2025.
[YPC Weizhe Yuan, Richard Yuanzhe Pang. Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. Self-rewarding language models. In Proceedings of the st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 57905-57923, PMLR, 21-27 Jul 2024.
[YPF*24] Zhaorui Yang, Tianyu Pang. Haozhe Feng, Han Wang, Wei Chen, Minfeng Zhu, and Qian Liu. Self-distillation bridges distribution gap in language model fine-tuning. arXiv preprint arXiv:2402. 13669, 2024.
[1BH*21] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
[WWMG22] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning. In Advances Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims made in the abstract are repeated in the introduction and
justified by theoretical proof as well as real experiments with LLMs.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed throughout the paper, which gives the reader the appropriate context to understand them.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: The main body contains a proof sketch which gives high-level ideas. However, the full proofs are contained in complete detail in the supplemental. Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [No]
Justification: Unfortumately, we are not at liberty to disclose all of the details our
computing infrastructure.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
cases, in which
5. Open access to data and code
16

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: While our experiments use many open resources (Gemma model and publically available data), the code itself is not open sourced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
/ooeeeee//ooeeSmmm ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe our experimental setup a level of detail that is sufficient for fully appreciating and understanding our empirical results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars not included for all results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Unfortunately, we are not at liberty to disclose all of the details our computing infrastructure.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the code of ethics and do not believe this work falls under the areas of concern.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Our paper does not introduce any new potential societal impact (positive or negative).
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not train models that have high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We credit all assets used and explicitly mention and respect their licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, thcoe com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our research did not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our experiments do not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
We did not use LLMs in non-standard way to conduct this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy huteesas/eerrpppcccc///ofoeeeees for what should or should not be described.
21

---

A Theoretical Analysis
A.1 Proof of Theorem 5
Throughout the proof, we will write S to denote a dataset where all of the labels were generated synthetically (i.e., by an LLM), D to denote a dataset where all of the labels were provided by labelerg, and D to denote dataset containing mixture of these kinds of data. Also, only datasets denoted by 'D will contain elements whose weights can differ from and All other datasets will be ordinary sets.
We adopt few simplifying assumptions and conventions. Assume that the given prompt set P is non-empty. Assume that the initial LLM go returns an incorrect response to every prompt. Assume that any LLM returns at most one correct response any prompt. Removing the latter pair of assumptions would only speed up the convergence of Algorithm@ to an optimal LLM,, but would also further complicate its analysis. Finally, we adopt the convention that oo 0 0. This convention is needed when Algorithm constructs Dr via the weighted union operation, since it can happen that A oc, but this only occurs when D; is empty.
Let P = fx E P: (I. y) E S be the correct prompts selected by filter. By definition P and Pt form a partition of P. Furthermore, D, pairs each prompt in P.- with the label it was assigned by labeler,, and S pairs each prompt in P2 with the (synthetic) label it was assigned by the previous iteration's s LLM, It-1- Observe that P. (x) = Σ, Dz y) and P,t E, S (z. v). For all 2/ 1 and x E P let
(x(x)= = Σ (((ylo)((z,1))
V
at (x) = Σ st (vlz))(Z.)))
(xx) = Σ ((())=))(,,1))
be the average quality of the responses to prompt x in datasets D., S and D,, respectively, For convenience we also define (xx Ly v)]. Note that (xx) 0 for all E X by assumption. Lemma 6. For all t 2 and x E P
Dr A.P. (z) + P. (z) (x)
(x))
E Po (1) (x)
Proof. We have
De(ulz) = Di(2, v)
Eg (1)
E's + St
Eg Ei de () (t,
E. d Po (z) P7)
Σ X.P, (z) (z)(z) (1)
E A. Po (x) (x)
and therefore
(z() = Σ De(y)z)a(z,9) v)
8
E, E'z +
22

---

Σ X.P (z)ga(a) + P (2)g. (x)
Ex=1>A (z) P (x)
Lemma 7. For all t 2 and x E P we have (:(x) = 0 if and only if ())(()) = and (c()) 0.
Proof. Suppose de (z)) 0. By Lemma 6 this implies A.P. (x) + (x)gg (x) = for s E (1)...,t), and therefore ( = O. This implies that a cannot be correctly labeled in S. and therefore Pt (x) = 1. Since > 0 we have d > O. And since A.Pr (x)):(x) = we must have (x(x) =.
Now suppose q (2) and (x()) O. Since @-1(2) 0 then again by Lemma
have
A,P. (x) + (x)g, (x) = 0 for s E f1..... .t 1}. The fact that (x) = 0 also implies
that x cannot be correctly labeled in St, and therefore Pt (1) 0. And since ge (ar) = 0 we have
Xe PE (x)) + (x))a (x) = O, which implies g (a) = 0.
Lemma 8. Lert 2 1 and x E P. 1f P. then ((x) E {0, 13. If P. then qt (x) = 1.
Proof. Note that D, contains each prompt only once (by Definition of labeler and St contains only correctly labeled prompts (by Definition 2 of filter). The lemma follows from the definitions of P. P.P. (z) and q." (z).
Lemma 9, a b, c, d 2 0 satisfy a S b, 2 d and > 0 then
+ + d b+c
b+d
Proof. If then clearly the lemma holds with equality. Otherwise if c > d then
a atc a a+d d
btc b+d
c)+ d) 2 ddbb+c)
b>
ab bcb ad + ed 2 ab + bdl + ac cd
+ bo+ + ac
a blc d) 2 a(c-d) d)
bba
c
Our analysis relies conditioning on the fact that once the quality of a particular prompt, is high enough, always selected by filter and is never sent to labeler Formally, fix the number of iterations, T, the set of prompts, P. and the quality of the weak data, 33. We define event E, as follows:
Event E' = For all & [T] and a E P (-)(2z) 2 then PT.
and to simplify notation, we drop the dependence of E on T, and 3.
Lemma 10. the repeat parameter Hog Pl then event E occurs with probability least 1-1.
By
Proof. By the definition of generate (Definition 4. each x is labeled times by 9c-1 1 in iteration t. with each label drawn independently from distribution (z)1 Thus we know that if
(x-( 2 3 then x E P. with probability at most (1 - 8²) This is because the probability that synthetic label does not prevent a prompt from being added to P. is at most 1 1-3+3(1--)) = 1- which is the probability that the label is low-quality that it is high-quality but is not recognized by f filter, (Definition 2 Therefore
Pr[-E] Pr [3 E [T] and x E P such that (x)) 2 B and E PT]
Pr(g.-1(x) 2 and PT]
t=
23

---

< [1 - By)
$ P exp(-8yk)
$ P exp( (-2 T log PI)
Assumption about At
P
7 Pl
1 T
The next result is our key lemma. It says that if event E occurs then (a) PT contains all and only the prompts that must have been incorrectly labeled by the previous iteration's LLM, (b) the size of P. shrinks exponentially over time, (c) once prompt is outside P." it remains that way. and (d) prompts outside of Pt are correctly labeled by the previous iteration's 's LLM with a probability that bounded above zero.
Lemma 11. Fix T. Let SIST and P. If event E occurs then all of the following hold:
(a) x E P. if and only if 4: (3)) =.
(b) s (011--y--1110 PP foor all r, 8 E [6] such that r 2 8.
(c) There exists r E [t] such that a E P. for all [r] (n x & P5 for all 8 (4) (r).
(d) Let [f] satisfy the conditions of part (c). I < then
a +
(x(x) 2
2 3.
(1- By)
3
Proof. The proof will proceed by induction. We begin by proving the base case, t = 1. To prove have (x()) Ey-sn(=))g((: y)] = 0, so only need to show that x E PT- Since Ey-gg(==]99 (g(2, (9 y)]
part (a), note that by assumption
= O, we know that i cannot be correctly labeled S1, which
implies x E PT. Part (b) follows immediately from the observation that when t = we have since in this case [t] \ [r] is empty, and have already shown PT in part (a). Part (d) holds vacuously because t must be false when t 1.
1. Part (c) holds immediately by letting
Now assume for induction that the lemma holds for a fixed t 2 1. We will prove the lemma for the case + To prove part (a), first assume ge (xx) O, which the premise of the 'if' direction. By Definition I we have
Ey~a(())l(e, v) Σ Di q(()) y
((())
which implies that x cannot be correctly labeled in St+1, and therefore x E P.+1- Now assume PT,, which is the premise of the 'only if' direction. To force contradiction, assume that (x) > 0. By part (d) of the inductive hypothesis, this implies z (x) 2 3. Since event E occurred, we have that x & Ptt1, which is contradiction. This completes the proof of part (a).
To prove part (b), choose any r; 8 E [t. [++1 1] such that r 2 8. If 8, part (b) follows immediately. If < ++ 1 and ø < t + 1 then part (b) follows from the inductive hypothesis. Henceforth assume s<r=tH Let
DJ ((,, v) E Dr g(x,z) 1}
be the subset of D. that is correctly labeled. We have
- (3-B) (1 - D
2 1121-1201 |
Definition of labelerg
Σ Dz( - Di(a,
2.9
Σ D.(a, (1(11199(_1)))) q(x,9))
24

---

Σ -L
Σ
Σ P(x)1 = 0}(1 (x()))
Inductive hypothesis, part (a)
Σ ((0)) ()1(((()) = 0}
Lemma
(())
Lemma
Σ (z)(()
Part (a)
and therefore
Pitl (1 B)TT (1- -1) (1-8)++-01P-1 (1-1=11337)) -1-
where the second inequality follows from the inductive hypothesis. This completes the proof of part (b).
To prove part (c), we must prove the existence of satisfying iteration r E [t + 1]. Let E [t] be iteration that satisfies part (c) of the inductive hypothesis. If = and x E P:+1 then we can let r t. If = t and E Pi+1 then we can let = t 1. If < t then we only have to show & P because in that case can let r'. Since r' < l we have & PT. and by part (a) we have (x:) 0. By Lemma we have (xx) > 0, and thus by part (a) again we have P This concludes the proof (c)
To prove part (d), let [t + be the satisfying iteration from part (c). Note that < t + by the premise of part (d). first prove that
PT (z)),(z) 0 for all E [r- 1].
(2)
Suppose for contradiction that Eq. (2) is not true, which implies that (r(xrg > 0 for some E (r 1]. By Lemma and the fact that P s+1 which contradicts part (c). Thus have proved Eq. We next prove that
have (x)) >> which implies by part (a) that
P. (x)
Suppose for contradiction that Eq. is not true, which implies by part (c) and Lemma that P. (z)q, (z) = 0. Thus by Eq. 2 we have P. (x)) = for E [r]. We also have by part (c)
that Pt (z) = 0 for E [r). Thus by Lemma 6 we have (z) 0, and this implics by part (a) that
E r+ which by r < t + 1 contradicts part (c). Thus we have proved Eq. (3). We are now ready
to complete the proof of part (d). We have
141 X, P. (x)).(x) + Pf (x) (x)
(+1(())
Lemma
D+ A.P. PP)+
(+1
P. (r)g, (x) (x)g
s=
141 a
Pr (x) + P (x)
D.
12, + E
Part (c), Lemma Eq. 2 and Eq.
2 [.. Et=+tt
1D, +t +1
Σ 0 +t-r+1
1== D.]
D, +t-
D. +1-r+1
s=
25

---

+1
Definitions of Pt and P.
2 m
+1 -
+t -t-r+ r++
Pr
Part (b)
8))- +
==
+1 +1-r+1
8)8
PP + 1 +
Geometric series formula
a11-(1-01)
+ r +1
P
|P (1 - r
af (1- Pr (t r
1 1-++1
Lemma and [P- (by choice of r)
(1-88) +l-r+1
which proves the first inequality of part (d). Continuing from above
+
r+1
(+)
From above
(1--118))
+
0
a
Lemma
a11-(1-8)7
8
a >
1 (1 - B)
28 which proves the second inequality of part (d).
3
We are ready to complete the proof of Theorem 5 Assume that event E' occurs, which by Lemma happees with probability at least 7. For each prompt P let Ta be the iteration that satisfies Lemma () when the lemma is applied prompt and iteration T. Letr log(2/c) and note that by assumption T. We have
8
[g(2,)) y)
Paw (x)
EENP Σ (r(el=)((2,))
Definition! D
Ex- [ār(z))
2 E (x)
r2 < r| Prewp s]
(rxr(ār(z))
Tx 5 r| Pr of P.11
Lemma I
(nwr((r(x))
Tx Sr]
-
iP [P]
1- 1P
En~p(ūr(x)
Ta S]
Lemma 11
|PT|
> Em-p(ir(z) Ta < (1)
Lemma
a 7 -T Tr
> min
(1 - (1-3)")
Lemma d)
((1-(1)))')) +T - Tx
zr.Sr
a T Ts
min
(1 - (1 - 3))
+T T
26

---

a+T -
+ (1 - (1 B)))
Lemma g
a+1
3 - - (1 (1-0-8)) Ar)
a+ 1
[1 1
3 +
2
Since
log
2a
2a
T
r +
A
it is easy to show via algebra that
+ -
2
and plugging this into the final expression above proves Pre--Py-erro 2 (1 (1-5)² 1 c, which proves the theorem.
B Additional Plots
Percentage of labeled problems with correct answer
0.8
correct
Boosting (GSMBK)
0.4
Boosting w/o focusing (GSMBK)
Boosting (MBPP)
0.3
Boosting w/o focusing (MBPP)
Round
Figure 3: Labeler accuracy across rounds. These results use weak data (A). Since training accuracy increases across rounds, the weak labeler gets more queries per question in both cases. Despite this, for Boosting we see that accuracy is relatively constant for GSM8K and decreasing for MBPP. This is because we focus on increasingly harder problems. Boosting w/o focusing, we observe labeler accuracy increasing because we do not focus labeler efforts on the highest difficulty problems.
In Figure we plot average response length on the test function of training rounds. Average response length does not change significantly throughout training, and incorrect answers are longer on average.
27

---

Dveral Cornect Incomee:
125
Oveenll Correct
Auorage response langth (COAI1))
115
120
Incorrsci
rapporta (argh
115
1.5
110
105
100
95
05
80
90
Round
Round
(a) Average response length, Boosting (A)
(b) Average response length, Boosting (B)
Figure 4: Average length of responses GSM8K test problems across rounds for Boosting
experiments.
C Examples
In Tables 3 and we present specific examples of the model improving its answer to particular question across rounds.
Question
Bill starts on the 3rd floor. He rides elevator up to the floor that's equal to times his starting floor plus What floor is Bill on now?
Response (round 1)
Let's use algebra to solve this problem. Let x be the number of floors Bill is on now. The number of floors Bill is on
is equal to 4x + know that starts on 3rd floor, so = 3. Therefore, Bill on 3rd floor now.
The answer is 3. x
Response (round 2)
Let's use algebra solve this problem. Let x be the floor Bill is The floor Bill is now is cqual 4x + We know that Bill starts on the 3rd floor, so 3. Therefore, Bill is on the 3rd floor now.
The answer is 3. x
Response (round 3)
Let solve using algebra. Let x be the floor Bill on now.
The floor he rides up to 4x 6. Therefore, the floor he rides up is equal to the floor he starts on plus 6. So, we have the equation: =
=
Therefore, Bill is on the 9th floor now.
The answer is 9. x
Response (round 4)
Let salve using algebra. Let be the floor Bill now:
6. We know that he starts on the 3rd floor, so x = Therefore. the floor he is now is 4(3) 6= 12 + 6 18. Bill is on the 18th floor now. The answer 18.
The floor
now
Response (round 5)
Let us solve it using algebra. Let x be the floor Bill is on now. The floor he is now is 4x 6. We know that he starts the 3rd floor, so 3. Therefore, the floor he Bill is on the 18th floor now. The answer is 18. /
now 4(3) 6 12 18.
Table 3: Tracking responses to problem id=253 from the GSM8K test set, across rounds of a Boosting (A) run. The model always sets up the right expression (Ar + 6), but in the first two rounds reports I as the answer instead of substituting it into the expression. The correct substitution is done in round 4, and the correct answer is maintained in round 5.
28

---

Question
The Kennel house keeps 3 German Shepherds and 2 Bulldogs. If German Shepherd consumes kilograms of dog food and bulldog consumes kilograms of dog food per day. How many they need a week?
kilograms of dog food
Response (round I)
Let's solve this problem using algebra.
Step I: Find the total amount of dog food consumed by the German Shepherds in a week:
* German Shepherds: 3 dogs
* Dog food consumed per day: kilograms
Dog food consumed in week: dogs 5 kilograms/dog 15 kilograms
Step 2: Find the total amount of dog food consumed by the Bulldogs in week
Bulldogs: dogs
Dog food consumed per day: killgrams
Dog food consumed in week: dogs kilograms/dog kilograms
Step 3: Find the total amount of dog food consumed in week:
kilograms 21 kilograms Therefore, the kennel house will need 21 kilograms of dog food in week.
Total dog food consumed: 15 kilograms
The answer 21.
Response (round 2)
Let's calculate the total amount of dog food needed in week.
**German Shepherds:
Number of German Shepherds:
Dog food consumption per day: kg
Dog food consumption a week: = kg
BBlldogss
Number of Bulldogs: 2
Dog food consumption per day: kg
Dog food consumption a week: 7 kg
*otat dog food consumption in week:
Total dog food consumption in a week: 21 kg 14 kg 35 kg
Therefore, the kennel house will need 35 kilograms of dog food in week.
The answer 35.
Response (round 3)
Let's solve this problem using algebra.
**German Shepherds:
Number of German Shepherds:
Dog food consumed per day: kilograms
Dog food consumed in week: 7 21 kilograms
*Bulldogs:*
Number of Bulldogs: 2
*
Dog food consumed per day: kilograms
* Dog food consumed in week: 7 14 kilograms
Totaa dog food consumed a week:**
Total dog food consumed in week: 21 kilograms 14 kilograms 35 kilograms
Therefore, the Kennel house will need 35 kilograms of dog food in week.
The answer 35.
Response (round 4)
Let's calculate the total dog food needed per day: 15 kilograms
German Shepherds: 3
Bulldogs: * 3 6 kilograms
Total dog food needed per day: 15 + 6 kilograms
Now, let's calculate the total dog food needed a week:
7 days 21 kilograms/day = 149 kilograms
Therefore, the kennel house will need 149 kilograms of dog food in week.
The answer is 149.
Response (round 5)
Let's calculate the total dog food consumption per day:
German Shepherds: 3 5= 15 kilograms
Bulldogs: * 6 kilograms
* Total: 15 + 21 kilograms
Now. let's calculate the total dog food consumption in week:
7 days 21 kilograms/day = 147 kilograms
Therefore, the Kennel house will need 147 kilograms of dog food in a week. The answer is 147.
Table 4: Tracking responses to problem id=1266 from the GSMSK test set, across rounds of Boosting (B) run. The round response gets the correct amount per day (21), but does not multiply to the correct value for the week. Rounds 2 and 3 fail to incorporate the amount of food eaten per dog. Round 4 gets the correct approach, but fails the final multiplication step, which is corrected in the final round.
29

---

D Prompts
D.1 GSM8K
As expert problem solver solve step by step the following mathematical questions.
Q:
Tina makes $18.00 an hour. If she works more than 8 hours shift, she is eligible for overtime, which is paid by your hourly wage 1/2 your hourly wage. If she works 10 hours every day for days, how much money does she make?
A:
Here's 's how calculate Tina's 's camiing:
* *Regular Time:**
Hours per shift: 8 hours
Wage per hour: $18.00
Regular per shift: 8 hours $18.00/hour $144.00
* *Overtime:**
Overtime hours per shift: 10 hours- 8 hours = hours
Overtime pay per hour: $18.00 ($18.00/ 2) $27. $27.00
Overtime pay per shift: 2 hours $27.00/hour $54.00
* "Total per day:**
Regular pay + overtime pay: $144.00/shift $54.00/shift $198.00/day
**Total for days:**
days $198.00/day $990.00
**Therefore, Tina will make $990.00 in 5 days.** **
The answer is 990.
Q:
Abigail is trying new recipe for cold drink. It uses 1/4 of cup of iced tea and and 1/4 of a cup of lemonade make one drink. If she fills pitcher with 18 total cups of this drink, how many cups of lemonade are the pitcher?
A:
## Ambiguity in the Problem Statement:
There one main ambiguity the problem statement:
**Total volume vs. Number of servings:** The statement "18 total cups of this drink" could be interpreted in two ways:
* cups of the combined volume
18 individual servings
Let assume the interpretation "18 cups of the combined volume".
# Solution assuming 18 cups of combined volume:
30

---

*Step I: Find the proportion of lemonade one drink:**
Lemonade: 1/4 cups
Iced lea: 1/4 cup
Total:: 1/4 =/4= 1/2 cups
Lemonade proportion: 1/4) 1/2) = 5/6
**Step 2: Calculate the amount of lemonade in the pitcher:**
Total volume: 18 cups
Lemonade proportion: 5/6
Volume lemonade: 18 (5/6) cups
Therefore, there are 15 cups of lemonade in the pitcher.
The answer is 15.
Q:
A deep-sea monster rises from the waters once every hundred years to feast on ship and sate its hunger. Over three hundred years, it has consumed 847 people. Ships have been built larger over time, each new ship has twice as many people the last ship. How many people on the ship the monster the first hundred years?
A:
Let solve using algebra. Let be the number of people the ship the monster ate in the first hundred years.
The number of people on the ship eaten the second hundred years 2x, and in the third hundred years is 4x.
Therefore, the total number of people eaten over three hundred years x 2x 847.
Combining like terms, we get 7x 847.
Dividing both sides by we find 121.
Therefore, there were 121 people on the ship the monster ate the first hundred years.
The answer is 121.
Q:
{problem}
A:
D.2
MBPP
are an expert Python programmer. Solve the following Python programming problems.
Q:
Write function to find the similar elements from the given two tuple lists. Your code should pass these tests:
assert simi atmilar_elemeea ele.eetttennt.... (3., 4, (5, 7, 4, 10)) == (4, 5)
sesert similar elenenta 2, 3, (5, 4, 3, 7)) (3,
sinilar elements((11, 12, 14. 13), (17, 15, 14. 13)) (13, 14)
A:
31

---

def similar_ elenente tupl, teat__tu2):: (uple(eetttest__uug set (ttess__tup)) return (rea)
O: Write python function identify non-prime numbers. Your code should pass these tests:
is_not_prine(() False
assert in is_not__ptiiie pot..prinn(1)) True
is po_.iot.priie88
import math def is_not_primees result for in int (g))(()) 1): if n 0: result True return resul:
Q:
Write function to find the largest integers from given list of numbers using heap queue algorithm. Your code should pass these tests:
assert gaap__ueue_largese [25, 35, 22, 85, 65, 75,
22,
68] .3) [86,
76,
66]
assert heap_queue. iurguee [25, 35, 22, 85, 14, 65, 75, 22,
58] (2) [855, 76]
[85,
sesert geepp_quee__argeee [25, 22, 14,
22, 58]
-- [85,
65, 58,
A:
import heapq def beap_.quee__ largest (ouns,u): largest_nuna hq. mlargest nuns) return largest_nums
Q:
{problem}
Your code should past these tests: {tests}
A:
E
Additional Discussion
Weighting Examples Theorem 5 requires a > 0 and > O, and it is easy to see that both assumptions are necessary. If 0 then each training D, contains no exogenously labeled data, and if 3 = 0 then all of the exogenously labeled data could be incorrect. In either case, and given worst-case initial LLM go that returns an incorrect response to every prompt, the learner has no way to determine the correct response to any prompt.
The astute reader will note that weakening the convergence guarantee. This curious property consequence of an idealized assumption that we made for the sake of analytical tractability. Specifically, learner can match the conditional response distribution of every prompt in the training data, no matter how infrequently the prompt appears in the data (see Definition []. In practice, constraints on training time and model size will prevent a learner from perfectly fitting the training data. So would be useful extend our results account for the possibility of imperfect learner, and expect that any such extension would imply non-zero lower bound on a. Nonetheless, our current results tell us something interesting computational limitations are the only barrier learning an arbitrarily good LLM, and not, as one might expect, the quality of the weak labeler.
also be arbitrarily close zero in Theorem without
assumed that the
Filtering Non-Synthetic Data Algorithm 2 has the property that it only applies filtering on L.MM- generated data. As discussed, this accurately models existing methods in the literature.
32

---

However, if we consider applying the quality function q on data produced by the weak labeler (that is, data that is not LLM-generated), then there is an alternate solution to the data generation problem. can be shown that ( //e)/ invocations of the weak labeler would suffice to correctly label all but e fraction of the prompts in P. and such a dataset could be given to a strong learner to produce an LLM that achieves (ss) error. It is worthwhile to reason about why such solution cannot be deployed in practice.
First, and most crucially - it has been repeatedly demonstrated empirically in the literature that training directly on 100% correct human responses is sub-optimal compared to training on self- generated synthetic data (SCA* 224, (WWGG22) Indeed this is confirmed in our own experiments where we find that finetuning on correct answers (Gold SFT) is dramatically outperformed by recursive training, hence motivating our theoretical study of the problem.
The weak data assumption specifies that 33 while arbitrary, is bounded away from zero. Just as the weak learning assumption might not hold in classical boosting, the weak data assumption might not hold in our setting. We argue that iteratively filtering the weak labeler's output should result in precipitous drop in the fraction of correctly-labeled examples. As an example, suppose human labelers provide good responses the 81 easiest coding prompts in some prompt set. One should expect that asking similarly-qualified labelers to respond to the remaining prompts results in a B2 < 81 yield of quality responses, as all but the easiest prompts have been answered. In contrast, a continually improved LLM endows human with more flexibility for future responses, such as rewriting nearly high-quality solutions provided by the last iteration of LLM, making a non-vanishing 3 a much more reasonable assumption.
While it keeps the setting simple presume that @ be evaluated on any labeled example, this is an overly permissive assumption. LLM-generated synthetic data can be made to include reasoning traces, and often produces responses that the LLM itself can verify as high quality. This facilitates the construction of automated quality checkers, which are much more difficult to construct when the labels are produced by a human, and therefore contain reasoning traces and responses that are unfamiliar to the current generation of LLM. This is born out in the literature, where quality verification of LLM-gennrated synthetic data is relatively easy to implement [SCA $ 24 YPF 24 ZWMG22).
Finally, and somewhat remarkably, Algorithm 2 with a = c achieves the same finite-time error rate as this baseline while only ever evaluating the quality of L.M--geneeteed data. Thus, the approach taken in practice matches the convergence rate that would be experienced under a much more powerful set of assumptions.
F Additional Experimental Results
In this section, we provide additional experimental results. Specifically, we conduct the following experimente experiments:
Off-policy boosting: We fixed the learner to be Gemma 2 2B and switched out the labeler from Gemma 2B (on-policy) to either Gemma I 2B or Gemma 7B, The stronger off-policy Gemma 7B model outperforms the on-policy Gemma 2 2B model. Results are pictured in Figure The weaker Gemma 2B significantly underperforms on-policy Gemma 2B, but after 5 rounds roughly approaches the performance of filter only. Note that the Gemma 2B model achieves only 11 % on GSM8K, and based on our theory, we should expect to require more iterations with weaker labeler.
Running longer and observing improvement plateau: In several prior works, it was reported that improvement plateaus after a few rounds. We run some experiments longer (>5 rounds). Indeed, in Figure 6 we see limited improvements in test accuracy (<2%) from additional rounds, despite persistent improvements training accuracy.
33

---

GSMBK test performance
GSMBK train performance
0.
0.-
This vouney d L998
0.6
0.3
0.2
9.4
01
0.0
0.0
found
Bound
Bosuling (on-swlko))
Boosking (on-policy)
Gord SFT
Buosting Gumma 75 labeler
GLtd 5F1
Brosking Gemma 7B lascker Brosking wi Gemmra 28 lanieer
Filter
Boosting Genmaa 28 labeler
Ritee only
Figure 5: We experiment with off- aff-policy labelers GSM8K, plotting performance across rounds. Boosting (on-policy) is the setting in all prior experiments, employing Gemma 2 2B PT as the labeler. We improvement from using the stronger Gemma 7B labeler. The weaker Gemma I 2B performs much worse, but approaches the results from Filter only after 5 rounds.
GSMBK test performance
0.40
accurac)
0.35
0.30
0.25
Round
Filter only
Gold
Doosting
GSMBK train performance
0.8
accuracy
0.2
2
4
9
PT
only Boasting
Gold SFT
Figure 6: We plot test and train performance of Filter only and Boosting GSM8K across more rounds (10). We see limited improvements test accuracy (222% despite persistent improvements in train accuracy.
34


Reviews:
[{"review_id": "5TW64HUjLa", "reviewer": "Reviewer_drWH", "summary": "This paper presents an effort to place the popular practice of LLM self-improvement—through rounds of synthetic data generation/pseudo-labelling, filtering, and fine-tuning—within a unifying theoretical framework. By inverting the traditional boosting framework from learning theory—shifting from weak learners with strong data to a setting with a strong learner and weak data, reflective of the current state of LLMs—they introduce a boosting-style algorithm that, upon convergence, offers probabilistic guarantees of producing correct responses to nearly all prompts. Experimental results display the effects of the combined algorithmic components on train and test error on mathematical and coding benchmarks, while further experiment in the appendix aims to empirically assess some relaxations of assumptions to the framework.", "questions": "- The impact of boosting towards the achievable test error remains unclear. Even though this is not set forth as a core concern of the paper it would be helpful for the reader that the authors seek a better explanation of the observed results. Moreover, can removing the focus on hard examples allow faster improvement over the first few rounds (as hinted in Fig.1 left), hence potentially be preferable under smaller computation budgets?\n\n- I think a valuable addition would be to explicitly compare the existing frameworks mentioned in the paper (STAR, REST, REST-EM) with the proposed framework, highlighting missing components or key variations. Moreover, including these frameworks as part of the experiments would provided useful baselines and expose which stages of the algorithm result in making a difference in practice.\n\n- A useful addition would be some further studies on the impact of the involved hyperparameters (e.g., the number of samples $k$, and fractions $\\alpha$ and $\\beta$).", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe paper comes up with an insightful theoretical framework for self-improvement and some theoretically-backed design principles for further developments in the area. The proposed boosting style algorithm is general enough to encompass commonly used frameworks and provides one of the first guarantees of convergence for self-improvement. Moreover, it allows diagnosing weaknesses of existing frameworks.\n\n**Weaknesses**\n\nThe work rests on some strong assumptions for the involved LLMs. Most notably the derivation assumes that the LLM can be considered as a strong learner powerful enough to interpolate a given conditional probability (which might be unrealistic in practice). Moreover, a perfect filter is also assumed. Thirdly, the verifier is considered sufficiently powerful to extract a (potentially tiny) fraction of high quality labels, which can be realized programmatically for some learning task categories, however again this might be facing limits in the wild. Empirical findings provide a mixed signal for the advantage that focusing on hard examples gives.", "was_revised": true, "final_justification": "Thank you for following up. My assessment remains the same based on the provided clarifications.\nMy main remaining point is the paper to be revised towards a better integration of the main part with the considerations/findings on weakening the assumptions for the learner and filtering, already attempted in the appendix.", "rebuttal": "We thank the reviewer for their positive assessment of our paper. We appreciate their recognition that (paraphrasing from their review) our algorithm is general enough to encompass existing frameworks, enabling diagnosis of their weaknesses, and provides one of the first guarantees of convergence for self-improvement.\n\nWe address their concerns and questions below:\n\n**Weakness:** *The work rests on some strong assumptions for the involved LLMs. Most notably the derivation assumes that the LLM can be considered as a strong learner powerful enough to interpolate a given conditional probability (which might be unrealistic in practice).* \n\n- **Response:** While our theory assumes an idealized strong learner, in our experiments we used a real learner instead of a strong learner, and our algorithm outperformed several baselines. So we think we have demonstrated that our theory is practically relevant, as it led to the development of a high-performing algorithm. Furthermore, our assumption is motivated by the widely observed phenomenon that LLMs memorize their training data. We agree that further weakening of the assumption would be an interesting extension of our theory, and view spurring research in that direction as a primary benefit of publishing our paper. \n\n**Weakness:** *A perfect filter is also assumed.*\n\n- **Response:** In fact, we relax this assumption in Appendix A.2 in the supplement, where we show that our results straightforwardly extend to a setting where the filter is imperfect.\n\n**Question:** *I think a valuable addition would be to explicitly compare the existing frameworks mentioned in the paper (STAR, REST, REST-EM) with the proposed framework.*\n\n- **Response:** In fact, we explicitly compare to STaR/ReST. See line 275.\n\n**Question:** *Can removing the focus on hard examples allow faster improvement over the first few rounds (as hinted in Fig.1 left), hence potentially be preferable under smaller computation budgets?*\n\n- **Response:** We agree with the reviewer that the benefit of focusing on the hard examples appears to mostly be realized in the later rounds of our algorithm, and thus might only be worthwhile when computational resources are adequate.\n\n**Question.**: A useful addition would be some further studies on the impact of the involved hyperparameters (e.g., the number of samples $k$, and fractions $\\alpha$ and $\\beta$).\n- **Response:** As $\\beta$ is a quantity determined by the quality of the weak labeler, we implicitly vary this parameter in our *off-policy* experiments in Figure 7 of the Appendix, where we experiment with different labelers and find faster improvement with larger $\\beta$ as the theory predicts."}, {"review_id": "bVzOvuhigP", "reviewer": "Reviewer_Ckek", "summary": "The paper formalizes “training on synthetic data with minimal curation” as a boosting-style process. Experiments fine-tune Gemma-2 2B on GSM8K and MBPP. “Boosting” improves training accuracy and slightly increases test accuracy compared with (a) “do nothing” and (b) “filter-only” baselines, but the gains over the best supervised baseline are small", "questions": "1. In practice the learner minimizes cross-entropy for limited epochs. How sensitive is convergence if the model only approximates the empirical distribution?\n2. What is the wall-clock time and GPU budget per round for k = 8 on GSM8K? How would this scale to a 70B model on 1M prompts?\n3. Does the algorithm still converge if the labeler’s errors are systematic (biased) rather than random?", "limitations": "See above.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* Unifying formalism that connects recent self-improvement heuristics (STaR / ReST) to classical boosting.\n* Empirical section reproduces the model-collapse phenomenon and shows the proposed variant can mitigate it.\n\nWeakness:\n* The proof requires every generation to exactly fit the weighted dataset (Definition 1). This is unattainable for billion-parameter LLMs trained for a few hundred steps, undermining the practical relevance of Theorem 6.\n* Experiments simulate the labeler with an already pretrained Gemma 2B, i.e. they assume access to an LLM stronger than the initial model. No evidence is given that a genuinely weak.\n* No error bars, confidence intervals, or significance tests are reported.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their engagement with our paper. We appreciate their recognition that (paraphrasing from their review) we provide a unifying formalism that connects recent self-improvement heuristics (STaR / ReST) to classical boosting, and that our experiments reproduce the model-collapse phenomenon while showing how our proposed variant can mitigate it.\n\nWe respond to their concerns and questions below, **nearly all of which seem to be based on misunderstandings**:\n\n**Weakness:** *Boosting improves training accuracy and slightly increases test accuracy compared with (a) do nothing and (b) filter-only baselines, but the gains over the best supervised baseline are small.*\n\n- **Response:** In fact, the gains are large. Table 1 (“GSM8k test” column) shows that boosting leads to substantial improvements in test accuracy over (1) do nothing baseline (29.3% absolute, 191.5% relative); (2) filter-only baseline (7.5% absolute, 20.2% relative); (3) supervised baseline, i.e., Gold SFT (7.7% absolute, 20.9% relative).\n\n**Weakness:** *Experiments simulate the labeler with an already pretrained Gemma 2B, i.e. they assume access to an LLM stronger than the initial model. No evidence is given that a genuinely weak...*\n\n- **Response:** In fact, none of our experiments use a labeler that is stronger than the initial model. In Section 7, both the initial model and labeler are Gemma 2 2B PT, which gets a test accuracy of 21.9%. In Appendix F in the supplement (Figure 7), the initial model is also Gemma 2 2B PT, but the labeler is Gemma 1 2B PT, which gets a test accuracy of 11%. We see improvements across rounds compared to the initial model (red dotted line). This demonstrates the success of our approach with genuinely weak labeler.\n\n**Weakness:** *No error bars, confidence intervals, or significance tests are reported.*\n\n- **Response:** We do report them. In Appendix F in the supplement (Table 4 and Figure 5) we report mean and standard deviation for all results in the main text (Table 1 and Figure 1). We ran 3 trials per experiment, which is more trials than the STaR/ReST_EM papers.\n\n**Weakness/Question:** *The proof requires every generation to exactly fit the weighted dataset (Definition 1) ... How sensitive is convergence if the model only approximates the empirical distribution?*\n\n- **Response:** While our theory assumes an idealized strong learner, in our experiments we used a real learner instead of a strong learner, and our algorithm outperformed several baselines. So we think we have demonstrated that our theory is practically relevant, as it led to the development of a high-performing algorithm. Furthermore, our assumption is motivated by the widely observed phenomenon that LLMs memorize their training data. We agree that further weakening of the assumption would be an interesting extension of our theory, and view spurring research in that direction as a primary benefit of publishing our paper. \n\n**Question:** *Does the algorithm still converge if the labeler’s errors are systematic (biased) rather than random?*\n\n- **Response:** We do not assume that the labeler’s errors are random. In Section 4.2 (line 181) we say that they can be arbitrary.\n\n\n**Question:** What is the wall-clock time and GPU budget per round for k = 8 on GSM8K?\n\n- **Response:** For GSM8K, and k=8, we sample 112k responses each round, which dominates the accelerator time compared to training, and requires approximately ~16 H100-equivalent hours. Scaling up to a 35x larger model and 142x larger dataset would suggest a 5000x increase in required compute, applying the flops = 2 * params * tokens estimate (assuming inference is batched enough to be compute bottlenecked)."}, {"review_id": "Q9J6CENoHO", "reviewer": "Reviewer_QYsv", "summary": "This paper addresses \"model collapse\" in LLMs trained on synthetic data by proposing a theoretical framework inspired by \"boosting\". It argues a \"strong learner\" (a powerful LLM) can continuously improve using only \"weak data\"—a minimal, external signal. The proposed algorithm iteratively identifies prompts where the model fails (\"hard examples\") and focuses a \"weak labeler\" on them. This feedback is mixed with high-quality synthetic data for retraining. The work proves this method prevents collapse and ensures convergence to an optimal model, outperforming baselines in experiments.", "questions": "N/A", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper addresses the critical and urgent problem of \"model collapse\" when training on synthetic data for the sustainable development of Large Language Models (LLMs).\n2. The interesting theoretical framework offers a deep and insightful explanation for why the method is effective.\n3. The experiments are well-structured to validate the theoretical claims.\n\nWeakness:\n+ The framework assumes a binary (good/bad), unambiguous, and efficiently evaluable quality function q. This is true for the tested tasks of math and coding, but for more subjective tasks like creative writing or summarization, quality is nuanced and hard to define, limiting the direct applicability of the theory.", "was_revised": true, "final_justification": "My concerns are resolved.", "rebuttal": "We thank the reviewer for their positive assessment of our paper. We appreciate their recognition that (paraphrasing from their response) our paper addresses a critical and urgent problem, offers a deep and insightful explanation of our method, and contains experiments which validate our theoretical claims.\n\nWe respond to their main concern below.\n\n**Weakness:** *The framework assumes a binary (good/bad), unambiguous, and efficiently evaluable quality function q. This is true for the tested tasks of math and coding, but for more subjective tasks like creative writing or summarization, quality is nuanced and hard to define, limiting the direct applicability of the theory.* \n\n- **Response:** In fact, we relax this assumption in Appendix A.2 in the supplement, where we show that our results straightforwardly extend to a setting where filtering according to the quality function can only be performed imperfectly."}, {"review_id": "7Y5z93hsx1", "reviewer": "Reviewer_BnJF", "summary": "In this paper authors introduce a theoretical framework on understanding an important phenomenon - model collapse - as well as ways of escaping it. The authors connect the ideas of weak data and strong learner with a classic ML algorithm - boosting. Experiments validate theoretical results.", "questions": "For the filter-only setup, is the model trained on less samples (than w/ boosting) since we filter the bad ones out?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- Well-written and solid theoretical grounds.\n- Fully-accessible model and datasets which makes it fully reproducible.\n- Experimental results are in line with the theoretical analysis.\n- Two datasets - GSM8k (math) and MBPP (coding). \n\n### Weaknesses\n\n- When T increases it seems from the experimental data (Appendix F) after T>5 improvements are diminishing. No explanation or analysis is given.\n- Only one model (Gemma2 2B) and only one prompting method (when fine-tuning) - 3-shot prompting.", "was_revised": true, "final_justification": "Thank you for providing additional clarifications. I will hold to the rating of 5.", "rebuttal": "We thank the reviewer for their positive assessment of our paper. We appreciate their recognition that (paraphrasing from their response) our paper is well-written, theoretically solid, and contains experiments that confirm the theoretical analysis.\n\nWe respond to their concerns below.\n\n**Weakness:** *Only one model (Gemma2 2B) and only one prompting method (when fine-tuning) - 3-shot prompting.*\n\n- **Response:** In Appendix F in the supplement (Figure 7), we experiment with other models, including Gemma 1 2B and Gemma 7B.\n\n> For the filter-only setup, is the model trained on less samples (than w/ boosting) since we filter the bad ones out?\n\nNo, in all settings considered, there are more correct examples than (training steps * batch size). The difference between the setups is not in the quantity of examples, but the source (either from the weak learner or from the model itself)."}]

Author rebuttal:

Meta-review: Paper studies the use of synthetic data to iteratively improve an LLM. Specifically they study how much curation needs to be applied on the synthetic data for the model to not collapse and continually improve. They study an algorithm which subsumes known successful methods and provide a novel analysis inspired by classical boosting literature. Under certain assumptions they show that focusing the synthetic labeling on challenging samples leads to continual improvement. These allow the authors to propose improvements to existing methods.

Most reviewers considered the contributions important, novel and well supported by promising empirical results. Reviewer drWH and Ckek pointed out that the assumptions are too strong, but the authors responded that their empirical performance is evidence that their theory is useful. Reviwer QYsv brought up that proposed method may only work in domains like math and coding where the correctness of a label is exact. Authors point to Appendix A.2 which relaxes this assumption in theory, but doesn't any provide empirical validation. Reviewer BnJF raises that performance drops for T>5 number of model iterations, but authors do not provide any explanation. Authors are encouraged to provide more experiments with diverse set of datasets and longer iterations in the next revision.
Decision: Accept (poster)