Spend Wisely: Maximizing Post-Training Gains in Iterative Synthetic Data Bootstrapping
Authors: Pu Yang
Yunzhen Feng
Ziyuan Chen
Yuhang Wu
Zhuoyuan Li
Venue: NeurIPS 2025 spotlight ()
Keywords: synthetic data
self supervised fine-tuning
iterative learning
boostrapping
budget allocation
Primary area: theory
TL;DR: We analyze optimal budget allocation strategies for iterative bootstrapping with synthetic data to maximize model performance.

Abstract: Modern foundation models often undergo iterative ``bootstrapping'' in their post-training phase: a model generates synthetic data, an external verifier filters out low-quality samples, and the high-quality subset is used for further fine-tuning. Over multiple iterations, the model performance improves, raising a crucial question: How should the total budget for generation and training be allocated across iterations to maximize final performance? 
In this work, we develop a theoretical framework for analyzing budget allocation strategies. Specifically, we show that constant policies fail to converge with high probability, while increasing policies---particularly exponential growth policies---exhibit significant theoretical advantages. 
Experiments on image denoising with diffusion probabilistic models and math reasoning with large language models show that both exponential and polynomial growth policies consistently outperform constant policies, with exponential policies often providing more stable performance.

Spend Wisely: Maximizing Post-Training Gains in Iterative Synthetic Data Bootstrapping
Pu Yang" Peking University
Yunzhen Feng" New York University
Ziyuan Chen' Peking University
Yuhang Wu
Zhuoyuan Lit National University of Singapore
UC Berkeley
Abstract
Modern foundation models often undergo iterative "bootstrapping" in their post- training phase: model generates synthetic data, external verifier filters out low-quality samples, and the high-quality subset is used for further fine-tuning. Over multiple iterations, the model performance improves, raising crucial ques- tion: How should the total budget for generation and training be allocated across iterations maximize final performance? In this work, we develop theoretical framework for analyzing budget allocation strategies. Specifically, we show that constant policies fail converge with high probability, while increasing policies particularly exponential growth policies- -elibies-exhibit exhibit significant theoretical advantages. Experiments on image denoising with diffusion probabilistic models and math reasoning with large language models show that both exponential and polynomial growth policies consistently outperform constant policies, with exponential policies often providing more stable performance.
Introduction
Supervised fine-tuning is critical stage in training large language models (LLM). Pre-training supplies them with broad linguistic and factual knowledge, but task-specific skills-such as tool use (Schick al. 2023), reasoning pattern (Gandhi et al. (025). and agentic behavior (Shao et al.) 2023)-mergge when the model further refined on carefully curated, supervised examples. Securing these high-quality human-annotudee data, however, remains major bottleneck, as it requires domain expertise and substantial resources.
To address this limitation, synthetic data have emerged as promising alternative that offers scalability and cost-effectiveeess. Despite concerns about potential risks of model collapse (Shumailov et al.. 2024, Dohmatob et al. (2024b), synthetic data is typically selected and verified before use in post- training (Feng et al.J 2024 Setlur et al., 2024). ensuring its quality. A common paradigm for leveraging synthetic data employs an iterative bootstrapping process (Zelikman al. 2022 Trung jet al. 2024): the model generates synthetic data, rewards or verifiers are used to filter and select high- quality data, and the model is then fine-tuned selected data. This process is repeated iteratively to fully improve performance. An illustration of this approach is provided in Figure I
However, for practitioners who implement this approach, an important question arises: How should fixed computational budget be allocated to decide the amount of synthetic data to generate and select at each iteration to maximize performance?
"qqual contribution.
Corresponning Author. zy. 110nus edu. sg
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Generator f (,,0(0)
fine-tune
selected dataset { 1M
74
Policy
generate
the t-th iteration
2; P 669
How to set across itcrations?
p: Rize)
generuted dataset
R
D, {
p=1 R((s)
reward model
Figure 1: Iterative learning with synthetic data. In this framework, synthetic data is generated, filtered using a reward model, and the selected data used further train the generator. The budget policy is defined as the quantity of data retained after selection, 74. Our goal is to identify the optimal policy across iterations to achieve the best final performance, given fixed budget.
In this paper, we establish some foundational principles for crafting optimal strategies for synthetic data generation across iterations. To the best of our knowledge, this is the first attempt address this problem. We begin with theoretical analysis of policies that control the amount of selected synthetic data used each iteration, as illustrated in Figure [ simplified setting with Gaussian data and exponential reward functions, we could identify the optimal policy. Contrary to common strategies that maintain fixed number of synthetic data across iterations (constant policies), our analysis shows that the optimal strategy requires exponentially increasing the amount of data at each iteration.
Furthermore, in more general settings with only mild assumptions about regularity, we demonstrate that constant policies fail to converge with high probability. In contrast, policies that increase converge optimal performance. Among such
the amount of synthetic data across iterations increasing policies, we prove that exponential growth policies guarantee exponential convergence, and in the worst-case scenarios, they achieve no larger computational costs compared to polynomial growth policies when attaining similar near-optimal performance.
Building these theoretical insights, validate findings with two experiments: image- denoising task using diffusion probabilistic models (DPMs), and math math-reassonng task with large language models (LLMs). Across these experiments, exponential and linear (polynomial) growth policies outperform constant policies. Moreover, the best exponential policies match or exceed the performance of linear policies in every case, confirming their theoretical robustness.
We summarize our contribution as follows.
We formulate the problem of optimizing post-training gains under limited budget for iterative learning in Section 3.1
In solvable Gaussian setting, we prove the optimality of an exponential growth policy in Section 3.2 and validate through numerical simulations.
In more general setting, we demonstrate in Section that constant policies fail to converge to the optimal reward with high probability, while increasing policies ensure convergence. Among these, the exponential growth policy achieves an exponential convergence rate and outperforms polynomial growth policies in the worst case.
Experiments on image denoising (diffusion probabilistic model) and math reasoning (large language model) in Section outperform constant policies, with exponential policies exhibiting greater stability.
[ confirm that exponential and polynomial growth policies
Related Work
Synthetic Data and Iterative Bootstrapping. The iterative bootstrapping framework, illustrated in Figure [ is widely used for training foundational models both text and vision domains. In this approach, synthetic data are generated and then iteratively refined filtered to produce new

---

training sets that more closely align with the target objectives. For large language models (LLMs) and vision language models (VLMs), this approach has been applied to instruction following (Touvron
et al.
2023) and alignment tasks (Dong et al.) 2023), and vision question answering YYng & Dong
2025 wherein reward models filter synthetic data to retain only high-quality samples. Recently,
has been particularly effective in improving reasoning abilities, as first demonstrated in Zelikman
ict al. (2022). Subsequent works (Lehnert et al.) 2024] Su et al. 2024; Trung et al. 2024; Guan
al. 2025 Singh et al. 2025) have extended this approach to generate diverse reasoning paths with search algorithms. Beyond text, iterative bootstrapping has also been utilized to generate training data
for image segmentation (Kirillov al. 2023), create image- pairs (Fan al.
2023 2024), and
synthesize images to address distribution shifts (Hemmat et al. 2024, Azizi et al. 2023). However, in
these empirical works, principled method for controlling generation each iteration is still missing.
To our knowledge, Ferbach et al. (2024) provides the only theoretical analysis of this paradigm, proving convergence to the optimal policy under the infinite-sample assumption. Crucially, their framework fails characterize finite sample requirements, offering little actionable guidance to prac- titioners implementing this framework. We address this gap by establishing optimal computational budget allocation strategies across iterations to achieve maximum performance gains.
Model Collapse. A series of papers has shown that the incorporation of synthetic data into the training corpus can lead to performance degradation, either single training instance iteratively over time (Shumailov et al. 2023; Alemohammad et al. 2024, Dohmatob et al. 2024b a 2025
Feng et al. (2024) has explored the use of data selection overcome model collapse by employing synthetic data in a single generation, combined with verification. In our work, we also leverage verification, but focus on addressing the question of how to optimally generate synthetic data during learning.
3 Problem Formulation and Preliminary Analysis
In this section, we formalize the iterative learning setting in Section 3.1 and then analyze optimal budget allocation strategies with simple Gaussian case Section 3.2
3.1 Problem Formulation
The setup is illustrated in Figure Our goal is to train a parameterized generative model f(=;0), for example, given by DPM or LLM, that generates data x ~ Po. A reward model R evaluates the quality of the generated data, serving as a measure of the model performance. We aim to train the generative model maximize the expected reward
r(0) Eg [R(x)].
where we assume R(x) E [0, 1].
The iterative training process leverages the generative nature of the model perform self-generation and refinement based reward signals. Specifically, start with initial model g(0), At the (f + 1)-th iteration, the model undergoes improvement through the following three steps:
Generation step: Generate N, data by f(; ((1)).
Selection step:
For each synthetic data point x, include it into dataset D, with probability R(x). Suppose of the N₂ data are selected to form the dataset D,, formally, [Del ne-
Updating step: Update g(t) g(++1) with the dataset D.-
This iterative approach generates additional training data using synthetic samples, with the reward- guided selection process ensuring its quality. The selected samples are then used to further enhance the model's 's generative performance (Feng et al.) 2024). This process, carried out iteratively to maximize performance gains, is particularly effective in scenarios where high-quality specialized data is scarce, for example, in mathematical reasoning (Zelikman et al. 2002 Guan et al.) 2025) Singh et al. 2025). The algorithm framework is formalized in Algorithm | In the selection step, the data may be selected with noise. We present simple form here for ease understanding, while our
results extend to the noisy case in Appendix B.5
3

---

When practitioners use this algorithm, a key challenge is determining how to set Nr and across iterations. This decision is particularly important, as the generation process of foundation models unually involves multiple forward propagations, and consequently, the generation cost often exceeds the training cost. Given limited budget, it is crucial design the process in a way that maximizes post-training gains.
Specifically, we focus on identifying the optimal policy of N2 and n, correlated with the generation cost and the training cost, respectively. Given the reward model and generator, .N, can be viewed as random variable dependent on ny and the selection rate. Therefore, we define the policy over ny as {na 1, and our objective is to identify the best scheme for
Algorithm Iterative learning with synthetic data
Input: An initial generative model f(=; (()))))), reward model R, and a policy (nehe with termination time T.
2: Output: A fine-tuned generative model f(: B(T)).
3: for t + 0 T - do
4: Initialize selected synthetic dataset D;
5: while DD] < The do
6: Generate sample x ~ Parn.
7: Add D, with probability R(x).
8: end while
9: Update g(t) to ((++1) using the dataset D.-
10: end for
3.2
Gaussian Case Study
To begin, we present warm-up example with Gaussian data, where the optimal policy can be derived analytically. This example provides theoretical insight into general iterative learning framework. The setup includes Gaussian data, exponential reward function, and maximum likelihood estimation (MLE) as the updating algorithm.
Gaussian Generator and Exponential Reward Model. To simplify the setting, we consider the generative model as one-ooneeee one-dimensional Gaussian distribution Pe ((, o²). where the mean is the parameter to learn, and the variance o² is fixed. The reward model is an exponential function (x:)) exp (-x² (2xx)). where is also fixed.
Updating via MILE In Algorithm 1 we employ MLE as the updating algorithm at line 9. Specifi- cally, the parameter ((++1) updated in the t-th iteration is obtained by
g(++1) arg max, II po(x),
æED.
where po( is the density function of the distribution Po-
Optimization Problem. We aim to find the optimal iterative learning policy within this setup. We assume that the cost of sampling the Gaussian distribution is negligible, while the training cost for MLE is proportional
the size of the dataset, 74. The objective maximize the expected performance of the generator in iteration T', as measured by the reward. Given a fixed budget C, the optimal policy will be the solution
max [n+]7
Egcry [r [B(T))
s.t.
Σ 1=0
m
C.
(1)
For the above setup, we can theoretically identify the optimal policy in the following theorem.

---

constant
linear
exponential
2 =2
K = 4
Gap to Optimal Reward
Gap Optimal Reward
200
9-6
13K
139
Computational Cost
Computational Cost
Figure 2: Empirical results of the toy example. We compare the exponential, constant, and linear policy, and show the gap to the optimal expected reward as a function of the computational cost (E, nz). All the results are averaged over 1,000 runs.
Theorem 3.1. If the initial parameter satisfies ( S (1+02//22) n22//22 then the optimal iterative policy, i.e., the solution of the optimization problem in Equation (T. is given by
(1+ R
I+
n2 x
The proof deferred to Appendix A and extends naturally the high-dimensional case, detailed in that the optimal policy in this setting follows an exponential scheme. The optimal strategy follows this rationale: an iterative optimization algorithm, the contribution of an early update to the overall eтrоr decays exponentially with the total number of iterations T. As a result, achieving a low final error increasingly depends on the accuracy later updates. To keep those updates precise enough, each iteration must therefore draw an exponentially larger number of samples. We will further analyze this exponential scheme in the general analysis in Section 4
Appendix E. (..joremm...ddmoonssst
Experiments. We include an empirical experiment to confirm Theorem 3.1 with data in two- dimensional space. The generator is set to (N, I2). The reward function is R(x)) exp
which favors data concentrated near the origin. controls the flatness of the reward. Full details are deferred to Appendix E.2 Figure plots the gap to the optimal reward as function of the cumulative cost, defined as the total computational cost incurred up that point. The results demonstrate that the exponential policy consistently approaches the optimal reward faster than the other schemes. While the linear policy converges slightly more slowly, the constant policy, as shown in the left figure, clearly fails to converge. These findings validate our theoretical proof. Additional results with various parameters are left in Appendix F.
Main Theoretical Results
We now proceed to analyze the general setting. We will show that constant policies fail to converge with high probability, whereas there exists an exponential growth policy that achieves an exponential convergence rate and outperforms polynomial alternatives in the worst case. The assumptions and setup are outlined below, while detailed proofs for the subsequent theorems are provided in Appendix B.4
Assumptions. We consider a general function class for the generative model, a general loss with regularity assumptions (Assumptions B. and B.2). and a reward model with mild assumptions about its relationship with the loss (Assumption B.3) The formal descriptions are provided in Appendix B The warm-up example in the previous section serves as a special case that satisfies all these assumptions. Our results also naturally extend to noisy rewards, as discussed in Appendix B.5
5

---

Updating via Gradient Descent. In Algorithm | we use gradient descent with a learning rate ? > as the update algorithm at line Specifically, the update for the t-th iteration writes
g
p(t) 17 The DED,
Σ
Vol (2; t= 0=0,,,...
where ((x; 0) the loss function. For any policy n, let (o0) denote the iterative trajectory of parameter.
Reward and Cost. Similarly to the warm-up example, we evaluate the generative model using the expected reward r(0). The optimal performance is the supremum of the expected reward over the whole parameter space
sup (0).
We consider general computational cost that accounts for both generation and training, with constant coefficients Cg and for each data point, respectively. Then, the total cost is given by
C(a,T)
Σ t=0
Nx + Cna.
The first policy class that consider is
constant policy: Teonst
72g = Ro. 0, 1, ....
This is the most standard policy, where can be set either as the total number of prompts in the dataset (Zelikman et al., 2022), or as a multiple of the batch size an online setting (Guo al., 2024). However, in the following theorem we show that constant policies fail to converge to the optimal reward with high probability.
Theorem 4.1 (Bounded Reward for Constant policy). Under Assumption B.3 there exists a constant
> 0 such that, for any T and any constant policy Teonst, with probability at least 1/4,
(T(T) Tecnest
cmo 1/2
The theory establishes the non non-convergence of constant policies, which stems from persistent random noise in the gradient. Specifically, the noise term is proportional to n. and remains non-decaying under any constant policy, resulting in suboptimal performance.
To address this limitation, we introduce increasing policies, where n, grows monotonically over iterations. The following theorem shows that increasing policies ensure convergence to the optimal reward with high probability.
Theorem 4.2 (Optimal Reward for Increasing Policies), Under Assumption B. for any increasing policy and > 0, there exists sufficiently large T such that, with probubility greater than
Assumption B.3
1 1--TT-
-r (O(T)) Se.
This theorem suggests that increasing policies should be used in practice to maximize performance. Hence, we consider the following two classes, namely,
polynomial growth policy: Polly 74 n1 1)o, exponential growth policy: Texp no (1 + w)²,
0, 1 0, 1,
0, ( = 0,
For the exponential policy, we can identify a specific policy that achieves an exponential convergence rate follows.

---

Table 1: A summarization of all experimental setups and implementations.
Experiment
Setup Type
Generator
Reward
Dataset
Image Denoising
Theoretical Validation
DPM
PSNR
MNIST Deng 2012
Math Reasoning
Practical Scenario
LLM
Accuracy
GSM-Symbolic
Mirzadch
2025
Theorem 4.3 (Convergence Rate for the Exponential Policy). Under Assumptions B.1 to exists an exponential growth policy Texp such that with probability at least (1-
B. there
- n
( (11+))
where Ç > O a constant defined in Theorem B.6
This theorem also applies to noisy rewards, as discussed in Appendix B.5
Notice that the above three theorems analyze the convergence of the respective policies (constant, polynomial, and exponential) as the number of iterations I grows without imposing any limit on the total computational budget. We now introduce the budget constraint and compare the exponential policy with the polynomial policy. Specifically, for any policy π,, we introduce
(T min {T [TIr- (
denoting the minimum number of iterations needed for the policy π to achieve a reward that is within of the optimal r". The cost (TT, 7 (T,, E)) is then the minimum cost attain the target performance of the policy т. The following theorem compares the total cost incurred by the exponential growth policy and polynomial policies in a worst-case scenario.
Theorem 4.4 (Worst-Case Optimality of the Exponential Policy, informal). For any constant or polynomial growth policy π and sufficiently small we have
sup C(Tixp: (TTxps <)) < sup C(H,T((T,C)
with probability at least (1-E Et-o (T n, a) Here, the supremum is taken over all feasible problem settings, including parameter spaces, loss functions, and reward functions under mild assumptions.
For the formal version, please refer Theorem B.7 This theorem shows that, in worst-case analysis, the exponential policy T from Theorem B.6 incurs a total cost no greater than that of any polynomial growth constant policy to reach specified performance threshold, establishing its efficiency and robustness. This analysis theoretically highlights the exponential policy as the preferred choice for our iterative learning tasks.
In summary, we draw the following conclusion:
Constant policies lead to rewards that are consistently below optimal by constant margin.
Increasing policies such as polynomial and exponential ones can achieve optimal performance.
In the worst case, there exists an exponential growth policy ensuring the same performance at equal or lower computational cost compared to any constant or polynomial growth policy.
Experiments
Our theoretical results show that, in iterative bootstrapping. synthetic data curation with exponential scheme outperforms both linear and polynomial schemes, while linear and polynomial schemes outperform the constant scheme. In experiments, evaluate constant, linear (as simplified polynomial scheme), and exponential schemes.
We conduct two experiments evaluate our schemes: image denoising with DPMs, and mathematical reasoning with L.MMs, covering both image and text domains. The setups and implementation details summarized in Table

---

cotallrk, ft, 1-B8
lincer, 1-
constors, 10-000000
Unner 36-17
exponcttial
100 8
linder, 100-8
443
PSNK
PSNNR
01-
620
0.00
6:50
Compussional Cost (FLOPs X
Computationl Cost (FLOPs. x10
Figure Empirical results of image denoising. The figure shows the average PSNR of the generated denoised images as a function of computational cost. Computational cost is measured in floating-point operations (FLOPs) during iterative learning, including both generation and training. The training batch size is set to B 640. We adopt n, 1.1* B and n, 1.05' B the exponential growth policies for = 10 and s = 20, respectively.
In these settings, we emphasize that our goal is not to propose a novel task-solving method or generation technique. Rather, we focus on optimizing the allocation generation and training costs in iterative learning with synthetic data-an objective that is method-agnostic and can be applied any generation approach. Here, validate our theoretical results from Section 4 and conduct performance comparison among exponential, linear, and constant schemes on image data (using DPMs) and text data (with LLMs).
5.1 Image Denoising with Diffusion Models
We consider applying iterative learning to the image denoising task, a representative problem in image processing, where the goal is to recover from noisy observation y 7 with unknown Gaussian noise v7.
Setup. We choose diffusion probabilistic models (DPMs) (Ho et al. 2020 Song et al. 202 as
denoisers due to their intrinsic design, which aligns closely with the denoising process (Xie et al. 2023) since the reverse process of a DPM gradually estimates and removes this noise using a trained network. effectively performing denoising. Specifically, we fine-tune a pre-trained diffusion mode for denoising the MNIST (Deng 2012) dataset, and readers may refer to Appendix C for more details.
We follow the iterative learning framework to annotate synthetic "clean" images to improve the denoising process. each iteration, the diffusion model first generates synthetic denoised images from the noisy observations. These are then filtered with reward model linear to the Peak Signal-to- Noise Ratio (PSNR) values compared with the real images. The filtered synthetic data are then used to fine-tune the model through a standard noise-prediction training procedure, iteratively improving its denoising capability. Although an oracle reward model may not be practical in many real-world scenarios, its use here helps validate the generality of our theoretical results. Furthermore, existing models or correction functions with prior knowledge could be utilized as the reward, as demonstrated in Gillman et al. 2024
Implementations. We perform denoising for two noise levels corresponding to the diffusion step 10 and 20, where the PSNR of noisy images are 33.25 and 28.40, respectively. A summary of the algorithm provided Algorithm 2 We consider three constant and three linear policies, each with small, medium, and large no- For the exponential policy, we have performed small hyperparameter search over no and u, selecting good configurations for two settings. The complete results of the hyperparameter search are presented as an ablation study on different exponential schemes in Appendix F22 demonstrating its robustness. The configurations for all schemes are presented in the caption and legend of Figure 3
htttp:://uuggingfacccoo////e

---

Table 2: Comparison of each policy on the image denoising and math reasoning tasks. The final accuracy are determined using a held-out validation set. We use boldface for the best result while the underline for the second best. For the exponential policies, we only exhibit a single set of configurations for the comparison. We plot for all the policies in Figure 3 and Figure respectively.
(a) Image denoising
(b) Math reasoning
POLICY
Fte
10
20
SYMBOLIC
PRE-TRAINED
N/A
40.92
36.96
N/A
55.60
34.98
15.
1 B
42.75
38.91
10 B
60.28
38.50
16.93
CONSTANT
10 B
43.15
39.45
30 B
60.16
39.68
17.73
100
44.18
39.91
100 B
64.04
41.66
18.81
B
43.90
39.90
3t B
62.54
38.88
17.01
LINEAR
3t 10t B
44.23
40.08
10t
61,96
39.94
17.29
44.64
40.26
30t B
64.24
40.84
19.17
EXPONENTIAL
44.84
40.14
65.66
47.26
20.65
Results. We summarize the final performance different schemes and baselines in Table and show the trade-off between performance and computational cost in Figure We observe that the exponential policy yields the best performance for s 10, and performs comparable to the best linear policy for 20. This observation demonstrates both the benefits of an increasing policy (Theorem 4.2) and the robustness of the exponential growth policy (Theorem 4.44). In contrast, the constant policy only matches the exponential policy in the early stages of training, but soon reaches performance plateau and eventually decays due overfitting, corroborating Theorem 4.1
5.2 Math Reasoning with LLMs
natural language processing. Iterative learning with synthetic data is also emerging method for LLMs. Previous research (Zelikman et al.] 2022) leverages the model to generate synthetic data, which then filtered using a reward model or accuracy metries and subsequently utilized to fine-tune the model. This framework is employed in various settings, including instruction tuning (Touvron al. 2023), alignment (Dong al. 2023 and recent efforts enhance reasoning
Now
capabilities through ohain-of-thought (CoT) approaches (Hosseini et al. 2024 Lehnert et al. 2024 Singh et 2025).
Setup. We select mathematical reasoning to evaluate the effectiveness of our strategy in iterative learning. Let q and a represent the question and its corresponding answer, respectively. We use the LLM to generate synthetic solutions with CoT and in-context examples. These synthetic data are filtered according to the correctness R(āa) l a). The filtered data used to further train the model in iterative process, without filtering based on the correctness of CoT itself. In summary, synthetic data with the correct final solutions generated and used for training in each iteration, closely mimicking the STaR method (Zelikman et al. 2022).
the final generated answer å, defined as the reward
Many contemporary LLMss are known to overfit to GSM8k (Cobbe et al.) 2021 or exhibit certain degree of data leakage (Zhang et al. 2024 Mirzadeh et al. 2025). where even minor changes to numbers or names can result in significant performance degradation. This issue renders the evaluation results on GSM8k unreliable. To address this, regenerate three math datasets- ASSM_symboll GSM_p1, and one of the most outstanding papers evaluating math reasoning. Each dataset consists of questions paired with their correct answers. GSM 1SM_symbolll retains similar difficulty level to GSM8k, while GSM_p1 and GSM_p2 _p introduce one and two additional clauses per question, respectively, making them progressively more challenging. Together, these datasets complete robust evaluation set.
Implementations.
symbolic math templates from Mirzadch et al. (2025),
In our iterative learning framework, we generate synthetic data using mixing ratio of 7:2:1 across these three datasets, filter the data based on the correctness of final answers, and fine-tune model on the selected data. We use the Llama-3-0 3-8B-Bas model (Dubey et al. 2024) with full-

---

orssunt, Re 10-B
liner, no 3t-
lacar,
ortscati, R2 100-8
linor, by 90t-B
(SM_eyabolic
GSN_p1
GSM_p2
Acorac (50
Acarro (5)
(5)
Compentioal (FLOPs xI0")
Conputational ConOTOLPS x19"))
Comporatonal (FLOPs x10*))
Figure 4: Empirical results of math reasoning. The figure shows the accuracies with respect to the computational cost, measured in FLOPs as well. The training batch size set to B 256. We adopt пı = 10 2 B as the exponential policy. We set the temperature to 0.3 during the generation step and include the results for a temperature of 0.7 in Appendix F.4. which lead the same conclusions.
weight fine-tuning. For evaluation, the model is tested on held-out datasets from all three difficulty levels. During data generation and evaluation, in-context examples and CoT reasoning are used, but in-context examples are excluded during fine-tuning.
summary of the algorithm is provided in Algorithm 3 Similarly to image denoising, we consider three constant and three linear policies, each with small, medium, and large no. For the exponential scheme, we have selected one configuration and left the full ablation results in Appendix F.3 The configurations for all schemes detailed in Figure and its caption. We also include the performance of the pre-trained model as a baseline.
Results. Figure S shows the performance versus computational cost across the three datasets under iterative training. while Table 2 summarizes the final performance for all policies. Across all three difficulty levels, the exponential growth policy demonstrates the steadiest improvement compared to other schemes. Although the linear policy ranks second overall, it fails to match the exponential growth policy when the training budget is large. In particular, with the exponential growth policy, iterative learning increases accuracy on GSM_p1 p1 from 35% to 47% and on GSM_p2 from 16% to 21 %. Based these results, recommend the exponential scheme for iterative learning.
6 Conclusion
In this work, we take the first step toward understanding how to design optimal budget allocation policies iterative bootstrapping for supervised fine-tuning. Through combination theoretical analysis and empirical validations, we demonstrate that constant policies are insufficient for achieving convergence, whereas increasing policies offer a more effective alternative. Among increasing policies, the exponential growth policy emerges as a robust and efficient method.
Although our focus here is SFT in iterative learning framework, recent research has highlighted the potential of Reinforcement Learning with Human Feedback (RLHF) (Yuan et al. 2024 Pang al. 2024 Setlur et (024) for similar iterative setups. Extending our framework incorporate RLHF is a promising direction for future work. By answering these open questions, we aim to lay foundation for more efficient iterative training approaches that can improve the post- post-training of foundation models.
10

---

References
Alemohammad, S., Rosriggzd J., Luzi, L., Humayun, A. L,, Babaci, H., LeJeune, D., Siahkoohi, A., and Baraniuk, R. Self-consuming generative models go MAD. In The Twelfth International Conference on Learning Representations, 2024.
Azizi, S., Korblith, S., Saharia, C.. Norouzi, M., and Fleet, J. Synthetic data from diffusion models improves imagenet classification. Transactions Machine Learning Research, 2023.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H.,, Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint 1rXiv:2110.1416. 2021.
Contributors, O. OpenCompass: A universal evaluation platform for foundation models. //github. com/ open- compass/ opencompass
https
htt.p///..pppp.aaooaao.ppppoaapp 2023.
Deng, L. The MNIST database of handwritten digit images for machine learning research. lEEE Signal Processing Magazine, 19(6):141-144 2012.
Dohmatob, E., Feng, Y., and Kempe, J. Model collapse demystified: The case of regression. In Advances in Neural Information Processing Systems, volume 37, pp. 46979-47013, 2024a.
Dohmatob, E., Feng, Y., Yang, P., Charton, F., and Kempe, J. tale of tails: Model collapse as a change of scaling laws. In Forty-first International Conference Machine Learning, 2024b.
Dohmatob, E.,, Feng, Y., Subramonian, A., and Kempe, J. Strong model collapse. In The Thirteenth International Conference on Learning Representations, 2025.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., KaShun, S., and Zhang, T. RAFT: Reward rAnked FineTTuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A.. Yang, A., Fan, A., et al. The Llama herd of models. arXiv preprint arXiv:2407.2178. 2024.
El Firdoussi, A., Seddik, M. E. A., Hayou, S., ALAMI, R., Alzubaidi, A., and Hacid, H. Maximizing the potential of synthetic data: Insights from random matrix theory. In The Thirteenth International Conference on Learning Representations, 2025.
Fan, L., Krishnan, D., Isola, P., Katabi, D., and Tian, Y. Improving CLIP training with language rewrites. In Advances in Neural Information Processing Systems, volume 36, pp. 35544-35575, 2023.
Fan. L., Chen, K., Krishnan, D., Katabi, D., Isola, P., and Tian, Y. Scaling laws of synthetic images for model training... for now. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7382-7392, 2024.
Feng, Y., Dohmatob, E., Yang, P., Charton, F., and Kempe, Beyond model collapse: Scaling up with synthesized data requires reinforcement. arXiv preprint arXiv:2406.. 07515, 2024.
Ferbach, D., Bertrand, Q., Bose, A. J., and Gidel, G. Self-consuming generative models with curated data provably optimize human preferences. In Advances in Neural Information Processing Systems, volume 37, pp. 102531-102567, 2024.
Gandhi, K., Chakravarthy, A. K., Singh, A., Lile, N., and Goodman, N. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective STars. In Second Conference on Language Modeling, 2025.
Gillman, N., Freeman, M., Aggarwal, D., Chia-Hong, H., Luo, C., Tian, Y., and Sun, C. Self- correcting self-consuming loops for generative model training. In Forty-forst International Confer- ence on Machine Learning, 2024.
Guan, X., Zhang, L. L., Liu, Y., Shang, N., Sun, Y., Zhu, Y., Yang, F., and Yang, M. rStar-Math: Small L.LMs can master math reasoning with self-evolved deep thinking. Forty-second International Conference Machine Learning, 2025.

---

Guo, D., Yang, D., Zhang, H., Song. J., Zhang, R., Xu, R.,, Zhu, Q., Ma, S., Wang. P., Bi, X., et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.1290 129448 2025.
Guo, S., Zhang. B., Liu, T., Liu, T., Khalman, M., Llinares, F., Rame, A., Mesnard, T., Zhao, Y., Piot, B., al. Direct language model alignment from online Al feedback. arXiv preprint 27Xi::2402.04792, 2024.
Hemmat, R. A., Pezeshki, M., Bordes, F., Drozdzal, M., and Romero-Soriano, A. Feedback-guide data synthesis for imbalanced classification. Transactions on Machine Learning Research, 2024.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, 2020.
Hosseini, A., Yuan, X., Malkin, N., Courville, A., Sordoni, A., and Agarwal, R. V-STaR: Training Verifiers for Self-Taugh Reasoners. In First Conference Language Modeling, 2024.
Hu, J.. Wu, X., Wang. W. Zhang, D., Cao, Y., et al. OpenRLHF: An casy-to-use, scalable and high-pent high-performance RLHF framework. arXiv preprint arXiv:2405.11143, 2024.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L.., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Lehnert, L., Sukhbaatar, S., Su, D., Zheng. Q. Mc P., Rabbat, M., and Tian, Y. Beyond *#; Better planning with transformers via search dynamics bootstrapping. In First Conference Language Modeling, 2024.
Mirzadeh, S. I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., and Farajtabar, M. GSM- Symbolic: Understanding the limitations of mathematical reasoning in large language models. In The Thirteenth International Conference on Learming Representations, 2025.
Pang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, Iterative reasoning preference optimization. In Advances in Neural Information Processing Systems, volume 37, pp. 116617- 116637, 2024.
Schick, T., Dwivedi- Yu, J., Dessi, R., Raileanu, R., Lomeli, M.. Hambro, E., Zettlemoyer, L.. Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. In Advances Neural Information Processing Systems, volume 36, pp.
2023.
Setlur, A.. Garg, S., Geng. X., Garg, N., Smith, V., and Kumar, A. RL on incorrect synthetic data scales the efficiency of LLM math reasoning by eight-fold. In Advances in Neural Information Processing Systems, volume 37. pp. 43000-43031, 2024.
Shao, Y., Li, L., Dai, J., and Qiu, X. Character-LLM: A trainable agent for role-playing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13153-13187, 2023.
Shumailov, I., Shumaylov, 7., Zhao, Y., Gal, Y., Papernot, N., and Anderson, R. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305. 1120055174999 17493, 2023.
Shumailov, I,, Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., and Anderson, R. Al models collapse when trained on recursively generated data. Nature, 631, 2024.
Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P. J., Harrison, J., Lee, J.. Xu, K., al. Beyond human data: Scaling self-training for problem-solving with language models. Transactions on Machine Learning Research, 2025.
Song. Y., Sock-DDik,,ein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2021.
Su, D., Sukhbaatar, S., Rabbat, M., Tian, Y., and Zheng, Q. Dualformer: Controllable fast and slow thinking by learning with randomized reasoning traces. arXiv preprint arXiv:2410.09916 2024.
12

---

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale. S., et al. Llama 2: Open foundation and fine-tuned chat models, arXiv preprint arXiv:2307.09228
2023.
Trung, L., Zhang, X., Jie. Z. Sun, P. Jin, X., and Li, H. ReFT: Reasoning with Reinforced Fine- Tuming. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7601-7614, 2024.
Xie, Y., Yuan, M., Dong, B., and Li, Q. Diffusion model for generative image denoising. arXiv preprint arXiv:2302. 02398, 2023.
Yang. P. and Dong. B. Mocoll: Agent-based specific and general model collaboration for image captioning, arXiv preprint urXiv:2501.01834, 2025.
Yuan, W., Pang, R. Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., and Weston, E. Self-rewarding language models. In Forty-first International Conference on Machine Learning. 2024.
Zelikman, E., Wu, Y., Mu, J., and Goodman, N. STaR: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, volume 35, pp. 14476-15488, 2022.
Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., Zhao, T., Raja, P., Zhuang, C., Slack, D.,
Lyu. Q. Hendryx, S., Kaplan, R., Lunati, M., and Yue, S. A careful examination of large language model performance on grade school arithmetic. In Advances in Neural Information Processing Systems, volume 37, pp. 468 19-46636, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have clarified our contributions in both the abstract and introduction parts.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations our work in the last paragraph of Section and in Appendix G.3
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification: We have briefly exhibited the assumptions and theorems in Section 3.2 and Section The detailed assumptions as well as the corresponding proofs are provided in Appendix and Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The experimental settings are described in Section including the setup for each individual experiment and the hyperparameters. More details for the experiments are provided Appendix F as well.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
15

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have included our codes for all the experiments in the supplemental materials. All the data are open-sourced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co.
wow.bmmmssiiooooooooooocc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips. www.cc/pwwpcppccccceceoeooooeeee http://wiisss/eeeeee www.eeeuiisssoonnooooooooooooeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All details for the experiments provided in Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report error bars for the synthetic experiments. For the large-scale experi- ments with diffusion models and language models, the computational cost of independent runs is very high. Instead, we include the performance curve with respect to the training to observe that the statistical variance among steps is small, Ablation studies are included to strengthen the claim.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: I have reported the experiments compute resources in Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:p///.aaappp..cpppooottttcta
Answer: [Yes]
Justification: Our research matches the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: I have stated the social impacts in Appendix G.I
Guidelines:
17

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work focus on the fine-tuning strategy with synthetic data, so poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the original paper and included proper license in our code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
18

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Our research does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We only leverage LLMs for writing and editing on sentence level.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
20

---

A Theoretic Intuition
A.1 Discussion on the Gaussian Setting
In the Gaussian case, we use simplified toy example that omits the generation cost only to provide clear, introductory setup for the reader. In our complete framework (see Section D however, we assume generation cost for standard estimators and incorporate this cost into the objective using the hyperparameter Cg. This hyperparameter offers flexibility: setting Cy generation, while a large C reflects expensive generation. We also provide general theorems that hold for all values of cg. and explain how the Gaussian case satisfies all assumptions of the general theorems in Appendix B.2 Thus, the Gaussian case can be viewed special instance, and the theoretical results still remain valid regardless of whether the sampling free or costly.
corresponds to free
A.2 Proof of Theorem
We first calculate the distribution of ((T) given an initial parameter (()) and a specific policy æ (nete with the terminal time T in the following lemma.
Lemma A.1. With the MLE updation and Gaussian distributions, given an initial parameter g(0) and specific policy {nehe with the terminal time T, we have
((0)
p(012) | 1 ;T) -N
(1+02/12))
nz n1
0² (2((++002)00000-1)
Suppose that there are N₂ data { generated from the i-th step generator such that the selected sample size is n,. Let b, ~ (zx:) as selecting sign where b, indicates that æ; is selected and placed into D., otherwise b, 0 indicates that x; D..
Proof.
We first consider one-step updating. We have the distribution of a data æt conditioned on 2q D,,
( 24 Dz) (æx b, =)
R(x;) .N
(z:0011 0(1) .02
x7
(x;
0(1))2
α exp
2x²
20²
((1)
o²
W
11 02 /122
1 1 02 02/12
Since ((++1) is the solution the MLE satisfying
((++1) arg max
II
NZ 8, 0z)
1
Σ K
DED.
rED.
we have the distribution of ((++1) conditioned on the parameter ((9) and the sample size of the previous iteration, written as
g(++1) (0()), ~ No ( (- ((()
σ²
(1+02/n² n1 + /2/
Considering the decomposition
((1)
@(++1)
0 1/2 + 100++22/222 &
12
with an independent auxiliary random & ~ N(0, recursively, we have
((0)
Φ(T)
E
O1
1/2 +
1=
21

---

Therefore,
(
o²
O(7)
000) x N
(1+02)n2)T
n₂(1
12
t=0
Back to the proof of the theorem, we define the mean and variance
(
pr (1+ 02/33/13177
and
o o²
(1 + 0² 112)
t=0
According to Lemma A.1
we have that
x²
Egery R.g [R(x) Egery Rx
exp
2x²
x22
(x
B(T))2
Egers
dr
VTT02
2k²
202
1
(B(T))2
/
02/k2
exp
2( + x2)
1
1
exp
01 10/2 R22) v1 11+ 02/R222
2(
x2 + o
K
1
+ R2 +
exp
2(0² + x2 o
With the assumption that nT < + the right term of the above equation is increasing function with respect to +o
Therefore, the objective function gets its maximum value when
minimize of. Furthermore, we have the optimal iterative learning policy from Cauchy-Schwarz inequality,
T-1
TT
2
n4
2
T2e (1 + 2² (2))
n4
t=
1=0
2 21+0//22
(1 01
where the equal sign is established if and only if
1+
Reindexing shifting t will complete the proof.
B
Main Theoretic Results
B.1
Notations
Consider a probability space = (S2, B(2), P), where the sample space, B(2) is the Borel o-algebra on Ω. and P is the probability measure. Let θ C IRP be parameter space, and define
Ag (f(w;0) 0 θ}
22

---

as parametric family of random variables mapping from f2 to measurable outcome space T. Each 0 induces& distribution Pa on where
Px(X) P (f((w:)) E X)
for any measurable set X CT.
Any function R: T [0: 1] induces additional distribution Pr:0 [0, 1J. where
PR. (B) Pa (R(x) E B) for any Borel set B [0,1]. 1].
Given reward model R(x), the reward r0)) for 0 E O is defined as the expectation of the distribution PR.0, i.e.
r(0) = / Joo11
rdPR.0(F) (o))(())(())))
Define the loss function lx;; 0) T x θ + [0, oc). Now we consider the optimizing algorithm that starts with an initial parameter (()) and updates the parameter ((() at the t-th iteration using gradient descent with a learning rate > and data D, E TwIP:1, formulated as
(++ (()) 1
Σ Vol (1;)(0)) t= 0,
DED.
B.2 Assumptions
We always consider the general case where the initial value (()) is sufficiently accurate. The distribution Poros is assumed non-degenerate and there exists constant G, such that
dP r:::00 2 c
Assumption B.1 (Basic properties of the optimization space and loss function). The parameter space for 9 and the loss function ((:; 8) indexed by 9 E θ satisfy the following requirements.
1. The covering number (e,, 0, (|2) for the parameter space O is bounded, which means the inequality
((e, o, l1a) s ces -Co 1 Ve >
holds for some positive constants ce and Ce-
2. The loss function [(x; 0) is Lipschitz continuous. Formally, there exists a constant 0 independent of x such that
(<=;0) 1(x; 01)1 < C 11112, v8, 01 E O.
(4)
3. For any E O there exist positive constants c and / satisfying
Vol(x; 0) cl(x;; 0),
Ve 0) llop B.
(5)
Remark. This assumption governs both the complexity of the parameter space + and the smoothness of the likelihood function (2;; 0). Here, we provide two examples to show that these assumptions are reasonable. Firstly, if θ is compact finite-dimensional space or functional space sufficiently smooth, then the coverage number requirement (Equation (3)) is satisfied. Secondly, for special case when [zx; 0)) is the negative log-likelihood for Gaussian distribution, the derivative condition (Equation (5) holds with
and
We define the post-selection distribution 2e via its Radon-Nikodým derivative relative Pe
dQo(x)
R(x)
dP (z)
ExnPe [R(x)]
which indicates that Qa just the original distribution Pa weighted by the reward function R(x). Equivalently, we can think of Oo the conditional distribution of given that is "selected" in proportion to its reward. For convenience, let po. 40, and pr:0 the density functions (with respect to some appropriate base measure) of Pa, Qe. and Pr.g. respectively.
23

---

Next, we define the expected likelihood functiin Pla, (02) as
Pla. (02) 2x, [1x;; 82)] Bu-ge [- log Po, Po, (x)]
Intuitively, Pla, (02) measures how well the model pea "fits" data drawn from the post-selection distribution Qa.
We define
e" arg min (l) B5
as the optimal parameter set that minimize all possible Pl) Finally, to introduce the sampling error, we define
(x)((; = Σ 7 (ag;
as an empirical evaluation for the negative log-like likelihood based on samples ( Assumption B.2 (Local reducibility of the expected loss function). There exists a constant (0, such that for any 0, 01 E ④ with ||0 01 ll I 2/2/ and Ple, (0) S Pll,, (01), ) we have Plg (8) Ple, (01) - (Ple (0) - Pl,, (0)))
Remark. This assumption ensures that, within local neighborhood of 8, any parameter update that reduces the expected likelihood under 0₃ 's distribution also decreases the expected likelihood under O's distribution, relative to Pl,, (01). Concretely, each iteration of the algorithm samples from 0 but then updates the parameter 0. Since the data remain the same while changes, this condition guarantees that a reduction in the loss is preserved across iterations. For instance, if Po is Gaussian distribution with mean / and standard deviation 1, and R(xx) is the normalized standard Gaussian density, one can verify that Assumption B.2 holds with y = 1/2.
Assumption B.3 (Relationship between the loss and the reward function). For any 0 e e holds for some C2 C3 s 1 that
(l) sr' S c2
Remark. This assumption establishes a direct relationship between the performance gap (r* r(0) and the expected loss (f). Specifically, the upper bound shows that if Plo(0) is small, then (r*- -r(0)) must also be small, which is crucial for proving convergence of our proposed policy. For example, in the Gaussian setting discussed in the remark of Assumption B.2 one can verify that 22 = 1/12. Conversely, the lower bound indicates that a small performance gap necessarily implies a correspondingly small expected loss, thereby capturing the sensitivity of the loss to improvements in the expected reward. Since both (7* - r(0)) and (0) quantify how well the policy performs, it is natural that each can bound the other. This relationship serves as a key underpinning for the theoretical results presented subsequent sections.
B.3
Main Theorems
Our theoretical results start from the defect of constant policies by showing that the final reward remains bounded away from the optimum with negliggli probability when using constant policies.
Theorem B.4 (Bounded Reward for Constant policy). Under Assumption B.3 there exists a constant 0 such that, for any T and any constant policy Weonst, with probability at least 1/4,
(TT)
cng -1/2
Recoms
On the contrary, any increasing policy eventually closes the optimization gap within arbitrarily small range as long as the training iteration goes to infinity.
Theorem B.5 (Optimal Reward for Increasing Policies). Under Assumptions B. to B. for any increasing policy and 0, there exists a sufficiently large T such that, with probability greater than [ ET- n2
r" r (0fT))
S
24

---

Next, we demonstrate that the exponential- -rowth policy from Theorem B achieves this convergence more effectively than polynomial growth policies, including the constant policies.
Theorem B.6 (Convergence Rate for Exponential Policy). Under Assumptions B.
to
B.3
there
exists exponential growth policy Texp such that with probability least Σ 21=0 n
r -r
(0
( Heep
O (11+5)
with constant Ç 7c/(83--1//))) 7c) >
Consider the minimum number of iterations (=x needed for a policy æ to achieve reward that is within e of the optimal r" as defined in Equation (2. then the cost C(T,, TT (x,,e) is the minimum cost to attain the target performance for policy я. The following theorem compares the total cost
incurred by exponential growth policies and polynomial growth policies (including constant policies) worst-case scenario.
Theorem B.7 (Worst-Case Optimality of Exponential Policy). Under the same conditions as in Theorem B.6 for any constant polynomial growth policy and sufficiently small e, have
sup 0,8 ((=:))()))
C (Texp: 1 (Texp:
(Teep:
sup 8,8 (u:0))
C (
with probability at least (1 (1-ETh (M=s)((aa n The supremum is taken over all possible param- eter spaces O O*, loss functions ((x; 0), and reward models R(x) satisfying the same conditions as in Theorem B.6
Such worst-case perspective shows that the exponential growth policy remains robust and outper- forms (or at least matches) any constant or polynomial policy. (Although the policy ' fixed in advance, the iterative sequence (0(h) T-1
is still random because data are sampled and selected probabilistically at each iteration. The stated probability bound reflects this inherent randomness and guarantees high-probability performance under the specified assumptions.
B.4 Proofs
In this subsection, we present proofs for all statements our theoretical results. We begin with the proof of Theorem B.6, proceed to prove Theorems B.4andd and B.5 and conclude with the proof of Theorem B.7
Proof of Theorem B.6
We begin by noting the following decomposition
Placo) (0())
-Placon (D(T) Plac0o (o())
Pnoo (x:01))
+ Pmo! (x;O(1)
Pno (x:
+ Pnoo (x;(00)
Plaros (810))
which gives
Placo (8(2))- -Plo0 (BLO)) < P (x; -no (x:
+ Plgcoo) (D1)) -Poo -Pro!(X;000) (x:00)) (6)
PPgg0o (or")) - Pno (x:0(1)
25

---

By Assumption B.1 and a Taylor expansion, choosing the step size 1) = 1/(28) gives
P. Png (X; pno (x; (Vol no (x::
(xoPo(((:000) T (p(U) - + 10000 - 0000
-01000000
77 VoPnoo (x: 0(0)) T
VolP no (x:
+ Bn² VolFno (x: 1
S 1
Volnol (x):0
43
Pnol (x;
S 43
Define
Rg Eg
sup A
Σ <x; 8)
where E; (1<is n) are i.i.d Rademacher random variables. By Dudley's 's entropy integral and Assumption B.1 we then get
sup Plaro (0) P.,(X;:) < 2Ro AE
2Ln 1/2 2Ln Jo Viog 0, (12) de
(8)
2LC log can 1/2 1/2
By Hoenofff- theorem,
P 2(moo-Nor 1Nor-ror Nor (D.0) 2 11) S exp
2t
Nor (o(o))
Combining Equations 7 and (8) with Equation 6 and choosing
== 2 Nor (01)) 1/2
log No:
see that, with probability at least (1-No² N)
(B(1))
(0(0)
Placo
Placo
S 46 Placo)
(
+
2
C
sup 8ee
PP (8) - (x:
43
s 43 Plawo
(0101)
+ 2
c
2LCs log comp 1/2
43
Plato (Bro)) + 2
2LC logce Noo (0(0) -2 (Nor (B1O)) 1/2
-1/2
S
log No
43
We choose No to satisfy
Nor (B10)= 2 22(Nor (BLO)) 1/2
-1/2
(B10)
2LCe log co
log No
S
Pl
43
88
By Assumption B.2
we have
Placs (0()) Plaro (010) 5/ [Placo (D(1) PPllco (BLO))
(0
<
83
which implies
Placs (0()) < Plaro: (010)) (1-
7c
8,
26

---

Define Plg (0). This can be rewritten as
Placo
Placo S
76
10000
88
and by Assumption B.3 we have the inequality
(Orl) r- c2lPlaus 2 co2l1p00
-ca-o2 1
70
8,3
By induction, that
(o(c)) S Flacos (DLW)) (1-
YC 8,3
Plaee
For each with probability at least No²
Placo
(g(et2)
Places
(0())
(p())
Plaue
sup
Placo (0) Pn. (X::)
48
48
sE0
Playe
(D())
2
- 43
Ce con, 1/2
43
S
Places (p(c))
+
2
c
20 logce Ner (Dl.) (Ner (Dlct)
log No.
-1/2
48
4/
We now choose N₂ so that
2
CCrCo log
Ner Nor (0(0) - 2 Ner (0(0)) 1/2
log No
112
Placo (0l0)) (1-
70
43
85
8,
Then
Plaisi (oltt1))
Place) (o(t))
Placos (0
(1
yC
+
43
8,3
8,3
< 1
c
Placo
(ol0))
L
c
+
c
Plaros (010)) (1-
70
43
8,9
83
8,
(+1
(D10))
$ Placoo)
8,3
Similarly,
(1- 7c 1+1
(o(ctt)) 2 2- 372-c2Placcs - -c2PPl:0:
8,
Consequently, we choose
N₂ r (0())
c (iP1
-2
(1 (1
7C
-2t
8,3
then for the initial value (00) which is sufficiently accurate, we have
r (p(0))
1/2 2 (B(o))
1/2 2 AN, 1/2 log No.
ensuring that our choice of N2 meets the needed condition.
Finally, we define Ç 7c/(88 7c), then
N,r (0()) = (2 (163 2c) c logeo) (Pi:))) (1 +++)) -2-
27

---

By the inequality
7-1
T-1
1
Σ
N₂
1
Σ
Rt
1=0
t=0
we obtain the results stated in Theorem B.6
Proof of Theorem B.4
By Assumption B.3 we have
sup PP [x)) [x)] oce
EEWP (P))
[R(x), 2 c3 6m o
OlT)
2 cy Feuest
(lagr
Pno
Xx (x:O1t) Mocama
)) + Pmo (x; o tcocn))
C3 Ro
1i o
((T)
-xxx
I (xx
Recmms )1)
4/24(0.75) VarxwP ((ar0
(x::
(x:0% 0 Reooun
)}
2 cg
with probability larger than 0.75 0.25, where we invoke the central limit theorem for the last inequality.
Proof of Theorem B.5 Let be given. We first show that there exists some satisfying
(
2
2
Following the proof of Theorem B.6
we have
P1 p
Pl
000
76C Pl
+
2-
2C2Ce log con, 1/2
48
48
There exists sufficiently large 7 such that
(2 4,3 2C log cont 1/2
76
1
min
e,
1633c2
202
For any 1 2 T", consider the following two cases:
(a) Ploge (202).
By Assumption
B.3 we have
((()) 2 2r ogPl p(y) 2 r*
2
(b) Playo git > e/(22). Since 14 l - xo, for all 2 T have
7 43
2C, log com, 1/2 -
7C
S
70
P7
16882
88
Therefore,
Pl Pl - Pl g() <
7C Pl
88
050.
Repeating this argument for = T², + 1, whenever Pl > e/(20z) shows that there must exist finite K (so T T K) where either
r ((())
Pc P.
2 C2
43
T-T"
02 7c P
1
7C 83
2 r"
Pl
(()))
43
or else PlaTT S S2((2))) in which case (a) implies
(())
1
2 E.
28

---

Thus, there exists some T such that
(
2 r"
2
Next, we prove that for any t 2 T.
(0())
2 r"
For 2 T, we have
sup T2
Pl (0+1) - Plato]
- sup
127
4,3
Yc
Pl
04
+
(2 2
48
c
20 Co log cont 1/2
20 2Cr.Ce log can. 1/2
sup 12T
2
43
8
202
From part (b) above, whenever P1 p0 2 E/ e2(22), we have
Pl Ploge1) - Pl S
70
P.
88 o
Hence,
sup P7 12T
+ sup
Pl Poge
19/ Pl/
202
tz
C2
It follows that for all 2 T,
r (
2 r E,
which completes the proof.
Proof of Theorem B.7 We prove this theorem by showing that for given policy x', there exists a specified parameter space & and O*, a loss function (x;; 0), and a reward model R(x), under Assumptions B. to B.3 such that
(r' -)-1
(Z(=)= =
(g + cnl)
(9)
CexpE Cexp
> CTTexp: T (Texp; E))
with probability at least (1 (1- (1- 1==
T (Tepp -1 n n
To begin, we use Theorem B.6 to obtain
(1 (1-
7c
(loge log c2pl1:0)) ): Ti
T (Texp; 8) S log"
83
with probability at least (1 - D T" (Feep
n, (T
Then the equality
C(Texp: T (Texps 8)) < CTTexpi Ti)
n-1
Σ (crr) + (2) CC (B1)0) -2
(11
11- 7e
-2t
83
-2 Ti
-21
5 (cor"' + +c) (2(163 - CCrCCe (P!.
Σ
70
88
1=0
< (crr + +a) aa) (2 20) Cr (p1)0)
-2
(1 + 2(T
1)
2 156 22
< Cospe= - Cexp.
29

---

holds for some positive constants Cexp and Ceep that do not depend on E. So we have proven the last inequality in Equation (9).
Next, we consider the second inequality Equation When n' is a constant policy, Theorem B.4 implies that for sufficiently small (*(x' and the ineguality holds vacuously. Otherwise, for a polynomial growth policy x' under Assumptions B.l to B.3 there exist θ Or, [(2; 0), and Rax)) such that for 04)
0 + PY 2 20 logco co 1 . Vnl" vm:
Pl (442 - Plag min
yc
4,8
43
for some < 1, and by Assumption B.3
r" r--r (
2 c3 cylPlgges. pio).
Since x' is a polynomial growth policy, let n' (n/y ni, nb.....) with n². n/((1 + t)a
b
definition. A decrease in expected loss occurs only
p7 2 (2-48 43
2 c
7c
CC log ce
S
48 9i0).
v (11++0 t)a
Besides,
Pl Plograan Scg -1
EEWP p (r [[x)]]
S Scg E,
it follows that
c 43
S 7c 4303
By (2-43)
20 Corgologoe ce
E,
Vni (1 T (x',
so
-2
1/a
T (T)
2csp(83- Ce log co
n1
8 -2/a
2/a == T2
Thus.
T (r)
T2-1
Σ
Ng + +am)) 2
Σ
Ng + ank)
1=0
To 1
(eg + cn)n((1 + +
1=
T2
Ts = Opoly 2(a+1)/aa 2
2 (cg + 4)nó
Opoly
2
for some positive constants Opoly and Cpoly independent of e. If e is small enough, follows that
T Ca' s)-1
(cg, + c.n) 2 C 2(a+1)/a
Cpoly > Cexpe 2
Cexps
t=
so the second inequality in Equation also holds.
B.5 Extension to Rewards with Random Noise
Our theoretical conclusions directly generalize to cases with noisy rewards. Consider a perturbed reward function R(xx) R(x)) + E, where is zero-mean noise independent of R(x). Crucially, the expected reward r(0) remains unchanged, preserving all key theoretical results. Although noisy rewards affect the sample selection process,
concentration bounds for selected sample sizes (Appendix [.4] and the expected reward under the selected distribution remain valid asymptotically. This robustness stems from the exponential policy's inherent adaptivity by progressively increasing sample sizes (Theorem 4.3 naturally compensates for reward noise variance.
30

---

C
Diffusion Probabilistic Models
diffusion probabilistic model (DPM) progressively adds noise clean data sampled from the target distribution until the data is fully corrupted. The generative process then employs a noise prediction model transform noisy samples into progressively cleaner ones auto-regressively. This reverse-time evolution naturally resembles a denoising process, making DPMs suitable for image denoising, especially when the noise modeling known (Xie al., )
Formally, a DPM defines a forward stochastic process with conditional probability
P= [o) = (~ 20, I) T 7 [0, 1],
where zo is drawn from the target distribution that the model aims to learn. The parameters ot and 3, are typically chosen such that ogg = 31 and 01 3g = This process progressively adds noise to 2o, resulting in fully corrupted sample æ1 ~ (O.
To model the reverse process, the time interval [0, 1] discretized into steps TO < TI <
TK = 1. Starting at TK. the reverse diffusion process gradually removes noise step by step until reaching Tg. At each step k. the noise added during the forward process is estimated and removed using a trained noise prediction model Eø( Tx). This model learns to predict the noise e based on the noisy sample Try, enabling the reconstruction of cleaner samples at each step.
The standard training procedure, which is also adapted in our experiments, attempts to solve
min Eanng gf (o.. ||Ea(a, B+e, r) e||²
(10)
0
rEU(0,s)
where denotes the distribution of training data.
D Algorithm for Realistic Experiments
We follow the algorithm in Algorithm|] | Empirically, in the two realistic settings (image denoising and math reasoning), our iterative learning policy satisfies that ne is an integer multiple of the batch size. Specifically,
For the constant scheme,
T4 = [m] B:
For the linear scheme,
n1 [n(t + B:
For the exponential scheme,
= [n(1 + BB
where n and hyperparameters, and B is the generation batch size.
E
Implementation Details
We provide the code for all the experiments in the GitHub repository: seliikkusppend-iiie
https://github. com/
E.1
Toy Example
In this subsection, we extend the theoretical intuition from Theorem 3.1 to high-dimensional settings and then derive the expected reward of each iteration, as well as the corresponding optimal expected reward.
We begin with the following theorem, which shows that in a high-dimensional Gaussian setting, the same exponential growth strategy emerges in the one-dimensional case.
Theorem E.1. Under the MLE update, suppose that the generator is d-dimensional Gaussian distribution (00 d² Ia), the reward model is an exponential function R(x) exp (-1|=|| (2x²)) and we start from
|000|| s (1 (1=02/12)² (d + xx))
31

---

Considering the optimization problem
T-1
max
Bgrr)
Eoop orT; R(x))
s.t.
Σ k=U
n
SC
then the optimal policy satisfies
1 o2)
n2 x
Proof. Since each component of (()) is ii.d and its distribution conditioned the initial parameter g(0) and the policy æ is given by Lemma A.1 it follows that
(T ((0), TT =N
(N(utomaa
( (1+02
(
T-1
n1 - 0² /2/n2))200 ()) Ia
for specific policy {me)e. By the definition of the reward model, R(xx) factors
R(x) II R(x,),
(I1, 22, x2).
Repeating the argument in the proof of Theorem 3.1 shows that
E Egrr R(x(()))
Egery E. II R(x)))
II Egcry B R(xi)
(0,0)2
Σ
exp
V 0
+ R
2
2( k² + oz) (1 + 0² /13/2T
i=1
0
K
n2
+ 0
exp
2( x² + + + 02 /2/
With assumption || < (d(a² + x²2)) 1/2
the right side increases as (a2 x² + of) -1/2
1/2 in-
creases. Hence, figuring with the proof of Theorem 3.1 in Appendix A one obtains
o2 t-
n4 1+
Reindexing or shifting t completes the proof.
Theorem therefore shows that the optimal iterative policy in the high-dimensional case mirrors the one-dimensiona scenario. Next, we derive the expected reward in the eigh-dimessionl case. Specifically, if æ ~ N( o² Ia)), then
||||
||
0
(2mo²)aa22 exp
da
2k
20
||0||2
(1 (1+02/x2)d77) exp
2( + x2)
Clearly, the maximum of this expression is attained 0 0. and the optimal expected reward is (1+ + o² /x2)/=4//
32

---

E.2 Synthetic Experiment with Gaussian Distributions
Setup. We begin with the synthetic setting introduced our warm-up (Section 3.2). which involves optimizing a generative distribution parameterized as a Gaussian to maximize the expected reward. Specifically, we focus on a two-dimensional space using MLE updates each iteration. The generator is set to (0 I2), with fixed variance. The reward model is defined R(x) = exp (- P((22)), favoring data concentrated near the origin. controls the flatness of the reward.
exponential function
All theoretical results from the warm-up extend naturally to the high-dimensional case, as detailed in Appendix E.I We now validate that with equal computational budgets and iterations, the exponential scheme outperforms the linear scheme, which outperforms the constant scheme.
We consider two settings with x² E (2, 4). Based on theoretical analysis, the exponential scheme is set as n, = [ng (1 + 1/ where no 10. For fixed total budget and number of iterations, the constant scheme is then
Implementation.
no (1+ 11+1//2)
while the linear scheme
2(t + 1)
T-1
Σ no (1 + 1/x3)*
T(T 1)
The initial parameter is set to ( = (1, 1) and the optimal solution for the reward function is 0- = (0,0) The optimal reward value is then
(o = (1 + 1/x²)=1
We evaluate how fast the three schemes approach this optimal reward value.
E.3
Image Denoising
The pre-trained model we choose is public checkpoint that can be accessed via https:// huggingface. htt.n//faac...ooooleeeeeeoo wwogooogle/ddmmciiiieieeeeeeeeee hitpdddiammmm-ooo--eeaaaa
For all denoising experiments, we fix batch size
B 640 and learning rate 5 x 10
At each iteration, synthetic samples are generated the fly, scored by reward model
R( x)) PSNR(#. à. æ) Tmin
Tmax -'mi
and accumulated into chunk until it reaches size T4. The full training procedure is given in Algorithm 2 Each configuration requires around 100 GPU-hours on an NVIDIA A800 cluster. Note that our setup is much more computationally intensive than the usual training of generative model in that it takes s times of network inferences to generate synthetic data for network training.
To compare increasing-schedule policies (particularly exponential growth), we report the computa- tional costs in terms of floating-point operations (FLOPs), which scale linearly with the number of forward passes. For diffusion models, generating one sample uses s denoising steps (i.e., forward passes), and each optimization step introduces additional forward pass.
E.4 Math Reasoning
We summarize the iterative math- reasonins algorithm in Algorithm Each configuration requires approximately 400 GPU-hours
NVIDIA A800 cluster.
Hyperparameters are as follows: one epoch per iteration on the selected data; constant learning rate of 10 10-"; batch size B = 256; and roughly 1.000 total generator update steps. During generation we use temperature 0.3 for diversity, and temperature 0 at evaluation for accuracy. The maximum generation length is 512. Prompts follow the few-shot setting of the GSM8k dataset in OpenCompass (Contributors) 2023). Our implementation builds on the public OpenRLHI framework (Hu al.
2024).
33

---

Algorithm 2 Iterative learning for image denoising
1: Input: a pre-trained generative model [(;; ((;;(()), policy æ (n+}r with a terminal time 7, reward model R. and a dataset 'D == {xi
2: Output: generative model f(::0()) fine-tuned with synthetic data.
3: for to T' do
4:
Initialize the set of selected synthetic data D, Ø.
5: while Del < do Sample mini-batch { randomly. 7: Create noisy yi P- (-1 x;) for each z,,
6:
8:
Generate synthetic 2; as a denoised version for each y, based on the generative model
f(: o(+));
9:
For each é;, add it into D. with probability R(2,, æz) clipped within the interval [0, 1].
end while
Truncate D, that it contains exactly ne samples.
for each batch { in Dj do
13: Take a single optimization step with the loss
(0(t)) = 1SS- Σ
| (ar x Br. Ex, Tx) - Ex
EES
to update O(0, where &o ~ NN(,,) I) and To U(0,s).
14:
end for
15:
((1-1) + ((t)
16: end for
Supplementary Empirical Results
F.
Toy Example
Additional results for the toy example, obtained under various parameter settings, are shown in Figure In all cases, the behavior is consistent with the conclusion in Section 3.2
F.2 Ablation on Exponential Growth Schemes in Image Denoising
We conduct an ablation study for exponential growth policies to understand the impact of the ni on the performance of the iterative learning framework, as measured by PSNR in Figure The results reveal that a smaller n, during the early stages of training leads to a rapid increase in PSNR and earlier convergence. In contrast, larger n, results in a slower initial increase in PSNR but achieves higher final PSNR when trained long enough. These findings underscore the trade-off between rapid early improvements and the potential for superior overall performance with prolonged training.
F.3 Ablation on Exponential Growth Schemes in Math Reasoning
We further explore the effects of varying the parameter n, within exponential growth policies on the performance of mathematical reasoming tasks. As shown in Figure [ configurations with higher ne demonstrate the potential for superior performance, improving the model's 's reasoning capabilities and ability to solve mathematical problems. However, these configurations also exhibit some instability, characterized by abrupt changes in accuracy as n4 increases. This highlights the need to find balance between achieving rapid learning and mitigating the risk of unstable training dynamics. Based on these findings, adopt n. 10 2
B in the main text.
F.4 Additional Empirical Results in Math Reasoning
In the main paper, we use temperature 0.3 for the math reasoning experiment shown Figure Here, present additional empirical results with temperature of 0.7 in Figure g The general
34

---

Algorithm 3 Iterative learning on math reasoning
1: Input: a pre-trained L.MM f(; ); policy a {me) with a terminal time T, and dataset D {(ss 0z)
2: Output: f(:;0(T) fine-tuned with synthetic data.
3: for 4- 0 - do
4: Initialize the set of selected synthetic data D; Ø.
// Generate answers and select out correct ones.
5: while Dj < n4 do
6: Sample mini-batch B = {(g 9,)}; C
7: Generate answers {ā,} based on f(- (g;);00)) with few-shot examples and CoT strategy.
8: Add (q, az) into Dr if â = aj for each j. end while
9:
10:
Dr subset of Dr containing exactly n, data.
// Update ((0) to ((++1) by the selected dataset D, with auto-regressive loss via Adam optimizer. for each B as a mini-batch of D, do
11:
Take a single optimization step with the auto-regressive loss
I(O)) =
log q; f(, 0))
B
(8
update ((*) 13: end for
14: ((++1) + g())
end for
Table 3: Additional empirical results of mixed policies in the math reasoning task. For the mixed policies, we use a linear policy initially and switching to an exponential policy when the linear phase ceased to grow. The results show that mixed policies outperform linear policies and are comparable to the exponential policies.
POLICY
SYMBOLIC
P1
P2
LINEAR
64.24
40.84
19.17
MIXED
65.98
45.56
20.16
EXPONENTIAL
65.66
47.26
20.65
trends across different curves and policy classes remain consistent when comparing these two figures. This ablation study further demonstrates the superiority of the proposed approach.
E,5
Mixed Policies
We also explore mixed policies empirically. Specifically, we use linear policy initially, switching to exponential policy when the linear phase ceases to yield growth. We present the results in Table 3 where we directly combine the linear and exponential schemes. The results show that mixed policies do not outperform the pure exponential policy. While this approach could be general strategy, the increased degrees freedom in parameter tuning make hyperparameter selection considerably more challenging. We leave further exploration of this direction to future work.
more
G
Diseussion
G.1
Impact Statement
This research advances the field of machine learning by addressing the strategic allocation of compu- tational resources in the post-training phase using synthetic data. We introduce novel thcoretical
35

---

Opized Ricwed
Conprnadaraa Cort
Compuriionl Cort
10, T = 15, 0 = (1, 1)
oxpeecsie
2
Ghe Opine Rowaal
Oongradaaaa Cor
Compuraional Cal
(b) Tt 15,T T
10, 0 (1, 1)
lanr
oxpecciia
1410"
Oytroal
Oog to
127
20
750
10, T 10,0 0 (0.5, 0.5)
x:
Ohg Opined Boomd Boood
150
25
250
Cumpradaeal
Compuriional
(d) n 10, T 10, 2)
Figure 5: Additional experimental results of the toy example with diverse parameters.
framework to optimize the distribution of a fixed budget across multiple training iterations, an ap- proach that demonstrates significant improvement in model performance. Our findings challenge conventional constant allocation strategies, revealing that dynamically adjusted policies, particularly those with exponential growth, not only converge more stably but also do at a reduced compu- tational cost. This work has broad implications for the development of Al models, especially in domains where the generation of synthetic data critical for enhancing model capabilities without the extensive need for expensive or hard-to-obtain real-world data. The practical applications of this research extend to fields such as autonomous driving, medical image analysis, and any other area where Al must perform with high stability and efficiency under constraints of limited labeled data. This study thus provides foundation for more sustainable AI development practices that prioritize both performance and computational efficiency.
36

---

Ablation
Ablatica for Rg
PSNK
PONNI
100
01-
020
1.00
1.
100:
10:
B
1.2
1000
2.00
0.00
1.0
539
925
125
1.0
Conpurational Cost LLUPPS x10"))
Computatioal Cost (HLOPs, xI0'
Figure 6: Ablation study for the image denoising task. scheme under the 10 setting with different parameters.
We show the performance of exponential
expoeemtall nn
10 1.5
exponsstial n,
expeasstial.. & 10-0- 30-8'-0
GSM_ symbolici
GSN_pl
GSM_p2
Accrray 240
Alcuaay
Compuazional (FLOPs x
Congataaional (FLOS, x13P)
Compracissal (FLOPs 1
3--2"
20 2
GSM_ symbolic
GSM_pl
GSM_p2
Accracy (8)
Accuracy (50
Compuaaional Cos (ELOPs x10
Compataiooall CoLL,, x10
Compuratooal Coa (FLOPs, x10"))
Figure 7: Ablation study for the math reasoning task. We show the performance of exponential scheme with different parameters.
G.2 Connection with Model Collapse
Prior work has introduced verification (Feng et al., 2024 Firdoussi et al. 2025) and correction (Gillman et al. 2024) mechanisms for synthetic data to mitigate model collapse. They have theoreti- cally derived one-step conditions under which collapse avoided or triggered. Our methodology crucially relics the reward model-based data verification process, using reward signals to filter data to prevent model collapse.
Empirically, we observe that increasing policies are generally more resistant to collapse. For constant policies, even with reward-based curation, the model's performance initially improves but then degrades over successive iterations. Increasing polices yield smoother, generally upward trajectories, though some decline under very extensive finetuning. These observations extend beyond the scope of the analyses in (Feng et al. 2024 El Firdoussi et al. 2025) and suggest the need for a deeper theoretical analysis. We will add this discussion on model collapse the paper and leave for future work.
37

---

orssunt, Re 10-B
liner, no 3t-
Bazar,
ormsoati, Ry 100-8
linor, by 90x-B
(SMeeybblic
GSN_p1
GSM_p2
2
A00030 (50
Acarna
12
Comprntioal xI0")
Computationll ConOTOOPS x19")
Comperzion x10*))
Figure 8: Additional empirical results of math reasoning where the temperature is 0.7. Other hyperparameters remain the same them in Figure 4 i.c., the training batch size still set B 256 and the exponential policy is still n4 = 10 2 B.
G.3 Limitation
In this paper, we focus achieving efficient SFT improvement within iterative learning framework. Recent trends (Guo et al. 2025) incorporate reinforcement learning into the entire post-training phase, while SFT remains important component for enabling the model to exhibit certain behaviors and skills necessary for successful RL training later (Gandhi et al.) 2025). We aim to extend the theory to develop efficient training scheme for RLHF and RL from verifiable rewards.
We also examine learning with access to reward model with (Appendix B.5) or without independent random noises. An empirical reward, such using an LLM as a judge, could produce biased results and lead to reward hacking. We leave the exploration of biased rewards to future work.
We conduct two experiments to validate our theoretical results. However, incorporating the SFT scheme design into production scale is beyond our resources at academic institutions. We present theoretical results and experimental findings across modalities that support the generalizability of our results.
38


Reviews:
[{"review_id": "wOYApbmrmX", "reviewer": "Reviewer_5fr2", "summary": "Multiple studies have shown that generative models can be retrained on their own (carefully curated) outputs in order to enhance their performances, which can be done iteratively multiple times. This paper tackles the question of how to split your training budget across the retraining iterations in order to maximize the performances of the generative model. The paper proposes experiments on toy data, image generation and coding tasks.", "questions": "- I have a question on the theoretical setup and the experimental protocol:\n    - The differences between the policy only lies in the number of sample at each step? If this is the case, could the variation  in performance of each method only be du to the number of samples with large reward in the training set?\n- Do the empirical and theoretical results suggest adding samples with large rewards as soon as possible (over the retraining iterations) in the training set?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The study tackles a very timely problem: how to spend you retraining budget?\n- The paper is clearly written\n\nWeaknesses:\n- As mentioned by the authors themselves, experimental evaluation can be seen as limited in this setup: except for the code generation experiment, for which a natural meaningful reward exists. IMO, this weakness is not critical", "was_revised": true, "final_justification": "I think this is a good paper, hence I keep my score: 5", "rebuttal": "We sincerely thank the reviewer for their thoughtful evaluation and for highlighting the key strengths of our work. We are especially grateful for the recognition that our study tackles a timely and important problem concerning the optimal allocation of training budgets in iterative synthetic data bootstrapping. We also appreciate the reviewer’s positive remarks on the clarity and quality of our manuscript.\n\n> [W1] As mentioned by the authors themselves, experimental evaluation can be seen as limited in this setup: except for the code generation experiment, for which a natural meaningful reward exists. IMO, this weakness is not critical\n\nWe agree that our study is limited to scenarios in which a natural or oracle reward signal is directly available. Our contribution is deliberately scoped: given such a reward model, we investigate how to maximize bootstrapping gains. By contrast, learning a high-quality reward proxy when no natural signal exists is, in our mind, an orthogonal problem. Recent work continues to push reward modelling forward on that front by training judges and leveraging rubrics.\n\n> [Q1&Q2] The differences between the policy only lies in the number of sample at each step? If this is the case, could the variation in performance of each method only be du to the number of samples with large reward in the training set? Do the empirical and theoretical results suggest adding samples with large rewards as soon as possible (over the retraining iterations) in the training set?\n\nIndeed, the primary difference among our proposed strategies lies in the number of synthetic samples curated at each iteration. Our theory shows that constant policy will suffer from the noise introduced by the fixed-size sample set, preventing reliable convergence. Therefore, we propose increasing policy, which generates more and more samples across iteration to control the noise down in order to finally converge stably.\n\nThe resulting performance differences are not due to the number of samples with high rewards, and our empirical and theoretical results do not suggest simply \"adding samples with large rewards as soon as possible in the training set\". To clarify, we cannot **actively** or **directly** inject high-reward samples into the training set in our setup. As highlighted in the box at line 103, the curation pipeline is identical across all settings: the model under training generates candidates, and the same reward model filters them to curate good examples. This mechanism is fixed; the sole variable is the total number of training samples per iteration, not the number of large-reward ones. To be optimal in budget, the sample size should grow progressively with each iteration rather than start out high \"as soon as possible\".\n\nWe thank the reviewer once again for the time and effort. Please let us know if we have addressed all the questions."}, {"review_id": "sXrggk3Tn7", "reviewer": "Reviewer_mn1i", "summary": "The paper explores in iterative supervised fine-tuning where the generated samples from the current model are filtered with certain reward functions and then used for the next iteration of training, actually needs a linear or exponential number of samples as more iterations are performed given a fixed overall budget. Empirical studies also support the theoretical findings in the math reasoning and umage denoising tasks.", "questions": "1. In theorem 4.2 and 4.3, it implies the error with go down as we increase T which is natual, but doesn't it imply the number of samples/total budget will grow for polynomial growth or exponetial growth policies? How can they use the same budget as the constant policy? Or is $n_0$ from growing policies much smaller than the constant policy?\n2. Since getting a large number of synthetic datapoints is easy, have you tried not limiting the budget and just using more examples in grwoth policies and does that give even better performance?\n3. I am not aware of similar arguments in RL that the fixed number of samples per iter is suboptimal. Could you comment on the intuition about the differences?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper has very interesting findings both theoretically and empirically. It basically says that the model will need more samples for later stages in the iterations when doing self-bootstrapped training via theoretical insights. \n2. It seems novel since I am not aware of other works doing this analysis.\n3. The writing is generally clear and easy to understand. \n\n\nWeaknesses:\n1. The experimental results have no error bars.\n2. The range of the tasks in the experiments is somehow limited.\n2. In theorem 4.2 and 4.3 the role of the constraint of the overall budget is confusing. See Q1.", "was_revised": true, "final_justification": "Thank you for the rebuttal. It mostly solved the concern and I will keep my score.", "rebuttal": "We sincerely thank the reviewer for their comprehensive and highly encouraging comments. We are particularly grateful for the recognition of our paper's very interesting and novel findings, both theoretically and empirically. We appreciate the acknowledgment of our core insight that models benefit from an increasing number of samples in later stages of iterative self-bootstrapped training, supported by rigorous theoretical analysis. \n\n> [W1] The experimental results have no error bars. \n\nWe agree that including error bars would further strengthen the analysis. However, providing error bars for iterative bootstrapping experiments, especially for generating data and training 8B models, consume more compute than we can access in an academic lab.\n\nWithin these constraints, we chose to spend our budget on breadth rather than repetition: we benchmarked the methods across a diverse set of tasks and carried out an extensive ablation study in Figure 6, 7, and 8. We believe this strategy still offers compelling evidence of the effectiveness and generality of our approach.\n\n> [W2] The range of the tasks in the experiments is somehow limited. \n\nWe would like to respectfully clarify that our empirical validation was deliberately designed to cover diverse settings, thereby providing robust support for our theoretical findings:\n\n1. **Theoretical validation**: The Gaussian toy experiment provides controlled verification of Theorem 3.1 under multiple parameter configurations.\n\n2. **Image denoising**: The image denoising task evaluates two different noise levels ($s=0.01$ vs $0.02$) with comprehensive ablation studies on exponential policy parameters (Fig. 6).\n\n3. **Math reasoning**: Crucially, our LLM experiments employ *three progressively challenging math datasets* (GSM-symbolic, GSM-p1, GSM-p2) specifically designed to test generalization across difficulty levels. We further validate robustness through ablations:\n   - Multiple policy hyperparameters (Fig. 7)\n   - Different generation temperatures (T=0.3 vs 0.7 in Fig. 4 vs Fig. 8)\n   - Full-weight fine-tuning of Llama-3-8B (non-trivial computational scale)\n\nWe hope this combination of cross-domain validation (vision and language) and the wide coverage of ablations provide strong empirical evidence for our conclusions.\n\n> [W3 & Q1] In theorem 4.2 and 4.3 the role of the constraint of the overall budget is confusing. ... In theorem 4.2 and 4.3, it implies the error with go down as we increase T which is natual, but doesn't it imply the number of samples/total budget will grow for polynomial growth or exponetial growth policies? How can they use the same budget as the constant policy? Or is $n_0$ from growing policies much smaller than the constant policy?\n\nThank you for raising this point. We would like to clarify how the overall budget constraint comes into our theoretical development.\n\n**Theorems 4.1–4.3.** These three results analyze the convergence of the respective policies (constant, polynomial, and exponential) as the number of iterations (T) grows. Theorem 4.1 shows that constant policies generally fail to converge with high probability. Theorems 4.2 and 4.3 demonstrate the convergence of polynomial and exponential policies, respectively. At this stage, the analysis tracks error decay without imposing any limit on total compute budget.\n\n**Introducing the Budget.** Only after obtaining the unconstrained convergence results, we introduce the budget constraint from line 205 and present the theory in **Theorem 4.4**. Equation (2) defines the minimum number of iterations each policy requires to achieve a target reward. Theorem 4.4 compares the computational cost of different policies and shows that, under a fixed budget, the exponential schedule reaches the target reward most efficiently, outperforming both the constant and polynomial schedules.\n\nWe will improve the presentation of this section, emphasising that the budget constraint enters only after the initial convergence analysis.\n\n> [Q2] Since getting a large number of synthetic datapoints is easy, have you tried not limiting the budget and just using more examples in growth policies and does that give even better performance? \n\nBuilding on the previous point, Theorems 4.1–4.3 show that, in the absence of any compute constraint, growth policies converge better.\n\nIn reality, \"getting a large number of synthetic datapoints\" online in iterative bootstrapping is not easy: at every iteration, the current model needs to generate new synthetic data. For both language models (using next token prediction) and diffusion models, the cost of generation can, in fact, significantly exceed the cost of model training:\n\n* For Diffusion Models, generating each sample necessitates a sequence of many denoising steps, each involving a forward pass through the model. \n\n* For LLMs, the serial nature of inference (multiple forward propagations are required to generate each token) results in an overall time cost that may instead be slower than training [1,2]. \n\nTherefore, we consider the realistic setup where there is a budget constraint. No matter how large the budget, Theorem 4.4 shows that the exponential schedule yields the best performance for a fixed total compute.\n\n[1] Kim, Kyoungmin, et al. \"Optimizing LLM Inference for Database Systems: Cost-Aware Scheduling for Concurrent Requests.\" arxiv preprint arxiv:2411.07447 (2024).\n\n[2] Shashank and Neal. \"Mastering LLM Techniques: Inference Optimization\" Technical Blog of Generative AI\n\n> [Q3] I am not aware of similar arguments in RL that the fixed number of samples per iter is suboptimal. Could you comment on the intuition about the differences?\n\nThis is a really good question. Please allow us to restate our insights: we demonstrate that constant policies fail to converge to optimality with high probability, while increasing policies converge. This crucial difference arises directly from **the noise introduced by finite samples in the gradient estimation step**, and the constant sample size brings noises in gradient estimation whose variance does not decrease as the number of iterations increases. \n\nA similar issue also exists in RL. Many theoretical RL works often assume access to a \"ground truth\" / noiseless policy gradient at each step, without considering the direct impact of finite sample noise [1,2]. Empirical RL often employs techniques such as replay buffers to enlarge the sample size with offline data to help stabilize the optimization. However, this approach also has its limitations and requires extensive tuning.\n\nIn our self-improvement scenario, we find that the noise contributed by finite samples is particularly significant. An increasing policy can progressively reduce this noise and stabilize the learning process, leading to convergence. \n\n[1] Agarwal, Alekh, et al. \"On the theory of policy gradient methods: Optimality, approximation, and distribution shift.\" Journal of Machine Learning Research 22.98 (2021): 1-76.\n\n[2] Xiao, Lin. \"On the convergence rates of policy gradient methods.\" Journal of Machine Learning Research 23.282 (2022): 1-36.\n\nWe thank the reviewer once again for the time and effort. Please let us know if we have addressed all the questions."}, {"review_id": "KcU4Wh9KGm", "reviewer": "Reviewer_hQEz", "summary": "This paper addresses the budget allocation problem for iterative synthetic data bootstrapping. Iterative synthetic data bootstrapping is a training process where models generate synthetic data, filter the data using reward models, and fine-tune the model iteratively to maximize expected rewards. The authors develop a theoretical framework to analyze optimal allocation of computational resources across iterations to maximize model performance. Their main contribution is proving that constant allocation policies fail to converge with high probability, while exponential growth policies achieve optimal performance with exponential convergence rates. The theoretical analysis is validated through experiments on Gaussian toy problems, image denoising with diffusion models on MNIST, and mathematical reasoning with LLMs on synthetic arithmetic datasets.", "questions": "* Did the authors explore mixed policies—for example, using a linear policy initially and switching to an exponential policy if the target performance is not achieved? This could offer a more flexible strategy in practice.\n\n* Have the authors evaluated the performance of the exponential policy in scenarios where the assumptions underlying the theoretical analysis are violated? It would be useful to understand how robust the method is in such cases.\n\n* The caption for Figure 1 appears to be merged with the image. Could the authors correct this formatting issue in the next revision?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\n* The paper presents a clear and well-motivated problem formulation for an important task. It appears to be the first systematic evaluation of budget allocation in iterative synthetic data bootstrapping. However, I cannot fully assess the novelty of the contribution, as this is not my primary area of expertise.\n\n* Given that training on synthetic data is a key component of modern generative modeling pipelines, the proposed method has the potential for substantial practical impact.\n\n* The experiments span both vision and language domains, demonstrating the generality and versatility of the approach.\n\n* The paper is well-written and easy to follow.\n\n* The theoretical analysis is thorough and well-structured, offering insights into both the benefits of the exponential policy and the convergence issues with constant policies.\n\n### **Weaknesses**\n\n* The primary weakness is the limited scope of experimental evaluation. The vision experiments are restricted to MNIST, and the language experiments focus on basic arithmetic tasks. Additionally, the use of oracle reward models reduces the realism and practical relevance of the setup. While this limitation alone is not a reason for rejection, the paper would be significantly strengthened by experiments on more complex and realistic tasks.\n\n* It is not clear to me how model collapse is prevented when fine-tuning on synthetic datasets using this method. The experiments suggest that additional fine-tuning continues to improve performance, but it would be interesting to see more analysis on when, and under what conditions, the model begins to degrade due to excessive fine-tuning on synthetic data.", "was_revised": true, "final_justification": "This paper tackles the budget‐allocation problem in iterative synthetic‐data bootstrapping—a critical component of tuning models to optimize a given reward function. In light of its solid theoretical and practical contributions, I vote to accept and encourage the authors to include results on more complex scenarios—such as reward‐based optimization of text‐to‐image models—in the final version.", "rebuttal": "We sincerely thank the reviewer for their comprehensive and thoughtful comments. We highly appreciate the recognition of our paper's clear and well-motivated problem formulation for a timely task, and its systematic evaluation of budget allocation in iterative synthetic data bootstrapping. We are also grateful for the acknowledgment of our demonstration on the generality and versatility of our approach. \n\n> [W1] The primary weakness is the limited scope of experimental evaluation. The vision experiments are restricted to MNIST, and the language experiments focus on basic arithmetic tasks. Additionally, the use of oracle reward models reduces the realism and practical relevance of the setup. While this limitation alone is not a reason for rejection, the paper would be significantly strengthened by experiments on more complex and realistic tasks.\n\nWe thanks the reviewer for acknowleding our experiments \"span both vision and language domains, demonstrating the generality and versatility of the approach\". \n\nRegarding the choice of datasets. Our guiding principle was to select datasets that allow a rigorous test of the theory while remaining tractable under academic-scale compute: MNIST is one of the image benchmarks on which diffusion models have not been extensively pre-trained, making it a clean test bed for iterative bootstrapping effects. GSM-Symbolic is a de-contaminated math dataset that remains challenging for 8B instruct models. These settings let us demonstrate our method across two modalities (images, text) and model families (diffusion, LLM) in a controlled environment.\n\nRegarding the oracle reward. We work in regimes where a reliable reward (oracle or natural) is available. Our contribution is to show how, given such a reward model, one can maximize bootstrapping gains. Learning a high-quality reward proxy when no natural signal exists is, in our mind, an orthogonal problem. \n\n> [W2] It is not clear to me how model collapse is prevented when fine-tuning on synthetic datasets using this method. The experiments suggest that additional fine-tuning continues to improve performance, but it would be interesting to see more analysis on when, and under what conditions, the model begins to degrade due to excessive fine-tuning on synthetic data.\n\nPrior work has introduced verification [1,2] and correction [3] mechanisms for synthetic data to mitigate model collapse. They have theoretically derived one-step conditions under which collapse is avoided or triggered. Our methodology crucially relies on the reward model-based data verification process, using reward signals to filter data to prevent model collapse. \n\nEmpirically, we observe that increasing policies are generally more resistant to collapse. For constant policies, even with reward-based curation, the model's performance initially improves but then degrades over successive iterations. Increasing polices yield smoother, generally upward trajectories, though some decline under very extensive finetuning. These observations extend beyond the scope of the analyses in [1,2] and suggest the need for a deeper theoretical analysis. We will add this discussion on model collapse to the paper and leave it for future work.\n\n[1] Feng, Y., Dohmatob, E., Yang, P., Charton, F., and Kempe, J. Beyond model collapse: Scaling up with synthesized data requires reinforcement. arXiv preprint arXiv:2406.07515, 2024.\n\n[2] Firdoussi, Aymane El, et al. \"Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory.\" arXiv preprint arXiv:2410.08942 (2024).\n\n[3] Gillman, Nate, et al. \"Self-correcting self-consuming loops for generative model training.\" arXiv preprint arXiv:2402.07087 (2024).\n\n> [Q1] Did the authors explore mixed policies—for example, using a linear policy initially and switching to an exponential policy if the target performance is not achieved? This could offer a more flexible strategy in practice. \n\nThanks for this suggestion. Mixed policies, such as initiating with a linear strategy and adaptively switching to an exponential one, are indeed flexible practial solution. We will try the following hyperparameter tuning strategy: we begin with a modest growth rate, then gradually increase the growth rate when performance (evaluated on a small validation dataset) stops growing. Given the computational demands of this experiment, we will include the results in the final version.\n\n> [Q2] Have the authors evaluated the performance of the exponential policy in scenarios where the assumptions underlying the theoretical analysis are violated? It would be useful to understand how robust the method is in such cases.\n\nThanks for this question. We appreciate the reviewer’s interest in understanding the robustness of our method beyond the theoretical assumptions. Our current theoretical framework is built on a relatively mild set of assumptions.\n\nThe core dependence of our theoretical results are on Assumption B.1. This condition is standard in prior learning-theory analyses, e.g., on the application of covering numbers [1,2,3], and on the continuity and convexity of the loss function [4,5,6].\n\nFor general settings, it is inherently difficult to verify whether the assumptions are satisfied, or to analytically characterize their violation. This challenge is not unique to our work, but reflects a broader limitation in theoretical research, where it is often necessary to make simplifying assumptions in order to derive nontrivial insights. We believe this is a reasonable tradeoff, especially since our assumptions are not tailored to any specific experimental design.\n\nAll of our empirical results—from both vision and language domains—align with the theoretical insights, further supporting their practical robustness.\n\n[1] Zhou, Ding-Xuan. \"The covering number in learning theory.\" Journal of Complexity 18.3 (2002): 739-767.\n\n[2] Suzuki, Taiji, Hiroshi Abe, and Tomoaki Nishimura. \"Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network.\" International Conference on Learning Representations. 2020.\n\n[3] Jin, Ming, et al. \"On solution functions of optimization: Universal approximation and covering number bounds.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 7. 2023.\n\n[4] Bassily, Raef, et al. \"Private stochastic convex optimization with optimal rates.\" Advances in neural information processing systems 32 (2019).\n\n[5] Zou, Difan, et al. \"Gradient descent optimizes over-parameterized deep ReLU networks.\" Machine learning 109 (2020): 467-492.\n\n[6] Yang, Long, et al. \"Policy optimization with stochastic mirror descent.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 8. 2022.\n\n> [Q3] The caption for Figure 1 appears to be merged with the image. Could the authors correct this formatting issue in the next revision?\n\nThanks for pointing it out. We will correct it in the next version.\n\nWe thank the reviewer once again for the time and effort. Please let us know if we have addressed all the questions."}, {"review_id": "Hts1F0obsp", "reviewer": "Reviewer_9EDy", "summary": "This paper investigates iterative synthetic data bootstrapping for post-training of generative foundation models such as LLM, utilizing synthetic data and existing reward models. In this setting, synthetic data generated from the generative model is input into the reward model at each iteration, and the decision to adopt it as training data is made based on its reward score. Subsequently, the model is updated through supervised fine-tuning (SFT) using the synthetic training data. This bootstrapping method has already been theoretically analyzed, and it is known to converge optimally under the assumption of infinite samples. This paper theoretically analyzes the behavior of iterative synthetic data bootstrapping in a more realistic problem setting constrained by a fixed budget. Specifically, it addresses the research question of how many synthetic samples should be generated at each iteration. To this end, the paper conducts (i) a theoretical analysis of a Gaussian distribution toy model and (ii) a theoretical analysis under realistic settings with mild assumptions, demonstrating that exponentially increasing the number of synthetic samples per iteration is optimal in terms of convergence to optimal performance. Experiments provide observational results supporting this theoretical finding through diffusion models and LLM post-training.", "questions": "Could you answer or comment on the concerns in the weakness sections?", "limitations": "The paper discusses the application of paradigms other than SFT and the application to production scale as limitations in Section B. These discussions are adequate.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n+ **S1.** The paper is well structured and clearly describes the research questions and contributions. Post-training is an important phase in operating foundation models such as LLM, and reducing its cost is a topic of interest. The theoretical results of this paper are not limited to toy models but are also demonstrated in practical settings, indicating a wide range of applications.\n+ **S2.** The empirical experiments to validate the theory are appropriately designed.\n+ **S3.** There is potential for further development toward more practical settings. While this study focuses on SFT, expanding this discussion could advance theoretical analysis of post-training using RLHF or GRPO in the future.\n### Weaknesses\n- **W1.** The practicality of the experimental results is somewhat weak, as discussed in Section B. In particular, the paper would have had a greater impact if there had been experiments suggesting the extent of the effect on general post-training of LLM rather than on specialized tasks. However, I understand that the main contribution of the paper is theoretical analysis and its verification, and please note that this comment does not request these experiments for the rebuttal.\n- **W2.** In order to properly implement an exponential growth policy, hyperparameter tuning and a validation dataset are necessary. Investigating the extent to which this affects performance in realistic settings with a small validation set would further highlight the strengths of this paper.", "was_revised": true, "final_justification": "I appreciate the authors' responses. The explanations in the rebuttal have largely resolved my concerns regarding hyperparameters. I recommend accepting this paper.", "rebuttal": "We sincerely thank the reviewer for their thorough and encouraging assessment. We greatly appreciate the recognition of our paper's clear structure, well-defined research questions, and significant contributions to the important field of post-training cost reduction for foundation models. We are also grateful for the acknowledgment that our theoretical results are not limited to toy models but are demonstrated in practical settings, indicating a wide range of applications.\n\n> [W1] The practicality of the experimental results is somewhat weak, as discussed in Section B. In particular, the paper would have had a greater impact if there had been experiments suggesting the extent of the effect on general post-training of LLM rather than on specialized tasks. However, I understand that the main contribution of the paper is theoretical analysis and its verification, and please note that this comment does not request these experiments for the rebuttal.\n\nWe thanks for the reviewer's understanding. Our aim is to conduct rigorous, scientifically sound comparisons at scales our academic resources allow, articulate the resulting insights clearly for the community, and leave it for the engineering teams at large companies to integrate them into frontier-scale post-training.\n\n> [W2] In order to properly implement an exponential growth policy, hyperparameter tuning and a validation dataset are necessary. Investigating the extent to which this affects performance in realistic settings with a small validation set would further highlight the strengths of this paper. \n\nWe appreciate the reviewer’s thoughtful comment. We fully agree that in practice, implementing a good exponential growth policy requires choosing an appropriate increasing rate, and it is hard to find the optimal value a priori. \n\nThat said, our results still recommend an exponential schedule (Theorem 4.4). Both constant and polynomial policies also require careful tuning of parameters (e.g., number of samples for the initial iteration, growth rate), yet the exponential policy carries a provable compute-efficiency advantage.\n\nIn our ablation studies, we observe that exponential policies is reasonably robust to variations in hyperparameters in Figure 7 and 8: increasing the base from 2 to 3 shifts the peak performance by less than 1 %. This suggests that while tuning is necessary, the exponential strategy does not require fine-grained search to yield strong performance in practice.\n\nWe thank the reviewer once again for the time and effort. Please let us know if we have addressed all the questions."}]

Author rebuttal:

Meta-review: In post training of generative models (LLM or Diffusion Models), synthetic data training with Verifier selection is a key paradigm. This paper answers the question that what should be the scheduling policy of number of generated synthetic data points given a total budget. Theoretical analysis are made, from simple Gaussian data to practical reward models, and concluded exponential policy on the scheduling works the best. The theories are also supported by empirical results on both image generation and LLM math reasonings. All reviewers uniformly agreed that this paper has good contribution and should be accepted.
Decision: Accept (spotlight)