Title: Spend Wisely: Maximizing Post-Training Gains in Iterative Synthetic Data Bootstrapping
Venue: NeurIPS 2025
Accepted: yes
Keywords: synthetic data

self supervised fine-tuning

iterative learning

boostrapping

budget allocation
Abstract: Modern foundation models often undergo iterative ``bootstrapping'' in their post-training phase: a model generates synthetic data, an external verifier filters out low-quality samples, and the high-quality subset is used for further fine-tuning. Over multiple iterations, the model performance improves, raising a crucial question: How should the total budget for generation and training be allocated across iterations to maximize final performance? 
In this work, we develop a theoretical framework for analyzing budget allocation strategies. Specifically, we show that constant policies fail to converge with high probability, while increasing policies---particularly exponential growth policies---exhibit significant theoretical advantages. 
Experiments on image denoising with diffusion probabilistic models and math reasoning with large language models show that both exponential and polynomial growth policies consistently outperform constant policies, with exponential policies often providing more stable performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates iterative synthetic data bootstrapping for post-training of generative foundation models such as LLM, utilizing synthetic data and existing reward models. In this setting, synthetic data generated from the generative model is input into the reward model at each iteration, and the decision to adopt it as training data is made based on its reward score. Subsequently, the model is updated through supervised fine-tuning (SFT) using the synthetic training data. This bootstrapping method has already been theoretically analyzed, and it is known to converge optimally under the assumption of infinite samples. This paper theoretically analyzes the behavior of iterative synthetic data bootstrapping in a more realistic problem setting constrained by a fixed budget. Specifically, it addresses the research question of how many synthetic samples should be generated at each iteration. To this end, the paper conducts (i) a theoretical analysis of a Gaussian distribution toy model and (ii) a theoretical analysis under realistic settings with mild assumptions, demonstrating that exponentially increasing the number of synthetic samples per iteration is optimal in terms of convergence to optimal performance. Experiments provide observational results supporting this theoretical finding through diffusion models and LLM post-training.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n+ **S1.** The paper is well structured and clearly describes the research questions and contributions. Post-training is an important phase in operating foundation models such as LLM, and reducing its cost is a topic of interest. The theoretical results of this paper are not limited to toy models but are also demonstrated in practical settings, indicating a wide range of applications.\n+ **S2.** The empirical experiments to validate the theory are appropriately designed.\n+ **S3.** There is potential for further development toward more practical settings. While this study focuses on SFT, expanding this discussion could advance theoretical analysis of post-training using RLHF or GRPO in the future.\n### Weaknesses\n- **W1.** The practicality of the experimental results is somewhat weak, as discussed in Section B. In particular, the paper would have had a greater impact if there had been experiments suggesting the extent of the effect on general post-training of LLM rather than on specialized tasks. However, I understand that the main contribution of the paper is theoretical analysis and its verification, and please note that this comment does not request these experiments for the rebuttal.\n- **W2.** In order to properly implement an exponential growth policy, hyperparameter tuning and a validation dataset are necessary. Investigating the extent to which this affects performance in realistic settings with a small validation set would further highlight the strengths of this paper.\n\nQUESTIONS:\nCould you answer or comment on the concerns in the weakness sections?\n\nLIMITATIONS:\nThe paper discusses the application of paradigms other than SFT and the application to production scale as limitations in Section B. These discussions are adequate.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the budget allocation problem for iterative synthetic data bootstrapping. Iterative synthetic data bootstrapping is a training process where models generate synthetic data, filter the data using reward models, and fine-tune the model iteratively to maximize expected rewards. The authors develop a theoretical framework to analyze optimal allocation of computational resources across iterations to maximize model performance. Their main contribution is proving that constant allocation policies fail to converge with high probability, while exponential growth policies achieve optimal performance with exponential convergence rates. The theoretical analysis is validated through experiments on Gaussian toy problems, image denoising with diffusion models on MNIST, and mathematical reasoning with LLMs on synthetic arithmetic datasets.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n* The paper presents a clear and well-motivated problem formulation for an important task. It appears to be the first systematic evaluation of budget allocation in iterative synthetic data bootstrapping. However, I cannot fully assess the novelty of the contribution, as this is not my primary area of expertise.\n\n* Given that training on synthetic data is a key component of modern generative modeling pipelines, the proposed method has the potential for substantial practical impact.\n\n* The experiments span both vision and language domains, demonstrating the generality and versatility of the approach.\n\n* The paper is well-written and easy to follow.\n\n* The theoretical analysis is thorough and well-structured, offering insights into both the benefits of the exponential policy and the convergence issues with constant policies.\n\n### **Weaknesses**\n\n* The primary weakness is the limited scope of experimental evaluation. The vision experiments are restricted to MNIST, and the language experiments focus on basic arithmetic tasks. Additionally, the use of oracle reward models reduces the realism and practical relevance of the setup. While this limitation alone is not a reason for rejection, the paper would be significantly strengthened by experiments on more complex and realistic tasks.\n\n* It is not clear to me how model collapse is prevented when fine-tuning on synthetic datasets using this method. The experiments suggest that additional fine-tuning continues to improve performance, but it would be interesting to see more analysis on when, and under what conditions, the model begins to degrade due to excessive fine-tuning on synthetic data.\n\nQUESTIONS:\n* Did the authors explore mixed policies—for example, using a linear policy initially and switching to an exponential policy if the target performance is not achieved? This could offer a more flexible strategy in practice.\n\n* Have the authors evaluated the performance of the exponential policy in scenarios where the assumptions underlying the theoretical analysis are violated? It would be useful to understand how robust the method is in such cases.\n\n* The caption for Figure 1 appears to be merged with the image. Could the authors correct this formatting issue in the next revision?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper explores in iterative supervised fine-tuning where the generated samples from the current model are filtered with certain reward functions and then used for the next iteration of training, actually needs a linear or exponential number of samples as more iterations are performed given a fixed overall budget. Empirical studies also support the theoretical findings in the math reasoning and umage denoising tasks.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper has very interesting findings both theoretically and empirically. It basically says that the model will need more samples for later stages in the iterations when doing self-bootstrapped training via theoretical insights. 
2. It seems novel since I am not aware of other works doing this analysis.
3. The writing is generally clear and easy to understand. 


Weaknesses:
1. The experimental results have no error bars.
2. The range of the tasks in the experiments is somehow limited.
2. In theorem 4.2 and 4.3 the role of the constraint of the overall budget is confusing. See Q1.

QUESTIONS:
1. In theorem 4.2 and 4.3, it implies the error with go down as we increase T which is natual, but doesn't it imply the number of samples/total budget will grow for polynomial growth or exponetial growth policies? How can they use the same budget as the constant policy? Or is $n_0$ from growing policies much smaller than the constant policy?
2. Since getting a large number of synthetic datapoints is easy, have you tried not limiting the budget and just using more examples in grwoth policies and does that give even better performance?
3. I am not aware of similar arguments in RL that the fixed number of samples per iter is suboptimal. Could you comment on the intuition about the differences?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

comments: SUMMARY:
Multiple studies have shown that generative models can be retrained on their own (carefully curated) outputs in order to enhance their performances, which can be done iteratively multiple times. This paper tackles the question of how to split your training budget across the retraining iterations in order to maximize the performances of the generative model. The paper proposes experiments on toy data, image generation and coding tasks.

STRENGTHS AND WEAKNESSES:
Strengths:
- The study tackles a very timely problem: how to spend you retraining budget?
- The paper is clearly written

Weaknesses:
- As mentioned by the authors themselves, experimental evaluation can be seen as limited in this setup: except for the code generation experiment, for which a natural meaningful reward exists. IMO, this weakness is not critical

QUESTIONS:
- I have a question on the theoretical setup and the experimental protocol:
    - The differences between the policy only lies in the number of sample at each step? If this is the case, could the variation  in performance of each method only be du to the number of samples with large reward in the training set?
- Do the empirical and theoretical results suggest adding samples with large rewards as soon as possible (over the retraining iterations) in the training set?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02