Improving the Scaling Laws of Synthetic Data with Deliberate Practice
Authors: Reyhane Askari-Hemmat
Mohammad Pezeshki
Elvis Dohmatob
Florian Bordes
Pietro Astolfi
Melissa Hall
Jakob Verbeek
Michal Drozdzal
Adriana Romero-Soriano
Venue: ICML 2025 oral (2025)
Keywords: Synthetic Data
Deliberate Practice
Active Learning
Sample Efficiency
Scaling Laws
Data Curation
Diffusion Models
Dataset Pruning
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: Deliberate Practice for Synthetic Data Generation (DP) dynamically generates informative samples to improve scaling efficiency, reducing sample requirements and training iterations while achieving superior performance.

Abstract: Inspired by the principle of deliberate practice in human learning, we propose Deliberate Practice for Synthetic Data Generation (DP), a novel framework that improves sample efficiency through dynamic synthetic data generation. Prior work has shown that scaling synthetic data is inherently challenging, as naively adding new data leads to diminishing returns. To address this, pruning has been identified as a key mechanism for improving scaling, enabling models to focus on the most informative synthetic samples. Rather than generating a large dataset and pruning it afterward, DP efficiently approximates the direct generation of informative samples. We theoretically show how training on challenging, informative examples improves scaling laws and empirically validate that DP achieves better scaling performance with significantly fewer training samples and iterations. On ImageNet-100, DP generates 3.4x fewer samples and requires six times fewer iterations, while on ImageNet-1k, it generates 8x fewer samples with a 30% reduction in iterations, all while achieving superior performance compared to prior work.

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
Reyhane Askari-Hemmat
Mohammad Pezeshki" 1 Elvis Dohmatob 123 Florian Bordes Pietro Astolfi
Melissa Hall Jakob Verbeek Michal Drozdzal
Adriana Romero-Soriano 1345
Abstract
refining learning through feedback, leads to real progress. This principle highlights that effective learning requires ex- posure to informative and difficult examples rather than passive repetition.
Inspired by the principle of deliberate prac- tice in human learning, we propose Deliberate Practice for Synthetic Data Generation (DP), a novel framework that improves sample efficiency through dynamic synthetic data generation. Prior work has shown that scaling synthetic data is in- herently challenging, as naively adding new data leads to diminishing returns. To address this, prun- ing has been identified as key mechanism for improving scaling, enabling models to focus on the most informative synthetic samples. Rather than generating a large dataset and pruning it after- ward, DP efficiently approximates the direct gen- cration of informative samples. We theoretically show how training on challenging, informative examples improves scaling laws and empirically validate that DP achieves better scaling perfor- mance with significantly fewer training samples and iterations. On ImageNet-1000 DP generates 3.4x fewer samples and requires six times fewer iterations, while on me- it generates 8x fewer samples with a 30% reduction in iterations, all while achieving superior performance com- pared to prior work.
In contrast, most machine learning models are trained on pre- collected data that remain static throughout training, limiting their ability to dynamically adapt to their own weaknesses. One promising source of data for visual recognition tasks is large-scale pre-trained text-to-image models (Rombach et al., 2022). They provide an essentially infinite source of synthetic training data, presenting an alternative to real- world datasets, which
often expensive or infeasible to curate (Hemmat et al., 2023; Shin et al., 2023; Zhang et al., 2024). With the great promise of text-to-image models, a natural question arises: what is the potential of learning using only synthetic data? Empirical studies show that increasing the volume of synthetic training data often leads to diminishing returns, with performance gains following a power law stagnation (Fan al., 2024; Tian al., 2024a). Instead, pruning to remove uninformative examples has proven effective in improving the effectiveness of training with real or synthetic data (Sorscher et al., 2022; Kolossov et al., 2024; Feng et al., 2024).
Inspired by human learning principles and recent advances in generative image models, we propose the Deliberate Prac- tice (DP) for Synthetic Data Generation framework. Unlike static approaches that generate all synthetic training data upfront (Fan et al., 2024; Shin et al., 2023; Hemmat et al., 2023), our framework incorporates a dynamic loop between a diffusion model and a downstream learner throughout the training. More concretely, rather than generating an entire dataset at once and irrespective of the learner and then prun- ing it to remove uninformative samples, we propose DP to efficiently generate data directly from the pruned distribu- tion of informative samples. By leveraging the learner's s prediction entropy to guide the generation process, our ap- proach generates only the most challenging and informative training examples.
1. Introduction
A key principle underlying learning in human is deliberate practice proorssog made not by repeating what is already known but by continuously engaging with tasks that stretch the limits of one's abilities (Friesson et al., 1993). For example, when learning to play the guitar, simply prac- ticing songs that one has mastered does little to improve skill. Instead, targeted practice on challenging tasks and
'Equal contribution 'FAIR at Meta Montreal, Paris, and New York City labs "Concordia University 'Mila "MeGill Uni- versity 'Canada CIFAR Al chair. Correspondence to: Rey- hane Askari Hemmat <reyhancaskariee motta.ommmm...mm Mohammad Pezeshki <mpezeshki@ meta.com?.
Our framework operates dynamically: we begin with an initial set of synthetic data and train learner until perfor- mance on real validation set plateaus. At this point, the learner's 's entropy is used to guide the diffusion model to gen-
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
erate new challenging examples. These examples are added to the training set, and the process repeats, ensuring that the model is continually exposed to increasingly informative data throughout training.
generative model synthesize training examples.
Formally, let y' denote the set of class labels. Our goal is to train classifier fo X' + V/ parameterized by o. which maps inputs x E X (e.g., images) to labels y E y. We are given a predefined label set Y a fixed (small) validation set D = {(2,, yi)) consisting of real data for evaluation, and generative model go capable of sampling synthetic data conditioned on label, i.e., I go However, no real training data is available, i.e., D = @. The objective is to train Sa using few generuted examples possible while maximizing generalization to real data as measured by performance on D The key challenge is to generate minimal yet effective training data, requiring a principled mechanism to select/generate informative examples.
This approach aligos with broader goals in machine learning, such as interactive learning environments, continual learn- ing (Kirkpatrick et al.. 2017), and active learning (Settles, 2009). By leveraging a dynamic loop, Deliberate Practice reduces inefficiencies from redundant or already learned data, thereby improving the scaling laws of training with synthetic data.
Our contributions are summarized as:
We introduce the Deliberate Practice for Synthetic Data Generation framework, which dynamically adds new data points when the learner's validation accu- racy plateaus [Section 3]. Our framework leverages the learner's prediction entropy to generate challeng- ing synthetic data, improving the scaling behavior of synthetic data (Figures 1 and 4).
The Need for Informative Examples. Not all synthetic samples contribute equally to learning. Prior work shows that simply increasing the synthetic dataset size leads di- minishing returns, as many generated samples are redundant or too easy (Fan et al., 2024). Instead, training should focus on examples that maximize learning efficiency.
We provide a theoretical analysis of the scaling behav- ior of simple model trained on selected examples (Section 4). Using random matrix theory, we charac- terize the test error as a function of data size and the example selection function, showing improved scaling when prioritizing hard and informative examples. We show that entropy-guides sampling approximates generating from an entropy-pruncd distribution (Sec- tion 2). We empirically validate that DP can improve the validation accuracy compared to direct pruning while being remarkably cheaper in compute up to 5x (Figure5) 5).
Given a measure of informativeness for a synthetic sample xt, one approach is to generate a large dataset and prune un- informative examples. Formally, let Dewal {(.., y: ) be large set of N generated samples. We define a prumed dataset as D
{ U.) i [N], = 1}. where qi E {0, is a selection variable determining whether a data point (æx, y;) E Desol is retained. The subset size is constrained by m -i=l= to as the over-sampling ratio.
Σ q.. The quantity N/m is referred
Let P and Q denote the distributions of the original and pruned datasets, respectively. The pruning process operates as an importance sampling scheme:
We demonstrate that DP outperforms prior work on both ImageNet-100 and ImageNet- Ik while requiring significantly less data and fewer training iterations. On ImageNet-1000 our approach generated 3.4x less samples and completed training in only one-sixth of the iterations used in prior work, yet still achieved superior performance. Similarly, on ImageNet- 1k, we generated 8x less samples and reduced the number of iterations by 30,, while outperforming previous results (Table 1).
dQ = π dP,
(1)
where π is a normalized weighting function that retains the informative samples. The generate-then-pee ensures that only informative examples are kept, is com- putationally inefficient, as many generated samples are discarded. This motivates the need to devise mechanisms to directly sample the informative examples.
approach
Furthermore, DP exhibits strong performance out- of-distribution (OOD) datasets, even outperforming models trained with real data on ImageNet-R and ImageNet-SSetch, with improvements of up to 15% (Table 1).
Approximate Sampling of Informative Examples. Sup- pose that Drool is generated using a diffusion model with induced probability P. The generative process is governed by reverse SDE (Song & Ermon, 2019):
dx = [v(x, - g(1) V (x) di + g(() dW())
2. Problem Formulation
(2)
Problem Setup. Standard supervised learning relies on a large real labeled training set. Here, however, we assume no real training data is available, and instead, we must rely on
where W(t) is a Wiener process, modeling stochastic noise, o(x, t) is a drift term, g(t) is a coefficient controlling the noise level at time f, and log Pe (x)) is the score function.
2

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
Conventional methods: Expensive one-time selection of challenging samples
13M synthetic data
synthetic data
1. 3M synthetic data with DP
Massive amount of Data
Challenging samples
Selection
Prompt
Generator
Selector
Criterion
10x loss data
50.
(Acording thelcomert
Learnr
Top- Accuracy
Deliberate Practice: Efficient continuous generation of challenging samples
Challenging
Selection Criterion
Prompt
Criterion-Guided Generator
samples
(Dynamic)
Mocced to leanted)
Learner
Figure 1: (Top): Conventional approaches generate (or collect) massive static dataset and then select challenging examples in one-time filtering step based on the learner's s selection criterion. This is inefficient, as most generated data is discarded. (Bottom): DP continuously generates only the most challenging examples based on continuous feedback from the learner, eliminating the need for large-scal data pruning. This iterative process ensures that training focuses on progressively informative examples, improving efficiency and performance. (Right): Top-1 validation accuracy on ImageNet-Il with models trained solely on synthetic data. DP (orange) achieves higher accuracy than the 13M synthetic data setup (blue) while using 10x fewer samples, significantly outperforming the 1.3M baseline (gray).
Instead of sampling from P. we aim to sample directly from Q as in Eq. (1). By Girsanov's 's theorem (Oksendal, 2013), modifying the probability measure from P to Q introduces a correction term in the reverse SDE:
in which 6." (zz, v) approximates the conditional score fune- tion using pretrained denoising network (Ho & Salimans, 2022):
cg (Z y) Rd (1+ A)εs(z, (x (x)
(5)
dr = [u(x, (VV) log V (logr(z,()) di + +1(()
where À is called the classifier-free guidance coefficient which controls the strength of conditional sampling on the label.
(3)
The term V log T (2x f) effectively modifies the score func- tion and biases the sampling distribution according to the weighting function =(x,t). t). This modification allows ap- proximating direct sampling from the pruned distribution Q, eliminating the need to first sample uniformly from P and later prune the data.
An efficient way of sampling from a modified diffusion mode as described in Eq. 3 was proposed by Hemmat et al. (2023), where the weighting function derived from the entropy of the downstream learner, such that,
log π x H(fo(zo)) = Σ foly xo) log foly mo). nEY
2.1. Efficient Guided Sampling with DDIM.
(6)
We leverage denoising diffusion implicit models (DDIMs) (Song et al., 2020) for efficient sampling. At each step I, the reverse update for generating conditional sample is:
To compute the entropy as in Eq. 6, we need the denoised sample zo- The term &o, can be used to cheaply approxi- mate entropy mid- dedd-generation. This allows direct sampling of high-entropy examples by modifying the score function:
xt-1 = ---1-0 Ve-1100 + V1- 61- - of (t) (xe, v) + J+Ct
(xz. (x,, y) (2) (xt, v) + W (ff())))))
(7)
direction pointing
midiom noise
where controls the contribution of the entropy entropy-guidance
where 6 is random noise and a, and &-1 are time- dependent coefficients. The term Zo, approximates the final denoised sample:
In (Hemmat et al., 2023), real data is used to pre-train the learner, enabling an accurate estimation of V., (fo(o.))) However, when real data is unavailable, alternative ap- proaches are needed to assess sample informativeness. In the next section, we propose to leverage the learner itself
VI E (II.V) v)
0
(4)
VEi
3

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
Algorithm 1 Deliberate Practice for Synthetic Data Genera- tion
validation accuracy A(ʃs: D is evaluated. If no improve- ment is observed for Tiax intervals (patience threshold), the framework triggers new data generation.
Input: Class labels V, Generative model go. Valida- tion set D Initial dataset size N, New data size P, Patience Tmax. Evaluation interval
Entropy guided sampling, When the patience mechanism triggers, P new examples Deww = [(Tg U are gener- ated. We directly generate samples from the entropy pruned distribution through entropy guided sampling, The entropy is computed based on the current stage of the classifier fø. The coefficient controls the effect of entropy-guidance guidance. With = O. we fall back into regular sampling of diffu- sion models, while w > 0 results in generations that have a higher entropy under the classifier.
2: Output: Trained classifier fø
3: Initialize: Generate Dí with N examples from go-
Start training fo with learning-rate warm-up.
4: Set patience counter T + 0.
5: while training do
6: Update So mini-batch drawn uniformly from
7: if (every iterations) then
8:
Evaluate validation accuracy A(f )
Training resumption. The newly generated examples are added to the dataset, D D Drew. After augment- ing the dataset, training resumes with a constant learning rate until the patience mechanism is triggered again. Mini- batches are drawn uniformly from the updated pool, which grows dynamically from size N to N + +k after k iterations of augmentation. This cycle is continued until we reach the cool-down phase where the learning rate is decreased and no more new data is added. See Figure for training dynamics of a classifier training with DP.
9:
Reset T < 0 if accuracy improves; else increment
+.+11
10:
end if
11:
IT 2 Timax then
12:
Generate P new examples Dnew with feedback:
13:
Vx, log
v) 21, log
log po (2+) +
=V., (ff(((((
14:
Augment training set: Die + DK. U Drew.
15:
Reset + O.
16:
end if
In Section 4, we provide intuitive theoretical framework to study the scaling behavior of simplified DP. In Sec- tion 5, we validate the effectiveness of DP in large-scale experiments.
end while
18:
Finalize: Apply learning rate decay.
during training to evaluate entropy and determine the infor- mativeness of generated samples dynamically.
4. Training on Informative Examples Improves the Scaling Laws
3. The Deliberate Practice Framework for Synthetic Data Generation
Before presenting empirical results, we first analyze how selecting informative examples affects the scaling of syn- thetic data. We study a high- high-dimensional linear classifier trained with uniform vs. selective sampling and derive an analytic expression for test error using random matrix the- ory (RMT). Our results show that selecting hard examples improves scaling laws, providing theoretical justification for our approach.
In this section, we describe our Deliberate Practice frame- work, in which we efficiently train the learner with synthetic data in absence of any real data. In particular, we move to a setup where we dynamically expand the dataset throughout the training. Our framework summarized in Algorithm
The initial training data. The framework begins by gen- erating initial set of N synthetic training examples D = {(2,, y:) J.N using pre-trained generative model go. For each class y/ E y. the generative model samples images x; ~ (o())) in a class-conditional manner. The clas- sifier fo starts training on this dataset, with a learning-rate warm-up phase.
4.1. Theoretical Analysis under an Idealized Setup.
Consider a simple generative model for training data:
I N, E), y y= ww (w) x),
(8)
where Wp E Rd is the ground labeling function. This gives a distribution on R2d TR.
Iterative training and additional data. Training proceeds iteratively with a mechanism to dynamically augment the dataset whenever the classifier's performance stagnates. The process alternates between training the classifier and gener- ating new synthetic examples.
We study the impact of uniform sampling versus selective sampling of informative examples on generalization. To formalize this, we assume pool of n i.i.d. training pairs:
Patience mechanism. At regular iteration intervals, 7, the
X EE Raxd YER".
(9)

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
4.5
Sie:: 120K
Total Data Size: 130K
Norm up phose
dation Accurary
Cool phase herations
Taining Loss
3.5-
added
50
w
Date Sire: 130c 130k
Total Data Size: 130k
2.0
Worm-up phase
Conkdoan which
10k 20k 30k
40k 50k 10k 20k
30k
40k
50k
Iterations
Rerations
Figure 2: Training loss (left) and validation accuracy (right) of Deliberate Practice on 100 The classifier begins training on an initial static dataset (130k samples) until validation accuracy plateaus. At this point, additional samples are generated using entropy-guice guided sampling, focusing on hard/informative examples. The two dashed vertical lines indicate points where new data is added. We compare three setups: (1) Orange: No additional data is added, training only on the initial dataset. (2) Purple: One round of antropy-guidee data generation adds 130k samples. (3) Blue: Two rounds of entropy-guide data generation, adding 260k samples total. Each data addition leads to accuracy boost, demonstrating the effectiveness of DP in improving performance with fewer training iterations. For clarity, this figure shows only two rounds of data addition, but in practice, more rounds occur based on the allowed maximum patience. Notably, while training loss increases with new data, validation accuracy steadily improves, showing that the model benefits from progressively challenging examples, ultimately reducing the generalization gap.
linear classifier u is trained using the following loss:
et al., 2024) to characterize the test error in Eq. (13), Our analysis based on the spectral density of the resolvent matrix R in Eq. (12), allowing us compute the first two moments of yæe i for test sample and derive pression for the test error. For simplicity, we assume an isotropic setup where = Ia and defer the general case to Appendix A.
A Σ VallwTrs,, vi) + 2 [w/l (10)
à arg min w
where ((2,)) = (=- u) 2 is the squared loss, À > 0 is regularization parameter, and q: qa ws) is a selection strategy that determines whether an example is included in training based on its projection in given direction w, E Rd and an arbitrary measurable binary function R + 1)
We shall work in the following so-called high-dimensional proportionate scaling regime
which encodes the selection strategy.
d, xo, d/n o,
(14)
The selection/pruning ratio is given by:
in which the input-dimension d and the sample size n di- verge to infinity at the same rate. The scalar á E (0, oo) cap- (11) tures the effective dimensionality or over-parametrizztioo rate of the problem.
p E[q(x' w.)] for (O, Σ).
The resulting classifier has closed-fome solution:
Key Scalars. WLOG, assume ||w,||| 1. It turns that the for fixed, pruning, p. the asymptotic test error is fully captured by the following scalars:
i RR R: (xxXXxXxxXX
(12)
n
p = w wo/ Soo/l/woll.
P VI VI- r: -[=G]GG))
where E Raxn is a diagonal matrix with Dús = qs-
Our objective is to analyze the asymptotic test error of di:
2Egg(G))(TG)),
3 3 (TG(G))(
(15)
Frass i) (siign(x T () + v),
(13)
where G ~ N( 1) with pdf oo and cdf T. Note that p quantifies the alignment between the pruning direction w, and the ground-truth labeler wp. while 3 and y capture statistical properties of the pruning strategy
where (xr a test example,
4.2. Asymptotic Behavior of the Test Error.
We leverage random matrix theory (RMT) techniques (Couillet & Liao, 2022; Liao & Mahoney, 2021; Firdoussi
Spectral functions. The Stieltjes transform m of the limit- ing spectral density of the resolvent matrix shown in
5

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
90
4.2.1. EXAMPLE: SELECTING INFORMATIVE EXAMPLES.
85
Consider selection function of the form q: g( w.) for all i, where,
Accuracy
75
70
1,
i
<() = 104 S 0, else,
(18)
85
Select top
Select top
Select top 80%
for some threshold 5 2 0. Such selection strategy selects only the examples near the decision boundary of w,, analo- gous to using classifier entropy
55
Select top 90%
Select
2'
2
selection criterion but simpler to study, Lemma and 2 derive explicit expres- sions for (7, B, (). Figure presents theoretical predictions for test accuracy across different degrees of example selec- tion, showing that selecting hard examples improves scaling laws, reducing the number of training samples needed for the same performance. However, beyond certain point, excessive pruning degrades performance, illustrated in Figure 5.
Size of Selected Dataset
Figure 3: Theoretical prediction for scaling behavior of ac- curacy (Theorem 1) for simple classifier in d = 512 dimensional input space, as a function of dataset selection strategy. The classifier is trained on synthetic data with different pruning probabilities, where higher pruning prob- ability corresponds to keeping only the most challenging examples (those closer to the decision boundary).
4.2.2. ADAPTIVE SELECTION STRATEGY.
Lemma 3 to be given by the exact formula (with X)
Data selection relies on pruning direction w, to select informative/hard examples: +> q(x] w.) E {0, 1), but these examples are ultimately used to train
m(z)
2
(p Ø 2)
402
(16)
w, and ú are misaligned, what is considered hard by w, may be hard for w, reducing the effectiveness of selective sampling, In fact, hard examples change over time: an example that was identified hard, might not remain hard are more train- ing is done. To ensure alignment, Ws should periodically update to reflect the evolving decision boundary w. This adaptive selection mechanism motivates the continuous data generation process of DP, as presented Section 3.
2Øz
and will play an important role in our theory. The above formula represents a somewhat distorted Marchenko-Pase law. Indeed, the classical MP (Marčenko & Pastur, 1967) corresponds to p > (i.e. no data pruning).
We further define the following auxiliary functions:
Data selection relies on pruning direction w, identify informative hard examples: i + 9(2; w.) {0, 1}. However, these selected examples are ultimately used to train w. and if w. and v are misaligned, what is considered hard by to may not be hard for vē, reducing the effectiveness of selective sampling. other the more i is trained on these examples. Moreover, the definition of "hard" changes was initially difficult may become casier training pro- gresses. To maintain alignment, w., should be periodically updated to reflect the evolving decision boundary of ιb This adaptive selection mechanism underpins the continuous data generation process in DP, as presented in Section 3.
s(z) = I+ <m((z)
m(z) = a(=)
r(z) = w² m(2) + :2 mm(),
(17)
with w = /I - B²B, w=== p3.
fact, W. and t deviate from each
time-an example that
Main Result: Test Error Scaling w.r. Selection Strategy.
Theorem 1. In the limit Eq. (14), the classification test error satisfies: (v) > arccos (Imal/ /vn) /T,, where
ma == wm(->) + @mm(-)
20m' (-2) No = pom' (->) + r' (-A) 1 vimll (-))r(AA)
5. Experiments
For all the experiments, we use the LDM1.5 (Rombach et al., 2022) as the pre-trained text-to-image (T2I) model. We studied four different T2I models and found this model outperforming the rest. For more details see Appendix D.
The scaling behavior of test error is fully determined by the six scalars d. P. 3, 3,3). 3). Importantly, the choice of the data point selection strategy i + qi wx) only influ- ences performance through P. 7, B, and
Datasets.
We validate our framework on two datasets.
6

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
ImageNet-100 (Tian et al., 2020; Samyildız al., 2023), subset of ImageNet-1k (Deng et al., 2009), containing 100 classes and 5k validation examples, where the real val- idation set is used for evaluation and the real training set (126,689 examples) serves as a held-out test set. We also conduct experiment ImageNet-Ik, Ik, using the 50k validation examples to monitor performance and reserving the real training set (1.3 million examples) as a held-out test set.
names for prompting and publicly available models for sample generation. Performance is assessed real Im- ageNet (held-out) training and validation sets, well as on ImageNet- V2 (Recht et al., 2019), SeageNet-Sketch (Wang et al., 2019), ImageNet-R (Hendrycks et al., 2021a), and ImageNet-A (Hendrycks et al., 2021b) to measure out-of- distribution (OOD) generalization.
The results in Table show that DP outperforms prior bench- marks both ImageNet- 100 and ImageNet-1K 1k while re- quiring significantly less data and fewer training iterations. On ImageNet-1000 DP generated 4.6 million fewer samples and trained for only one-sixth of the iterations compared to previous works, yet achieved superior performance on the real data. Similarly, on ImageNet- Ik, DP reduced sample generation by 56.2 million and cut training iterations by over 30%, while still outperforming previous results.
5.1. Scaling Laws of Synthetic Data
We train a Vision Transformer (ViT-B) (Dosovitskiy et al., 2021) classifier with synthetic data. We study two scenarios: 1) Static data generation and 2) Deliberate Practice (DP). In all the experiments in this section we have a fixed and controlled setup. We train the models for 100k and 50k iterations for ImageNet- Ik and ImageNet- 100 respectively. For additional details, see Appendix D.5.
Furthermore, models trained with DP exhibit strong per- formance on -off--i-istiittionn datasets, even surpassing models trained on real data on ImageNet-R R and ImageNet- Sketch, with improvements of up to 15%.
Static data generation. In this setup, all data is generated before training. and the classifier is trained on a fixed dataset. We experiment with different dataset sizes see its impact on accuracy.
5.3. Connection Between Pruning and DP
Deliberate Practice data generation. Hyperparameters w and are tuned on ImageNet-100 and found effective for ImageNet-Ik Ik as well (see Section D.5 for details). We track validation accuracy throughout training and use it to determine when to generate new data, following patience- based criterion. To ensure the model has not over-fitted to the validation set, we also report accuracy on the full real training sets of ImageNet- 100 and ImageNet-Ik Ik, used as held-out test sets.
In Section 2, we discussed how DP approximates direct sampling from a pruned distribution. Here, we validate this experimentally on ImageNet- 100 using two setups:
1. Oversampling then Pruning: Generate a large pool and select high-entropy samples.
2. Direct entropy-guided generation: Generate only informative samples (a special case of DP with single step of data addition).
Figure 4 compares the scaling laws of the Static and De- liberate Practice (DP) on ImageNet- 100 and ImageNet- 1k. On both datasets, we note that DP scales well with dataset size and it consistently outperforms the Static setup, achiev- ing higher validation accuracy at any given dataset size. On ImageNet- 100 we observe that DP can reach the best ac- curacy of the static setup (with 3 million examples) using only 400k examples. This means that DP requires 7.5x less data to reach the same performance. On ImageNet-Ik Ik, we observe that DP can outperform the best accuracy of the static setup (with 13 million examples), using only 640k examples. This translates to DP requiring 20x less data to outperform the Static setup. For additional details on the hyper-parameters of these experiments, see Appendix 1.5.1. Refer to Figure 13 for a visualization of how the dataset evolves from the start to the end of training.
We start with 130k generated samples (regular vanilla sam- pling), train for 17k iterations, then add a one-time addi- tional 130k samples, increasing the total data size to 260k and training for an additional 33k iterations.
In setup 1, we vary the pool size, ranging from no pruning (130k pool) up to an oversampling ratio of 18 (2.4M pool), selecting the top 130k high-entropy samples. In setup 2, we generate exactly 130k entropy-guided samples, varying the cntropy-gauiddan coefficient.
Figure 5 (a, b) shows that both methods improve perfor- mance up to a point, after which excessive selection of high-entropy samples leads to degradation due to selecting high-entropy but harmful outliers. This aligns with our theoretical predictions in Figure 5 (c).
5.2. Comparison with Previous Work
Regarding computational costs, generating single image with entropy-guidance on an Nvidia H100 takes 1.82x longer than standard vanilla sampling. However, achiev- ing similar performance through oversampling requires sig- nificantly more data, leading to a linear increase in cost.
We compare DP with prior works on synthetic data gener- ation for image classification (Sanyıldız et al., 2023; Fan et al., 2024). Specifically, we evaluate setups that use class-
7

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
98
50
66
Top-1 Accurary
Top-2 Accuracy
64-
58
DP Static
DP Stati:
56
(los)
(log)
Figure 4: Scaling laws of synthetic data. Real Validation accuracy total dataset size for the Static (pink x). and Deliberate Practice (blue o) setups ImageNet- 100 (left) and ImageNet- 1k (right). DP significantly outperforms Static data generation, achieving higher accuracy with fewer synthetic examples. DP achieves the same accuracy the static setup using 7.5x x less data on ImageNet- 100 and 20x less data while outperforming it on ImageNet-IK.
100
Table Comparison with previous work. DP outperforms other models on both ImageNet-100 and ImageNer- Ik while requiring significantly less data and fewer training iterations. Note that DP experiments reported in this table are trained longer than models reported in the previous section and, consistent with other work, use smaller classifier free guidance scale of À 2.
Task
# Iters
Data size
IN real Val.
IN real tr.
IN-v2
IN-Sk
IN-R
IN-A
Real
IN-100
100k
130k
76.4
37.1
60.8
33,5
Syn. Static (Saryyildieed et al., 2023)
IN-100
13k
130k
63.5
62.7
41.8
64.2
13.7
Syn. Static (Sarıyıldized al., 2023)
IN-100
635k
6.5M
73.3
72.3
42.0
59.4
17.1
Syn. DP (ours)
IN-100
100k
1.9M
74.3
75.0
66.3
52.0
76.6
25.9
Real
IN-IK
200k
1.3M
82.6
70.9
32.5
44.6
29.4
Syn. Static (Saryyılıi al., 2023)
IN-1K
130k
1.3M
42.9
43.0
16.6
26.3
3.6
Syn. Static (Fan al., 2024)
IN-IK
210k
2M
42.2
27.2
45.7
Syn. Static (Fan et al., 2024)
IN-1K
315k
64M
54
46.0
32.4
52.5
9.4
Syn. DP (ours)
IN-IK
200k
6.5M
54.1
54.84
48.5
34.7
56.0
12.3
Syn. DP (ours)
IN-IK
200k
9.1M
55.1
55.73
49.3
36.0
57.2
13.4
As result, DP is 5x more efficient while also providing higher absolute improvements compared to pruning-base baase selection. See Figure for details and Figure 11 for some visualizations.
the selection of informative examples throughout training, as done Deliberate Practice (DP). See Figure 12 for some visualization generations through training.
6. Related Work
5.4. The Evolution of Hard Examples Over Time
Synthetic data for training neural networks. Synthetic data has become powerful tool for training machine learn- ing models across various domains. For instance, text-to- image diffusion models have been successfully used for visual representation learning (Astolfi et al., 2023; et al., 2025; Tian al., 2024a;b; Sanyıldız al., 2023). How- ever, limitations of synthetic data highlighted by Fan et al. (2024), emphasizing the importance of generating more challenging and informative examples. Addressing distribution shifts between synthetic and real data, Hemmat et al. (2023) and Yuan et al. (2023) propose synthesizing training data that matches real data distributions or condi- tioning on real examples reduce this gap. Expanding small-scale datasets has also been studied, see e.g. Zhang
"Does the sample hardness change as training progresses?"
To answer this question, Figure tracks the error on exam- ples that were misclassified the time they were added. As expected, once introduced, the model gradually Iearns to classify them correctly. However, an interesting trend emerges: even before these examples were added, their error was lower than at the moment of inclusion. This suggests that the notion of hardness is dynamic- whatt is considered challenging at one point may become easier over time. Con- easy might later become
versely, examples that were difficult due to shifts in the learned decision boundaries. This highlights key limitation of static pruning approaches and underscores the importance of dynamically adapting
8

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
(8)
(b)
(c)
(d)
65.0
65.0
65.0
Explicit
DP
Theory prediction
Explict: pruning
64.5
64.5
82.5
64.5
5x
Accuracy
63.5
Accuracy
63.5
Accurac
77.5
Accuracy
63.5
63.0
63.0
75.0
63.0
62.5
62.5
72.5
62.5
62.0
61.5
67.5
61.0
61.0
65.0
61.0
10x
15x
0.00
0.04 0.06
Over-sampling ratio
DP coefficient
Over-samp Ing ratio
Cost (gpu.hour)
Figure 5: Plots describing the performance of DP compared to explicit pruning and theory prediction while changing the oversampling ratio or the DP coefficient. (a) Over-sampling with entropy- selection Generate a large pool of samples (ranging from 130k 2.4M) and select the 130k highest-entropy examples.
(b) Generate 130k high-entropy examples directly using DP with varying entropy guidance strength through w. (c) The theory prediction the accuracy based the over-sampling ration. (d) Comparing the compute cost of DP vs oversampling then pruning. We observe that DP exhibits similar accuracy curve compared to explicit pruning and theoretical prediction when changing the over wver-samllnggDD coefficient. However, DP is computationally remarkably more efficient while gaining more accuracy delta.
prioritize iterative model updates with tailored data. These methods highlight the importance of selecting informative samples based on the model's s current state. (Sorscher et al., 2022) showed that pruning static datasets using metries like margin scores improve scaling laws by retaining the most informative examples, albeit in non-adaptive manner.
Hard Exomples Error
0.6
Challenges and risks of synthetic data. The challenges of training models on synthetic data, have gained significant attention. Dohmatob et al. (2024a;b) studied "model col- lapse", a phenomenon where iterative training on synthetic data degrades performance. They emphasize that data verifi- cation mechanisms can mitigate this risk and enable scaling with synthetic data. Similarly, our framework by generating informative examples through dynamic loop, improves sample efficiency.
nhtll
0.4
addud
Data wdited 27k
Data added at 26k
0.2
Data acded 10t
Data added 34k
Data acded 36k
Whin phose
10k
22k
20k
30k.
34836k
50k
Training Iterations
Figure 6: Error trajectories of hard (misclassified) examples added at different training stages. The red curve highlights the first batch of added data for better visibility, but the same trend applies to all batches. Notably, even before being trained on, these examples exhibit a lower error rate than at their point of inclusion, indicating that hardness is static, it evolves throughout training.
7. Conclusion
We introduced Deliberate Practice for Synthetic Data Gen- eration, framework that improves scaling laws by dynami- cally generating challenging and informative training exam- ples. Unlike traditional methods that rely on static datasets, our approach approximates generating data directly from pruned distribution, reducing inefficiencies and ensuring models continuously training on informative samples. We provided theoretical insights into the benefits training on pruned distributions and empirically demonstrated that method significantly improves performance while requiring fewer training iterations. Our results on ImageNet-100 and ImageNet- IK show that Deliberate Practice achieves supe- rior accuracy with far less data and compute, outperforming previous tfate-of-the-art. Our work highlights the potential of structured synthetic data generation in advancing efficient and adaptive learning.
et al. (2024). Another related line of work involves using VLMs and LLMs to generate descriptions for augmenting datasets (Dunlap et al., 2023).
Synthetic data increasingly used to train (LLMs). For example, LLaMA3 (Grattafiori et al., 2024) employs AI- generated data for fine-tuning. Similarly, self-play ap- proaches, e.g., Yuan et al. (2024), align with our framework by generating increasingly difficult examples for training.
Continual learning and active learning. Our work is also closely related to principles from active learning (Bang et al., 2024; Evans al., 2023) and continual learning, which

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
Impact Statement
Evans, T., Pathak, S., Merzic, H., Schwarz, J., Tanno, R., and Henaff, J. Bad students make great teachers: Ac- tive learning accelerates large-scale visual understanding. arXiv preprint. 2312.05328, 2023.
This work introduces a method for improving the sample efficiency of synthetic data generation through deliber- ate practice framework that prioritizes the most informative training examples. By enabling more efficient use of gener- ated data, our approach may reduce the computational and environmental costs associated with training large models, and make data generation more accessible in low-resource settings.
Fan, L.,, Chen, K., Krishnan, D., Katabi, D., Isola, P., and Tian, Y. Scaling laws of synthetic images for model training... for now. In CVPR, 2024.
Feng, Y., Dohmatob, E.,, Yang. P., Charton, F,, and Kempe, J. Beyond model collapse: Scaling up with synthesized data requires reinforcement, 2024. URL https:/ //axxiv. org/ abs /400 240..4007077 07515.
References
Astolfi, P., Casanova, A., Verbeek, J., Vincent, P., Romero- Soriano, A., and Drozdzal, M. Instance-conditioned gan data augmentation for representation learning. arXiv preprint arXiv:2303.09677, 2023.
Firdoussi, E., Seddik, M. E. A., Hayou, S., Alami, R., Alzubaidi, A., and Hacid, H. Maximizing the potential of synthetic data: Insights from random matrix theory, 2024.
Astolfi, P. Careil, M., Hall, M., Mañas, O., Muck- ley, M.. Verbeck, J., Soriano, A. R., and Drozdzal, pareto fronts con-
Grattafiori, A. et al. The Ilama herd of models, 2024. URL https:/ / waxii..orro wwtp/oo///abba////ee
M. munsissecy-ddiversityyeee ditional image generative models. arXiv preprint arXiv:2406. 10429, 2024.
Hemmat, R. A.. Pezeshki, M., Bordes, F., Drozdzal, M., and Romero- -ooero-Sooiano, A. Feedback-guided data synthesis for imbalanced classification. arXiv preprint, 2310.00158, 2023.
Bang, J., Ahn, S., and Lee, J.-G. Active prompt learning in vision language models. In CVPR, 2024.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M.. et al. The many faces of robustness: critical analysis of dut-o--distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1340-8349) 2021a
Couillet, R. and Liao, Z. Random Matrix Methods for Machine Learning. Cambridge University Press, 2022.
Deng, J., Dong, W., Socher, R., L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 228..5..00 Ieee, 2009.
Hendrycks, D., Zhao, K., Basart, S., Steinhardtt, J., and Song, D. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15262-15271, 2021b.
Dohmatob, E., Feng, Y., Subramonian, A., and Kempe, J. Strong model collapse. arXiv preprint 24X0v:2410.04840, 2024a.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12549 2022.
Dohmatob, E., Feng, Y., Yang, P., Charton, F,, and Kempe, A tale of tails: Model collapse as change of scaling laws. arXiv preprint 2740:33402.07443, 2024b.
Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao W., et al. Minicpm: Unveil- ing the potential of small language models with scalable training strategies. arXiv preprint, 2404.06395, 2024.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Trabska-Barwinge
A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13):3521-3222 2017.
Dunlap, L., Umino, A., Zhang, H., Yang, J., Gonzalez, J. E.,, and Darrell, T. Diversify your vision datasets with automatic cufuusionbbase augmentation. In NeurIPS, 2023.
Kolossov, G., Montanari, A., and Tandon, P. Towards a statistical theory of data selection under weak supervision. The Twelfth International Conference on Learning Representations, 2024. URL https::/openrevieee net: wtrumorrmmmmmmm d= fohghneg.....
Ericsson, K. A., Krampe, R. T., and Tesch-Römer, C. The role of deliberate practice in the acquisition of expert performance. Psychological review, 100(3):363, 1993.
10

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
Li, X., Yang, Y., Li, X., Wu, J., Yu, Y., Ghanem, B., and Zhang, M. Genview: Enhancing view quality with pre- trained generative model for self-supervised learning. In European Conference on Computer Vision, pp. 306-325. Springer, 2025.
Tian, Y., Fan, L., Isola, P., Chang, H., and Krishnan, D. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. In NeurIPS, 2024b.
Wang, H., Ge, S., Lipton, Z., and Xing, P. Learning robust global representations by penalizing local predic- tive power. Advances in Neural Information Processing Systems, 32, 2019.
Liao, Z. and Mahoney, M. W. Hessian eigenspectra of more realistic nonlinear models. In Advances in Neu- ral Information Processing Systems, volume 34, Curran Associates, Inc., 2021.
Yuan, H., Chen, Z., Ji, K., and Gu, Q. Self-play fine-tuning of diffusion models for text-to-image generation. arXiv preprint, 2402. 10210, 2024.
Marčenko, V. A. and Pastur, L. A. Distribution of eigenval- ues for some sets of random matrices. Mathematics of the USSR-Sbornik, 1(4):457, apr 1967.
Yuan, J., Zhang, J., Sun, S., Torr, P., and Zhao, Real- fake: Effective training data synthesis through distribu- tion matching. arXiv preprint, 2310.10406 2023.
Oksendal, B. Stochastic differential equations: intro- duction with applications. Springer Science & Business Media, 2013.
Zhang, Y., Zhou, D., Hooi, B., Wang, K., and Feng, J. Expanding small-scale datasets with guided imagination. In NeurIPS, 2024.
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize imagenet? In Interna- tional conference on machine learning, pp. 5389-5400. PMLR, 2019.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Sarıyıldı, M. B., Alahari, K., Larlus, D., and Kalantidis, Y. Fake it till you make it: Learning transferable rep- resentations from synthetic imagenet clones. In CVPR, 2023.
Settles, B. Active learning literature survey. 2009.
Shin, J., Kang, M., and Park, Fill-up: Balancing long-tailed data with generative models. arXiv preprint, 2306.07200, 2023.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint, 2010.02502, 2020.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Mor- COS, A. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35: 35:19523-19536 2022.
Tian, Y., Krishnan, D., and Isola, P. Contrastive multiview coding. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part 16. pp. 776-794. Springer, 2020.
Tian, Y., Fan, L., Chen, K., Katabi, D.,, Krishnan, D., and Isola, P. Learning vision from models rivals learning vision from data. In CVPR, 2024a.

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
A. Further Theoretical Analysis and proofs
A.I. The Unregularized Regime
We now consider our theory in the limit - 0+. Thus, the parameter vector for the classifier is the least-squares estimate for to, i.e vē irs = x y', Recall the definition of the constants 7/, 8, 3, w, and ß from equations (17). Recall that p E (0, 1] is the proportion of training data left after pruning the original dataset (X, Y) containing n examples. We have the following important corollary to Theorem 1.
Corollary 1. In the (ordered) limit n, d + oo, d/n o, A - 0+, it holds that Exest arccos((al//b)) where the constants and b are given as follows:
(A) If Φ < p, then
b:= p²ó + (r/- 2
= (w + wp/h)/(p------- 6).
(19)
(p 6)3
with ro = f² + p/7
r6 = p- (3² 32 ((p 6)p// + 0/11)
(20)
(B) 1f > p, then
a = (w + w/c)eeo
b = (pé ro)co:
(21)
with co =- 1 p/d,
ai == (p 7)/
ro =- 32 + B/B//1
(22)
The result is empirically verified in Figure 7(a).
250
600
1000
Clossrication orror
0.4
0.3
0.2
keep hard
0.1
0.0
0.2
0.4
0.6
0.8
1.0.0
0.2
0.4
0.6
0.8
1.00.0
0.2
0.4
0.6
0.8
1.0
(a) Regularization parameter
10-6
0.5
Clocofication orter
0.3
easy keep hard
0.1
0.8
0.6 1.00.0
0.8
(b) Regularization parameter
10
Figure 7: Empirical verification of Theorem 1 and Corollary For this experiment, the input dimension is d 350, and each subplot corresponds to different value of the original sample size The experiment for À 10 is proxy for the unregularized case À + O+ Solid lines correspond to observed values of the test error (w), while broken lines are the theoretical prediction of Theorem 1 (bottom row) and Corollary (top row). Notice the excellent match between the experimental results and our theory. Also, observe the multiple-descent patterns, reminiscent of a non-trivial effect of different pruning strategies in different regimes of the pruned training dataset size no = np; the vertical line corresponds to an interpolation threshold p Φ, i.e., ng d.
12

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
A.2. Some Important Examples of Pruning Strategies
Keep Hard Examples (KH). Consider the case where the pruning strategy is given by qi ( w.) for all i, where
(
1, if [t| S E
(k = l|lt| -
(23)
0, clse,
for some { 2 Define a = E//|w,/l We have explicit formula for the constants 3 and 3 appearing in Theorem 1. Viz, p²)
Lemma 1. With p/ / 1 p², (a / p-p - L, and eg 24(Ta) 1, it holds that
2(pp(0))) = - 3(Ąкн) 20(()) /1 p² E1.
(24)
Example 2: Keep Easy Examples (KE). Here, the pruning strategy is qi qKε(o, w.), where
(Kε(1) == 1[14 E = 0, s
(25)
1, else
Lemma 2. With 1 = -
1/0111100a///11111 p²)), c2 20((Ta) - 1, it holds that
Blgke) 2(pp(0)c 2(pp()))( p(a))2), ((Ąke) = -2((0) v p² 61.
(26)
Interpolation between Keep Hard and Keep Easy Strategies. proposed in (Kolossov al., 2024)
Example 3:
Consider the following pruning strategy
(t)) x (1-))(--((t)
(27)
for some mninng parameter w. Here, is the sigmoid function. We can associate q( w.) with the probability the auxiliary classifier æ + sign(x" w,, assigns to an example z;. Thus, positive values of w correspond to keeping examples considered uncertain (i.e hard) by this classifier, while negative values correspond to examples considered easy.
A.3. Main Ingredients of Proofs
A.3.1. DETERMINISTIC EQUIVALENT FOR THE RESOLVENT MATRIX R.
Definition (Deterministic Equivalents). Given sequence of random N x N matrices (Rx) deterministic equivalent thereof is a sequence of deterministic x matrices (Rx)), N such that
tr AN - Rx) 00
(28)
for all sequences of N x N matrices (An)N with bounded Frobenious norm.
Let II (resp. 11, La II) be the projection onto the span (resp. orthogonal complement of the span) of w.. Define the following auxiliary vectors and scalars
1 Wt |
11/2
W.:
11
81
(29)
Note that tn is (d 1) dimeenniia and ||11| Henceforth we make the replacement
1 of.
= - < O. so that the resolvent matrix R now writes
R(s) = (x DX/n -Ia)
(30)
13

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
Let 6(2) be the unique positive solution the fixed-point equation
m(2) d' 1 tr R2),,,
8(2) - tr ERo(=),
(31)
((xTw,
-1
R²(2) (E
E E==Iaa)
zTd
(32)
1 + g(x² wo )
Note that the inner expectation evaluates to
g(x T w.) 1 10 w. (²))
p 1(2),
(o,E)
1+ 5(2)
and so Ro(x) (t(z) - = Observe that Ro(z)(t(2)Z- -2da) = =a. and so ((z)iRo(( = La + =R((2). We deduce that
n - = = + 0. =m(2)))
Thus the equations defining m=)) and 6(=) can be rewritten
m(z) d tr(t(2)E- 2Id)
(33)
1(=) I+ ++))))
P
(34)
6. (1 2m) t(=)((2) = t(2)
1) =p-t(z). t(z)
P
(35)
Solving for (zz((2) in terms of t(2) in the last equation gives
Φzm(2)=
pō(z) -=
p
= p p 10-00-10 t(z).
1 8(2)
1 1+ 8(2)
Plugging this into the first equation gives the following fixed-pointt equation for t(z)
p- = =n =n (r((())) =la)
(36)
The following result shows that R is a deterministic equivalent for R. Proposition Recall the function t(2) as the unique positive solution to the equation (36). Then,
R~ R, with R -/ + m(x) 1/2
(37)
where m(2) t(2) - x
m(2) s(2) - 7 = G() for G ())
1
s(2) 1 d+z(2)=
7
(1/p)t(2),
(38)
(39)
A.4. Isotropic Case
Consider the special case where the covariance matrix is Σ = Ta- It is not hard to see that we must have m(z) = m(2) = 8(2)/ó. Let us now compute m(z). Lemma 3. For every z - m(z) given by formula (16).
Proof. Indeed, observe that in the isotropic case the equation (36) reduces to p- -- t(2) oz/(t(2) 2), or equivalently
Φz + (1(=) - + (1(=) - =) = 1(2)² - (p - o + =)((=) + pz.
The discriminant of this quadratic equation evaluates to
1 2)² 4pz = (p - x - 2 - 4pz
(p - 0 10--) z)² + 42z + 4z(p 0- - 2) -4p
(p- 0 - 2)² 40z,
14

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
and because =- < O, the positive solution is
((z) - 0 +
(p
2)
102
(40)
2
We deduce that
Φ
2 +
V(
(p
2) 402
m(=)
((z)
2
2 p- 2 v
(p 0 2) 400
(p Φ 2) ((p
2) 40z))
p
(p 0 2)² 402
2Φz
which is precisely the claimed formula given in (16).
The following result then follows directly from Proposition 1.
Corollary 2. In the isotropic setting, we have the following deterministic equivalents:
R R. with R m(2)1I + s(2III
(41)
R² ~ m' (2) + m II)
(42)
where mmz):= = 1/(s(z) - 2), s(2) 7/(1 pm and 7 2 0 is as given in (43).
W, WD
8 = E
7 7 B |
(43)
w. wool
A.5. Test Error Representation ("Scaling Laws")
We are now ready to state our main theoretical results, which is a generalization of Theorem 1.
Remark 1. For simplicity of presentation. all our theoretical results only consider symmetric pruming strategies for which ((-t) = qt)). This includes the "keep hard" and "keep easy" pruning strategies considered in (Sorscher et al., 2022).
Proposition 2. Define the following quantities:
mg
C REWD
m = 1+ 8"
ma
/|
(44)
2c T Re
::= (1 + 8)22
Up
1/ = P tr EE' + +CT E'e n
tr EΣ',
(45)
1 8
with c:= E[g.y:0;] = B(z.))~pa(e wo)ya), E' : E [RER].
(46)
Then, in the limit (14), the test error of à is given by
Erest (1)) + - arccos (Imol/ /vo)
(47)
B. Proof of Proposition? 2
The proof follows standard (Couillet & Liao, 2022; Firdoussi et al., 2024) "leave-one-out" techniques which are now standard for analyses based on random matrix theory.
15

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
B.1. Main Idea
For random test point (I. Ps, we can write
yo à = yz T /// sign(2 /x,1/20)) /.
Write _1/20 a.)1/2 +r, where r = / // and a 2 is to be determined. Observe that r perpendicular 1/ wo iff /21/// T Eup aa|55/1// ///l////////// woll 0 iff
a = wo/ Σ wol 1|2
(48)
With this choice a, one computes
yo i = oyz ///2 uo + yz r.
(49)
Because r is perpendicular to w', we know that the above is sum of two independent random variables.
For the first summand in (49), observe that
yz T 1/ yx wo = sign(r' (r T wn) wn = |x | wol,
which has the same distribution as w/ 11 for G1 (0, 1). For the second summand, it has the same distribution as distribution ||r|[22 where G2 ~ (O, 1) and |||| = || a 03|1/2// 1 follows that if o > O,
Exest (w) Pz, (yx à - 0) P(a| (/ol= |r| 0)
P(al|E
Wo /oolE G1 + ||r| < 0)
P([G2//1| 11 G2 0) with 17 == a=]l wo |||l
P(G2 < (] 2 n) with T == G2/G1 Cauchy(0,11
1
2P 2 1) = (T 2 n)
2
1
1
(
+ arctan 11) π
(m/2- /2 arctan 77)
2
π
1
a: a||EE//2 wol
arccos(
arccos(
|
Similarly, if < O, we get the same expression with replaced by -a. Therefore, irrespective of a, have
Etest (1) = arccos( |a |a|||E//3
(50)
|
It remains estimate the random quantities |a| and || in the asymptotic limit (14).
B.2. Leave-One-Our Arguments
We start with the Woodbury identity tells that
Rr; = (x DX/N Xaa) -x; = Σ
yjoz /n Ala)
j=1
R_ i
-RR] + qııaa T /n) -x;; 1 qı R_,,//1
where R_; = En j (f + Ala) is a version of the resolvent matrix constructed without the ith data point. This "Icave-one-out" trick is well-known in random matrix theory calculations.
On the other hand q, R_xi//n concentrates around its mean which is
E [q:0 R_jx, [n] tr (E(g,z,0) 1aliua,,RR__00 RRii/n) tr ER
n
with 8 = P Lr ER, p== Elgg).
n
16

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
Therefore, have the following identities holding for every i,J [n] with + j:
R_
RI;
(51)
1+6"
R 21 R-is
R- = R
(52)
1+0
Now, let x: be random test point from class y, independent of training data, For later use, note that
yz
qyyyz
Ro,
qoy.z Rx
71
galay'' R.
(53)
(1 d)m
B.3. Asymptotics of //
Note that
// 10
Ex, [yyx u) E((x"
[..... [[))).
Squaring (53) gives
(x x)² (1 + 8)
Σ qi (a R=xx))² (1 8)2
Σ Aoguay ( R= (R,)()))RR R-zzz)
(t)
For the expectation first some, note that
E [ (x R_ [o Elggx R.o R_ tr (E [Ier |R Ra R-l)) P. E>Y,
with E' = RE We deduce that
1
E
Σ 4s (x R
EE [RER]
(1 (1+
2)
(1 8)2
P (1 8)2
nn-1 lr E [RAEE hard life!
if isotropic, otherwise.
Now, let i.j E [n] with + j. One computes
1
E [aggyvr (x (x Rmzwz))
E (... ET )ior)
1 +
+ 5 (Ai A2 - Aa + A4),
where Tij
R-ij Sij/n,
R R. [
Si
1 1 8
A = E 91
R ER_
1
A2 (1 + 8)n
Ay:=
[a. R DS 5
(1 + 8)n
A4
[ S E5
(1 + 8)2n2
17

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
We now compute the terms A1, A2, A3. A1.
A1 R. (1gg,iyfa R- ( RE)s,
tr (E (1 [KER]] = c ''c,
where E' = E|
Similarly, A3 = A2 with
A2 B (a:g Siz (g
E Blagwayarr R-jjayz)) R-i ER-
(1 + 5)m
Ir (E (ag R Regoiga |B RR- ER-i)))
(1 5)m
Now, computes
E 400g;000 R ( R
( c Rc.
E| B| =;
from which it follows that
c Rc 1
A3 A
8 tr ΣΣ'
(54)
Finally, it is easy to show that A4 = O(1/)) = o(1)
Putting things together, we deduce that
E|||>//3 ~ル:= (1 + 6)22
(55)
where vi 2 0 is as Proposition 2.
B.4. Asymptotics of a
Proceeding as in the computation of the asymptotics of ||| above, can show that
(cT RRwo))
// w EEa2
W6w T2w))² = 1E T Wwy Ew
REwgw ERC
(1+ 8)2
(1 + 6)2
On the other hand,
wo0l2 = Evd Ewo ~
E- Σ
1 5
Bla.y:o R_
1 5
Blq.yand RE
1 5
c REwn
1 8
Thus, the variance of is vanishing, and deduce that
T REw
mo
a Ea
(56)
(1 /200
(1 6) || /
where mp is as given Proposition 2.
18

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
B.5. Final Step (Proof of Proposition
Combining (50), (55), and (56) completes the prove of Proposition 2.
B.6. An Important Lemma
The following result computes the mean vectors μ and c.
Lemma 4. Let E [-1,1] be the cosine of the angle between w; := / and Wg == / Let u be the unit-vector the direction of w, and let be its completion to an orthonormal basis for the span of ws and wo (if wy and wy are parallel, ie if +1, we simply set 0).
= ((..))>-pp [yz],
c: (a. (oo0xxxwwwy))) w,yyzl
(57)
Then, p 22/T Ewa/llwalss and e = Bu + Bu, where
= 21 = :
[2(1a((1 (| (TG) G), B 21 [[((00]10)))((]]]
with G ~ NN 1).
(58)
In particular, when #1 (i.e pruning along the data generator).
31 (. GI], 82 = O.
(59)
Proof. Observe that by instead considering Σ-1/2 1-1/2c, and defining == /1/2w, and := ///2 when computing ,r, and then En/2wn when computing c, we reduce the problem to the isotropic case (0, Ta)-
So let wp and WLOG, assume is aligned with the first canonical axis in IR,, i.e ||u|ler. Write (x1, 20 and u (v1, v_), where _ 2j=2 Zjez gd-1, and t := Σj= Uyej E Rd-1 It is clear that x u ||u||=r and T v 01 + g. where = xI U1. Furthermore, 4i and g are independent with distributions (0, 1) and and.N(0 (0, ll²) respectively. follows that
1/2 1/2 4 = [sign(x' ")x]] = E [sign [sign(Iu[laa)aneee E [
2
2 π π |ul
2 /2
61
π Wolll
from which we deduce the prescribed formula for the vector μ. This proves the first part of the claim.
We now establish the formula c = f1u + 32v. The proof for the formula for μ follows a similar (but simpler) path.
Observe that by instead considering Σ 1/2,, we reduce the problem to the isotropic case ~O0 Ta). We can explicitly write
Ws
TH Wp |
W Wo |0, ip
V =
P ==
(60)
Tp
where IT = wu? and IT- = Ia II. One can decompose
Giu + G2 + G and we = +2v
G1 := r u, G=:= =T G1 := p+x, C1 wh u,
(61)
02 ;= = ITe c1:= ply1/2w0 wo.
(62)
where P is the projector onto the span of and v. Note that G1, G2, and G_ forms set of independent random variables. Moreover, G1 and G2 have distribution (00 1), while G1 has distribution N/00, I2-2). We obtain
Eig(x T w..) wi)ssgg(x wo)oa = [a(x w.)sign(x ( wo)a] = E [g(x "s)sgin(x" Dobign(TTwp)er wp)xr]
(63)
la E [o(loog|llG)ssggpc(cCcc++.o
(64)
(65)
EE [ellllelllG))ssmmaaaaaaaaa
(66)
19

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
Now, due independence, the third term decomposes as
E [ Gi)sign((Gi [G]] [ 0.
We deduce that
Blg(x' w2) ws)sign(x wo)a] 31u. 320,
where 31 and B2 are as specified in the lemma and we have used the fact that
a p:
/2/l| /1 0².
In particular, if =1 (meaning that wp and w, are parallel), then
=
if otherwise.
Bg = E3 [ll GG)G
(67)
0,
We now compute the coefficients 31 and 32. Observe that thanks to Lemma 5, one has
E(sign(Gs)) Gi] [psign(pGi v1- PG2) Gi]
24 (TG1) -1,
E G()
G1] = E[sign(pGi + (1-PGG)GG Gi] = 2p((r1).
Therefore, with |. we have
81 == E(g(rGi)ssin(GG))G = 2E ( (TG1) G.I - (g) = 2E (g(+))) Gi].
82 =
= 2E (d(rGi))(((G))
where we have used the oddness of the function + (g(rt) in the last equation on the first line.
Lemma 5. Let G ~ 1), and let b E IR with a + 0. Then,
Elsign(aG b)] (bbb//a - 1, E[sign(oG + (p(b//).
(68)
Furthermore, holds that
lim [[sign(aG + b]] sign(b), e
lim []C 0. e+0
(69)
Proof. Indeed, one computes
E Elsign(aG b)] = P(aG + +(aG+b>00)
P(aG +bl -P
= 2P -b)
2P(G -b/a) - 20(b/a) - 1, if > 0
22P <b/a) 1 20(-b/a) 1, if
We deduce that E[sign(aG + b)] (bblaa))) as claimed.
C. Proof of Lemma 1 and Lemma 2
"Keep Hard" Examples (Lemma 1). Let b - r, t= - = - formulae for the anti-derivative of the function + zz((bz)((z)
p². Using Lemma 4 and standard
2E 2E((()G)(((G)) 2 1" p((z))p())dz = 2 (00)000((2)
a
= (2/1-p2p(0)) (20(a/ /1 p?) -1) -2p(0) /1 p26z.
'For example, funstions.
Wikipedia
20

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
On the other hand, we have B 81 [B = (f)= with
(TZ2)(((2)2 = =0 - [
(b/0))((0(20(0)) (p()))2(((()))) - = 1) (( - 1)
-P0(0)e1-p(p))
which proves Lemma
"Keep Easy" Examples (Lemma 2). Indeed, since 9K 9KH, we know from the previous lemma (KH strategy) that
((хKε) 2T ((kE((G))(FGGG 2 [+(rG)G] 2 (oxx(1C)0((T)G)
- = 2(pp(0) - v((a) - (((k)) -pp((0) - (p0(0))-2((()(((((x))) + 20(())2((KH)
=((pp(0)cr + =((xp(0)a+++((0))) (())
The computation of (2(1K)) uses completely analogous idea:
B(αKe) = 2Klgge = 2E[p(TG)]
2.p(0) -222gk(k)())))
= 2 (v((0) /1 1- 11-0p--p(00) -0((0)) (
-2p(0) 0² (1=c((KH))
-2p(0)/1-
This proves Lemma 2.
C.1. Proof of Proposition 1
Using Theorem 4 of Liao and Mahoney's "Hessian Eigenspectra of More Realistic Nonlinear Models" https://arxe org/abs/22 21 03. 01519 9 and some basic manipulations, we can write
R~ R,
(70)
9 /11 E1/2 + eaT) =Id,
where R- Eg
(71)
+ qō(2)
Since q is Bernoulli with mean = P(q 1), it is clear that where q = q- w.) for x ~ NO E), a = I1/
(ws)
Eg
1+ 0(z) = =(()).
P
+ (0)))
This further gives
R- = =1(2)////2 1//2 =Ta /TKTTE
q(u Te) wu
(72)
with K E. 1 1 qqu Tv)((2)
where u := 1/2 (0, Ia) and o: 1l/2
Now, to determine the matrix K, we first rewrite u = (uz, u1) and v 01), where
Ws
U1 = v w R, R.
Uy [wa1
(73)
|w
1 11 11 1
U1 11 dd11
(74)
21

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
The advantage of this representation is that
T and tu are orthogonal Ws.
3 and tn are statistically independent.
" has distribution WO0,1 1).
W. has distribution (0, Ta-1).
By symmetry of the situation, know that
K = +1)
where s2)) = Elh(w G s1(2) : E'h(w' 9
w = (uz, [[) R² g (G1, GL) ~ ~(0,1
I())) = 1 + q0(2)
9
Combining with (72), we get
3-1=\//2(a(3)) ====\"(aaaaa+<<mmm=""" where a(z) = t(2) - 2, t(2) = 1 0(2)
(75)
p
b(2) = s(2) - ((z).
(76)
Now, using the Interii-nnvesiion Lemma, one can obtain R from R-1 as follows:
1 La b(z)/a(2 m1 a(z) II + b(2) a(=) II. a(z)
b(=)/a(z)
suffices to notice that 1/(b(=) + a(2)) = 1/(s(2) - 2) = =m(z)= and 1/a(2) = m=) by definition, and the result follows.
C.2. Proof of Theorem
Set z ーA Recall from Lemma 4 c = 3ru + S2c. In Theorem I, we have the identification 8 = .3 and 3 = 31. We know that R = m(=)II' + mIIIII, where II uu One computes
1
= (wo/ll Rc
wi (m(2))I* + m(=)II) (Biu B20),
|woll
1
wp (8rm(z)u + 32m(2)u).
woo
Wp u
w v
wp wo/ |woll - pl| wolllllu T wo woo/lwwl) wl
with
[wol
= P.
|
|| V p²
p p²
V1- =: w/B2.
Putting things together gives mo wm()) + (=) as claimed.
Likewise, one computes
Ir R2 (m'
ml (=) (m)) pmm
n
n
c Rc = c (m(2))l M1) = (B Bzv) (m{2) [B1u u + 820)
{m((2)
- ()) =: (+)(()=)(()) r:)(;),
- Ty' = e E T (m)()) + (=)) c= 3 (2) + (z) = (2))
from which the claimed formula for follows.
22

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
C.3. Proof of Corollary 1
As usual, set == == < O.
(A) For & < p, it is easy to see from formula (16) and Lemma 6 that in the limit -> 0, one has
m(z) + 1
p "²
m(z) + 0,
mmz)) + p/7
0¹
m'(z) - (p- 6)3
p
m'(x) + 1
p- p-6²
m2)x) + (p- 6)3
p/7²
(p(p - o) +
p
((p - 0)p//> 0/77.
=
(p - +)3
m2(2) 1 1+ 6m((2)
6)2
Furthermore, with my and up as defined in Theorem 1, one computes
r(z) = Pm(=)) + +m)=)) + 13 + 32 p/7
ro
P
Φ p 0
(=) = (m)(s)++++++)))))>)))))))))))3 + ( + 8 (p 1p- 0)3
p
+ 32
p/72 2
ró (plp - x) + 04) = (p Φ)
(p 3)
where ro and ró are as defined in the claim. We deduce that mo/ / vo- my = a/vb Theorem 1.
Vbb a² and the result follows from
(B) Now consider the case & > p. Observe that mo //0 my -m v²² 2 =mg- On the other hand, from (16)
we know that
=m(2)= v(p (p 0 - 2)2 40z (p- - x - 2)
(77)
2Φ
Combining with Lemma 6, we deduce the following limits
-zm(z), 23 =2 (z) - CD := - p/b
m ml(2) + p/d
p-p' p
-zm(2), 23 m' (x) - -2m (2)
CO
7/66 + co
-
+ mm())
0
Furthermore, one computes
-2r(z) = 52 (-2m (2)) + 87 (-2m(x)) = 53.0 + + 8D
= coro,
7/0 + co
*+(())= = 33=²2m
(33=))=3=='m))) (=) + 33 (z) = 2200 + 8
80
CoTO,
7/0 + 60
-2mp 22 (- (-2m(2)w-2m(3
T/O00 (w + 1//////7/////// 10 + co)) =a a,
2
2210 = p622mm (2) + 27((2) 26
(-2r(z))
1 + dm (z)
+ +pcc + Toco 2 = co (pp ro) =; b.
23

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
We deduce that
ma/ V Mo m = - - V /2v00 =2m - a/ sa/Vb-/ a²,
and the result follows from Theorem 1.
Lemma 6. We have the following identities:
m(2)²
m'(x) = 1- (1 m(2))
m (z) (= (b)m())) /m(=//m(22 pp
p
(0 1/m(2))2 pd
7am (=)
(z) = m(=)² ( +1+ (m(=))2 + +1)
r'(z) = mm (z) + (z).
D. Additional Experimental
D.1. Choice of Generative Model
We evaluate the capabilities of four open-source large-scale pre-trained text-to-image models (Rombach al., 2022) in a controlled setup to determine which one performs best for the maagecclassificttinn task. Each synthetic image is generated with a simple prompt (class name). We create dataset of size 130,000 examples and train a ViT-B model on the synthetic data. Our results (Table 2) show that LDM-1.5 outperforms its more recent counterparts, LDM-XL and LDM-2.1, despite being an older model. We hypothesize that this is due to the lower diversity of generations in more recent models. This finding is consistent with previous work (Astolfi et al., 2024), which observed lower diversity in more recent latent diffusion models. For all of our experiments, we use L.MM-1.5 as it is the best performing model.
Table 2: Study the choice of generative model for the task of ImageNet- 100 classification with synthetic data. All experiments are trained for 50k iterations and the dataset size is static size of 130k.
Syn. Data Source
Real Val. Acc.
LDM-1.4
59.06
LDM-1.5
59.24
LDM-2.1
55.92
LDM-XL
52.8
D.2. Ablations
== 0 w 0 To understand the effect of different components of our framework, we ablate the case where data generated through the DP framework, but with a coefficient of zero for the term w. We also report results using different values of w. See results in Table comparing row 1 with rows 2, 3 and 4.
Incremental patience In experiments, setting the maximum patience value (Tmax) to fixed number resulted in the model requesting too much data when the size of the dataset was grown too big. For example, with fixed patience of Timx 7, for an experiment with initial dataset of size 130k samples, monitoring the validation accuracy every 130k iterations, meant that in the beginning every example was seen on average times before the patience reached Tmax- But as we generate more examples throughout the training, with a fixed patience value, each example would not be able to be seen even at least once. When the dataset grows to be 1.3 million, each example is seen on average of 0.7 times. This resulted in the model hitting the maximum patience very often. As result, we incrementally increase the maximum patience value as the dataset increases in size. See Table 3 for the result that compares the two scenarios (comparing rows I and row 5). Note we found using incremental patience to be significantly easier to tune. We often start with patience of and continue training. However, fixed patience requires more tuning depending on the size of the dataset and the number of training iterations.
Dataset sampling probabilities One can assume that newly generated examples could be more valuable then previously generated examples. As a result, we experiment the case where every newly generated example has twice as much probability
24

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
Table 3: Ablation study on ImageNet-100. Given a baseline method with DP, we modify each component of the framework one-by-one and study the effect of each change. All the experiments are trained for 50k iterations and have the same final size.
T Tma
Sampling
Real Val. Acc.
Real tr. Acc.
0.05
inc.
uniform
68.04
69.25
2
0
inc.
uniform
61.58
63.09
3
0.03
inc.
uniform
66.70
68.00
4
0.07
inc.
uniform
66.88
68.66
0.05
fixed
uniform
67.22
70.38
6
0.05
inc.
non-uni.
68.01
69.11
Figure 8: Intermediate stages of reverse sampling, Samples of the xy approximation using the DDIM sampler. While blurry, these intermediate samples provide sufficient gradients for entropy guidance, with key features like color and shape discernible even in early stages.
be selected when sampling the data batch for a given iteration. We observe that having higher probability does not lead to statistically significant improvements. See results in Table 3 comparing rows 1 and 7.
D.3. Intermediate stages of reverse sampling
In section 2 mentioned that DDIM's zy approximation is good approximation to guide the sampling process. In this section we plot these intermediate examples which are fed the classifier to compute the entropy of the sample and use it for guidance in the sampling process. Figure 8 shows that although intermediate samples are noisy, they contain the key features.
D.4. Studying the effect of w
In this section study the effect of dynamically generating the data without entropy guidance versus generating it uniformly from the beginning. In the first scenario, we use the DP framework, with monitoring the patience variable but using an w = 0 which effectively generates with naive sampling. In the second case all the data is generated in advance and no data is added during training. As it can be see in Figure 9, there is close to no difference between generating all the data in advance or generating dynamically if we allow for enough iterations for training.
In this experiment, we evaluate on ImageNet-100 validation set and train all the models for 50,000 iterations.
D.5. Experimental Details
1. SCALING PLOTS
We have used the Warmup-Stable-Deeay (WSD) learning rate scheduler (Hu et al., 2024), which stabilizes the learning rate throughout most of the training, ensuring effective adaptation to newly generated data. For ImageNet-1000 we train on nodes, each with 8 GPUs with batchsize of 64. For ImageNet-1k, of 128. For all the experiments, initial 10% of the iterations is done with linear-warmup and the last 20% of the iterations is for cool-down with Cosine Annealing. The intermediate steps are constant learning rate. For all these experiments we use A = 3 and = 0.05.
train 4 nodes, each with 8 GPUs with batchsize
For ImageNet 100, the learning rate is 0.003 with an EMA momentum of 0.001. For ImageNet-Ie Ik, the learning rate is set to 0.0016 with an EMA momentum of 0.001. We also use label smoothing with a value of 0.11. We use Mixup with an alpha
25

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
DP, oreea=0
60
Accuracy
56
54
10a
10c
Total Data Size
Figure 9: there is close to difference between generating all the data in advance generating it dynamically if we allow for enough iterations for training. Both cases have the same scaling behavior.
of 0.5 and CutMix with an alpha of 1.0. Furthermore, we use the AdamW optimizer.
Furthermore, for each setup in our experiments, we apply branch-outs. branch-out is the same experiment as an initial setup except that it does not allow additional data starting from specific epoch. The epoch selected based on the times that the T max was hit. Meaning branch out is just before additional data is added the training
N
N + kP
Init. Tmaz
Branch out Epoch
IN Val.
IN-Sk
IN-R*
32000
16000
80000
0.05
662
59.48
31.49
58.92
32000
16000
4
96000
0.05
6
701
60.54
33.69
59.97
32000
16000
112000
0.05
767
61.80
35.03
61.24
32000
16000
128000
0.05
859
62.68
35.95
62.55
32000
16000
160000
0.05
951
64.40
38.08
63.87
64000
32000
256000
0.05
469
65.52
43.42
67.32
64000
32000
320000
0.05
606
66.28
44.33
67.94
64000
32000
11
416000
0.05
782
66.92
44.99
68.81
64000
32000
18
640000
0.05
1001
67.80
45.25
68.46
130000
130000
6
910000
0.05
14
68.28
45.06
70.87
I 130000
64000
27
1794000
0.05
494
68.46
46.33
71.04
I 130000
64000
47
3138000
0.05
618
68.88
45.76
71.26
64000
64000
0
inf
56.56
27.86
52.97
130000
130000
0
inf
59.44
33.32
55.95
260000
0
260000
0
inf
60.02
33.79
56.74
400000
0
400000
0
inf
61.92
36.03
59.75
2000000
0
2000000
0
inf
62.16
34.97
60.15
4000000
0
4000000
0
inf
62.32
36.43
60.89
Table 4: Details of the results reported in Figure 4 for the ImageNet- 100 dataset. All the experiments are trained for 50k iterations. The variables are based on the notations defined in Algorithm Note that Tmaz incremental.
D.6. Visual examples
Below provide additional examples of generations throughout time with different coefficients (x-axis) of [0.0001, 0.1, 0.3, 0.5, 0.7]. All samples are generated with the same seed. As from top to bottom the epoch number increases.
26

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
P
k
kP
3
Init. T Tmax
Branch out Epoch
IN Val.
IN-Sk
IN-R
160000
160000
320000
0.05
134
42.572
39.363
20,987
320000
160000
480000
0.05
191
44.880
41.987
23.095
320000
320000
640000
0.05
71
47.910
46.887
27.568
654000
654000
1308000
0.05
124
50.226
49.867
29.843
654000
654000
2
1962000
0.05
156
50.670
51.027
29.944
1300000
650000
10
7800000
0.05
246
50.908
49.820
31.217
654000
654000
19
13080000
0.05
51.198
16.776
320000
0
320000
0.0
inf
39.334
32.653
18.495
654000
0
654000
0.0
inf
42.514
33.883
21.303
1300000
1300000
0.0
inf
44.116
37.337
23.653
2600000
0
2600000
0.0
inf
45.006
38.667
24.298
10000000
0
10000000
0.0
inf
45.614
40.050
24.762
13000000
13000000
0.0
inf
45.628
40.357
Table 5: Details of the results reported in Figure 4 for the ImageNet-Ik, dataset. All the experiments trained for 100k iterations. The variables are based on the notations defined in Algorithm 1. Note that T is incremental.
27

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
(a) Class snail.
(b) Class daisy.
(c) Class seat belt.
(d) Class volcano.
Figure 10: Examples of generated samples for different class prompts across training epochs, with varying entropy guidance coefficient (w) (left to right) the training progresses (top to bottom).
28

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
Pool of generations:
Uniform Selection
High-entropy Selection
DP: Direct
Generator
High-entropy Generation
Figure 11: Efficient and Diverse Sampling with DP: Instead of inefficiently over-sampling and selecting high-entropy examples, DP directly generates high-entropy samples. This not only improves computational efficiency but also results in greater visual diversity.
29

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
Initial samples with no intropy-guidange
DP samples with entropy-guidance Earlier during training
DP samples with entropy-guidance Later during training
Figure 12: Evolution High-Entropy Samples During Training: Early-stage generations show mainly color diversity, while later stages exhibit richer transformations, aligning with the classifier's evolving uncertainties.
30

---

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
Initial data at the beginning of training:
Final accumulated data the end of training:
Figure 13: Comparison of Initial and Final Training Data: The initial training data lacks entropy guidance, as the classifier is untrained. By the end of training, the accumulated dataset contains progressively harder/diverse examples.
31


Reviews:
[{"review_id": "WrrHZ59Cl8", "reviewer": "Reviewer_rVX6", "summary": "This work addresses the challenge of improving data size scaling laws for models trained on synthetic data. In particular, it uses the intuitive idea of generating more synthetic data where the model being trained (referred to as the learner) has high entropy. To do so, the paper relies on a setting where the synthetic data is generated using a 'Denoising Diffusion Implicit Models' & modifying the generation to prefer samples where the learner has high entropy. This approach is illustrated to work in a toy theoretical setting as well as through real-world experiments on IN-100 & IN-1K.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful and encouraging review, we are very pleased that you found the paper intuitive, practical, and effective. \n\n\n### **Related Work and Data Pruning:**\nThanks for pointing out these relevant papers. While our method focuses on improving training via generation of informative synthetic data rather than selection or pruning of real data, both lines of work share the goal of increasing data efficiency by concentrating training on examples that contribute most to learning. In particular, works like Coleman et al. (2019) and Mindermann et al. (2022) develop data selection strategies based on proxies for example utility, such as gradient norms or learnability. Similarly, Pooladzandi et al. (2022) and Yang et al. (2023) propose coreset-based methods that aim to retain the most informative or representative points. Joshi et al. (2024) further emphasize the importance of data quality over quantity, which aligns closely with our motivation, though we operationalize this through generative modeling and entropy feedback rather than dataset filtering as discussed in Sec 5.3. We view our approach as complementary: instead of selecting from a fixed pool of data, we dynamically generate examples in regions where the model exhibits high uncertainty, effectively synthesizing the types of data these methods might prioritize. We will include this discussion in the revised related work section.\n\n\n### **DDIM Background:**\nWe agree that the paper would benefit from a more complete and self-contained explanation of DDIM. Due to space constraints, we had to keep the background section concise, but we will expand it in the final version. We also want to clarify that while our experiments use DDIM, the proposed method is not limited to this specific sampler. Our approach relies on accessing an approximation of the denoised sample x_0 at each step, which is also available in other diffusion samplers such as DDPM and its variants. As long as the sampler provides a usable x_0 ​ estimate, the entropy-guided feedback mechanism remains cheap and can be applied efficiently. We will emphasize this more clearly in the revised version.\n\n\n### **Language Model Extension:**\nThis is a very relevant direction which we have been pondering about. While our current work focuses on image generative models, the principle of DP can be extended to language. One key reason this approach works efficiently with diffusion models is that we have access to an approximation of the clean sample x_0  at each step, which enables us **to steer the sampling process before the generation ends**. This intermediate visibility during generation allows us to influence the generation process. Extending this idea to autoregressive language models is less straightforward since generation typically proceeds token-by-token with limited ability to revise earlier outputs. However, the emergence of language diffusion models opens up a promising path [1]. These models offer a latent trajectory similar to image diffusion models and may allow for entropy-guided sampling. We see this as a compelling direction for future work and are actively thinking about how DP could be incorporated into language diffusion models for synthetic data training.\n\n\n[1] Nie, Shen, et al. \"Large Language Diffusion Models.\" arXiv preprint arXiv:2502.09992 (2025).\n\nWe are very open to further discussion and would be happy to address any other questions.", "claims_and_evidence": "Yes. \n\n1.  The claim of the proposed sampling scheme modifying the sampling distribution is supported by the steps illustrated in equations 3-6. \n\n2. The claim of this approach improving scaling laws is evidenced by the experiments on ImageNet. \n\n3. The toy theoretical example serves to further illustrate this point.", "methods_and_evaluation_criteria": "Yes, I believe the datasets & baselines used for this method are fair & comparing to SOTA.", "theoretical_claims": "Yes, I checked the theorem in the main paper and this looks correct.", "experimental_designs_or_analyses": "Yes, this is sound.", "relation_to_broader_scientific_literature": "I think this is a useful practical tool in the training of image models using synthetic data. The idea & implementation of modifying a diffusion model to generate examples in high-entropy regions for a classifier being trained is extremely intuitive, simple & effective.", "essential_references_not_discussed": "I believe there should be an inclusion of references to earlier works on data pruning including, but not limited to: \n\n- Coleman, Cody, et al. \"Selection via proxy: Efficient data selection for deep learning.\" arXiv preprint arXiv:1906.11829 (2019).\n- Mindermann, Sören, et al. \"Prioritized training on points that are learnable, worth learning, and not yet learnt.\" International Conference on Machine Learning. PMLR, 2022.\n- Pooladzandi, Omead, David Davini, and Baharan Mirzasoleiman. \"Adaptive second order coresets for data-efficient machine learning.\" International Conference on Machine Learning. PMLR, 2022.\n- Yang, Yu, Hao Kang, and Baharan Mirzasoleiman. \"Towards sustainable learning: Coresets for data-efficient deep learning.\" International Conference on Machine Learning. PMLR, 2023.\n- Joshi, Siddharth, et al. \"Data-efficient contrastive language-image pretraining: Prioritizing data quality over quantity.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2024.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "I think the paper could benefit from a more detailed background section on DDIM. Personally, since I was not familiar with this, I had to refer prior work to understand this, before I could fully understand the paper & it would be better if the paper could be read in a self-contained manner.", "questions_for_authors": "Have the authors considered a similar idea for training language models? Do they have any initial thoughts on how such an approach would be extended to that setting?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6dNIM76Z2G", "reviewer": "Reviewer_vJVZ", "summary": "The authors introduce a new methodology called \"Deliberate Practice for Synthetic Data Generation\" (abbreviated DP) to train a machine learning model for classification using entirely synthetic samples generated from a pre-trained diffusion model. Instead of generating many samples and pruning, the method uses an additive correction to the score to prioritize samples which are most challenging for the model; this correction is updated dynamically during training and is based on the entropy of the model on the sample at the time where the data is generated. The authors justify their approach of selecting difficult examples via theory for high-dimensional linear regression. Then, they perform some experiments: (1) comparing DP to another (\"static\") approach of pre-generating all data and training on a fixed synthetic dataset; (2) comparing DP to previous methods [1, 2]; (3) comparing DP to another (\"pruning\") approach of pre-generating data and then pruning via model entropy; (4) answering the scientific question of whether models trained with DP change their (entropy-based) difficulty estimates over the training trajectory. For experiments (1)-(3) the DP method generally performs uniformly better than the chosen alternatives; for experiment (4) it confirms that models trained with DP indeed change the difficulty estimates.\n\n[1] Sarıyıldız, M. B., Alahari, K., Larlus, D., and Kalantidis, Y. Fake it till you make it: Learning transferable representations from synthetic imagenet clones. In CVPR, 2023.\n\n[2] Fan, L., Chen, K., Krishnan, D., Katabi, D., Isola, P., and Tian, Y. Scaling laws of synthetic images for model\ntraining... for now. In CVPR, 2024.\n\n## Update After Rebuttal\n\nThe rebuttal clarified the theoretical/conceptual points I had raised. Because of that, I think the work is a good contribution and should be accepted. The other point I raised (which seems to be echoed by the other reviewers) is that the experimental section's scope is a little lacking; I had expected more datasets/diffusion models in order to verify the scientific claims, while other reviewers comment on specific evaluations and comparisons to other work. Still, the experimental results seem overall reasonable to me. For that reason I recommend acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed feedback. We're glad that you found the method conceptually clear. Also, thank you for the helpful proofreading suggestions. We will incorporate these corrections in the final version.\n\n---\n\n**Connection Between Theory and Practice:** We agree that the linear regression setup in Section 4 is a simplified version of the full setup used in practice. The goal of this section is not to precisely model the real-world setting, but to isolate and analyze the principle of prioritizing difficult examples based on entropy or uncertainty. We will include a short discussion highlighting the limitations and interpretability benefits of the theoretical analysis in the simple setup. Also, note that even though loss used in training the linear model is regression loss (squared L2), the learned model is used as a classifier and we analyze classification accuracy (see Section 4). The use of linear classifiers fitted with squared loss is common in ML theory (Couillet & Liao, 2022; Liao & Mahoney, 2021; Firdoussi et al., 2024).\n\n**Unpacking Theorem 1.** The point of the theorem is to showcase that the test error can be written analytically as a function all problem parameters: parametrization rate $\\phi$, regularization parameter $\\lambda$, pruning ratio $p \\in (0,1)$, cosine of angle between the pruning direction and the ground-truth labelling function $\\rho$, etc. Analytic expression provided is defined via concrete functions like $\\Phi$, and the spectral functions (linked to Marchenko-Pastur Law) introduced earlier and can be used to numerically compute / simulate the shape of the theoretical error curve in different regimes corresponding to different settings of the problem parameters. We agree with the reviewer that expanding the Phi function can provide some quantitative insights, at least in setting regimes of its argument, especially: large +ve, large -ve, and small values.\n\nFinally, note that in the unregularized (ridgeless) case $\\lambda \\to 0^+$, the aforementioned analytic expression simplifies drastically, as shown in Corollary 1 (Appendix).\n\nIn Appendix A.1, the order of the limits is: first let $d,n \\to \\infty$ such that $d/n \\to \\phi$. Then, let $\\lambda \\to 0^+$. \n\n**Proportional scaling in linear regression.** This is actually standard in high-dimensional statistics and learning theory. Here, such a scaling allows us to capture important regimes: interpolation threshold (corresponding to $\\phi=p$), under-parametrized (phi < p), over-parametrized ($\\phi > p$), extreme under-parametrized ($phi \\to 0^+$), extreme over-parametrized ($\\phi \\to \\infty$) regimes while keeping the analysis tractable (via the tools of random matrix theory).\n\nThe reviewer’s statement that we are “only scaling data” is inaccurate. Observe that $d,n \\to \\infty$ such that $d/n\\to \\phi$ can be equivalently written as $n = \\phi d$ (rounded to an integer), $d \\to \\infty$. Therefore, we can fix $d$ to a large value and study the effect of $\\phi$, as done in the experiments. Other large values of $d$ would yield a similar experimental picture. Everything about scale is completely captured in the $\\phi$ parameter. We shall make this clearer in the manuscript.\n\n\n---\n\n### **Experimental Diversity:**\nWe appreciate your suggestion to evaluate DP across a broader range of generative models and datasets. While we included comparisons in Appendix B.1 using other diffusion models, and have already reported results on two datasets (ImageNet-100 and ImageNet-1K) we acknowledge that this could have been more clearly emphasized in the main paper. We will revise the manuscript to better highlight the diversity already present in our evaluation.\n\n\nWe also want to clarify that LDM-1.5 was **not chosen because DP performs best with it**. Rather, we selected it because it consistently provides the strongest baseline performance in static data generation setups, independent of DP (as shown in Appendix B.1). This makes it a compelling and fair testbed. This model choice is in-line with prior work (e.g., Astolfi et al., 2024 and  Fan et al., 2024), which uses LDM-1.5 as it produces more diverse samples than alternatives which is a desirable property when training classifiers on synthetic data.\n\n\nMore broadly, we do not expect DP to be limited to a particular diffusion model or sampling strategy. Our approach is a lightweight modification of the sampling process, similar in spirit to classifier guidance, that leverages the model's entropy to influence which samples to be generated. As long as the sampler provides an approximation of the denoised sample x_0, DP can be applied by using the downstream classifier to adjust the sampling trajectory. We will clarify this generality in the revised version.\n\n---\n\nThank you again for your constructive comments. If there's anything in particular that’s holding you back from potentially increasing your score, let us know.", "claims_and_evidence": "There are several claims being made in the paper:\n\n(1) the asymptotics of the test error of the toy linear regression setup in Section 4;\n\n(2) the advantage of DP (in the classification setting) over more trivial methods for synthetic data generation like \"pruning\" and \"static\" approach;\n\n(3) the advantage of DP (in the classification setting) over previously proposed methods for synthetic data generation;\n\n(4) the claim that the same sample may be considered more or less difficult by a model over the DP training.\n\nOf these, (1) is a mathematical claim and it seems adequately backed up via proofs in the Appendix A. (2) and (3) are empirical claims about method performance, and hold on the provided testbed (trained on varying number of samples generated from the same diffusion model LDM-1.5, and evaluated on subsets of ImageNet 1K). Still, in order to back up the more general claims it may help to generate from different diffusion models and evaluate on different datasets. (4) is a scientific claim and straightforwardly backed up.", "methods_and_evaluation_criteria": "The methods are: training using DP (or other methods) on samples generated from a single diffusion model LDM-1.5 (except for a small ablation in the appendix) and evaluated on different subsets of ImageNet-1K. These methods and evaluations make sense and are standard, though (as remarked above) to truly back up the claims, it might help to use more evidence using different generative models (especially since LDM-1.5 was picked specifically because DP performs best with it relative to the other tested generative models) and/or different evaluation testbeds.", "theoretical_claims": "While I did not carefully check the proofs extremely carefully, they seem to be valid (and there is experimental evidence on toy datasets to confirm their correctness). A minor issue is that in Theorem A.1 it's not specified in which order the limits $\\lambda \\to 0$ and $n, d \\to \\infty$ with $d/n \\to \\phi$ are taken.\n\nSome slightly larger problems with the theoretical component of this work are:\n1) The rate in Theorem 1 the main body is not very \"clean\"; when I look at the rate I have no idea what order of magnitude I expect for the test error. It could be possible to work it out using asymptotic expansions of $\\Phi$ and the other terms, but it would be very helpful to understand the rate approximately in the main body (while keeping the more complicated part for the appendix, of course), and it could help make some qualitative insights very straightforward.\n2) The use of the proportional scaling regime for linear regression is slightly un-natural (to me); in the experiments you're only scaling data, not dimension, and so it might make sense to consider fixed $d$ and large $n$. What kinds of asymptotics are achieved there? Are they interesting? It would be useful to comment.\n3) More generally, the linear regression problem in Section 4 is a toy problem, and the practice uses cross-entropy losses and measures the difficulty totally differently from the sign-of-inner-product way that's given in Section 4. (Of course, the real data isn't generated by a Gaussian and linear classifier either.) While I appreciate the considerable difficulty of expanding the analysis to this more practical setting, these differences should be remarked. \n\nThese may cause the (considerable) difference between real and predicted accuracy in Figure 4 (c) --- it would be helpful to comment on that and bring the theory a little closer to practice.", "experimental_designs_or_analyses": "The experimental designs and mechanisms (including things like hyperparameters) seem well-described to me; while many details (and ablations, etc.), are not in the main body, they are in Appendix B. The analysis of the above experiments in Section 5 and Appendix B also seems sound.", "relation_to_broader_scientific_literature": "This paper broadly intersects with two different areas:\n- Synthetic data and its use to train neural networks (and associated with challenges such as model collapse, bias of the generator, etc).\n- Conditional diffusion models (for generating the synthetic data in this work).\n- Active learning and continual learning (which also involve tailoring data to improve learning trajectory).\n- High-dimensional statistics (corresponding to the theory component.)", "essential_references_not_discussed": "I don't know any significant omitted references.", "other_strengths_and_weaknesses": "- The main strengths are the improvement of DP over previous methods, as well as its simplicity. \n- The main weaknesses are the lack of diversity in experimental setups (discussed previously), and the lack of connection between the theory and practice, which operate in two different regimes (classification vs regression, proportional scaling vs not, etc) and so it is unclear what messages or guidance to take from the theoretical part.", "other_comments_or_suggestions": "Some nitpicks:\n- In equation (5) I think there should be a $(t)$ superscript for all model uses.\n- Section 3 title: \"Deliberate\" should be capitalized.\n- The paragraph titles have inconsistent formatting; some are capitalized (\"Asymptotic Behavior of the Test Error\") while most aren't.\n- Some Typos in Appendix B, such as the section title \"Additional Experimental\".", "questions_for_authors": "No questions at this time.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3u6gV3mR99", "reviewer": "Reviewer_gZy1", "summary": "This paper empirically demonstrates that \"delibrate practice\" is meaningful in improving the scaling law of synthetic data generation. Broadly speaking, this falls under the general umbrella of efficiently collecting as few data as possible, with the additional problem context being, a generative model is used in generating the data to be collected. Though delibrate practice was a notion related to \"curiosity\" and has been explored in the past, this work empirically re-examined such use in the above said context. What's encouraging is, this work presents a scaling law, not just a few case studies.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n\nThe concept of \"deliberate practice\" has indeed been borrowed from psychology and has thematic connections to \"curiosity\". However, to the best of our knowledge, we are the first to adapt and implement this principle in the setting of training classifiers entirely on synthetic data generated from diffusion models. Rather than generating a large static dataset or pruning post hoc, we propose a dynamic approach that modifies the generative process itself to prioritize informative, high-entropy examples.\n\n\nOur work goes beyond showing isolated gains, it demonstrates that deliberate practice shifts the scaling laws of training with synthetic data, enabling significantly better performance with fewer generated examples.\n\n---\n\nWe were puzzled by the low overall rating, especially given that the review doesn’t raise specific methodological, theoretical, or experimental concerns. If there are particular issues we may have missed, we’d be happy to address them. Otherwise, if the current score was entered in error, we would be grateful if you would consider updating it.", "claims_and_evidence": "They are fine.", "methods_and_evaluation_criteria": "They are fine.", "theoretical_claims": "They are fine.", "experimental_designs_or_analyses": "They are fine.", "relation_to_broader_scientific_literature": "This paper empirically demonstrates that \"delibrate practice\" is meaningful in improving the scaling law of synthetic data generation. Broadly speaking, this falls under the general umbrella of efficiently collecting as few data as possible, with the additional problem context being, a generative model is used in generating the data to be collected. Though delibrate practice was a notion related to \"curiosity\" and has been explored in the past, this work empirically re-examined such use in the above said context.", "essential_references_not_discussed": "I didn't check", "other_strengths_and_weaknesses": "I am not sure that novelty is high enough.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q0twyHIvts", "reviewer": "Reviewer_ovd8", "summary": "This paper focuses on the task of synthetic data generation for image classification. Specifically, traditional methods of synthetic data generation in classification tasks suffer from diminishing returns as the dataset size increases, leading to inefficient use of generated data. Inspired by the concept of deliberate practice in human learning, the authors propose a novel framework, Deliberate Practice for Synthetic Data Generation (DP). Instead of generating a static dataset upfront, DP iteratively trains a model, generates new data, and selects challenging examples to incrementally expand the training set. The authors provide theoretical justifications for their approach and conduct empirical experiments on two image classification datasets. The results demonstrate that their framework significantly improves data efficiency while reducing computational costs.\n\n### update after rebuttal:\nI thank the authors for their response and have taken into account the perspectives of the other reviewers. I will maintain my score, as it is already a positive one.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and encouraging review. We’re glad to hear that you found the method to be conceptually intuitive, the theoretical analysis to be sound, and the experimental setup and evaluation thorough.\n\n### **Qualitative Analysis of Hard Samples:**\nThank you for raising the point about qualitatively analyzing the generated hard samples. While we do include some visual examples in Figures 6 and 10, in response to your suggestion, we have conducted additional visual analysis available at this anonymous link (https://drive.google.com/file/d/1XbkVVsHDQyhfSJfqGkaFLC7Polb0dHZR/view) (can be access in incognito mode and requires no logging in):\n\nIn the newly added Figures 1 and 2, we start with a pool of examples and compare the following sampling setups visually:\n\n1. Uniform random selection from the pool\n2. High entropy selection from the pool\n3. Directly generating high entropy samples with DP\n\nWe visually observe that:\n1. Uniform selection does not change the distribution of the examples.\n2. High-entropy selection results in some ambiguous or atypical examples, often featuring occlusions, rare viewpoints, complex backgrounds, or low-contrast textures; factors that increase classification uncertainty.\n3. DP’s direct generation produces the most visually diverse and semantically rich set of samples. These include unusual lighting, texture variation, and sometimes near-failure cases; all of which challenge the model and encourage more robust learning.\n\n\nFigure 3 compares the following setups visually:\n1. Initial samples with no entropy-guidance\n2. DP samples with entropy-guidance earlier during training\n3. DP samples with entropy-guidance later during training\n\nWe observe that early-stage generations show mainly color diversity, while later stages exhibit a richer set of transformations, aligning with the classifier’s evolving uncertainties.\n\nFigure 4 compares the following setups visually:\n1. Random samples from the initial data at the beginning of training for the class ‘fox’\n2. Random samples from the final accumulated data at the end of training.\n\nThe initial training data (visually) lacks diversity. By the end of training, the accumulated dataset contains progressively harder/diverse examples.\n\n---\n\nThank you again for your thoughtful comments and for recognizing the contributions of this work. We hope the addition of qualitative examples will further strengthen the final version. If there's anything in particular that’s holding you back from potentially increasing your score, we’d really appreciate hearing about it. We're happy to clarify or address any remaining concerns.", "claims_and_evidence": "The main claim of this paper is that the proposed synthetic data generation framework improves data efficiency and reduces computational costs. I believe this claim is well-supported by evidence presented in the paper. The authors conduct experiments on two widely used datasets and show that their dynamic, multi-round data generation approach successfully produces more informative training examples. The results demonstrate that their method can achieve comparable to static methods, but with significantly fewer training samples. Additionally, comparisons with other recent synthetic data generation techniques highlight the advantages of the proposed method over traditional generate-then-filter methods in terms of data efficiency and model performance.", "methods_and_evaluation_criteria": "I find the proposed method to be conceptually intuitive and practically applicable. The idea of iteratively refining the training data through entropy-guided sample selection is novel and well-motivated.\n\nThe evaluation criteria, including commonly used benchmarks and metrics, are reasonable for assessing the effectiveness of the proposed framework.", "theoretical_claims": "The authors theoretically analyze in Section 4 how selecting difficult examples can improve the scaling laws of synthetic data. I have reviewed the proof and found it to be correct.", "experimental_designs_or_analyses": "I find the experimental setup and analysis to be well-designed and thorough. In particular, I appreciate the authors' analysis in Section 5.4, where they examine the evolution of hard examples over time to highlight how difficult samples dynamically change during training.", "relation_to_broader_scientific_literature": "The key contribution of this paper—multi-round training with dynamically selected hard samples—fills an important gap in synthetic data research. Many existing works focus on generating large-scale datasets and designing filtering mechanisms to remove low-quality samples. In contrast, this paper suggests that instead of generating a massive dataset and filtering it later, one can directly generate selective samples by leveraging uncertainty in the generative model.", "essential_references_not_discussed": "The paper discusses enough related work.", "other_strengths_and_weaknesses": "Overall, I find the proposed method to be novel, and the experiments and analyses are well-conducted. One concern I have is that the authors do not provide any qualitative analysis of the generated hard samples. It would be beneficial to investigate whether the generated difficult samples are truly challenging, and whether they follow specific patterns that could offer more insights into the model’s learning process.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Have you conducted any manual analysis of the generated hard samples?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3VoVqNNMiZ", "reviewer": "Reviewer_K1M2", "summary": "The authors propose a method for synthetically generating data based on an entropy-guided sampling of diffusion models. Their method is dynamic in that they call their entropy-guided sampling every time their model's validation accuracy plateaus. They show that their synthetically generated data is better than prior work on ImageNet-100 and ImageNet-1K. They present some theoretical motivation for their approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and detailed feedback.\n\n---\n\n### **On theory**\n\n**Relation to Active Learning.** Though related at a high level, our work differs from standard active learning, which focuses on querying labels for unlabeled data. We instead focus on generating useful training examples. The papers mentioned by the reviewer establish bounds which indicate that aggressive pruning restricted to a well crafted region leads to improved sample complexity for active learning. Our contributions differ: (1) We show that DP is equivalent to pruning. (2) We analyze this pruning in a toy setting using random matrix theory, deriving **exact** error curves.  (3) This analysis helps explain why DP improves sample efficiency in practice.\n\n**Connection between theory & algorithm.** The theory offers a principled explanation for DP’s success: it effectively samples from a pruned data distribution, without over-generating and discarding samples. We focus on high-dim regression as it’s analytically tractable yet captures the key aspects of the problem. While simplified, the goal of the theory is to isolate and analyze core components of the algorithm, helping explain its empirical effectiveness. We will clarify this in the revision.\n\n**It just seems to be a linear classifier over gaussian input data?** No. The theory involves linear classification over **\"pruned\"** Gaussian data, which significantly alters the data distribution. For example, pruning distorts the standard Marchenko-Pastur law (Lemma 4), making the analysis technically involved. This required careful use of non-standard random matrix theory tools (see Appendix). We'll clarify this in the manuscript.\n\n\n\n### **Use of the validation set**\n\nUsing a validation set for early stopping and hyperparameter tuning is standard practice [1-4]. We follow the same principle: validation is used only for model selection, not training. Like early stopping, we detect when performance saturates, but instead of stopping, we generate new data and continue. This can be seen as repeated early stopping with dynamic dataset expansion.\n\n**Fair evaluation.**\nIn Table 1, we report on **both** the standard ImageNet val set (IN real Val.: commonly reported, e.g., [1, 2] since the actual test set is not public) and on the **held-out training data** (IN real tr.), which is untouched by the selection or training process in DP and serves as a true test set.\n\n**Validation use is helpful but not essential.**\nWhile we use the validation set to determine when to generate more data, DP does not depend on it fundamentally. Alternatives include:\n\na. Internal signals, like training loss flattening (see Figure 5 in this anonymous link https://drive.google.com/file/d/1XbkVVsHDQyhfSJfqGkaFLC7Polb0dHZR/view).\n\nb. Predefined schedules, e.g., adding data every X epochs.\n\nc. A synthetic validation set\n\nIn short, our use of validation aligns with common practice and is just one possible design.\n\n[1] Sarıyıldız et al. 2023.\n[2] Fan et al. 2024.\n[3] Dosovitskiy et al., 2021.\n[4] He et al. 2022.\n\n\n### **Comparison with Hemmat et al.**\n\nWhile we build on feedback-guided generation, the two works differ significantly in their motivation, setup, and methodology. Our goal is to improve scaling laws in a **zero-shot** setting with **dynamic data generation**. In contrast, Hemmat et al. focus on class imbalance, using a **static**, one-time rebalancing step with **image-conditioned** generation.\n\n| Aspect                    | This Work                            | Hemmat et al. (2023)                  |\n|---------------------------|---------------------------------------|----------------------------------------|\n| **Primary Goal**         | Improve scaling laws of synthetic data | Handle imbalanced classification       |\n| **Use of Real Training Data**     | None           | Real examples used for conditioning    |\n| **Generation Type**      | Dynamic, repeated                     | Static, one-time                       |\n| **Diffusion Conditioning** | Class labels only                   | Image and class labels                      |\n| **Scaling Analysis**     | Theoretical + empirical               | Not studied                            |\n\n\n\n### **Table 2 and Figure 2**\n\nIn Tab 1, columns 2, 3 report baselines with varying data sizes and iterations. We include these to ensure a fair comparison with our dynamic setup, which increases data over time.\n\nIn Fig 2, the x-axis shows the final data size. E.g., to train with 2^10 examples using the \"top 10%\" (red), we generate 2^11 and keep the hardest 10%. The \"select all\" (black) strategy generates and uses all 2^10.\n\n---\n### **DP as the acronym**\nThanks for the suggestion. We are happy to adopt the acronym to SDP (Synthetic data with Deliberate Practice). It has a nice ring to it!  :)\n\n---\n\nThank you again for your constructive comments. If there's anything in particular that’s holding you back from potentially increasing your score, let us know.", "claims_and_evidence": "Yes the empirical claims are supported by clear evidence, but I do not see how their theory is related to diffusion models and synthetic data generation which is the main topic of their paper.", "methods_and_evaluation_criteria": "I am confused as to why the authors use the validation set for their algorithm instead of a subset of the training set. It makes comparison to prior work more difficult as prior work reports the model performance on the test set and not the training set.", "theoretical_claims": "The proofs are in the appendix and I did not carefully check them, but I do except the claims to be correct.", "experimental_designs_or_analyses": "The experimental design is valid for the task they are trying to solve.", "relation_to_broader_scientific_literature": "Synthetic data is becoming an integral tool for the broader scientific community, offering solutions for settings that are data scarce, have privacy concerns, and more generally to improve model performance.", "essential_references_not_discussed": "The theory effectively claims that selecting examples along the decision boundary is beneficial and that one should continue to update the decision boundary.  These two facts have been shown theoretically and empirically in the active learning literature in much more general settings than the ones analyzed here. See for example \"Margin Based Active Learning\" by Balcan et al and \"Improved algorithms for efficient active learning halfspaces with massart and tsybakov noise\" by Zhang et al. (for theory) and Batch Active Learning at Scale by Citovsky et al. (for empirical evidence).", "other_strengths_and_weaknesses": "Strengths\n* Leveraging the diffusion mechanism by using entropy-guided sampling presents an interesting algorithmic approach. \n* Their method avoids the need to over-generate and then prune strategy that priori work has often used. \n* Empirical results show positive improvement using their proposed methods. \n\nWeaknesses\n* Theory seems to be disconnected from the algorithm they present. \n* Theory doesn't provide novel insights than prior theoretical work and seems to be in more restricted settings. \n* Main differences with Hemmat et al 2023 are not explained in detail. \n* Dynamic nature of the algorithm seems a bit ad-hoc and it would be much better if it didn't need to use an external set to decide whether to generate more synthetic data.", "other_comments_or_suggestions": "DP is not a great acronym as it is usually used for differential privacy.", "questions_for_authors": "1. What is the main difference between this work and Hemmat et al 2023? \n\n2. The theory section seems disconnected from the setup the authors are interested in. In particular, how is the theory related to diffusion models and synthetic data generation? If I am understanding correctly, it just seems to be a linear classifier over gaussian input data? \n\n3. I am confused by Figure 2. The black line is the strategy that selects all data, but then the x-axis is the size of the selected data? \n\n4. For Table 2, the \"IN real Val\" column is the model performance on the ImageNet validation set that was used in the algorithm? If it is used in the algorithm, then we should not be comparing against it. Why didn't the authors just use a subset of the training data as their validation set so that we could more easily compare to previous work? Also why does a static method from previous work appear multiple times? Is the difference only the number of iterations run? I don't see a discussion or explanation in the main text. \n\n5. Is there other ways to make the algorithm dynamic without having to use the train/validation set? Most dynamic selection algorithms don't make use of the validation set and are solely based on the learner so it begs the question whether this is actually needed here.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper applies the principle of Deliberate Practice to effectively generate synthetic data with Denoising Diffusion Implicit Models for image classification tasks. There are five reviews received for the paper with a distribution of (1, 3, 4, 4, 5). The authors have the opportunity to engage with reviewers during the rebuttal phase, and clarify on several fronts, especially the contributions and experimental settings. The only negative review was abstract at the very beginning, and even after one round of rebuttal, the review can still improve on clarity and be more constructive itself. While I appreciate reviewer's efforts in reading the paper and discussing with the authors, I would have put more weights on the negative review if a few clear references are given on the main argument of limited novelty. As ICML this year particularly emphasize correctness over novelty, this paper is a clear acceptance. 

During the discussion, while reviewers acknowledge the paper is generally well written, it appears the presentation can be improved for clarifying contributions and experimental settings. In fact, there are some remaining concerns on the fairness of experiments, which has not been clarified with the limited number of communications during rebuttal. The paper can be further strengthened by considering generative tasks beyond classification, language and multi-modal tasks, while most of the reviewers acknowledge the current contribution and happily accept the paper.
Decision: Accept (oral)