Generative Data Augmentation via Diffusion Distillation, Adversarial Alignment, and Importance Reweighting
Authors: Ruyi An
Haicheng Huang
Huangjie Zheng
Mingyuan Zhou
Venue: NeurIPS 2025 poster ()
Keywords: Generative Data Augmentation
Diffusion Model Distillation
Primary area: deep_learning

Abstract: Generative data augmentation (GDA) leverages generative models to enrich training sets with entirely new samples drawn from the modeled data distribution to achieve performance gains. However, the usage of the mighty contemporary diffusion models in GDA remains impractical: *i)* their thousand-step sampling loop inflates wall-time and energy cost per image augmentation; and *ii)* the divergence between synthetic and real distributions is unknown--classifiers trained on synthetic receive biased gradients.
We propose DAR-GDA, a three-stage augmentation pipeline that unites model **D**istillation, **A**dversarial alignment, and importance **R**eweighting that makes diffusion-quality augmentation both fast and optimized for improving downstream learning outcomes. 
In particular, a teacher diffusion model is compressed into a one-step student via score distillation, slashing the time per-image cost by $>100\times$ while preserving FID.
During this distillation (D), the student model additionally undergoes adversarial alignment (A) by receiving direct training signals against real images, supplementing the teacher's guidance to better match the true data distribution.
The discriminator from this adversarial process inherently learns to assess the synthetic-to-real data gap. Its calibrated probabilistic outputs are then employed in reweighting (R) by importance weights that quantify the distributional gap and adjust the classification loss when training downstream models; we show that reweighting yields an unbiased stochastic estimator of the real-data risk, fostering training dynamics akin to those of genuine samples.
Experiments validate DAR-GDA's synergistic design through progressive accuracy gains with each D-A-R stage. Our approach not only surpasses conventional non-foundation-model GDA baselines but also remarkably matches or exceeds the GDA performance of large, web-pretrained text-to-image models, despite using solely in-domain data.
DAR-GDA thus offers diffusion-fidelity GDA samples efficiently, while correcting synthetic-to-real bias to benefit downstream tasks.

Generative Data Augmentation via Diffusion Distillation, Adversarial Alignment, and Importance Reweighting
Ruyi An
Haicheng Huang"
Huangjie Zheng²
Mingyuan Zhou'
'The University of Texas Austin
Shanghai Jiao Tong University 3 Applee
ruyi auyyxannutexs.eed mi ngyuann shou@mccombs utexas. edu
Abstract
Generative data augmentation (GDA) leverages generative models to enrich train- ing sets with entirely new samples drawn from the modeled data distribution achieve performance gains. However, the usage the mighty contemporary dif- fusion models in GDA remains impractical: i) their thousand-step sampling loop inflates wall-time and energy cost per image augmentation; and ii) the divergence between synthetic and real distributions is maknown-classineg trained synthetic receive biased gradients. We propose DAR-GDA, three-stage augmentation pipeline that unites model Distillation, Adversarial alignment, and importance Reweighting that makes diffusion-quality augmentation both fast and optimized for improving downstream learning outcomes. In particular, a teacher diffusion model is compressed into one-step student via score distillation, slashing the time per-image cost by > 100x while preserving FID. During this distillation (D), the student model additionally undergoes adversarial aligmment (A) by receiving direct training signals against real images, supplementing match the true data distribution. The discriminator from this adversarial process inherently learns to assess the synthetic-to- -e-arl data gap. Its calibrated probabilistic outputs are then employed in reweighting (R) by importance weights that quantify the distributional gap and adjust the empirical loss when training downstream models; we show that reweighting yields an unbiased stochastic estimator of the real-data risk, fostering training dynamies akin to those of genuine samples. Ex- periments validate DAR-GDA's synergistic design through progressive accuracy gains with each D-A-R stage. Our approach not only surpasses conventional non- coindation-moder GDA basclines but also remarkably matches or exceeds the GDA performance of large, web-pretrained text-to- eett-to-image immge models, despite using solely in-domain data. DAR-GDA thus offers diffusion-fided fidelity GDA samples efficiently, while correcting synthetic-to-real bias to benefit downstream tasks.
teacher's guidance better
1
Introduction
From early datasets of a few hundred instances [23 22 63 to today's web-scale corpora 17 74),
data has always been the engine of machine learning. Discriminative models are only good as the quantity, quality, and diversity of the samples they see. Data augmentation eases this dependency by synthesizing additional training examples, aiming improve the generalization and robustness of models by exposing them to a more varied set of training instances. Classic hand-crafted transforms, such geometric warps, color jitter, flips, exploit invariances of natural images [5I]88 but cannot create entirely new content. Generative data augmentation (GDA) closes this gap by sampling fresh, high-fidelity images that improve downstream performance and unlock learning under privacy [122
95 35,, security [108] 88), or copyright constraints [56] 76).
'Thee authors contributed equally.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Among generative models, diffusion models B1 85 have emerged as the state-of-the delivering However, two obstacles limit their practical use for augmentation: i) diffusion sampling requires hundreds to thousands of denoising iterations, significantly inflating computational cost per image [005;; ii) the generator distribution QG inherently differs from the true data distribution Pdata- Unmeasured discrepancies between QG and Pdatn can seed spurious artifacts [19] and coverage bias [94] 69]. and ultimately skew classifier gradients, leading to biased generalization.
unparalleled sample quality and mode coverage 18 32
To overcome these challenges and unlock the potential of diffusion models for GDA, we introduce DAR-GDA a unified framework that renders diffusion -oauiiiinnuullty...... efficient generative augmentation by intertwining diffusion model Distillation, Adversarial alignment, and Reweighting by importance in GDA. First, score-based Distillation compresses a full teacher diffusion model trained on the true dataset into an efficient, single-step student generator, drastically cutting wall-time per sample by more than two orders of magnitude while preserving the Fréchet Inception Distance. Second, when integrated with the distillation process, Adversarial training further aligns the student model with the underlying distribution: the student generator not only imitates the teacher's scores but also competes against real images, which directly minimizes the Jensen-Shannon divergence with the true data, further narrowing the gap between qa and Pdata- Crucially, the discriminator learned in this process doubles as a density-ratio estimator; its calibrated output approximates the density ratio between two distributions, yiclding per-sample importance weights. Third, for Reweighting, these importance weights are applied when training downstream learners on synthesized images from the fixed student generator, for which reweighting the classification objective with these importance weights yields an unbiased stochastic estimator of the true data risk. Furthermore, we show that the combined effect of adversarial alignment and importance reweighting tightens upper bound on generalization error relative to conventional GDA. While the proposed DAR-GDA framework is generally applicable
discriminative modeling, in this work, we focus on one specific task of classification as the fundamental benchmark for investigating its impact on model generalization. The source code is available at bittps://qithhn com/ ruyianry/ gda dar
Our contributions are as follows:
We introduce DAR-GDA, a unified framework enabling practical GDA with diffusion models by concurrently addressing i) their slow sampling speeds and ii) the synthetic-to-real data bias, integrated via adversarial training.
We demonstrate how integrating adversarial training with score distillation not only improves the distilled model through real-data guidance by minimizing JS divergence, but also yields a discriminator. This discriminator, in turn, enables sample importance reweighting to correct for bias present in empirical risk on synthetic samples, forming our three-stage DAR-GDA approach.
Experiments confirm progressive performance gains across D-A-R stages, with DAR-GDA simultaneously demonstrating superior classification performance and efficiency for GDA on CIFAR-10 and ImageNet-1K.
2 Related work
2.1
Generative Data Augmentation
Early GDA approaches rely on VAES and GANs 48 25
which proved valuable for niche
scenarios such as class imbalance, low-shot recognition, and domain shifts [120] 91 73
77 7
67 54
36 Their impact on large-scale, high-resolution tasks remained limited because those models struggled to capture complex data distributions. Diffusion models overcome that limitation: they faithfully reproduce fine detail and avoid mode collapse, enabling advances in
115
53 1
robustness 6 26
10
62
privacy preservation (63). data imbalance 33, and self- or semi supervvsd
learning 998 90
A growing line of work harnesses foundation models diffusion generators
pre- treitrained on web-scale corpora such as Stable Diffusion and GLIDE [68] 64 and then aligns
them to a downstream domain S 34 99 72 123 29 [09]. While providing strong priors, this strategy raises several concerns such as licensing and usage constraints and limited applicability in fields with no web counterpart like scientific diffraction imaging (97]. Crucially, the high inference cost of iterative diffusion sampling remains, rendering it economically challenging solution. These

---

limitations motivate task-centrie GDA pipeline that i) learns an expressive generator directly from the classifier's own training set; and ii) produces crffusion-qualiig samples at reduced cost.
2.2 Adversarial Training
GANs cast generation as two-player game in which a discriminator learns to distinguish between real and generated samples, outputting probability of a sample being real [25]. This adversarial process is recognized for minimizing the Jensen-Shannon divergence between the true data distribution and the generator's distribution, which is distinct from the reconstruction losses-based generative models 21. Beyond its role in training the generator, the GAN discriminator's output can be interpreted as likelihood-fres density-rate estimator [93] 27 8T 922, which has spurred various techniques such as post-hoc discriminator soniiiidnttt-gudeeddddeee rejection sampling weakly supervised discriminator trained on reference set to refine generator outputs, yielding fairer or higher-fidelity sample sets []] 44 [22 4.
Both rely on training GAN from scratch and therefore inherit mode-collapse and stability issues [05 that limit their usefulness for large-scale augmentation. More recent work attaches a separately trained, GAN-style discriminator to a diffusion backbone steer sampling [42]; this requires distinct secondary training phase for the discriminator, and the valuable density-ratin information learns is typically not propagated inform downstream tasks. Collectively, we need for more integrated adversarial mechanism that can be synergistically employed with diffusion models, and applied for enhancing downstream applications.
3 Preliminaries
3.1 Diffusion Models
Diffusion (or score-based) generative models 80 gradually coruppts data point Ed into noise, and learns to reverse this process generate new data. In the forward diffusion, a deterministic schedule (at, of ) mixes signal and Gaussian noise: 2t = 0g + deCt, C ~ (O,
31
85
34
construct a latent Markov chain that
(1)
with ot decreases and a, increases with / such that कт approaches standard Gaussian noise.
The reverse process is modeled by a parameterized conditional distribution pa (= whose noise-prediction network Eg is trained to predict the noise e, added t [31 85]:
(idillusion) ( Ego-P. [[ll - 6 (r,, 112]
The trained noise predictor eo is directly related to the score function Va, log pogps often approxi- mated as & (, I) -a, -1 Les (Ft 1). generate a new sample, the model iteratively applies this reverse process, starting from pure noise ær and gradually denoising it using the learned score over T steps to reconstruct an initial data point zg. While this iterative process yields high-fidelity fidelity samples, performing T (often hundreds to thousands) steps incurs considerable computational cost.
3.2 Score Distillation
Score distillation aims to accelerate the inference of the trained T Tsstpp teacher diffusion sampler by condensing them into efficient student generator, Gg. This student, often capable of few- or single-step synthesis, is trained to replicate the teacher's learned data distribution, typically by using the teacher's score predictions guide the student's 's outputs various noise levels. This is commonly obtained by training an auxiliary fake diffusion model, parameterized by t/. on student output,
yt = az,G((z) + orEt:
2, E,E,N((O,))
via the same objective as Eq. [59 122 106), i.e., L (fake scor))(a)) = [ cg (v., 12] Let (0) be the minimizer w.r.t. v for fixed 0. The generator is then updated by minimizing the score distill (SD) loss, the divergence between the fake score (0) and the teacher score ps:
(0) Ex.,, = (D(p (y: t) Po +))]
The divergence D can be instantiated as Kullback-Leibler (KL) divergence [101] 59 111. Fisher divergence [12 121), or score-constructed trajectory-level divergence (43). In practice, v and 0 are
initialized from & and alternately updated. This process can yield one-step generator reproducing the teacher's 's quality at significantly reduced sampling cost (59] 122 121,, offering a promising path to overcome the efficiency bottleneck of diffusion models in practical GDA.
3

---

3.3 Generalization Risk in Supervised Learning
Let P be the unknown underlying data distribution over input-label pairs (2e,y) E X x.V. Given a hypothesis h:: 2 XR E H. e.g., a u-parameterized neural network function hw,, the generalization risk R(;; P) quantifies the expected loss on unseen data using loss function E: Z x 3 IK(such as cross-entropy), quantifying the penalty for the predicted output h(x) given the true label
Since P is inaccessible, learning relies on finite i.i.d. sample set Sp = ((2,, 34 ) P. The true risk is then approximated by the empirical risk, an unbiased estimation Remp (h: Sp):
Rh;; P) Ep(e(((z)))))) v)] = nnn Σ (e(((z)).bb) Remp Sp).
(5)
Empirical risk minimization (ERM) 196 60 is fundamental principle of statistical learning that aims to find a hypothesis that minimizes this empirical risk, Remp. with the expectation that a small empirical risk translates into low true risk and thus good generalization. The ERM principle forms the operational basis for the vast majority of supervised learning algorithms [6]
4 Distillation, Adversarial Alignment, and Reweighting for GDA
Before presenting our unified framework in Section 4.4 we first motivate its design by detailing the logical progression that leads to its components. This section begins with a critical reflection
on the implicit biases in conventional GDA (Section 4.1). We then establish principled, bias- correction mechanism via density-ratin reweighting (Section 4.2). This solution's 's efficacy demands generator with high fidelity and mode coverage, pointing directly
offeeoo-thhe-ar diffusion models. However, their practical use is barred by prohibitive sampling cost and the lack of built-in density-ratio estimator. We then employ adversarial score distillation (Section 4.3) as the technique that simultancously resolves both challenges, providing the foundation for the DAR-GDA pipeline.
4.1 Rethinking Learning from Generated Data Augmentation
Given the unknown data distribution (x. y), the generator G defines a distribution Qg. from which we can draw sets of augmenting synthetic samples SQa (()) ~ QG. A common practice to train a hypothesis, e.g., a classifier, is to minimize the empirical risk on these synthetic samples:
|Seal" Σ ((h(x)).u)
E ((h()).))
(6)
This strategy implicitly assumes that the synthetic distribution QG is a faithful proxy for the true data distribution P. However, in practice, Qg inevitably differs from P. Consequently, optimizing the objective in Eq. 6 amounts to optimizing with respect to a biased approximation of the true risk Rh; P)- y) ((h(x))))))))))))))))(())) v)]. This inherent bias reflects the error in the learning objective that arises directly from the distributional misalignment between 2a and P:
A bias e ((h()),))) - (=)
This distributional misalignment can manifest in several ways. For instance, if certain features or modes are overrepresented in QG relative to P -h-that is, in regions where 9G (x() ) > Pamta (z), the hypothesis may overfit to these dominant synthetic patterns, thereby neglecting rarer but potentially crucial characteristies of the true data. This issue of imbalance is recognized challenge, even in
today's advanced diffusion models [94 69 [05]. Furthermore, the deviation exacerbated if Qa generates spurious, incoherent, or tow-quality samples, i.e., samples æ. for which ga (.. y) > 0 but Pdata (x.,y) y) & O. Training on such artifacts, also reported in recent diffusson models [19 37 45), can lead the hypothesis to learn incorrect correlations, thereby undermining the effectiveness of GDA.
Given these inherent distributional discrepancies, naively treating all synthetic samples as perfect and equally valuable representatives of Paas implied by Eq. 6 can misguide the learning process. Therefore, to obtain more faithful estimate of the true risk R(h; P). and thus mitigate Abias (h), is crucial to account for variations among individual synthetic samples in their fidelity and alignment with P. A principled approach involves reweight the contribution of each synthetic sample in the empirical risk according to its estimated alignment. This naturally gives rise to the need for mechanism that can quantify the degree of alignment between each synthetic sample and the true data distribution, enabling the corresponding correction of the loss terms {((x)..))

---

4.2 Density-Ratio Reweighting
A natural way to perform this quantification is to compare the marginal densities of synthetic samples under the two distributions. However, since Pima(.) is only known through its empirical distribution and (c(+) is defined implicitly by the generative process, directly quantifying their discrepancy-and thus correcting for the induced learning bias-is challenging.
To circumvent the need for explicit likelihood estimatiin, we draw inspiration from the Generative Adversarial Network (GAN) framework (25), which provides a likelihood- mechanism for
distinguishing between real and synthetic samples, In GANs, discriminator D and a generator G
engage minimax game: ming maxp lE (t) [log ((x)
(x=) [Io[[] --)))()) For
fixed generator G, the optimal discriminator D satisfies 25 114]
Puara +
(*)x)= =
(8)
Piata (a) (c(x)
This optimal discriminator D captures the probability that sample originates from Pdata (x) rather than qa (x), thereby providing an implicit comparison between the two densities.
Leveraging this, the density ratio r(æ), which compares the likelihoods of a sample æ under Pdata versus 9G can be derived from D* (z):
D7 1- (x) 1- Pdata (x) (Ptata (x) + (x))
Pasaa (3:) (Pana ax) (x)) Piama (x)
r(z)
ga (a
This ratio acts an importance weight briding the expectations of the involved distributions: for any integrable f:
(( ((e.))-e(((()) ((x (xx,
(10)
so weighting synthetic samples by r(z) converts expectations under qG into those under Pdat- Intu- itively, an r(æ)> amplifies regions underrepresented by qg. whereas an ratio r(æ) < 1 downweights overrepresented or atypical synthetic xxamples-herrbb thereby correcting distributional misalignment.
To apply this approach to GDA, we estimate r(æ) via the discriminator D (denote the estimate (n)))) and reweight the synthetic loss contributions:
C(reweight) (h; D) ==|Sqal" -1 Σ rD(())((((z))))
=Bea (FD())(((())) v)] ]
(1I)
Optimizing i on synthetic samples under this reweighted objective aligns training on the synthetic distribution Qa with learning directly from the true data distribution P, thereby mitigating the bias Apias()) that arises from their distributional discrepancy.
In practice, r r(æ) is estimated using parameterized discriminator (x) x D* (x) which introduces additional variance into the loss computation. To manage the inherent bias-variance trade-off, we employ two variance reduction techniques:
Truncation: To prevent large r(z) values from dominating the loss and destabilizing training, clip the importance weights as F(x) min(r(x), r), for a threshold y 2 1.
Self-normalization: Within each mini-batch of & synthetic samples (zy)}-1. we apply batch-wise self-normalization to the truncated weight: r(æ) = f(x)/E/ F(æ,). This normalizes the sum of weights a mini-batch, further stabilizing the updates.
The efficacy of importance reweighting hinges on a well-hehaved density ratio r r(x), which requires
significant overlap between the generative (Qc) and (P) distributions (24). Generators trained
with purely adversarial objectives often fail this prerequisite, suffering from mode collapse or support
mismatch, especially in the early, prodecng training stages [T7 86 87
38 2 49] This
makes their discriminators poor estimators for r(x). Diffusion models, on the other hand, with emerge as ideal candidates for qa- However, they present two fundamental obstacles: i) Efficiency: they are computationally prohibitive for augmentation sampling; and ii) Mechanism: they are trained without an adversarial component, leaving no mechanism to estimate the r r(æ) required for importance reweighting. These challenges motivate unified approach that simultaneously addresses the efficiency problem and the density- ratio-estimation problem. We therefore bridge this by integrating adversarial training directly within
their high sample quality and excellent mode coverage 83 47,
5

---

the score distillation framework to compress pre-trained diffusion teacher model into fast student generator, while concurrently learning the discriminator needed for the bias-corg reweighting
4.3 Adversarial Score Distillation for Diffusion Models
The Reweighting stage of our DAR-GDA framework necessitates discriminator capable of estimat- ing the density ratio r(z). As established, vanilla diffusion models not only lack this adversarial component but are also prohibitively slow due their iterative sampling process, leading to high per-sample costs. To address both the need for density-ratio importance estimator and the demand for computationally efficient augmentation, we therefore bridge the score diffusion distillation proce- dure, detailed in Section 3.2 with adversarial training. drawing inspiration from methodologies that integrate GAN-like objectives with diffusion processes [10).
Critically, the student generator Go is initialized from and guided by the high-fidelity, pre-trained teacher. This provides a powerful starting point, allowing the adversarial training to bypass the unsta- ble, error-prone early stages-where a generator starting from noise struggles to produce informative samples-thal that typically plague purely adversarially trained models [86 87).
In this process, the trained iterative teacher parameterized by e is compressed into a one-step, fast student generator Go. while discriminator D,, is learned simultancously. This is framed as the following minimax objective:
min max A1 (SD) (0) + A2 Vradvs(0, 0,,1),
(12)
where C is the score- distiliitillationn loss defined in Eq. compelling the student Go to match the teacher's is score estimates. The adversarial component is defined by the value function Vindv) (0, m):
Diades (0. (0.n) =EE co.n D2 + ofe;t)] E=V((,, co -,, t)
(13)
where y: = Go (z) + a noisy sample from the student generator Gg at l, and ~ r is sampling distribution over timesteps. This design allows the discriminator to compare real and fake samples
across various noise levels-a design shown to enhance training stability [100] [07]. In the specific case where the adversarial game operates only the clean image space, π simplifies to a Dirac delta distribution centered at = O. The adversarial gradient of the student is obtained by minimizing the
c(adv:) (0) EE,,, [log (36: (Me:))] non-saturating negative of the second expectation in Eq. 13 i.e., L
The discriminator Dr can be realized as a standalone network that processes the entire input to produce single global probability [43], or as an integrated component sharing the student's s score U-Net encoder [119] In this latter, integrated approach, a final probability is obtained by applying patch-wise aggregation to the encoder's 's output logits. The parameter vector v) denotes all the weights of this discriminator, including any parameters shared with the student generator.
The adversarial score distillation strategy offers several synergistic advantages. The student generator Go produces samples in single step, drastically reducing computational costs compared to its iterative teacher and making diffusion models practical for GDA. The synergy of adversarial training and score distillation also drives the student distribution qG, closer to Pdata by directly minimizing the Saaenn--hnnonn divergence, while preserving the teacher's high-fidelity details. Empirically. adversarial distillation has shown stable behaviour, with one-step students generally matching-an and
frequently reported to surpass- tthiir teachers' quality FID 121 110 59. Crucially, this process yields the discriminator Dr, as concurrent byproduct, eliminating the need for a separate training phase. This discriminator provides the exact mechanism required for the reweighting stage, furnishing the density-ratio estimate in Eq. 9 needed to debias the downstream classifier's 's training.
4.4 The DAR-GDA Framework
Having introduced the modular components of Distillation, Adversarial alignment, and Reweighting, we now synergistically combine them into the DAR-GDA framework. The DAR-GDA framework provides unified pipeline: an efficient student generator Go (from Distillation) that is aligned with Paara (via Adversarial training), and a co-trained discriminator D., that enables bias-correcting reweighting. This integrated design enhances generalization by mitigating the key error sources inherent in learning from synthetic data. Furthermore, initializing Go from a proficient diffusion teacher significantly stabilizes adversarial training and mitigates classic GAN pathologies such

---

sampler
One-step
Score
simpling
Distillation
Student Generator (One-step)
Teacher Diffusion Model (Multi-step sampling)
+
efficient generative data ougmentation
Past
Importance Reweighting
dversorial
Algoment
Reverss DiFFusion Step
Dataset
1o [e] -F - [e)
a bota Soeples
Discriminator
Classifier
Figure Overview the DAR-GDA framework. (D) A multi-step teacher diffusion model is
distilled into an efficient one-step student generator. (A) The student is adversarially aligned with
the true data distribution using a discriminator. (R) The discriminator's outputs are then used as
importance weights to train downstream classifier.
as mode collapse and catastrophic forgetting [89]. We illustrate the overall idea in Figure | and summarize the complete algorithmic procedure Appendix
This integrated design enhances generalization by mitigating the key error sources inherent in learning from synthetic data. We can formalize this improvement by analyzing the expected risk Ep [(()] following
10 118 75). This true risk can be decomposed as two summands:
Ep [p() [Ep[ [{+)]] Ep [((() ]] + [Ep[<(h)e.,)
(14)
The first summand reflects the impact of distributional mismatch. For bounded loss |4| s L:
Ep - [((h) ]| 2L Drv (P @G.) s 2L, Qc.),
(15)
by variational characterization of total variation (TV) distance and Pinsker's inequality. Adversarial score distillation directly minimizes the Gensen-Shannol (JS) divergence between P and Qg. hence tightening the upper bound on the distribution mismatch compared to using non-adversarially trained generators or the original teacher diffusion model. The second term, the biased synthetic risk, the objective naively optimized by conventional GDA (Eq, (6). Our Reweighting stage directly corrects this. By employing the objective Eq. [1] we replace the minimization of this biased term with an unbiased stochastic estimator of the true data risk, Ep [p()]]]
Thus, DAR-GDA enhances generalization through dual mechanism: the Adversarial component minimizes the distributional bias (Term 1), while the Reweighting component provides an unbiased risk estimator for hypothesis training (correcting Term 2). Coupled with the practical benefit of > 100x reduction in per-sample generation cost from Distillation, DAR-GDA is positioned as powerful and practical tool for GDA.
5 Experiments
We evaluate DAR-GDA on CIFAR-10 and ImageNet-IK, comparing it with i) standard data- augmentation pipelines, ii) strong diffusion- based GDA baselines, and lii) state-of-the-art GAN. The analysis proceeds in two stages. We benchmark it against conventional data augmentation methods and existing GDA baselines. We also assess its performance adaptability with various underlying generative models. We additionally probe how DAR scales under dynamic augmentation smaller datasets for practical feasibility.
Datasets. We use the canonical train/val splits of CIFAR-I0 (50] and ImageNet-IK KK [17]: no external or test data are introduced at any stage, including teacher training, distillation, or classification.
Evaluation. Classification performance is reported as top-1 accuracy for CIFAR-10 and top- 1/top5 accuracy for ImageNet- IK Generator quality is measured with Fréchet Inception Distance (FID) (30].
Generative Models. For diffusion- GDA, we adopt the publicly released checkpoints of DDP for CIFAR-10 (39 and DDPM++EDMM2XX
for ImageNet- 1K [40]. On CIFAR-
10 we include R3GAN 333, GAN state-of-the-are achieving the best reported FID. We omit GANs on ImageNet because no open-source 512 x 512 model matches diffusion quality, and de novo training is
7

---

Table 1: CIFAR-I0 classification accuracy with ResNet-18 and VGG-16, using training sets aug- mented by synthetic data equal to 1x the original dataset size (additive augmentation) on various generative models and adversarial-ddistilattoo methods under static versus dynamic data generation schemes. * indicates the use of web-pretrained text-to-image model, such as GLIDE [64] and Stable Diffusion (SD) [68] which are to be carefully compared to other generative models trained only the original data. The best result is shown in bold.
DAR progression
static (gen. once)
dynamic (re-gen. each epoch)
Generative Model
Distill
Adv. ulign Reweight
ResNet-18
VGG-16
ResNet-18
VGG-16
Real only
95.00=0.15
93.76+0.20
95.00+0.45
03.76+0.50
18.18=0.50
93,90=0.42
95.61+0.98
14.1
R3GAN (F96=1.96)
95.32.0.47
93.8610.39
95.7211000
93.7311.26
GLIDE real guidance SD ActGen
95.77=0.25
94.50+0.41
95.92=0.42
94.68+0.39
X
95.37=0.23
93,95+0.32
16.16+0.76
94.98+0.79
X
95.3220.39
93,9740.29
95.8410.86
95.08&1.02
CTM +
Diffusion Model (FID==181)
1.73=
X
95,43
94.0910.26
96.3510.98
95.15 1.13
95.88=0.22
94,28+0.24
96.57+
95.33+ 1.07
X
95,48=0.25
94.1810.28
96.2910.68
95.2220.79
SiDA 121
(FID=11.39
X
95.84=0.36
94.53+0.36
96.40+ 1.16
95.40+ 1.02
96.21=0.25
94,64+0.20
96.73+ 1.15
95.73+ 1.08
notoriously unstable at that scale. We also include selected finetuned text-to-image models pre-trained on web-scale image data. Please note that comparisons with these models may be subject to concerns like additional information from pre-training data and potential leakage.
Implementation Details.
Table 2: ImageNet- classification accuracy with ResNet- 50 and ViT-S/16, using training sets augmented by synthetic data equal 1x the original dataset size (additive augmenta- tion) on various generative models under the static generation. denotes the use a web-pretrained text-to-image model, Imagen 770 or Stable Diffusion (SD) 68 which are to be carefully compared to other generative models trained only on the original data. The best result is shown in bold.
We instantiate the (D) and (A) steps of our DAR-GDA with two re- cent leading algorithms: CTM [43). a rajectory-basse score distillation with a standalone discriminator, and
SiDA 1.2 a -isher-divergence- minimizing score-based distillation with encoder-sharing discriminator operating in noise space. Both are tested on CIFAR-I0; SiDA alone is used on ImageNet-Il IK owing to CTM's current memory footprint. On CIFAR-10, 10, we set 1.2 for SiDA and train with Adam (Ir=le-5) [46] optimizer. CTM is trained for 256 steps per batch with student Iearn- ing rate of 3e-4 and the discrimina- tor learning rate 2e-3 (batch size 128). For ImageNet- 1K we distill EDM2- XXL with SiDA across 8 A 1100-80GB
DAR progression
Generative Model
Distill Ady, align Reweight Top-1 Acc. Top-5 Acc.
ResNet- 50 classifier
Real only
76.37=0.03
92.8620.05
Tmagen finetune
18.17
SD + ActGen
34
78.34.0.32
94, 12.0.33
x
77.12.0.15
93.95.0.20
Diffusion Model 50
X
77.15-0.27 93.7420.19
(FID~1.91)
SIDA I
(FID-L.7 .37)
x
77.8940.18
93.90-0124
78.0340.23
9..13.0.32
VIT classiteer
Real only
793120004
94.4820005
Imagen linetune
81.00
SD + ActGen H
81.17-051
95.32-0.11
80.5020.18
95. 18:025
Diffusion Model 4
80.46.0.23
19.09..
GPUs: a= 1.0, per-GPU batch size 64, gradient accumulation every 128
(FID=1.91)
SIDA 121
(FID=1.37)
80.9920.32
95.38.0.43
81.17-0.45
95.36-0.40
iterations using Adam (Ir=5e-5) optimizer. For CIFAR-10 we train ResNer-18 (28 and VGG-16 [79]:
for ImageNet-IRP 1K we use ResNet-50 (28) and ViT-S/16 20 to evaluate the performance with and without the (R)eweighting component with self celf-noomalization and = 1. CIFAR-10 models are trained for 300 epochs with batch size 128 using momentum SGD (Ir=0.1). The hyperparameters A and X2 for the adversarial alignment objective follow the settings from prior work. On ImageNet- ResNet-50 is trained for 90 epochs with batch size 4096 and initial learning rate 1.6, while ViT-S/16 is trained for 300 epochs with batch size 1024, Adam W (58), and initial learning rate 3e-3. We apply self-normalization and set 7 to be for obtaining r(æ) for reweighting. Diffusion outputs are generated at 32 32 for CIFAR-10 10 and 512 x 512 for ImageNet; the latter are down-sampled to 224 x 224 before classification to match baseline protocols. Experiments use NVIDIA 100-80GB GPUs-single-GPU for CIFAR- 10, 8-GPU (DDP) for ImageNet- IK implemented in PoTorh 2.1 (66).
8

---

Table 4: CIFAR-10 classification accuracy with ResNet-18 and VGG-16, 16, using training sets sub- stituted by synthetic data equal to 1x the original dataset size on various generative models and adversarial-distillanee methods under static and dynamic data generation.
The best result bold.
DAR progression
static (gen. once)
dynamic (re-gen. cach epoch)
Generative Model
Distill
Adv. align Reweight R8sNet-18
VGG-16
ResNet-18
VGG-16
Real only
95.0010.15
93,7620.20
95.0010.45
93.76.0.50
90.0910.65
87.75.0.39
9..2610.84
89.7910.58
R3GAN 33 (FID=1.96)
90.29a0.71
87.97.0.58
92.9110.72
88.0320.61
X
X
91.78+0.25
90.60=0.22
95.88=0.19
94.26=0.21
X
90.83+0.39
89.65=0.30
93.70=0.24
93.18=0.28
CTM 43
Diffusion Model 30
FID==173
X
91.03a0.44
89.78=0.46
94.00=0.70
93.27=0.67
(FID=1.81)
91.37+0.43
9..26=0.52
94.32+0.72
93.38+0.52
93.20+0.28
91.71=0.33
0.33
94.46+0.27
SiDA 121
FID=1.39
X
93.50+0.35
91.88=0.38
96.39+0.60
95.25+0.52
93.77+0.37
92.31=0.48
96.68=0.63
95.72+0.48
More detailed experimental setups are provided in Appendix C
DAR-GDA thus offers a practical, drop-in GDA solution, achieving diffusion-lewe fidelity with efficiency bias correction.
5.1 Additive Augmentation Results
We evaluate DAR-GDA by supple- menting the original training set with an equal volume of synthetic samples that replicate the original label distri-
Table 3: GPU hours to generate 1:1 augmenting training data
replica. SD involves downsampling for CIFAR-1O.
Dataset
R3GAN EDM/EDM2 CIM-EDM SID-EDM
SD
CIFAR-10 26.0s
3102s
58.38
39.1s
6.8h
bution, using both static (one-time)
INIK
29th
8.7h
1290h
and dynamic (per-epoch) generation strategies. Tables and 2 present classification results for
CIFAR-10 and ImageNet-IK. DAR-GDA consistently boosts accuracy across both datasets and hypothesis sets. On CIFAR-10, improves 18 by +1.7 pp and VGG-16 by +1.9 pp. Similar gains are observed on ImageNet- IK for ResNet-50 (+1.7 pp) and ViT-S/16 (+1.3 pp). Notably, our in-domain DAR-GDD (trained solely on task-specific data) matches or outperforms pre-trained large text-to-image models, particularly where baseline diffusion models without DAR are suboptimal, Dynamic generation consistently yields better results than static, underscoring the value of increased diversity. Also, SiDA leads to better classification performance than CTM, correlating with SiDA's superior FID score.
Table 5: ImageNet-IK IK classification accuracy with ResNet- 50 and ViT-S/16, using training sets substituted by synthetic data equal to 1x the original dataset size various genera- tive models and
Table 3 highlights the substantial GPU time savings for synthetic data gener- ation. Distillation achieves synthesiz- ing speeds comparable to fast GANs, positioning (D)istillation as key to achieve an economical solution for high-quality GDA.
methods under static versus dynamic data generation schemes.
Generative Model
DAR progression
Distill Adv, align Reweight Top- Acc. Top-5 Avc.
ResNet-30 classifier
Real only
76.3740.08
92 8620.0
X
66.414047
86.5840.35
5.2 Substituting Augmentation Results
Diffusion Model A0
(FID=1.91)
SIDA 121
FID=1.37
X
66.22.0.50
66.42.0:59 86.541033
86.301042
66.5020.87
86.82:0.66
We next consider evaluate DAR-GDA in a full data replacement scenario, where hypotheses are trained solely on synthetic samples equivalent in vol- ume and label distribution to the origi- nal training set. Tables 4 and report the classification performance for CIFAR- 10 and ImageNet-IK, respectively. While DAR-GDA consistently boosts accuracy over baseline synthetic data, perfor- mance gap to training on real data is observed, this drop being more significant on ImageNet- IK. Notably, on ImageNet- K, DAR-GDA enables hypotheses trained synthetic data to match the performance achieved with data from the original, don-diittlledd multi-step diffusion teacher model.
VIT-S/16 VIT-s/16classllee
Real
79.101008 94.431008
X
X
67.89+0.38 25.0502229
Diffusion Model 40 (FID=1.91)
SIDA
121
X
X
67.5440.3
85.0940.30
F1D=1.37
67.7340.50
85.30+044
68.01.0.64 05.89.0.6
On CIFAR-10, SiDA-generated data significantly outperforms CTM data, despite both models being distilled from the same teacher. Remarkably, fully-synthetic GDA with SiDA with dynamic generation can surpass the performance of GDA training on the real datasct alone.

---

E700 0
Penarago Traisi Semple Seieeedd (h0)
(b) Figure 2: Classification accuracy of SiDA-distilled EDM/EDM2 models on CIFAR-IO/IN-IK with
(a)
(c)
(d)
varying synthetic dataset sizes (measured as a % of the original training set). Performance is shown
under two schemes: additive augmentation (Figs. 2a 2b and full data substitution (Figs. 2c 2d
5.3 Further Empirical Studies
We present an ablation study on augmenting size in Fig. 2 For additive augmentation, performance increases with more synthetic data, and our method's 's components yield incremental gains. Conversely, under data substitution, performance generally declines with increasing synthetic data. This decline more significant for ImageNet, where generative models face greater difficulty matching the true data distribution. Notably, on CIFAR-1O, a dynamic generation schedule improves performance with up 75% of the synthetic data replacement, where diversity from new per-epoch samples appears to offsee the loss of real data. Additional empirical studies are presented in Appendix D
6 Discussion and Conclusion
Limitations and Ethical Considerations. Our methodology concentrates on GDA itself, rather than on the development of new generative models. Hence, we leveraged state-of-the-art EDM diffusion models as the teacher. While this approach entails a high, one-time pre-training cost, our framework does not address this; rather, our focus is making these powerful, pre-trained models highly efficient for low-latency applications like GDA. A related limitation is that our validation, while rigorous, was constrained to the two large-scale datasets for which these specific teacher weights are publicly available, though the observed patterns were consistent and supported our hypotheses. Ethically, our GDA framework contributes positively to content safety by enabling stronger and more reliable discriminative models for detecting and filtering harmful or NSFW material. Nevertheless, because the generator component is trained to replicate the visual characteristies of harmful content for the purpose of improving filtering, it inherently carries the capability to reproduce such material. To ensure responsible use, the trained generator should be securely stored to prevent misuse.
Future Work. Looking ahead, the core principles of DAR-GDA are fundamentally domain -agnssii. Our framework treats the teacher model as a "black box" for score predictions. This modularity allows the framework to be applied beyond vision, for instance, to emerging diffusion models in domains like protein [T02] and molecule generation [12], simply by replacing the U-Net backbone with a cemaii--ppecffc architecture. This is particularly promising for data-scarce fields like medicine (e.g., MRI or X-ray synthesis), where distilled student model could be shared to enable "pseudo data sharing" " without violating patient confidentiality. Beyond new domains, the framework is applicable to broader range of conditional tasks. For instance, in text-to-image generation, the discriminator component could be adapted to evaluate image quality and prompt alignment, egg., assessing Pimage prompt). The same principle applies to dense prediction tasks like segmentation, where the challenge shifts towards pseudo-labeling from foundation models or other forms of weak supervision create the necessary training signals.
Conclusion. In this paper, we addressed two primary hindrances to the practical application of current diffusion models in GDA: suboptimal sampling efficiency and potential misalignment be- tween generated and target data distributions. We introduced DAR-GDA, a three-stage synergistic framework where adversarial alignment (A) serves as crucial bridge connecting model distillation (D)-to to enhance sampling speed-avt and sample reweighting (R)-to correct distributional shifts. Our experiments demonstrated that the progressive integration of these DAR components leads to consistent improvements in GDA performance. Beyond the class-conditional models explored, this work may inspire the application of similar principled strategies to wider array of diffusion-baser generative models for effective data augmentation.
10

---

Acknowledgments
M. Zhou acknowledges the support of gift grant from Apple.
References
[1] Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprint arXiv: 2017.
[2] Martin Arjovsky and Leon Bottou. Towards principled methods for training generative adversarial networks. In ICLR, 2017.
[3] Reyhane Askari-Hemmat, Mohammad Pezeshki, Florian Bordes, Michal Drozdzal, and Adri- ana Romero-Soriano. saadback-guided data synthesis for imbalanced classification, Trans. Mach. Learn. Res., 2024.
[4] Samanch Azadi, Catherine Olsson, Trevor Darrell, Ian J. Goodfellow, and Augustus Odena. Discriminator rejection sampling. In ICLR, 2019.
[5] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. Synthetic data from diffusion models improves ImageNet classification. Trans. Mach. Learn. Res., 2023.
[6] Hritik Bansal and Aditya Grover. Leaving reality imagination: Robust classification via generated datasets. In ICLR Workshop on Trustworthy and Reliable Large-Scale ML Models, 2023.
[7] Christopher Beckham, Sina Honari, Vikas Verma, Alex Lamb, Farnoosh Ghadiri, R. Devon Hjelm, Yoshua Bengio, and Chris Pal. On adversarial mixup resynthesis. In NeurlPS, 2019.
[8] Yasser Benigmim, Subhankar Roy, Slim Essid, Vicky Kalogeiton, and Stéphane Lathuilière. One-shot unsupervised domain adaptation with personalized diffusion models. In CVPRW, 2023.
[9] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
[10] Zhen Cheng, Fei Zhu, Xu-Yao Zhang, and Cheng-Lin Liu. Breaking the limits of reliable prediction via generated data. Int. Comput. Vis., 2024.
[11] Kristy Choi, Aditya Grover, Trisha Singh, Rui Shu, and Stefano Ermon. Fair generative modeling via weak supervision. In ICML, 2020.
[12] François Cornet, Grigory Bartosh, Mikkel N. Schmidt, and Christian A. Naesseth. Equivariant neural diffusion for molecule generation. In NeurIPS, 2024.
[13] Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
[14] Ekin D. Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. AutoAug- ment: Learning augmentation strategies from data. In CVPR, 2019.
[15] Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. RandAugment: Practical automated data augmentation with a reduced search space. In NeurIPS, 2020.
[16] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge J. Belongie. Class-balanced loss based on effective number of samples. In CVPR, 2019.
[17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
[18] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs image synthesis. In NeurIPS, 2021.
[19] Anh-Dung Dinh, Daochang Liu, and Chang Xu. Representative guidance: Diffusion model sampling with consistency. In ICLR, 2025.

---

[20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition scale. In ICLR, 2021.
[21] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[22] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. Int. J. Compud. Vis., 88(2):003- 338, 2010.
[23] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories. In CVPRW, 2004.
[24] John Geweke. Bayesian inference econometric models using Monte Carlo integration. Econometrica: Journal of the Econometric Society, pages 1317-1339, 1989.
[25] lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde Farde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commun. ACM, 63(1 (3(11))139-144 2020.
[26] Sven Gowal, Sylvestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A. Mann. Improving robustness using generated data. In NeurIPS, 2021.
[27] Aditya Grover and Stefano Ermon. Boosted generative models. In AAAI, 2018.
[28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[29] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip H. S. Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? In ICLR, 2023.
[30] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In NeurIPS, 2017.
[31] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
[32] Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23:47:1-47:33, 2022.
[33] Nick Huang, Aaron Gokaslan, Volodymyr Kuleshov, and James Tompkin. The GAN is dead; long live the GAN! modern GAN baseline. In NeurlPS, 2024.
[34] Tao Huang, Jiaqi Liu, Shan You, and Chang Xu. Active generation for image classification. In ECCV, 2024.
[35] Häkon Hukkelás, Rudolf Mester, and Frank Lindseth. DeepPrivacy: A generative adversarial network for face anonymization. In ISVC, 2019.
[36] Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as a data source for multiview representation learning. In ICLR, 2022.
[37] Yazid Janati, Badr Moufad, Mehdi Abou El Qassime, Alain Oliviero Durmus, Eric Moulines, and Jimmy Olsson. A mixture-based framework for guiding diffusion models. In ICML, 2025.
[38] Simon Jenni and Paolo Favaro. On stabilizing generative adversarial training with noise. In CVPR, 2019.
12

---

[39] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-bedd based generative models. In NeurIPS, 2022.
[40] Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In CVPR, 2024.
[41] Vahid Kazemi and Josephine Sullivan. One millisecond face alignment with an ensemble of regression trees. In CVPR, 2014.
[42] Dongjun Kim, Yeongmin Kim, Se Jung Kwon, Wanmo Kang, and Il-Chul Moon. Refining generative process with discriminator guidance in score-based diffusion models. In ICML, 2023.
[43] Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ODE trajectory of diffusion. In ICLR, 2024.
[44] Ycongmin Kim, Byconghu Na, Minsang Park, JoonHo Jang. Dongjun Kim, Wanmo Kang, and Il-Chul Moon. Training unbiased diffusion models from biased dataset. ICLR, 2024.
[45] Yujin Kim, Hyunsoo Kim, Hyunwoo Kim, and Suhyun Kim. When model knowledge meets diffusion model: Diffusion-assistie data-free image synthesis with alignment of domain and class. ICML, 2025.
[46] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR, 2015,
[47] Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. On density estimation with diffusion models. In NeurlPS, 2021.
[48] Diederik P. Kingma and Max Welling, Auto-encoding variational Bayes. In ICLR, 2014.
[49] Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. On convergence and stability GANs. arXiv preprint arXiv: 2017.
[50] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Canadian Institute for Advanced Research, 2009.
[51] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet classification with deep convolutional neural networks. In NeurlPS, 2012.
[52] Jinhee Lee, Haeri Kim, Youngkyu Hong, and Hye Won Chung. Self-diagnosing GAN: diagnosing underrepresented samples in generative adversarial networks. In NeurIPS, 2021.
[53] Chongxuan Li, Taufik Xu, Jun Zhu, and Bo Zhang. Triple generative adversarial nets. In NeurIPS, 2017.
[54] Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Kreis, Sanja Fidler, and Antonio Torralba. BigDatasetGAN: Synthesizing ImageNet with pixel-wise annotations. In CVPR, 2022.
[55] Yuting Li, Yingyi Chen, Xuanlong Yu, Dexiong Chen, and Xi Shen. SURE: survey recipes for building reliable and robust deep networks. In CVPR, 2024.
[56] Chumeng Liang, Xiaoyu Wu, Yang Hua, Jiaru Zhang, Yiming Xue. Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Adversarial example does good: Preventing painting imitation from diffusion models via adversarial examples. In ICMI. 2023.
[57] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, 2014.
[58] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[59] Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff-Instruct: A universal approach for transferring knowledge from pre-trained diffusion models. In NewrIPS, 2023.
13

---

[60] Mehryar Mohri. Foundations of machine learning. MIT press, 2018.
[61] Andrea Montanari and Basil N Saeed. Universality of empirical risk minimization. In CoLT, 2022.
[62] Byeonghu Na, Yeongmin Kim, HeeSun Bae, Jung Hyun Lee, Se Jung Kwon, Wanmo Kang, and I1-Chul Moon. Label-noise robust diffusion models. In ICLR, 2024.
[63]
Sameer A Nene, Shree K Nayar, Hiroshi Murase, al. Columbia object image library (coil-20). 1996.
[64] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishknn, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In ICMI, 2022.
[65] Jinseong Park, Yujin Choi, and Jaewook Lee. In-distribution public data synthesis with diffusion models for differentially private image classification. In CVPR, 2024.
[66] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performanee deep learning library. In NeurlPS, 2019.
[67] Suman V. Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. NeurlPS, 2019.
[68] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[69] Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M. Weber. CADS: unleashing the diversity of diffusion models through condition-annealed sampling. In ICLR, 2024.
[70] Chitwan Saharia, William Chan, Saurabh Saxena. Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In NeurlPS, 2022.
[71] Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In NeurIPS, 2016.
[72] Mert Bülent Sariyildiz, Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake till you make it: Learning transferable representations from synthetic ImageNet clones. In CVPR, 2023.
[73] Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adversarially robust generalization requires more data. In NeurlPS, 2018.
[74] Christoph Schuhmann, Romain Beaumont, Richard Veneu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: open large-scale dataset for training next generation image-text models. In NeurlPS, 2022.
[75] Vikash Sehwag, Saeed Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang, Mung Chiang, and Prateek Mittal. Robust learning meets generative models: Can proxy distributions improve adversarial robustness? In ICLR, 2022.
[76] Shawn Shan, Jenna Cryan, Emily Wenger, Haitao Zheng, Rana Hanocka, and Ben Y Zhao. Glaze: Protecting artists from style mimicry by Texa-to-Image models. In USENIX, 2023.
[77] Konstantin Shmelkoy, Cordelia Schmid, and Karteek Alahari. How good my GAN? In ECCV, 2018.
14

---

[78] Patrice Y. Simard, David Steinkraus, and John C. Platt. Best practices for convolutional neural networks applied to visual document analysis. In ICDAR, 2003.
[79] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[80] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
[81] Jiaming Song and Stefano Ermon. Bridging the gap between f-GANs and Wasserstein GANs. In ICML, 2020.
[82] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[83] Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In NeurlPS, 2021.
[84] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurlPS, 2019.
[85] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[86] Akash Srivastava, Lazar Valkov, Chris Russell, Michael U. Gutmann, and Charles Sutton. VEEGAN: reducing mode collapse GANs using implicit variational learning. In NeurIPS, 2017.
[87] Ruoyu Sun, Tiantian Fang, and Alexander G. Schwing. Towards a better global loss landscape of GANs. In NeurIPS, 2020.
[88] Vajira Thambawita, Pegah Salehi, Sajad Amouei Sheshkal, Steven Hicks, Hugo I. Hammer, Sravanthi Parasa, Thomas de Lange, Pál Halvorsen, and Michael A Riegler. SinGAN-Seg: Syn- thetic training data generation for medical image segmentation. PLOS ONE, 1765):e0267976. 2022.
[89] Hoang Thanh- Thanh-Tung and Truyen Tran. Catastrophic forgetting and mode collapse in GANs. In IJCNN, 2020.
[90] Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. In NeurIPS, 2023.
[91] Toan Tran, Trung Pham, Gustavo Carneiro, Lyle J. Palmer, and lan D. Reid. Bayesian data augmentation approach for learning deep models. In NeurIPS, 2017.
[92] Ryan D. Turner, Jane Hung, Eric Frank, Yunus Saatchi, and Jason Yosinski. Metropolis- hastings generative adversarial networks. In ICML, 2019.
[93] Masatoshi Uehara, Issei Sato, Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Generative adversarial nets from density ratio estimation perspective. arXiv preprint arXiv:1610.02920 2016.
[94] Soobin Um, Suhyeon Lee, and Jong Chul Ye. Don't 't play favorites: Minority guidance for diffusion models. In ICLR, 2024.
[95] Boris van Breugel, Hao Sun, Zhaozhi Qian, and Mihaela van der Schaar. Membership inference attacks against synthetic data through overfitting detection. In AISTATS, 2023.
[96] Vladimir Vapnik. Principles of risk minimization for learning theory. In NeurIPS, 1991.
[97] Benquan Wang, Ruyi An, Jin- Kin-Kyu So, Sergei Kurdiumov, Eng Aik Chan, Giorgio Adamo, Yuhan Peng. Yewen Li, and Bo An. OpticalNet: An optical imaging dataset and benchmark beyond the diffraction limit. In CVPR, 2025.
15

---

[98] Yifei Wang. Jizhe Zhang, and Yisen Wang. Do generated data always help contrastive learning? In ICLR, 2024.
[99] Zerun Wang, Jiafeng Mao, Xueting Wang, and Toshihiko Yamasaki. Training data synthesis with difficulty controlled diffusion model. arXiv preprint arXiv:2411. 2024.
[100] Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion-GAN: Training GANs with diffusion. In ICLR, 2023.
[101] Zhengyi Wang, Cheng Lu, Yikai Wang. Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifieDreamer: High-fidelity and diverse text-to-3D generation with variational score distillation. In NeurIPS, 2023.
[102] Joseph Watson, David Juergens, Nathaniel Bennett, Brian L Trippe, Jason Yim, Helen Eisenach, Woody Ahern, Andrew Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with RFdiffusion. Nature, 62097976)1188--1100 2023.
[103] Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: study using woalwwrldd human annotations. In ICLR, 2022.
[104] Ruixuan Xiao, Yiwen Dong, Haobo Wang, Lei Feng, Runze Wu, Gang Chen, and Junbo Zhao.
ProMix: Combating label noise via maximizing clean sample utility. In I/CAI, 2023.
[105] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. ICI.R, 2022.
[106] Sirui Xie, Zhisheng Xiao, Diederik P. Kingma, Tingbo Hou, Ying Nian Wu, Kevin P. Murphy, Tim Salimans, Ben Poole, and Ruiqi Gao. EM distillation for one-step diffusion models. In NeurIPS, 2024.
[107] Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. UFOGen: You forward once large scale text-to-image generation via diffusion GANs. In CVPR, 2024.
[108] Zhexin Yao, Qiuming Liu, Jingkang Yang, Yanan Chen, and Zhen Wu. PPUP-GAN: A GAN-based privacy-protectinn method for aerial photography. Future Gener. Comput. Syst., 145:284 2928, 2023.
[109] Teresa Yeo, Andrei Atanov, Harold Lue Benoit, Aleksandr Alekseev, Ruchira Ray, Pooya Es- maeil Akhoondi, and Amir Zamir. Controlled training data generation with diffusion models. Trans. Mach. Learn. Res., 2025.
[110] Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang. Eli Shechtman, Frédo Durand, and Bill Freeman. Improved distribution matching distillation for fast image synthesis. In NeurIPS, 2024.
[111] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Frédo Durand, William T. Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In CVPR, 2024.
[112] Jinsung Yoon, James Jordon, and Mihaela van der Schaar. PATE-GAN: Generating synthetic data with differential privacy guarantees. In ICLR, 2019.
[113] Zebin You, Yong Zhong, Fan Bao, Jiacheng Sun, Chongxuan Li, and Jun Zhu, Diffusion models and semi-supervised learners benefit mutually with few labels. In NewrlPS, 2023.
[114] Jing Zhang, Zewei Ding, Wanqing Li, and Philip Ogunbona. Importance weighted adversarial nets for partial domain adaptation. In CVPR, 2018
[115] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, -ran-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. DataselGAN; Efficient labeled data factory with minimal human effort. In CVPR, 2021.
[116] Bo Zhao and Hakan Bilen. Synthesizing informative training samples with GAN, In NeurIPS Workshop on Synthetic Data for Empowering ML Research, 2022.
16

---

[117] Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song, Noah D. Goodman, and Stefano Ermon. Bias and generalization in deep generative models: An empirical study. NeurIPS, 2018.
[118] Chenyu Zheng, Guoqiang Wu, and Chongxuan Li. Toward understanding generative data augmentation. In NeurIPS, 2023.
[119] Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion probabilistic models and diffusion-based adversarial auto-encoders. In ICLR, 2023.
[120] Zhedong Zheng, Liang Zheng, and Yi Yang. Unlabeled samples generated by GAN improve the person re-identification baseline in vitro. In ICCV, 2017.
[121] Mingyuan Zhou, Huangjie Zheng, Yi Gu, Zhendong Wang, and Hai Huang. Adversarial score identity distillation: Rapidly surpassing the teacher in one step. In ICLR, 2024.
[122] Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In JCML, 2024.
[123] Yongchao Zhou, Hshmat Sahak, and Jimmy Ba. Training on thin air: Improve image classifi- cation with generated data. In ICML Workshop on Data-centric Machine Learning, 2023.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See the abstract and Section 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
The paper provides the full set of assumptions and complete (and correct)
proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Experimental settings, parameters, and environment are described Section to enable reproducibility of results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The experimental data are all from public datasets. Open access to the code provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips)) cc/
pub / gu des / CodeSuuom ss issionn onPol cy) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https httpsscccpppccccccccc.co.cccccca Swwpeiidee/odeeSeebieeea Sub.odeeSSussse... as ss ciiioooolll ooyyl for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
reproducible, they
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Data splits, hyperparameters, and training details are thoroughly described in Section 5
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars reported and the results are discussed comprehensively.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides sufficient information for the computational resources in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp/s///eeerrrr...c/oo//obbbp
Answer: [Yes]
Justification:
The research adheres to the NeurIPS Code of Ethics with ethical issues.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both potential societal impacts and negative societal impacts of the method proposed in Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not involve releasing models or data with high misuse risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All third-party assets used are properly licensed and cited, including datasets and model weights used.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paapersw paperswithoooe com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new asset is introduced.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or human subjects research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No human subjects research involved; IRB approval not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: Large Language Models are used as core component this research.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : : //nuurripss. cc /cc//onfeeeecee///000 2025/ LLM) for what should should not be described.
24

---

A More Mathematical Discussions
A.1 Detailed Deriavations for Eq. 11
By the law of large numbers, the empirical reweighted loss converges to its population expectation under Oc:
Seal
Σ (h))),)) a ((aw))-errr(())(((()))
(16)
(2.8)sSeo
Substituting the definition of the density ratio r(x) Pama (z)a(() gives (())))))drr))((((((()))))))))))) r(x) ((h())))()))))))))))))))())))))) qa y) do dy
ʃ Pdtaul (x) ((h(z). 3) qa ( y) dx dy
qa (z)
= f Pama ga(y x) ((h(2))))) y) do dy.
(17)
The term Paam (x) adjusts for the marginal discrepancy between Paat and ga. ensuring that the contribution of each sample in the expectation reflects its true data likelihood. Hence, Eq. represents the unbiased risk with respect to the true data marginal:
(17
B( ((h(z),)) = (<- [ w)]]
(18)
Assume the generator is sufficiently well-trained based on the strong conditionally pre-trained teacher and is class faithful, i.e., (xr,y) Qc. the inner expectation naturally approximates By-pan(yia (()ppo((()))((((())))))))))) [(((==))(((()).))))))))))))))))) ()) Consequently, the reweighted risk converges to the true expected risk:
[F (n)( (()()),))
(19)
This establishes that the reweighting mechanism based on the discriminator-sstiaane ratio rD (x) cor- rects for the marginal distributional shift between (α(x) and Paata (a)ta()),, thereby aligning optimization on QG with learning on P.
A.2 Detailed Derivation for 15
Let (9,F, be measurable space and assume P, Q admit densities p
dP and dQ dp w.r.t. μ.
Define the total variational distance:
Div(P1@)= = S
dp = gl
f, min(p,g) dp.
(20)
For the bounded loss |4| L,
[F([{()]] Bel Be(
2)
(21)
$ f (
(22)
s L / [d(p-))
(23)
(24)
By Pinsker's inequality
13 for any R.S with densities r,s,
Drv (R((R)) s V (R(S).
(25)
Equivalently, DKL. (R]() > 2D (R. 2
Fix π E (0, 1) and set the mixture M = xP + (1- with density m = +p+ (1- T)g. Define the Whenson-Shannon divergence
Dus (P [)) DKL (M) (1 -)) (
(26)
Applying Pinsker's to each KL. term and then expressing the two TVs through Drv (P,2):
Drv(PI|M) f Ip- m| du = f (11 (1 4) (1 mm) Drv (P,2),
(27)
(Iv( = ʃ 1/- f1a ml mldup= = f =
25

---

Hence,
(P1) 2. ((-))))()MM)))))
(28)
(1- Drv
(29)
= -Tv(1=))DTv((11))
(30)
For the standard Jensen-Shannon divergence, i.., -=
Das(PI 2 {
(31)
Drv(P) s (2D1)
(32)
Combining the results, have [p[[(h)] Ee[((h)] 2LDrv(PI0) 2L (1s2
(33)
Lastly, since any quantity is trivially upper-bounded by its absolute value, ie., x |ac/, relate Eq. 33 to the first summand in Eq. 15 by [E][(hh]] Ee[e(h)] Ep[[(h)] Be Be[((h)] giving
valid upper bound for the non-absolute risk difference.
B Algorithmic Description of GDA-DAR
We summarize the overall training scheme Algorithm| T which illustrates a dynamic, alternating adversarial optimization. For conceptual clarity, the algorithm presents the discriminator update as distinct step, corresponding to an architecture where D,, is standalone network [4] However, as detailed in Section 4.3 alternative formulations often integrate the discriminator into the student's score network (i.e., the "fake" U-Net) to leverage a shared encoder [119 [21]. In such cases, the parameters 7 are subsumed by the score network parameters v/; consequently, the separate discriminator step is omitted, and & is updated jointly to simultaneously maximize the adversarial value function and minimize the score matching error the perturbed generated data.
C Experimental Details
C.1 Details of the Datasets
CIFAR-10 consists 60,000 RGB images, each with resolution 32 pixels. The dataset is evenly distributed across 10 distinct classes, with each class containing 6,000 images. By default, these are split into 5,000 training samples and 1,000 test samples per class, offering a balanced and computationally efficient benchmark for evaluating machine learning algorithms.
I K is dataset comprising 1.28 million images labeled across 1,000 object categories. Each image is annotated with one or more class labels, enabling comprehensive studies of object recognition in diverse real-world scenarios.
C.2 Details of the Metrics
We assess model performance based on accuracy. For ImageNet-IK, which contains 1,000 possible classes, we report both Top-1 and Top-5 accuracies.
We evaluate the generative models based the Fréchet Inception Distance (FID) 30), metric to evaluate the quality of images generated by generative models. This is based on the distance between the feature distributions of real and generated images in the latent space of pretrained Inception network.
widely used
Table 3 details the GPU hours required to generate the augmenting dataset. For CIFAR-IO, computa- tions were performed on a single 80GB-A GPU. For other datasets, we utilized eight 80GB-A100 GPUs, employing the maximum possible batch size that is power of two.
26

---

Algorithm Generative data augmentation in supervised training with GDA-DAR framework.
Input:
Teacher diffusion model eş; real dataset Sp; hyperparametees: learning rates Ooe Cny Oh loss weights A1. A2, Auug. truncation threshold 7;; number of training iterations N Vgea> Netr-
Phase 1: Adversarial Score Distillation (D & A stages) Initialize student generator Gg and auxiliary score network Eú from teacher cΦ. Initialize discriminator Dy- for I...,, do Sample real batch x Sp and noise 2 (O.
// Discriminator Update
Compute adversarial value Vat (11) on real x and synthetic Go(z) by Eq. 13 Update discriminator: !i - 1) - on Vn (-Vat((1))
// Distillation
Construct perturbed synthetic samples: y): anGa(2) ore where 2,6,-N(0,
Compute fake score loss: (ffaee score) (v) F=z,e.. Co (vr,
Update score network: 1b w/- ag V (take (16)
Compute score distillation loss L (() (Eq.
Compute generator adversarial loss L (0) Ezllog D,
Update generator: 0 0 Vg (SD) (0) + /2 (0)). end for
Phase 2: Reweighted Classifier Training (R stage)
Initialize discriminative hypothesis (classifier) h. Freeze parameters 0 and rj- for i = 1, 1....,,, Nar do // Real Data Path
Sample real batch (z., Sp.
Compute real loss C(real) = 1 Σラ ((h(z)),)))
// Synthetic Augmentation Path
Let synthetic batch size Bung >mug B|.
Sample noise batch (= (2e, Bu N(0, (, I) Cat()).
Bus Go ( Put). Generate synthetic batch (æ.) j-1
// Compute Importance Weights for synthetic batch
for = I,.... Baug do Compute density ratio ry Truncate weight F3- min(r,,,). end for if self norm then
ry= = (r(0t.)/(11 (1-
Self-normalize: fj + P3/ Exe Fk for j 1,.... Bang
end if
Compute reweighted loss (rrwweight)
ESer f2 ((h((t.))) N...).
// Combined ppdate of the hypothesis
Compute total loss Liotal = L(real) (rewweighn)
Update hypothesis: h- h 04 Erotal- end for
Output: Trained hypothesis h.
C.3 Details of the Implementation
For CIFAR-10 10, we set 1.2 for SiDA and train using the Adam optimizer with a learning rate of x 10-5 CTM is trained with a batch size of 128 for 256 steps per batch, using a student learning rate of 3 x 10-4 and a discriminator learning rate of 2 x 10 3 For ImageNet- we distill EDM2-XXI. using SiDA across A100-80GB GPUs with a 1.0, per-GPU batch size 64, and gradient accumulation every 128 iterations, using Adam with learning rate 5 x 10 CIFAR-10
27

---

models are trained for 300 epochs with batch size of 128 using momentum SGD (learning rate 0.1). Hyperparameters A and X2 for adversarial alignment follow prior work.
For ImageNet- KK ResNet-50 is trained for 90 epochs with batch size 4096 and initial learning rate 1.6, while ViT-S/16 is trained for 300 epochs with batch size 1024, using AdamW (58] and initial learning rate 3 x 10-3. Self-normalization is applied with
to obtain r(x) for reweighting. Diffusion outputs are generated at 32 32 resolution for CIFAR-IO and 5 512 for ImageNet, with the latter downsampled to 224 x 224 before classification to match baseline protocols.
C.3.1 Classifier Training
The hyperparameters used for training the hypotheses. i.e. the classification models, on the CIFAR-10 and ImageNet-IK datasets detailed in Tables [ and A2 respectively. Models are trained using Stochastic Gradient Descent (SGD) optimizer across 90 epochs. The learning rate is programmed to initially ramp up from 0.0 0.4 over the first five epochs, followed by decrements of 0.1 at the 30th, 60th, and 80th epochs.
For CIFAR-10 10 training, utilize standard augmentation techniques including padding each image by pixels on all sides, then performing a random 32 x 32 crop from either the padded image or its horizontal flip. following the methods described in [28]. For ImageNet- 1K, the models undergo training using a 224 x 224 random crop, applied with bilinear interpolation, from either the original image its horizontal flip.
The detailed recipes of the hypothesis training are reported in Table A1 and Table A2 respectively.
Table A1: Training recipes for ResNet-18 and VGG-16 on CIFAR-10.
Model
R8
VGG-16
Batch size
128
128
Epochs
300
300
Optimizer Learning rate
Momentum SGD
Momentum SGD
0.1
0.1
LR scheduler
CosineAmea
CosineAnne
Nesterov
True
True
Weight decay
5e-4
5e-4
Table A2: Training recipes for ResNet-50 and ViT-S/16 on ImageNet- KK.
Model
ResNet-50
VIT-S/16
Batch size
4096
1024
Epochs
130)
300
Optimizer
Momentum SGD
AdamW W
Learning rate
1.6
0,003
LR scheduler
Cosineannealngg.R
CosineAmmellnggL.R
Weight decay
lc-4
0.3
Warmup epochs
5
30
Label smoothing
0
0.11
Mixup probability
0
0.2
Cutmix
0
1.0
C.3.2 Generative models training
For the base diffusion model, we adopt the public weight for EDM Diffusion on CIFAR-I0 and EDM2 on ImageNet-1 KK for their state-of-the-art performance.
For experiments on the CIFAR-10 dataset (see Tables A we set 1.2 for SiDA and train using the Adam optimizer with learning rate of le-5. The batch size is set to 256, and gradient accumulation performed every iteration (i.e., aceumulation round = 1). We do not use mixed precision training (fp16) for SiDA on 10. Dropout and data augmentation settings follow those used in EDM for CIFAR-10. All other SiDA-specific hyperparameters are kept consistent with the original implementation described in 121.
28

---

For CTM, CTM is trained with batch size 128 for 256 steps per batch, using student learning rate of 3 10- and discriminator learning rate of 10-3 Mixed precision training (fpl6) was employed to match the original setup. The exponential moving average (EMA) decay rate is 0.999. Other CTM-specific parameters adhere to the settings reported in [3].
On the ImageNet- IK dataset, we distilled EDM2-XXL with batch size of 2048 and gradient accumulation performed every 128 iterations. train using the Adam optimizer with learning rate of 5e-5. Mixed precision training (fp16) was used for SiDA on this dataset. Dropout parameters follow those of EDM2 on ImageNet-Il 1K. and all other SiDA SiDA-specific hyperparameters remain consistent with those in the original work [21]
C.3.3 Training Procedure
In our experiments on the CIFAR-IO dataset (shown in Tables [ndd 4), we investigated two distinct strategies for generating synthetic data: the static approach and the dynamic approach. For the static approach, we created 50,000 synthetic samples, equivalent in size to the original dataset, which remained consistent throughout the entire training process. In contrast, the dynamic approach involved generating a new set of 50,000 synthetic samples the start of each epoch, ensuring that the synthetic dataset used for training was entirely refreshed every epoch. For the experiments on the ImageNet- 1K dataset, we generated 1.28 million synthetic samples for training, which is equivalent to the size of the original dataset. Given the large scale of ImageNet- 1K, regenerating the synthetic dataset at each epoch would be computationally expensive and consuminn. Therefore, we opted for the static approach.
C.3.4 Implementation Details and Notes
While our framework comprises three stages, its implementation manageable due its modular design. The (D), (A), and (R) stages are sequential and decoupled. As noted in relevant work [22 43). stability is observed in the adversarial distillation. This can be attributed the design of initializing the student with a pre-trained teacher, which bypasses the unstable early stages of typical GAN training. Hence, for the student generator and discriminator training phase, a single, fixed set of distillation hyperparameters was used across all datasets. For the final classifier training (R) stage, we simply follow standard, existing recipes from the literature. addition.
The primary tuning consideration i) balancing the loss weights A and A2 remain at comparable magnitude; and ii) considering truncation threshold y and the use of self-normalization achieve favorable blasinariance trade-off in the estimated importance ratios.
Additionally, a key in- in-process indicator of successful D+A phase is the student generator's 's own performance. We (and others [12 43 110 59) observe that the student's FID score remains stable and often improves beyond that of the original teacher. This provides direct evidence that the discriminator providing high-quality gradients and has successfully captured the real-versus- generated distribution gap. We recommend users monitor the student's generation quality as a primary check for a healthy run.
D Additional Experiment
D.1 Ablation Study of Variance Reduction Techniques
We ablate on the two variance-reducing techniques used, namely self self-normalization and trucation, with respect to the hyperparameter that controls the upper bound of y in Table|A3] Our observations indicate that removing self- normalization generally leads to a decrease in classification performance. This decrease particularly significant on ImageNet- K, likely because weight variance can be greater with such massive dataset. Furthermore, we observe that 1 appears to be the optimal upper bound for truncation. At this value, synthetic image contributes equally to a real image the loss computation and subsequent optimization.
29

---

Table A3: Classification accuracies (top-1) under different weight variance reduction techniques for additive, static augmentation using EDM/EDM2- SiD distillation.
CIFAR-10
ImageNer-Il
IK
Self-norm
ResNet-18
VGG-16
ResNet-50
ViT-S/16
X
95.30
94.08
77.09
80.05
96.15
94.60
77.75
80.90
X
95.72
94.10
77.23
80.12
0.5
95.70
94.37
77.73
80.47
96.21
94.64
78.03
81.17
95.86
94.40
77.64
80.68
In addition, we conducted an empirical study reporting the sample variance of the weights for 100,000 generated ImageNet samples. As shown in Table A4 both truncation and self-normalization progressively reduce the variance.
Table A4: Sample variance of importance weights r(x) for 100,000 generated ImageNet samples, under different truncation (7) and nommaliomlintion (SN) schemes.
Truncation
Variance of r(a)) (no SN)
Variance after SN
(no clip)
0.42
2.0
0.37
0.12
1.0
0.18
0.05
0.5
0.07
0.02
D.2 Discriminator Noising Scheme
When employing the discriminator the encoder for the student generator's backbone, it's necessary to transform the clean input, Zelean- into a noisy version, Tnoisy. This is achieved by corrupting Zetaa specific timestep Lg. The choice of this noising
with predetermined level of noise corresponding scheme, specifically the selection of lo. can influence the features learned by the discriminator- encoder and subsequently the performance of the student generator. Please note that the original p(t) used for the computation of [ (lake score) (1)).
We investigated several approaches for selecting tu. as detailed in Table A5 We observe that the uniform sampling strategy is the best among all ablated ones with to 0.I7 having comparable performance on both datasets. The approach treating the shared encoder as the discriminator offers trade-off: it obviates the need to tune the noising scheme, while avoiding the potential complexities and additional computational overhead associated with employing separate, dedicated discriminator network.
Table AS: Classification accuracies (top- under different discriminator noising schemes for static additive augmentation using EDM/EDM2-SID distillation.
CIFAR-10
ImageNet-IP IK
ResNet-18
VGG-16
ResNet-50
ViT-S/16
(0.057
96.08
94.48
77.87
81.02
8(0.17)
96.18
94.66
77.89
81.10
8(0.57)
95.33
94.07
77.23
80.51
(00 0.57)
96.21
94.64
78.03
81.17
D.3 Noisy and Imbalanced Setting
While initial goal was to focus challenges like distribution mismatch and sampling in GDA, we further investigated how our approach would be suitable for these real-world scenarios such on imbalanced and noisy data.
Class Imbalance We conducted experiments on CIFAR-I0-LT 16 with an imbalance factor of 10 and 100 using the me training protocol as in the balanced- setting the dynamic scheme, with
30

---

ResNet- 18 as the backbone in Tables A7 A We also included long-tate long-tailed tailed-based method, SURE [55], for comparison. The evaluation tracks the progressive impact each stage in our DAR-GDA pipeline (+D), (+A), and Reweighting (+R) with SiD -2istlldd EDM [39] diffusion model. This performance likely stems from our method's ability to leverage the strong generative priors of the diffusion model teacher to populate tail classes, different paradigm from methods like SURE, which are designed learn directly from the imbalanced data.
Table A6: Results on CIFAR-10-LI with Imbalance Factor 10.
Method
Augment
Substitute
Baseline
86.75
86.75
SURE [SS]
95.15
95.15
EDM-SID (+D)
94.88
94.47
EDM-SID (+D+A)
95.53
94.98
EDM-SID (+D+A+R)
95.78
95.02
Table A7: Results CIFAR-IO-LT with Imbalance Factor 100.
Method
Augment
Substitute
Baseline SURE
69.76
69.76
86.75
86.75
EDM-SID (+D)
91.45
91.02
EDM-SID (+D+A)
91.84
91.35
EDM-SID (+D+A+R)
92.01
91.64
Noisy Label We also conducted an experiment on the CIFAR-106 R1-N dataset [103] using the "aggre" noise labels to simulate real-world scenario with untrusted data. We used ResNet-18 backbone with the augmenting data protocol. We compare result with noisy-label learning method, ProMix 104 in Table AS
Table A8: Results CIFAR-IO-N with ResNet-18.
Method
Static
Dynamic
Baseline
87.77
87.77
ProMix 104
97.65
97.65
EDM-SID (+D)
93.48
95.79
EDM-SID (+D+A)
93.80
96,26
EDM-SID (+D+A+R)
93.81
96.30
Our method contains no explicit mechanism for label correction; the discriminator is designed to down-weight visually atypical samples, not to detect label noise. Consequently, while the strong generative prior provides a robust foundation, the performance does not surpass that of specialist methods like ProMix, which are explicitly architected to handle label noise.
D.4 Comparisons with Non-generative Data Augmentation Methods
Our primary work focused challenges within the GDA paradigm. To provide a broader context, we conducted additional experiments comparing our DAR-GDA framework against two widely-used non-generative data augmentation basclines: AutoAugment [4] and RandAugment [15].
31

---

Table A9: Comparison with non con-generative augmentation on CIFAR-10 with ResNet-18.
Method
Static
Dynamic
Real-only
95.00
95.00
Autonugment
95.85 15.85
95.85
RandAugment [1
95.79
95.79
EDM-SID (+D)
95.48
96.29
EDM-SID (+D+A)
95.84
96.40
EDM-SID (+D+A+R)
96.21
96.73
EDM-SID (+D+A+R) + RandAugment
96.48
96.97
Table A10: Comparison with non-generative augmentation ImageNet- 1K with ResNet-50.
Method
Top-1
Real-only
76.37
AutoAugment
77.62
RandAugment 15
77.65
EDM-SID (+D)
77.15
EDM-SID (+D+A)
77.89
EDM-SID (+D+A+R)
78.03
EDM-SDD SID (+D+A+R)+ RandAugment
78.29
We notice that our full pipeline (DAR-GDA +D+A+R) outperforms both AutoAugment and RandAug- ment. Furthermore, we highlight that our generative approach complementary to these traditional, non-generative data augmentation techniques. These transforms can be applied synthetic images just as they are to real ones, further diversifying the training set. To demonstrate this, we applied RandAugment top of our generated data, which yields further improvement.
D.5 On FFHQ Dataset
To further validate our framework's robustness, we conducted experiments on gender classification task using the FFHQ 64x64 dataset [41]. Since the EDM teacher and our student generator are unconditional, we first created a high-quality labeled dataset by using an external classifier of ResNet- 50 pretrained on Celeb-A to generate raw predictions, which were then human verifeee We then applied our "static" scheme with results shown in Table A11 We observe monotonic performance gains as each stage of the DAR pipeline applied.
Table A A11: Gender classification accuracy FFHQ 64x64. The baseline was trained real data with human-verified pseudo-labels.
Method
ResNet-18
VGG-16
Real-only EDM-SID (+D)
94.08
93.10
94.20
93.25
EDM-SID (+D+A)
94.63
93.62
EDM-SID (+D+A+R)
94.87
93.70
E
Computational Complexity Analysis
We perform formal analysis of the computational complexity for our framework. define the following computational costs:
T:
The number of denoising steps in the teacher sampler, e.g., T 1000.
ey:
The wall-clock cost one forward pass of the diffusion model backbone (assumed the same for the teacher and the one-step student).
32

---

N:
The total number of synthetic instances to be generated.
Caisiun
The one-time, upfront cost of distilling the teacher into the student generator.
pretrain
The cost of pre-training the teacher model, which is shared/sunk for both methods and cancels out in the comparison.
The core of our efficiency gain comes from reducing the per-sample generation cost from the O complexity of iterative samplers, like DDPM [1] or DDIM [2]. to an (1)) complexity. Table A12 contrasts these approaches.
Table A12: Comparison of sampling time complexity and theoretical speed-up.
Sampler
# Reverse Steps
Asymptotic Cost
Theoretical Speed-up (vs, 1000-step DDPM)
DDPM
1000
O(T)
Baseline (Ix)
DDIM (fast)
50
O(T)
20x
One-step distilled
O(1)
With these variables, we can compare the total cost of generating N samples using naive iterative approach versus our DAR-GDA framework. The costs are summarized in Table A13
Table A13: Computational cost comparison for generating N synthetic samples.
Pipeline
Total Cost Equation
Naive Diffusion GDA Craive N-T-cg DAR-GDA
CDAR Cassill N ccy
Our framework (DAR-GDA) is faster than the naive approach whenever ODAR < Cnavee which implies:
Caisill Ney NTez
Cáisinl Ney(T-
(34)
We can estimate the one-time distillation cost as Caisill = P. N ks- cy, where P is the number of data passes (epochs) during distillation, N is the number of data samples in the training set, and & the number of model evaluations per sample per pass.
Empirically, recent works [ 43 suggest that one-step distillation converges in most P s 150 passes, with & = 3, i.e., teacher, fake, and student score network forwards. Assuming a teacher with T 1000 steps, the ratio of distillation cost savings is:
Caisill
PNkey
Pk 150 3
0.45 <
(35)
(T (T-(T-1) T-1
999
Thus, the inequality is comfortably satisfied. Furthermore. if synthetic data is dynamically refreshed each epoch. i.e., the "dynamic" scheme, the savings compound. In that case, the effective N becomes N. E. (where E is the number of classifier training epochs), making the efficiency gain of DAR-GDA even more significant.
33


Reviews:
[{"review_id": "RydmKF0201", "reviewer": "Reviewer_W9GD", "summary": "The authors introduce DAR-GDA, a three-stage framework for generative data augmentation using diffusion models that combines distillation, adversarial alignment, and importance reweighting to generate high-quality and unbiased synthetic training data efficiently. The proposed method compresses slow diffusion models into fast one-step generators while aligning them with the real data distribution via adversarial training. Finally, the resulting discriminator is used to reweight synthetic samples for accurate downstream learning.", "questions": "1. Do you anticipate any limitations of the proposed method in other domains such as healthcare or in extensions to other tasks such as segmentation, regression, etc.?\n2. Is there a reason why non-generative augmentation techniques [1,2] were not used as baselines?\n3. Is the proposed method robust to label and/or input noise? How about when there is significant class imbalance?\n\n[1] Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V. and Le, Q.V., 2018. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501.\n[2] Cubuk, E.D., Zoph, B., Shlens, J. and Le, Q.V., 2020. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (pp. 702-703).\n\nAddressing these questions effectively may help raise my score.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The authors tackle both issues with generative data augmentation: high sampling time and the divergence between real and synthetic distributions is unknown, so the proposed method is well-motivated.\n- As seen in the results, distillation, adversarial alignment, and reweighting seem to mesh quite nicely to perform generative data augmentation.\n- Strong performance with both the CIFAR-10 and ImageNet-1K datasets.\n- A 100x reduction in per-image costs while preserving FID is greatly beneficial.\n\nWeaknesses:\n- Only 2 general imaging datasets used - unclear if the method would be as useful in different domains.\n- Unclear whether the method would work under noisy or imbalanced settings.\n- Non-generative augmentation methods are not included as baselines.", "was_revised": true, "final_justification": "The authors have sufficiently addressed my concerns and I will raise my score accordingly from a 4 to a 5.\n\nThe authors have bolded the wrong number in the Substitute column in the \"CIFAR-10-LT with imbalance factor = 10\" table. They should also bold the best results in the table in the Noisy section in the rebuttal. Also, the authors use \"Addictive\" instead of \"Additive\" three times in the paper. These errors should all be fixed in the final version.", "rebuttal": "Thank you for your time and for providing a detailed and insightful review! We are grateful for your positive feedback and your appreciation of our method’s **motivation**, **proposed components mesh well,** and its achievement of **strong performance with significant efficiency gains**.\n\nWe also appreciate the constructive criticisms and questions you have raised. They have helped us identify key areas for clarification and improvement. Below, we address each of your points and outline the specific changes we will make in the revised manuscript.\n\n**[W1, Q1] Dataset Used and Extension to Other Domains**\n\nThank you for raising this important point regarding the generalizability of our method across domains and tasks. We chose these two benchmark datasets to rigorously validate the DAR-GDA framework under well-established, challenging settings. This aligns with prior work in generative data augmentation and allowed us to ensure fair standardized evaluation and scalability analysis. Given the complexity, we believe that a focused, controlled evaluation was necessary to first establish its stability and effectiveness. Additionally, as our core expertise lies in machine learning, we prioritized domains where we could ensure a thorough and responsible study.\n\nThat said, we would like to emphasize that the core principles of our framework are fundamentally **domain-agnostic** and do see promise in applications in other domains. Unlike traditional augmentation methods that rely on hand-crafted domain-specific transformations, our method only assumes access to a pretrained diffusion teacher capable of producing score estimates. This teacher is treated as a black box, which enables a good adaptation of our framework to other data modalities. For example, in the medical domain, where data is often scarce and governed by strict privacy constraints, DAR-GDA can be used to generate high-fidelity synthetic X-ray or MRI scans to augment limited datasets. Additionally, institutions could share distilled student model weights instead of raw data, enabling *pseudo data sharing* without violating confidentiality agreements. Beyond imaging, DAR-GDA is also applicable to medical use, such as drug molecular modeling and discovery, where diffusion models have recently achieved impressive results [1,2]. In these cases, adaptation involves replacing the U-Net backbone with domain-specific architectures while keeping the core framework of DAR unchanged.\n\nWe also acknowledge that adapting DAR-GDA to other tasks and modalities introduces new research challenges, exactly as you mentioned—particularly around the nature and structure of the conditional signal required by the teacher model. For instance:\n- Regression Tasks (continuous label): \nExtending to regression tasks, especially with multivariate or high-dimensional targets, poses difficulties in conditional sampling. While univariate conditioning is straightforward, complex regression settings require more expressive conditional teacher models. That said, this challenge lies in the construction of the teacher—not the DAR-GDA framework itself, which is agnostic to the conditioning structure.\n- Dense Prediction Tasks (e.g., Segmentation): For tasks like segmentation, where dense labels are needed for the generated instances, obtaining the clean labels required to build a high-quality conditional teacher model can be a bottleneck. In such cases, the framework could be effectively integrated with techniques like pseudo-labeling with powerful models like SAM or other forms of weak supervision to create the necessary training signals for the teacher. Often, these tasks face scarcity in labeled data, which makes generative data augmentation especially promising in this context.\n\nIn addition, we observed strong results in challenging in-domain scenarios (noisy, long-tail) *(See W2, Q3)*, providing some indirect evidence of the framework's robustness for generalization.\n\n**Action:** In our revised paper, we will expand the Discussion section to include this nuanced analysis of domain and task generalization. We will outline the anticipated challenges and opportunities for adapting DAR-GDA to other settings and tasks.\n\n**[W2, Q3] On Noisy and Imbalance Setting**\n\nThis is an excellent point. While our initial goal was to focus on challenges like distribution mismatch and sampling cost in GDA, we are happy to investigate how our approach would be suitable for these real-world scenarios.\n\n**A. Class Imbalance:**\n\nWe conducted experiments on CIFAR-10-LT with an imbalance factor of 10 and 100 (“significant class imbalance”) using the same training protocol as in the balanced-data setting the dynamic scheme, with ResNet-18 as the backbone. We also included a strong long-tailed learning baseline, SURE [3], for comparison. The evaluation tracks the progressive impact of each stage in our DAR-GDA pipeline (+D), (+A), and Reweighting (+R). Here is a summary of the results:\n\n**CIFAR-10-LT with imbalance factor = 10:**\n\n|Method|Augment|Substitute|\n|---|---|---|\n|Baseline|86.75|86.75|\n|SURE [3] |95.15|95.15|\n|EDM-SID (+D)|94.88|94.47|\n|EDM-SID (+D+A)|95.53| 94.98|\n|EDM-SID (+D+A+R)|**95.78**| **95.02**|\n\n**CIFAR-10-LT with imbalance factor = 100:**\n\n|Method|Augment|Substitute|\n|---|---|---|\n|Baseline|69.76|69.76|\n|SURE [3]|86.75|86.75|\n|EDM-SID (+D)|91.45| 91.02|\n|EDM-SID (+D+A)|91.84| 91.35|\n|EDM-SID (+D+A+R)| **92.01**| **91.64**|\n\nWe found that our proposed method still progressively improves performance. It gives performance close to the current SOTA under the static setting. Furthermore, with dynamic regeneration (fresh synthetic images each epoch), DAR–GDA surpasses the long‑tail SOTA without any class‑specific tuning.\n\nWe also note that our method's potential can be further unlocked by combining it with generators specifically designed for class imbalance, such as [4], which we defer to further research.\n\n**B. Noisy:**\n\nWe also had a run on CIFAR-10-N with “aggre” label on ResNet-18 with the current training protocol with a comparison to a SOTA method in noisy label, ProMix [5].\n\n|Method|Static|Dynamic|\n|---|---|---|\n|Baseline|87.77|87.77|\n|ProMix [5]|97.65|97.65|\n|EDM-SID (+D)|93.48|95.79|\n|EDM-SID (+D+A)|93.80 |96.26|\n|EDM-SID (+D+A+R)|93.81|96.30|\n\nOur discriminator only down‑weights visually atypical images; it does **not** correct corrupted labels. Still, our method improves, though not as much compared to the SOTA method, over the baseline in the static scheme, and still achieves a performance close to the SOTA method with its “dynamic” scheme. Nevertheless, it shows that further research is warranted in such real-world scenarios, where the application of noise-handling techniques (e.g., co‑training, small‑loss selection) is a promising direction.\n\n**Action**: In the revised manuscript, we will incorporate a new appendix section that outlines the experimental results for these scenarios and discusses the potential of fusing our method with specialized long-tail and noise-robust discriminative methods.\n\n**[W3, Q2] non-generative augmentation methods**\n\nThank you for bringing this important point up! Our initial work focused on the specific challenges within generative data augmentation (GDA), so we prioritized comparisons with other GDA methods. \n\nFollowing your suggestion, we have run experiments comparing our method against *AutoAugment* [6] and *RandAugment*[7], and present the results below:\n\n**CIFAR-10, ResNet-18**\n|Method|Static|Dynamic|\n|---|---|---|\n|Real‑only|95.00|95.00|\n|AutoAugment|95.85|95.85|\n|RandAugment|95.79|95.79|\n|EDM‑SID (+D)|95.48|96.29|\n|EDM‑SID (+D+A)|95.84|96.40|\n|EDM‑SID (+D+A+R)|**96.21**|**96.73** |\n|EDM‑SID (+D+A+R) + RandAugment|**96.48** |**96.97**|\n\n**ImageNet-1k, ResNet-50**\n|Method| Acc|\n|---|---|\n|Real‑only|76.37|\n|AutoAugment|77.62|\n|RandAugment|77.65|\n|EDM‑SID (+D)|77.15|\n|EDM‑SID (+D+A)| 77.89|\n|EDM‑SID (+D+A+R)|**78.03**|\n|EDM‑SID (+D+A+R) + RandAugment |**78.29**|\n\nWe observe that 1) our method, on its own, outperforms both non-generative data augmentation methods, AutoAugment and RandAugment, on the standard benchmarks.\n\nWe would also like to highlight that our generative approach is **complementary** to traditional augmentation techniques. Standard transforms can be applied on top of synthetic images to give even more diverse training instances to the classifiers. When we apply RandAugment on top of our method (\"EDM‑SID (+D+A+R) + RandAugment\"), performance is further boosted.\n\nThank you again for this valuable suggestion. And, we will include them in the table, which positions our paper better, and we will feature the complementary nature of generative and traditional data augmentation in a section in the appendix.\n\n**[formating]**\n\nThank you for pointing out the spacing concern. We will ensure the revision is properly formatted with standard spacing, strictly adhering to the style file. Having a presentable manuscript will be our top priority for the revision. Additionally, we plan to use the one extra page allowed for the cameraready version to incorporate improvements in visual appeal, organization, and formatting without compromising content.\n\n----\n\nThank you once again for your constructive review, which has pushed the work to strengthen its scope and evaluation! We hope this new evidence addresses your concerns.\n\n**References**:\n\n\n[1] Cornet et al., *Molecular Diffusion Models*, NeurIPS’24\n\n[2] Hoogeboom et al., *Equivariant Diffusion for Molecule Generation in 3D*, ICML'22.\n\n[3] Li et al., *SURE: SUrvey REcipes for building reliable and robust deep networks*, CVPR'24\n\n[4] Qin et al., *Class-Balancing Diffusion Models*, CVPR'23\n\n[5] Xiao et al., *Combating Label Noise via Maximizing Clean Sample Utility*, IJCAI'23\n\n[6] Cubuk et al., *Autoaugment: Learning augmentation policies from data*, CVPR'19\n\n[7] Cubuk et al., *RandAugment: Practical automated data augmentation with a reduced search space*, NeurIPS'20"}, {"review_id": "JZ0qR0X5Jb", "reviewer": "Reviewer_Fxjd", "summary": "Authors propose a three-stage framework (named as DAR-GDA) to make generative data augmentation (GDA) with diffusion models both efficient and unbiased. It combines Distillation (to compress slow teacher models into fast one-step generators), Adversarial alignment (to match the synthetic data distribution to the real one), and Reweighting (to correct for residual biases using density ratio estimates). The method significantly improves classification accuracy on CIFAR-10 and ImageNet-1K, even outperforming real-data augmentation in some settings. It also achieves GAN-level speed with diffusion-level fidelity. It showcases how real data can still not be substituted by synthetic data for competetive performance. While the approach is promising, their deployment is currently limited by  complexity of three stage implementation. Overall, DAR-GDA is an efficient, accurate, practical but complex-for-implementation method for generative data augmentation.", "questions": "The questions (1-5) are clearly mentioned in the weakness section above, and addressing them will affect my evaluation score. \n\n1-3 are major concerns.\n\n4-5 are minor concerns.\n\nApart from technical aspect, the writing in the paper could benefit from greater clarity. Some sections (like section-4) are dense and difficult to follow.", "limitations": "The limitations apart from the ones mentioned in the paper, are discussed in the weakness section. The major limitation are due to 1. Multiple interdependent components in the implementation and tuning complexity, and 2. dependence of Reweighting on a well-trained discriminator; where errors can mislead training.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths** :\n1.  The method distills a slow diffusion model into a one-step student generator, reducing image generation time by over 100×, making GDA feasible for large-scale tasks.\n\n2.  It introduces a better, principled importance __reweighting__ mechanism using a learned discriminator to correct for distribution mismatch between real and synthetic data, reducing bias in model training.\n\n3. By integrating score distillation from diffusion models with adversarial training from GANs, the proposed method maintains **high sample quality** while also learning a discriminator for density ratio estimation r(x).\n\n4. The results show a **decent improvement** in classification performance in Table 1, when synthetic data is added to the real training data across different model variants on  CIFAR-10 and ImageNet. Their method is also theoretically grounded as an added bonus.\n\n**Weakness**\n\n1. Lack of consistency in Paper Writing: \n\nManuscript is very difficult to read due to sentence level mistakes especially in section 4 -- Line 141: “*Empirical risk minimization (ERM) [82, 49] is a common xxxxx seeks a hypothesis that*”. Line 148: “from which we can draw an augmenting the sample set of synthetic samples”.\n\nLine 223: “is a distribution weighting timestep t\". Line 300: “On CIFAR-10, On CIFAR-10 we”\n\n2. Several tuning parameters for multi-component optimization: \n\nThe proposed multi-component optimization introduces complexity, both in implementation and training dynamics. This seems like a challenge for an end user for implementation sake, making it fragile and difficult to debug. How do authors address this challenge ?\n\n3. Dependence on the Discriminator Quality\n\nThe paper discusses that \"*Empirically, the student often attains lower FID scores than non-adversarially distilled counterparts or even the teacher model in some scenarios*\". This means that performance improvements are not consistent across all cases—the method may be sensitive to how well the adversarial discriminator is trained. How do authors analyse the robustness/accuracy of the trained discriminator to ensure this quality check?\n\n4. High variance from r(x) ? [Minor since it is a known issue]\n\nComputing per-sample importance weights r(x) introduces variance into the training objective, especially when there's a large gap between the real and generated data distributions. If r(x) becomes too large (which happens when q_{G} (x) << p_{data} (x) ) , training can become unstable or biased toward a few high-weighted samples. The paper does mention truncation and self-normalization to control this variance, but doesn't dig into discussion if this happens with the current setup?\n\n5. Clarification [Minor]: Even when the student generator is fast, it still depends on a pretrained diffusion teacher. Training that teacher model can be computationally expensive, especially on large datasets like ImageNet-1K. Does the paper acknowledge that it can still impose a significant compute burden before DAR-GDA becomes efficient ?", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive and detailed reviews! We are very grateful for your positive feedback and your recognition of our work’s key strength, including **significant improvement over efficiency and empirical performance**, the **principled reweighting mechanism**, and the **novel integration**.\n\nYour critique is exceptionally helpful, and we agree with your assessment, particularly regarding the issues with clarity and the perceived complexity of the method. Below, we address each of your concerns point-by-point and detail the specific revisions we will make to the manuscript to resolve them.\n\n**[W1] On Paper Writing and Formatting**\n\nWe take this piece of feedback very seriously.\n\n**Action:** We are conducting a thorough professional copy-edit and proofread of the entire manuscript. We will correct all errors and rewrite unclear sections (especially Section 4) to enhance clarity. We will ensure that the methods are easily understandable by having non-author colleagues read through the revision and confirm their accessibility. Additionally, we plan to make full use of the one extra page allowed for the camera-ready version to address the issue of *packed and visually cramped content*. This includes adjusting to sufficient spacing and fixing the locations of figures and tables.  Delivering a well-written and visually appealing paper is **our top priority** for the revision. \n\n**[W2] On Tuning Parameters for Multi-component Optimization**\n\nThis is a crucial practical concern, and we thank you for raising it. While the framework has multiple components, our design and the corresponding code implementation have made it more manageable than it appears through modularization.\n\nWe would like to highlight that the three proposed stages are sequential, not a single joint optimization. The optimization is decoupled, and each stage can be tuned and validated largely independently, which is not as complex as an end-to-end optimization. \n\nAlso, we observe the robustness in the adversarial distillation. This arises from our design of initializing the student with a pre-trained teacher diffusion model, which bypasses the early unstable stage of training to yield stability. For our implemented method, a single, fixed set of hyperparameters for distillation has been used for all datasets, including CIFAR-10, ImageNet-1K, and FFHQ.\nFor the final classifier training, we simply follow standard, existing recipes from the literature without introducing new hyperparameters.\n\n**Action**: To make our method more accessible and to address your concerns about implementation and debugging for end users, we will:\n\n1. Add a new discussion in the appendix on hyperparameters to provide a guide on the key parameters\n2. Release our full, documented source code upon cameraready, with modularized stages that allow for easy usage and debugging.\n\n**[W3] On Discriminator Quality**\n\nThis is a critical point, and we agree that the success of our reweighting stage hinges on a sufficiently accurate discriminator as a 100%-accurate discriminator is unattainable. Our confidence in this component stems from two factors: (1) the inherent stability of pre-initialized adversarial training, which produces a robust discriminator, and (2) evidence to validate the discriminator’s effectiveness.\n- (1) Our discriminator is not fragile because our framework is explicitly designed to avoid common GAN pathologies. Unlike traditional GANs that train with generators producing random noise at the start, our adversarial process begins with a generator initialized from a high-quality, pre-trained diffusion model teacher. This is a crucial distinction. It allows us to bypass the notoriously unstable early phases of GAN training, where issues like support mismatch is one of the key factors for poor or uncalibrated discriminators [1,2]. By starting from a strong baseline with already a considerable distribution overlap, our discriminator is tasked with a more stable and well-posed refinement problem, making it inherently more robust.\n- (2) We use the student generator's performance as a powerful, real-time indicator of the discriminator's health. The logic is straightforward: a student model distilled into a single step should, by principle, be of lower quality than its multi-step teacher due to information loss. However, we observe the opposite. During adversarial alignment, the student's performance, as measured by FID, not only improves from its initial distilled state but consistently surpasses the original teacher model. This observation is consistent with current adversarial distillation methods [3-5]. Such an outcome is only possible if the discriminator is providing high-quality and informative gradients. It is direct proof that the discriminator is successfully capturing the difference in real and generator distributions and guiding the generator to a state of higher fidelity than the teacher it was distilled from. \n\nThank you for also pointing out the confusing sentence on Line 227. It was poorly phrased.\n\n**Action:**\nWe will rephrase the sentence on Line 227 to \"Empirically, adversarial distillation has shown stable behaviour, with one-step students generally matching—and frequently reported to surpass—their teachers’ FID.\"\n\n**[W4] On High Variance from r(x)**\n\nThank you for this question and the opportunity for us to clarify this. While variance from importance weighting is indeed a known challenge, our combined approach was highly effective in our setup.\n\nThe raw importance weights r(x), derived from the discriminator output, can have a long right tail. It makes sense to truncate extremely large values, and intuitively, this limits the maximum contribution of any single synthetic sample relative to a real one. While this introduces a bias-variance tradeoff, it effectively curbs the influence of extreme outliers.\n\nWe can also normalize the weights by self-normalization (SN), such that weights are scaled to a fixed sum, where we empirically find improved downstream performance with SN used.\n\nWe conducted an empirical study reporting the sample variance of the weights r(x) for 100,000 generated ImageNet samples.\n\n| Truncation γ | Variance of r(x) (no SN) | Variance after self-norm (SN) |\n| --- | --- | --- |\n| ∞ (no clip) | **1.21** | 0.42 |\n| 2.0 | 0.37 | 0.12 |\n| 1.0 | 0.18 | 0.05 |\n| 0.5 | 0.07 | **0.02** |\n\nThis result confirms that our combination of truncation and self-normalization provides a robust and effective scheme for stabilizing the reweighting stage, and this control is a part of the stable, consistent performance gains with reweighting.\n\n**Action:**\n\nWe agree with you that it is a known issue. To provide more details, we will add this ablation study and a detailed discussion on our variance mitigation strategy to the appendix of our revised paper to provide readers with a clearer understanding of the effect of the variance control. \n\n**[W5] On Clarification of Pre-trained Teacher Cost**\n\nThis is a fair and important point for contextualizing our work. We acknowledge that a pre-trained teacher is needed, and the cost of pre-training can be expensive, especially on large datasets. Our work focuses on making these powerful models practical for widespread use.\n\n**Action:** We will add a sentence in the discussion to acknowledge this. For example: *\"While our method leverages a powerful pre-trained diffusion model, which entails a high one-time training cost, our focus is on making these models highly efficient for low-latency applications like GDA.\"*\n\n---\n\nWe are confident that these planned revisions will address all your concerns and improve the paper's clarity, accessibility, and overall quality. To that end, we want to emphasize that our method's sequential design makes it more manageable than it may appear, and we will work our best to address concerns about implementation complexity in our code release.\n\n**References**:\n\n[1] Jenni & Favaro, *On Stabilizing GAN Training with Noise*, CVPR’19\n\n[2] Arjovsky & Bottou, *Towards Principled GAN Training*, ICLR’17\n\n[3] Zhou et al., *Adversarial Score Identity Distillation*, ICLR’25\n\n[4] Kim et al., *Consistency Trajectory Models*, ICLR’24\n\n[5] Yin et al., *Improved Distillation for Fast Image Synthesis*, NeurIPS’24"}, {"review_id": "SYhemTbZtd", "reviewer": "Reviewer_2NvN", "summary": "This paper proposes DAR-GDA, a three-stage framework for generative data augmentation (GDA) using diffusion models. The key novel components include distillation, adversarial alignment, and importance reweighting, which accelerate and improve GDA. By combining adversarial training with diffusion distillation, the method yields a discriminator that estimates density ratios to correct biases. Extensive empirical validation demonstrating progressive gains at each DAR stage and surpassing prior generative augmentation baselines.", "questions": "- One concern is whether the process of distilling knowledge from large teacher models could result in training instability or model collapse.\n- Is it feasible to extend the proposed method to applications involving non-image data or complex conditional generation tasks?", "limitations": "- It would be helpful to include an analysis of the computational complexity to better understand the efficiency and scalability of the method.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- The proposed method effectively resolves the key practical issues in diffusion-based GDA, namely the slow sampling and synthesis bias via adversarial training.\n- The paper presents a theoretical analysis that links adversarial alignment with reweighting strategies, deriving stricter bounds on generalization performance.\n\n**Weaknesses:**\n- Some expressions are informal (e.g., the headings of section 2.1 and 2.2 should not contain “.”). Additionally, there are incomplete sentences, such as \"ERM is a common xxxxx seeks …\" in line 141. These issues indicate that this paper has not been thoroughly polished or carefully prepared.\n- The evaluation is limited to two visual datasets, which limits the validation of the generalization performance of the pre trained diffusion model.\n- The core novelty of this paper primarily derives from the integration of distillation, adversarial alignment, and discriminator-based importance reweighting, specifically tailored for diffusion-based generative data augmentation. However, these individual technical components are largely built upon existing research. The paper’s main contribution lies in their integrated application rather than fundamental redesign of these elements, which limits its degree of originality.\n- The accuracy of density ratio estimation and reweighting largely depends on the calibration of the discriminator, so instability or bias here may lead to incorrect importance weights.", "was_revised": true, "final_justification": "After reading the rebuttal and other reviewers' comments, I will raise my score from 3 to 4.", "rebuttal": "Thank you for your time and for providing a detailed review of our work! We appreciate your recognition of our method being **effective in resolving key practical issues** in GDA and **supported by theoretical analysis** that links the different stages.\n\nWe also appreciate your feedback and have developed a comprehensive plan to address it in a revised version.\n\n**[W1] On Editing**\n\nWe take this piece of review very seriously.\n\n**Action:** We are conducting a copy‑edit and proofread of the entire manuscript. We will correct the errors and expressions and re-edit Section 4 to enhance its clarity. Delivering a high-quality, well-written paper is our top priority for the revision.\n\n**[W2, Q2] On Limited Evaluation and Extensibility to Other Domains**\n\nOur primary goal for this initial work was to rigorously validate our three-stage framework on a well-established, well-understood frontier: large-scale image datasets. Given the complexity, we believed it was essential to first understand and validate the framework's stability, effectiveness, and scalability in a domain with standardized benchmarks. We believe this was the most scientifically sound way to build a foundation that could be reliably extended to other domains in the future. We indeed recognize that there is a meaningful application in other specialized domains where diffusion models show their promise. As our expertise lies in machine learning, we prioritized domains where we could ensure a thorough and reproducible evaluation.\n\nThat said, the motivations and core principles of DAR-GDA are fundamentally **domain-agnostic**. Unlike traditional data augmentation methods that rely on domain-specific transformations, our framework only requires a pretrained teacher diffusion model. Since the teacher is treated as a black box, the same distillation and reweighting procedures can be applied to any domain, with minimal adaptation, such as replacing the U-Net with backbones suiting the structure of the data\n\nFurthermore, recent successes of diffusion models in domains such as proteins [1] and molecules [2] demonstrate that high-quality pretrained teachers are indeed available beyond vision tasks, proving the \"generalization performance of the pre-trained diffusion model\" in other domains. Because our method treats the teacher as a black box, those models can be distilled and reweighted exactly as we do for images. Moreover, since the challenges of slow sampling and distribution mismatch are universal, we believe our method remains highly relevant, and perhaps even more impactful, in domains where labeled data is scarce or expensive to obtain.\n\nAdditionally, to provide further evidence of our framework's robustness, we conducted new experiments on static GDA-DAR on the FFHQ 64x64 for a gender classification task with EDM-distilled student augmenting the classifier training, and progressively applying the DAR stages (+D, +A, +R).\n\n**Gender classification on FFHQ 64×64**\n| Method| ResNet-18| VGG-16|\n|---|---|---|\n|Baseline|94.08|93.10|\n|EDM-SID (+D)|94.20|93.25|\n|EDM-SID (+D+A)|94.63| 93.62|\n|EDM-SID (+D+A+R)|**94.87**| **93.70**|\n\nWe observe monotonic performance gains as each stage of the DAR pipeline is applied, reinforce the effectiveness of our framework.\n\n**[W3] On Originality**\n\nWe agree that each component has prior art. We respectfully argue that our contribution is not the invention of these components, but the elegant, synergistic design into a unified framework. This integration is well-motivated by the core dilemma in GDA: no existing method could simultaneously offer the sample quality of diffusion models, the speed of GANs, and a principled mechanism for correcting synthesis bias. Our work introduces a new paradigm to solve this. \n\nWe argue that this design is not a trivial A+B+C, but an efficacious cycle where each stage creates a precondition for, enables, and enhances the next: the unified framework proposes a process of both improving and expediting the generator, while simultaneously creating the tool to correct its remaining flaws. \n\nAs you insightfully noted in your review, this framework \"*effectively resolves the key practical issues in diffusion-based GDA*\" and does so while achieving \"*theoretically stricter bounds on generalization performance*\". We believe the conceptual leap to this synergistic system is non-trivial, well-motivated, and original, with clear value to the community.\n\n**Action:** We apologize that the initial manuscript did not make the perceived novelty sufficiently clear. To correct this, we will revise the abstract and introduction and reframe the narrative to explicitly highlight the originality of our synergistic framework, ensuring it is understood as a conceptual advance for GDA rather than an incremental combination of techniques.\n\n**[W4, Q1]  On Framework Stability and Discriminator Validation**\n\nThese are critical points, and we thank you for the opportunity to clarify how our framework's design intrinsically ensures both training stability and the subsequent quality of the discriminator. Our approach is validated by a clear trail of evidence at each stage of the process.\n\nFirst, the foundation of our framework's stability lies in its initialization strategy. The one-step student generators are initialized from a strong teacher model trained by a stable, diffusion loss, meaning they already capture the target distribution well (possessing a degree of direct generation ability), and the distillation process refines this one-step ability.  This bypasses the notoriously unstable initial phases of traditional GAN training and mitigates well-known pathologies that arise from training from random noise, such as initial support mismatch, which can lead to mode collapse or catastrophic forgetting and result in poor discriminators [3-5].\n\nIn addition, our modular framework provides an in-process indicator of the discriminator's effectiveness: the performance of the student generator itself. While a one-step distilled model is typically expected to have a quality drop compared to its multi-step teacher, our experiments show the opposite. The student generator’s FID scores consistently improve during the adversarial alignment stage, empirically surpassing the original teacher. This improvement beyond the distillation baseline is direct evidence that the discriminator is providing high-quality, stable gradients and has successfully captured the nuanced differences between the real and generated distributions. This observation is consistent with current literature on adversarial distillation that also reports stable training and improved generator performance [6-8].\n\nFinally, while a perfect discriminator is unattainable, the practical question is whether it is accurate enough to guide the reweighting process effectively. The fact that it can guide a stronger student than a teacher generator suggests that the weighting it provides is meaningful. This can also be empirically observed from the consistent increase in downstream classification performance compared to their non-reweighting counterparts.\n\n**Action:**  Though we don’t observe them empirically and nor does the current literature, we agree with you that stability is a valid point of concern. To make this explicit for future researchers, we will highlight the cause of the stability and remind readers and users of our codes to pay attention to the student’s generation performance as a check.\n\n**[L1] On Computational Complexity Analysis**\n\nWe agree with you that a formal analysis of the computational complexity is crucial for understanding the practical benefits and scalability of our method. We will include a computational complexity analysis in the appendix.\nLet's define the computational costs:\n- $T$ denoising steps in the teacher sampler (≈ 1000).\n- $c_f$ - wall-clock cost of one diffusion model backbone forward (same for teacher and student).\n- $N$ – number of synthetic instances.\n- $C_{\\text{distill}}$ – one‑time cost of distilling the teacher into the student. (The teacher‑pretraining cost $C_{\\text{pretrain}}$ is shared by both methods and cancels out.)\n| Pipeline | Cost |\n| --- | --- |\n| Naive diffusion GDA | $$C_{\\text{naive}} = NTc_f$$ |\n| DAR–GDA | $$C_{\\text{DAR}} = C_{\\text{distill}} + Nc_f$$ |\n\nDAR–GDA is faster whenever$C_{\\text{distill}}<Nc_f (T-1)$\n\nWe estimate \\$C\\_{\\text{distill}}\\$ as: $C_{\\text{distill}} = P N k c_f$\n\nwhere:\n* $P$: number of data passes during distillation,\n* $k$: number of forward evaluations per sample per pass.\n\nEmpirically, recent works [6–9] suggest that one-step distillation setups require no more than \\$P \\le 150\\$ passes to converge, with \\$k = 3\\$ passes per sample (teacher, fake, and student's score estimations). Under these conservative values and a teacher with \\$T = 1000\\$ steps, we obtain:\n$\\frac{C_{\\text{distill}}}{N(T-1)c_f} = \\frac{P k}{T - 1} \\approx 0.45 < 1$\n\nThus, the inequality is comfortably satisfied even under the worst-case distillation settings.\n\nMoreover, when synthetic data are dynamically refreshed each epoch (the “dynamic” augmentation scheme), the savings compound across classifier training epochs. In that case, the effective $N$ becomes $N \\cdot E$, further improving the efficiency of DAR–GDA relative to naive diffusion-based generation.\n\n**References**:\n\n[1] Ingram et al., *Protein Generation with Diffusion*, Nature 2023\n\n[2] Cornet et al., *Molecular Diffusion Models*, NeurIPS’24\n\n[3] Jenni & Favaro, *On Stabilizing GAN Training with Noise*, CVPR’19\n\n[4] Arjovsky & Bottou, *Towards Principled GAN Training*, ICLR’17\n\n[5] Kodali et al., *On Convergence and Stability of GANs*, arXiv:1705.07215\n\n[6] Zhou et al., *Adversarial Score Identity Distillation*, ICLR’25\n\n[7] Kim et al., *Consistency Trajectory Models*, ICLR’24\n\n[8] Yin et al., *Improved Distribution Matching Distillation*, NeurIPS’24\n\n[9] Luo et al., *Diff-Instruct*, NeurIPS’23"}, {"review_id": "5ef75RsYnm", "reviewer": "Reviewer_BfFU", "summary": "This submission proposes an interesting idea of efficiently using Probabilistic Diffusion Models in data augmentation of discriminative models. A naïve use of generative synthetic sample-label pairs with diffusion models has limitations of ‘slow generation’ and ‘synthetic-real distribution gap’. The key idea here is to use score-distillation (a known concept) of the diffusion model to estimate a student model that can generate image in a single denoising step. The student generative model is also guided within a GAN-type framework that aims at aligning the synthetic data distribution to real data (available as training data of the discriminative model). The discriminator of the GAN-setup is also used to estimate a ‘probability ratio’ used as a measure of typicality of the synthetic sample in the real data distribution. This ratio is used to re-weight the synthetic samples in the training of the discriminative model for the eventual classification task.\nThe method is tested for CIFAR-10 and ImageNet-1K, showing favourable results against direct use of synthetic data augmentation, and demonstrating general performance improvements with GAN-type setup and reweighting along with distillation.", "questions": "-\tIn authors’ view, how critical is the availability of the teacher generative model on the dataset of the eventual discriminative problem? \n-\tWhy would the dynamic strategy still lead to a considerable gain over static generation in Table 4 when the training data is completely substituted with the synthetic data? There does not seem to be any additional feedback mechanism to alter the generation quality in that mode. Reviewer expected the numbers to be close to the static generation in Table 4. It will help if the authors can explain.\n-\tIt seems the ratio r(x) could also be used to filter synthetic samples falling far outside p_data. Could the authors comment on this thought.\n-\tDo the authors plan to also make the source code public? Since there are many details in the approach that can be better understood through the code, it will be helpful to release the implementation for replication.\n- The distillation and its embedding in GAN framework seems to be tailored for Markov chain based diffusion models, e.g., DDPM. Could the authors comment on what changes might be needed for extending the method to non-Markov models, e.g., DDIM. Also, how the compute gain of the proposed synthetic data generation would fare in that case.", "limitations": "Authors could also mention in the paper about the potential negative societal impact regarding efficient deepfake generation with their method, and generation of NSFW images.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n-\tThe key idea of the paper regarding adversarially aligning a distilled model and using the discriminator for estimating importance weights, is interesting.\n-\tExperiments and results systematically establish that the overall approach works and its constituents progressively add to the performance of the method.\n-\tDetails and discussion about results is comprehensive.\n\nWeaknesses:\n\n-\tThe paper suffers a bit from poor presentation. Whereas the method is interesting, it’s a bit hard to comprehend because no schematics, pseudo-code or algorithmic flow is presented. \n-\tThere are plenty of minor errors and typos, that further complicate understanding the paper. Line numbers of a few are mentioned: L46, L59, L140, L141, L217, L300.\n-\tResults are only provided on two data sets.\n-\tIt seems that the approach has a pre-requisite that a teacher generative diffusion model for the classification problem’s training data is available for the method to work.", "was_revised": true, "final_justification": "Reviewer concerns are addressed and queries well-answered.", "rebuttal": "Thank you for your detailed and thoughtful review of our paper. We are very encouraged that you found the core idea of our work to be “**interesting**” and appreciated that our experiments \"**systematically** establish that the overall approach works” with “**detailed discussion**”.\n\nYour feedback regarding the paper's presentation and clarity is particularly valuable. You have identified several key areas for improvement that we are eager to address. Below, we respond to each of your points and questions and outline our plan for revision.\n\n**[W1, W2] On Presentation, Clarity, and Typos**\n\nWe are very grateful for this direct and actionable feedback. We take the presentation and clarity issue very seriously.\n\n**Action:** To fundamentally improve the paper's clarity and presentation, we will make two major additions to the revised manuscript:\n\n1. **A new figure in the method section** to provide a step-by-step schematic of our proposed framework\n2. **A new algorithm box in the appendix** presenting the complete training procedure in pseudocode, detailing the inputs, outputs, initialization, and calculation with respect to equations.\n3. We have also done a **thorough proofreading** of the entire manuscript to correct all the errors you noted and any others we find. We are confident these changes will dramatically improve the paper's clarity. Delivering a well-written paper is our top priority in the revision.\n\n**[W3] On Limited Datasets**\n\nOur primary goal for this foundational work was to rigorously validate our three-stage framework on a well-established and understood frontier: large-scale image datasets. We believed it was essential to first prove its stability, effectiveness, and scalability in a domain with standardized benchmarks and felt this was the most scientifically sound way to build a strong foundation that could be reliably extended to other domains in the future.\nNevertheless, to demonstrate the versatility and robustness of our framework beyond these initial datasets, we have conducted additional evaluations on the FFHQ dataset (64x64), where we apply our DAR stages progressively (+D, +A, +R) on the EDM-SID model to statically augment the classifier for a gender classification task:\n\n**Gender classification on FFHQ 64×64**\n|Method| ResNet-18| VGG-16|\n|---|---|---|\n|Baseline|94.08|93.10|\n|EDM-SID (+D)|94.20|93.25|\n|EDM-SID (+D+A)|94.63| 93.62|\n|EDM-SID (+D+A+R)|**94.87**|**93.70**|\n\nHere, we observe the same monotonic gains from the D-A-R progression, as seen on CIFAR/ImageNet. Furthermore, at the suggestion of Reviewer W9GD, we have added a comprehensive evaluation on challenging real-world scenarios, including long-tail and noisy-label data. The detailed results of this analysis are presented in our full response to Reviewer W9GD.\n\n**[W4] On Dependence on Teacher's Data**\n\nThank you for pointing this out. We would highlight that given the strong ability of diffusion models to generate faithful outputs and the distillation procedures being stable and able to match and even surpass the teacher’s output quality. Therefore, we argue that the training is not necessarily reliant on the availability of the teacher’s data. As shown in Figures 1(c), 1(d), when a classifier is trained on a dataset entirely generated by our final student model (100% replacement), the classification accuracy can still be well maintained.\n\nMoreover, the \"dynamic\" scheme with 100% synthetic data even surpasses the performance of a classifier trained on only the static real data. This suggests that our efficient, high-quality generator can provide discriminative signals and diversity beyond what is available in the original fixed training set. This is particularly promising for data-constrained or privacy-sensitive scenarios.\n\n**Action:** We thank you for the opportunity for us to clarify this and will make this clearer in the experimental discussion.\n\n**[Q1] Teacher Model for the Target Dataset**\n\nThank you for this question. \n\nThis is indeed a crucial prerequisite for our current framework. If the teacher were trained on a different domain, the distilled student would generate out-of-domain samples, and the framework would likely fail. The goal of DAR-GDA is not to solve domain adaptation but to make augmentation for a *given target domain* highly efficient and unbiased.\n\n**[Q2] Performance Gain with the \"Dynamic\" Scheme**\n\nThis is a very sharp and interesting observation. You are absolutely correct that the generator itself does not change between epochs in the \"dynamic\" scheme. The performance gain comes from how the *classifier* uses the data.\n\nIn the “dynamic” scheme, the classifier is fed a brand new, unique batch of N synthetic images in every single epoch. Instead of repeatedly seeing the same fixed training set, it is exposed to a continuous stream of diverse samples drawn from the generator's learned distribution. This constant refreshment of data acts as a powerful regularizer, preventing the classifier from overfitting to specific training examples and forcing it to learn a more robust and generalizable representation of the underlying data manifold. \n\nThis provides strong evidence that our generator faithfully models the true data distribution. It shows that by sampling extensively from our generator, a classifier can learn a more generalizable model than one trained only on the finite set of real data.\n\n**Action:** We will highlight this key finding and its implications in the experimental discussion.\n\n**[Q3] Filter Alternative**\n\nThis is an insightful thought, and the answer is yes. Filtering is definitely another interesting idea. If a r(x) is very small, in our current approach, it is equivalent to filtering it off. We chose reweighting for two reasons: i) We avoid the setting another senstivie hyperparameter (the threshold) that needs to be tuned; we keep the reweighting scheme atumatic ii) Reweighting uses all samples but modulates their influence, which can be more stable than the \"hard\" decision of completely discarding, especially if the discriminator's estimates are prone to variance.\n\n**Action:** We will add a brief comment in our discussion acknowledging filtering as a valid alternative strategy and stating our rationale for choosing reweighting.\n\n**[Q4] Release of Source Code**\n\nYes, absolutely. We strongly believe in open and reproducible research. To this end, and to help the community build upon our work, we are committed to making our framework as handy and accessible as possible.\n\n**Action:** We will make our full, documented source code publicly available on GitHub upon cameraready.\n\n**[Q5] Extension to DDIM**\n\nThis is an excellent technical question. You are correct that DDPM and DDIM are two fundamental methods to reverse to sample from diffusion models. They typically use the same pre-training teacher, where DDIM speeds up the DDPM sampling by a constant factor (but still has the same asymptotic cost). Our method is agnostic to the specific solver used by the teacher model. The core requirement for our distillation is a pre-trained teacher model capable of estimating scores (or equivalently noise), which is a foundation for both DDPM (Markovian) and DDIM (non-Markovian) samplers.\n\nWe aim to replace the multi-step iterative solving process of DDPM or DDIM entirely with a direct, one-step generator. Therefore, there is a computational gain regardless of which solver the teacher uses. The gain is simply the cost of N solver steps vs. the cost of 1 generator step.\n\nTo make this concrete, we have tabulated a comparison of the inference time and asymptotic cost:\n\n| Sampler                      | # reverse steps $T$ | Asymptotic cost | Theoretical speed-up vs. 1000-step DDPM |\n| ---------------------------- | ------------------- | --------------- | --------------------------------- |\n| DDPM                         | 1000                | $O(T)$          | baseline                              |\n| DDIM (fast setting)          | 50                  | $O(T)$          | ≈ 20 ×                            |\n| **DAR–GDA** (1-step student) | **1**               | **$O(1)$**      | **≈ 1000 ×**                      |\n\nA single UNet evaluation is the theoretical lower bound on diffusion sampling cost, so the distilled generator, with a sampling complexity $O(1)$, cannot be slower than any multi-step scheme with a time complexity of $O(T)$. Ignoring practical overheads such as kernel launch and initialization, the expected computational speedup scales linearly with $T$.\n\n**[Ethical]**\n\nThis is a very important point. We agree with you that all efficient, high-quality generative models, including ours, present a dual-use risk that must be acknowledged.\n\nWhile the primary concern is the potential for misuse in rapidly generating harmful or deceptive content, we also see a promising positive application. Our framework could be used to train stronger discriminative classifiers that act as more effective filters for undesirable content (e.g., NSFW images) in content moderation. The only concern in this use case is that the generator trained on undesirable content must be kept secure to prevent it from being misused.\n\n**Action**: We will add such ethical considerations and their potential impact in the discussion section of our paper.\n\n----\n\nWe are confident that we have addressed your concerns. Thank you again for your insightful questions that allow us to further improve our work."}]

Author rebuttal:

Meta-review: This paper proposes DAR-GDA, a three-stage framework for generative data augmentation (GDA) using diffusion models. The method addresses two major limitations of GDA with diffusion models: namely inefficient computation, and bias due to distributional mismatch between real and synthetic data. DAR-GDA integrates distillation, adversarial alignment, and importance reweighting for diffusion-based augmentation. They show the proposed method achieved SOTA performance through experiments on CIFAR-10, ImageNet-1K, and FFHQ datasets. The reviewers pointed out several strengths of the paper, including Innovative Integration of known techniques, theoretical justification connecting adversarial alignment with generalization error bounds, and strong empirical evaluation results. However, the reviewers also raised several concerns: lack of clarity in paper presentation, implementation complexity in the three-step framework, and limited scope of evaluation. The authors provided comprehensive rebuttal, including additional experimental results, computational analysis, and ethical discussion, which demonstrates good responsiveness and well addresses the reviewers' concerns. The final ratings from the four reviewers after the rebuttal consist of 2 Borderline Accepts and 2 Accepts. The AC reads the paper, the reviewers' comments, and the authors' rebuttal, and agrees with the reviewers' suggestions, thus recommending accepting the paper.
Decision: Accept (poster)