Title: Generative Data Augmentation via Diffusion Distillation, Adversarial Alignment, and Importance Reweighting
Venue: NeurIPS 2025
Accepted: yes
Keywords: Generative Data Augmentation

Diffusion Model Distillation
Abstract: Generative data augmentation (GDA) leverages generative models to enrich training sets with entirely new samples drawn from the modeled data distribution to achieve performance gains. However, the usage of the mighty contemporary diffusion models in GDA remains impractical: *i)* their thousand-step sampling loop inflates wall-time and energy cost per image augmentation; and *ii)* the divergence between synthetic and real distributions is unknown--classifiers trained on synthetic receive biased gradients.
We propose DAR-GDA, a three-stage augmentation pipeline that unites model **D**istillation, **A**dversarial alignment, and importance **R**eweighting that makes diffusion-quality augmentation both fast and optimized for improving downstream learning outcomes. 
In particular, a teacher diffusion model is compressed into a one-step student via score distillation, slashing the time per-image cost by $>100\times$ while preserving FID.
During this distillation (D), the student model additionally undergoes adversarial alignment (A) by receiving direct training signals against real images, supplementing the teacher's guidance to better match the true data distribution.
The discriminator from this adversarial process inherently learns to assess the synthetic-to-real data gap. Its calibrated probabilistic outputs are then employed in reweighting (R) by importance weights that quantify the distributional gap and adjust the classification loss when training downstream models; we show that reweighting yields an unbiased stochastic estimator of the real-data risk, fostering training dynamics akin to those of genuine samples.
Experiments validate DAR-GDA's synergistic design through progressive accuracy gains with each D-A-R stage. Our approach not only surpasses conventional non-foundation-model GDA baselines but also remarkably matches or exceeds the GDA performance of large, web-pretrained text-to-image models, despite using solely in-domain data.
DAR-GDA thus offers diffusion-fidelity GDA samples efficiently, while correcting synthetic-to-real bias to benefit downstream tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission proposes an interesting idea of efficiently using Probabilistic Diffusion Models in data augmentation of discriminative models. A naïve use of generative synthetic sample-label pairs with diffusion models has limitations of ‘slow generation’ and ‘synthetic-real distribution gap’. The key idea here is to use score-distillation (a known concept) of the diffusion model to estimate a student model that can generate image in a single denoising step. The student generative model is also guided within a GAN-type framework that aims at aligning the synthetic data distribution to real data (available as training data of the discriminative model). The discriminator of the GAN-setup is also used to estimate a ‘probability ratio’ used as a measure of typicality of the synthetic sample in the real data distribution. This ratio is used to re-weight the synthetic samples in the training of the discriminative model for the eventual classification task.\nThe method is tested for CIFAR-10 and ImageNet-1K, showing favourable results against direct use of synthetic data augmentation, and demonstrating general performance improvements with GAN-type setup and reweighting along with distillation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n-\tThe key idea of the paper regarding adversarially aligning a distilled model and using the discriminator for estimating importance weights, is interesting.\n-\tExperiments and results systematically establish that the overall approach works and its constituents progressively add to the performance of the method.\n-\tDetails and discussion about results is comprehensive.\n\nWeaknesses:\n\n-\tThe paper suffers a bit from poor presentation. Whereas the method is interesting, it’s a bit hard to comprehend because no schematics, pseudo-code or algorithmic flow is presented. \n-\tThere are plenty of minor errors and typos, that further complicate understanding the paper. Line numbers of a few are mentioned: L46, L59, L140, L141, L217, L300.\n-\tResults are only provided on two data sets.\n-\tIt seems that the approach has a pre-requisite that a teacher generative diffusion model for the classification problem’s training data is available for the method to work.\n\nQUESTIONS:\n-\tIn authors’ view, how critical is the availability of the teacher generative model on the dataset of the eventual discriminative problem? \n-\tWhy would the dynamic strategy still lead to a considerable gain over static generation in Table 4 when the training data is completely substituted with the synthetic data? There does not seem to be any additional feedback mechanism to alter the generation quality in that mode. Reviewer expected the numbers to be close to the static generation in Table 4. It will help if the authors can explain.\n-\tIt seems the ratio r(x) could also be used to filter synthetic samples falling far outside p_data. Could the authors comment on this thought.\n-\tDo the authors plan to also make the source code public? Since there are many details in the approach that can be better understood through the code, it will be helpful to release the implementation for replication.\n- The distillation and its embedding in GAN framework seems to be tailored for Markov chain based diffusion models, e.g., DDPM. Could the authors comment on what changes might be needed for extending the method to non-Markov models, e.g., DDIM. Also, how the compute gain of the proposed synthetic data generation would fare in that case.\n\nLIMITATIONS:\nAuthors could also mention in the paper about the potential negative societal impact regarding efficient deepfake generation with their method, and generation of NSFW images.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes DAR-GDA, a three-stage framework for generative data augmentation (GDA) using diffusion models. The key novel components include distillation, adversarial alignment, and importance reweighting, which accelerate and improve GDA. By combining adversarial training with diffusion distillation, the method yields a discriminator that estimates density ratios to correct biases. Extensive empirical validation demonstrating progressive gains at each DAR stage and surpassing prior generative augmentation baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The proposed method effectively resolves the key practical issues in diffusion-based GDA, namely the slow sampling and synthesis bias via adversarial training.\n- The paper presents a theoretical analysis that links adversarial alignment with reweighting strategies, deriving stricter bounds on generalization performance.\n\n**Weaknesses:**\n- Some expressions are informal (e.g., the headings of section 2.1 and 2.2 should not contain “.”). Additionally, there are incomplete sentences, such as \"ERM is a common xxxxx seeks …\" in line 141. These issues indicate that this paper has not been thoroughly polished or carefully prepared.\n- The evaluation is limited to two visual datasets, which limits the validation of the generalization performance of the pre trained diffusion model.\n- The core novelty of this paper primarily derives from the integration of distillation, adversarial alignment, and discriminator-based importance reweighting, specifically tailored for diffusion-based generative data augmentation. However, these individual technical components are largely built upon existing research. The paper’s main contribution lies in their integrated application rather than fundamental redesign of these elements, which limits its degree of originality.\n- The accuracy of density ratio estimation and reweighting largely depends on the calibration of the discriminator, so instability or bias here may lead to incorrect importance weights.\n\nQUESTIONS:\n- One concern is whether the process of distilling knowledge from large teacher models could result in training instability or model collapse.\n- Is it feasible to extend the proposed method to applications involving non-image data or complex conditional generation tasks?\n\nLIMITATIONS:\n- It would be helpful to include an analysis of the computational complexity to better understand the efficiency and scalability of the method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose a three-stage framework (named as DAR-GDA) to make generative data augmentation (GDA) with diffusion models both efficient and unbiased. It combines Distillation (to compress slow teacher models into fast one-step generators), Adversarial alignment (to match the synthetic data distribution to the real one), and Reweighting (to correct for residual biases using density ratio estimates). The method significantly improves classification accuracy on CIFAR-10 and ImageNet-1K, even outperforming real-data augmentation in some settings. It also achieves GAN-level speed with diffusion-level fidelity. It showcases how real data can still not be substituted by synthetic data for competetive performance. While the approach is promising, their deployment is currently limited by  complexity of three stage implementation. Overall, DAR-GDA is an efficient, accurate, practical but complex-for-implementation method for generative data augmentation.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths** :\n1.  The method distills a slow diffusion model into a one-step student generator, reducing image generation time by over 100×, making GDA feasible for large-scale tasks.\n\n2.  It introduces a better, principled importance __reweighting__ mechanism using a learned discriminator to correct for distribution mismatch between real and synthetic data, reducing bias in model training.\n\n3. By integrating score distillation from diffusion models with adversarial training from GANs, the proposed method maintains **high sample quality** while also learning a discriminator for density ratio estimation r(x).\n\n4. The results show a **decent improvement** in classification performance in Table 1, when synthetic data is added to the real training data across different model variants on  CIFAR-10 and ImageNet. Their method is also theoretically grounded as an added bonus.\n\n**Weakness**\n\n1. Lack of consistency in Paper Writing: \n\nManuscript is very difficult to read due to sentence level mistakes especially in section 4 -- Line 141: “*Empirical risk minimization (ERM) [82, 49] is a common xxxxx seeks a hypothesis that*”. Line 148: “from which we can draw an augmenting the sample set of synthetic samples”.\n\nLine 223: “is a distribution weighting timestep t\". Line 300: “On CIFAR-10, On CIFAR-10 we”\n\n2. Several tuning parameters for multi-component optimization: \n\nThe proposed multi-component optimization introduces complexity, both in implementation and training dynamics. This seems like a challenge for an end user for implementation sake, making it fragile and difficult to debug. How do authors address this challenge ?\n\n3. Dependence on the Discriminator Quality\n\nThe paper discusses that \"*Empirically, the student often attains lower FID scores than non-adversarially distilled counterparts or even the teacher model in some scenarios*\". This means that performance improvements are not consistent across all cases—the method may be sensitive to how well the adversarial discriminator is trained. How do authors analyse the robustness/accuracy of the trained discriminator to ensure this quality check?\n\n4. High variance from r(x) ? [Minor since it is a known issue]\n\nComputing per-sample importance weights r(x) introduces variance into the training objective, especially when there's a large gap between the real and generated data distributions. If r(x) becomes too large (which happens when q_{G} (x) << p_{data} (x) ) , training can become unstable or biased toward a few high-weighted samples. The paper does mention truncation and self-normalization to control this variance, but doesn't dig into discussion if this happens with the current setup?\n\n5. Clarification [Minor]: Even when the student generator is fast, it still depends on a pretrained diffusion teacher. Training that teacher model can be computationally expensive, especially on large datasets like ImageNet-1K. Does the paper acknowledge that it can still impose a significant compute burden before DAR-GDA becomes efficient ?\n\nQUESTIONS:\nThe questions (1-5) are clearly mentioned in the weakness section above, and addressing them will affect my evaluation score. \n\n1-3 are major concerns.\n\n4-5 are minor concerns.\n\nApart from technical aspect, the writing in the paper could benefit from greater clarity. Some sections (like section-4) are dense and difficult to follow.\n\nLIMITATIONS:\nThe limitations apart from the ones mentioned in the paper, are discussed in the weakness section. The major limitation are due to 1. Multiple interdependent components in the implementation and tuning complexity, and 2. dependence of Reweighting on a well-trained discriminator; where errors can mislead training.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce DAR-GDA, a three-stage framework for generative data augmentation using diffusion models that combines distillation, adversarial alignment, and importance reweighting to generate high-quality and unbiased synthetic training data efficiently. The proposed method compresses slow diffusion models into fast one-step generators while aligning them with the real data distribution via adversarial training. Finally, the resulting discriminator is used to reweight synthetic samples for accurate downstream learning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The authors tackle both issues with generative data augmentation: high sampling time and the divergence between real and synthetic distributions is unknown, so the proposed method is well-motivated.\n- As seen in the results, distillation, adversarial alignment, and reweighting seem to mesh quite nicely to perform generative data augmentation.\n- Strong performance with both the CIFAR-10 and ImageNet-1K datasets.\n- A 100x reduction in per-image costs while preserving FID is greatly beneficial.\n\nWeaknesses:\n- Only 2 general imaging datasets used - unclear if the method would be as useful in different domains.\n- Unclear whether the method would work under noisy or imbalanced settings.\n- Non-generative augmentation methods are not included as baselines.\n\nQUESTIONS:\n1. Do you anticipate any limitations of the proposed method in other domains such as healthcare or in extensions to other tasks such as segmentation, regression, etc.?\n2. Is there a reason why non-generative augmentation techniques [1,2] were not used as baselines?\n3. Is the proposed method robust to label and/or input noise? How about when there is significant class imbalance?\n\n[1] Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V. and Le, Q.V., 2018. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501.\n[2] Cubuk, E.D., Zoph, B., Shlens, J. and Le, Q.V., 2020. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (pp. 702-703).\n\nAddressing these questions effectively may help raise my score.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}