Distribution-Aware Data Expansion with Diffusion Models
Authors: haoweiz
Ling Yang
Jun-Hai Yong
Hongzhi Yin
Jiawei Jiang
Meng Xiao
Wentao Zhang
Bin Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion models
data expansion
Primary area: diffusion_based_models

Abstract: The scale and quality of a dataset significantly impact the performance of deep models. However, acquiring large-scale annotated datasets is both a costly and time-consuming endeavor. To address this challenge, dataset expansion technologies aim to automatically augment datasets, unlocking the full potential of deep models. Current data expansion techniques include image transformation and image synthesis methods. Transformation-based methods introduce only local variations, leading to limited diversity. In contrast, synthesis-based methods generate entirely new content, greatly enhancing informativeness. However, existing synthesis methods carry the risk of distribution deviations, potentially degrading model performance with out-of-distribution samples. In this paper, we propose DistDiff, a training-free data expansion framework based on the distribution-aware diffusion model. DistDiff constructs hierarchical prototypes to approximate the real data distribution, optimizing latent data points within diffusion models with hierarchical energy guidance. We demonstrate its capability to generate distribution-consistent samples, significantly improving data expansion tasks. DistDiff consistently enhances accuracy across a diverse range of datasets compared to models trained solely on original data. Furthermore, our approach consistently outperforms existing synthesis-based techniques and demonstrates compatibility with widely adopted transformation-based augmentation methods. Additionally, the expanded dataset exhibits robustness across various architectural frameworks.

Distribution-Aware Data Expansion with Diffusion Models
Haowei Zhu', Ling Yang
Jun-HHai Yong',
Hongzhi Yin", Jiawei Jiang", Meng Xiao
Wentao Zhang
Bin Wang
Tsinghua University
PPeknng University
3 University of Queensland
Wuhan University CNIC, CAS
wentao. zhang@pku. edu.
wangbins@tsinghua.
www.gghuuaeeu...c..... edu.cn
Abstract
The scale and quality of dataset significantly impact the performance of deep models. However, acquiring large-scale annotated datasets is both a costly and time- consuming endeavor. To address this challenge, dataset expansion technologies aim to automatically augment datasets, unlocking the full potential of deep models. Cur- rent data expansion techniques include image transformation and image synthesis methods. Transformation-bae based methods introduce only local variations, leading limited diversity. In contrast, synthesis-based methods generate entirely new content, greatly enhancing informativeness. However, existing synthesis methods carry the risk of distribution deviations, potentially degrading model performance with out-of-din of-distributton samples. In this paper, we propose DistDiff, training-free data expansion framework based on the distribution-awan diffusion model. Dist- Diff constructs hierarchical prototypes to approximate the real data distribution, optimizing latent data points within diffusion models with hierarchical energy guidance. We demonstrate its capability to generate distribution-consiteen sam- ples, significantly improving data expansion tasks. DistDiff consistently enhances accuracy across diverse range of datasets compared to models trained solely on original data. Furthermore, our approach consistently outperforms existing synthesis-baase based techniques and demonstrates compatibility with widely adopted transformation-basee augmentation methods. Additionally, the expanded dataset exhibits robustness across various architectural frameworks. Our code is available at https//ggihbb..omm/moveeii//D//d
Introduction
substantial number of training samples are essential for unlocking the full potential of deep networks. However, the manual collection and labeling of large-scale datasets both costly and time-intensive. This makes it difficult to expand data-scarce datasets. Therefore, is of great value to study how expand high-quality training data an efficient and scalable way [6].
Automatic data expansion technology can alleviate the data scarcity problem by augmenting or creating diverse samples, it mitigates the bottleneck associated with limited data, thereby improv- ing model's 's downstream performance and fostering greater generalization [14, 78]. One simple yet effective strategy is employing image transformation techniques such cropping, rotation, and erasing to augment samples [59]. Although these methods prove effective and have been widely applied in various fields, their pre-defined perturbations only introduce local variations to the images, thereby falling short in providing a diverse range of content change. In recent
"Equal Contribution. 'Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

times, generative models have gained considerable attention [18, 44, 47, 50, 54, 45], exhibit im- pressive performance in various areas like image inpainting [36, 53], super-resolution [26, 55], and video generation [25, 41J. Generative models leverage text and image conditions to create images with entirely novel content, harnessing the expansive potential of data expansion [13]. Nevertheless, there is risk of generating images that deviate from the real data distribution. Therefore, when employing diffusion models for dataset expansion tasks, further research is necessary to ensure a match between synthetic data distributions and real data distributions.
(a) Conventional Expansion Methods
optimization
or filration
Zo
Z
Zy
There are several strategies aimed at mitigat- ing the risk of distribution shift, which can be broadly categorized into two groups: training- based and training-free free methods. The training- based methods [42, 52, 65] fine-tune pre-trained diffusion models to adapt target dataset, neces- sitating additional training costs and increas- ing the likelihood of overfitting on small-scale datasets. Other training- free methods [17, 22] eliminate potentially noisy samples by design- ing optimizing and filtering strategies, but they still struggle to generate data that conforms to the real data distribution.
(b) DistDiff
Data
Rc
Energy G uidiance
Set
transfomeed
Zy
Figure A comparison unveils distinctions be- tween conventional data expansion methods and our innovative distribution aware diffusion frame- work, benefiting from hierarchical clustering and multi-step energy guidance.
In this work, we propose training- frining-free data expansion framework, dubbed Distribution-Aware Diffusion (DistDiff) to optimize generation results. As shown in Figure 1, DistDiff initially approximates the true data distribution using class-level and group-level prototypes obtained through hierarchical clustering. Subsequently, DistDiff utilizes these prototypes to formulate two synergistic energy functions. A residual multiplicative transformation is then applied to the latent data points, enabling the generation of data distinct from the original. Following this, the hierarchical energy guidance process refines intermediate predicted data points, optimizing the diffusion model to generate data samples that are consistent with the underlying distribution. DistDiff ensures fidelity and diversity in the generated samples through distribution- aware energy guidance. Experimental results demonstrate that DistDiff outperforms advanced data expansion techniques, producing better expansion effects and significantly improving downstream model performance. Our contributions can be summarized as follows:
We introduce novel liffusion-based data expansion algorithm, named DistDiff, which facilitates
data augmentation without requiring re-training.
By leveraging hierarchical prototypes to approximate data distribution, we propose an effective distribution-awanl energy guidance at both class and group levels in the diffusion sampling process.
The experimental results illustrate that our DistDiff is capable of generating high-quality samples, surpassing existing image transformation and synthesis methods significantly.
2 Related Work
2.1
Transformation-Batt Data Augmentation
Traditional data augmentation techniques [5, 10, 23, 74, 76, 81, 82) typically involve expanding the dataset through distortive transformations, aiming enhance the model's ability capture data invariance and mitigate overfitting [59]. For instance, scale invariance is cultivated through random cropping and scaling, while rotation invariance is developed through random rotation and flipping. Region mask-based methods [5. 12, 82] enhance model robustness against target occlusion by strategically obscuring portions of the target area. Interpolation- based methods [23, 74, 76] generate virtual samples by randomly blending content from two images. Rand/Agmmett [10] further boosts augmentation effectiveness by sampling from diverse range of augmentation strategies. However, these methods induce only subtle changes on the original data through transformation, deletion, and blending, leading to a lack of diversity. Moreover, they are predefined and uniformly applied across the entire dataset, which may not be optimal for varying data types scenarios.

---

(a) Real Distribution A Approximation
(b) DistributionAware Sampling Process
Optimizaioon step within diffusion!
Image Encoder
Hierarchical
Prototype
Pe
Transfrrm
Guidance
Dacoder
Approximation
Pa
Z0/c
Z < (1 ee +b
Zt
Z
Pg
Zo14
Pc
Samping Optnization
V (2oje PexPo)
Hierarciicll prolotypeppprxxinng
Distrbution-Aware Energy Guidance
Figure 2: Overview of the DistDiff pipeline. DistDiff enhances the generation process diffusion models with distribution-aware optimization. It approximates the real data distribution using hier- archical prototypes P. and Py. optimizing the sampling process through distribution-awan energy guidance. Subsequently, original generated data point is refined for improved alignment with the real distribution.
2.2 Synthesis-Based Data Augmentation
Generative data augmentation aims to leverage generative models approximate the real data distribution, generating samples with novel content to enhance data diversity. GAN [18] excels at learning data distributions and producing unseen samples in unsupervised manner [1. 20, 33, 39, 79, 80, 67]. While their efficacy has been demonstrated across diverse downstream tasks, studies indicate that training existing models like ResNet50 [21] images synthesized by BigGAN [4] yields subpar results compared to training on real images. This disparity in performance can be attributed to the limited diversity and potential domain gap between synthesized samples and real images. Additionally, the training processes of GAN are notoriously unstable, particularly with low-data regime, and suffer from mode collapse, resulting in lack of diversity [3, 19, 48]. In contrast, diffusion model-based methods [69] can offer better controllability and superior customization capabilities. Text-to-image models such Stable Diffusion [50], DALL-E [47] and RPG [71] have demonstrated the creation of compelling high-resolution images [70, 72, 77]. Recently, large-scale text-to-image
models have been used for data generation [2. 15, 35, 58, 64, 63]. For example, LECF [22] utilized GLIDE [44] to generate images, filtering low-confidence samples to enhance zero-shot and few-shot image classification performance. SGID [32] leverages image descriptions generated by BLIP [34] to enhance the semantic consistency of generated samples. Feng et al. [17] filters out low-quality samples based on feature similarity between generated and reference images. GIF [78] creates new informative samples through prediction entropy and feature divergence optimization. However, it's crucial to note that datasets generated by existing methods may exhibit distribution shifts, impacting image classification performance significantly. Zhou et al. [83] address this issue by employing diffusion inversion to mitigate distributional shifts. contrast, we propose a training- approach, leveraging hierarchical prototypes optimization targets to guide the generation process, thereby addressing distributional shifts. This approach offers the advantage of avoiding additional computational costs and overfitting issues associated with fine-tuning diffusion models.
3 Method
In this study, we introduce distribution aware data expansion framework utilizing Stable Diffusson as a prior generation model. This framework guides the diffusion model based on hierarchical prototype guidance criteria. As illustrated in Figure 2, our DistDiff initially employs an image encoder 0 to extract instance features and subsequently derives hierarchical prototypes to approximate the real data distribution. Next, for given seed image and corresponding text prompts, we extract image's
3

---

latent feature and apply stochastic noise to it. Subsequently, in the denoising process, we optimize the latent features using a training-free free hierarchical energy guidance process. Our optimization strategy ensures that the generated samples not only match the distribution but also carry new information to enhance model training.
3.1 Task Definition
In the context of a small-scale training dataset, the data expansion task is designed to augment the original dataset D. = {æ,, y= with a new set of synthetic samples, referred to as D. = f 3/ Here æ; and y/, represent the sample and its corresponding label. where no and n, respectively denote the original sample quantity and the synthetic sample quantity. The objective is to enhance the performance of deep learning model trained both the original dataset D, and the expanded dataset D, compared to a model trained solely on the original D... The crucial aspect lies in ensuring that the generated dataset is highly consistent with the distribution of the original dataset while being as informative as possible.
3.2 Hierarchical Prototypes Approximate Data Distribution
Prototypes have been widely employed in class incremental learning methods to retain information about each class [40, 49]. In this work, we propose two levels of prototypes to capture the original data distribution. Firstly, the class-level prototypes Pe are obtained by averaging feature vectors within the same class. The class vector aggregates high-level statistical information to characterize all samples from the same class as a collective entity. However, as class-level prototypes represent the class feature space as a single vector, potentially reducing informativeness, we further introduce group-level prototypes to capture the structure of the class feature space. Specifically, we divide all samples from the same class into K groups using agglomerative hierarchical clustering, followed by averaging feature vectors within each group to obtain K group prototypes Pg (p& P .... p K Instances with similar patterns are grouped together. Transitioning from class-level to group-level, the prototypes encapsulate abstract distribution information of the class at different scales.
of Thanks to these hierarchical prototypes, we design two function Dí and D to evaluate the degree
distribution matching:
(x pe) 10(xx) -Pell2
(1)
D ax. P&) = [0(()) - -palla
(2)
s.t.j = arg max(cos (p')
where 0()) means feature extractor, which could be ResNet [21], CLIP [46] or other deep models.
Note that these two functions evaluate the score of distribution matching from two perspectives. The value will be lower when æ is more consistent with the real distribution. As shown in Figure (a), Dž gauges the distance of sample features to the class center, resulting in low scores for easy samples while high scores for hard samples that are situated at the boundaries of the distribution. On the other hand, DD assesses distance from the group-level, offering lower scores for hard samples, while still maintaining relatively high scores for outlier samples. Consequently, these two scores mutually reinforce each other and are indispensable.
3.3 Transform Data Points
Given a reference sample yz) the pre-trained large-scale diffusion model can generate new samples x' with novel content. We formalize this process as x' G( + d). where 4(x) represents the latent feature representation and d is the perturbation applied to latent features. Drawing inspiration from GIF [78], we introduce residual multiplicative transformation the latent feature z (xx) using randomly initialized channel-leve noise e ~ [4, 1) and b (0 1). We impose an ball constraint on the transformed feature control the degree of adjustment within reasonable range, i.e., || 21100 S e, and derive 2 as follows.
1 = Pz. (T())) = Ps.c((1 + e)2 b),
where T(-) represents the transformation function and P.,((.) denotes the projection of the trans- formed feature ž onto the e-ball of the original latent feature

---

Now, the key challenge lies in optimizing e and b to create new samples that align closely with the real data distribution. Another intuitive approach is directly optimize latent features instead of performing residual multiplication transformations. However, directly optimizing latent features leads to minimal perturbations, making it challenging to achieve performance gains. We discuss this alternative approach in Section 4.4.
3.4 Distribution-Awan Aware Diffusion Generation
In a typical diffusion sampling process, the model iteratively predicts noise to progressively map the noisy 2r into clean zg- While existing data expansion methods [17, 22, 32] treat the generative model as a black box, focusing on filtering or optimizing the final generated zg. The importance of the intermediate sampling stage is ignored, which plays crucial role in ensuring data quality, especially as the image begins take on a stable shape appearance. Diverging from prior approaches, we advocate for intervention at the intermediate denoising step for optimization.
Specifically, we first introduce energy guidance into standard reverse sampling process to optimize the transformation using Equation 4. As our energy guidance step is applied to the transformed data point, the transformed data point ž is denoted as 2₁ for simplicity.
é' e Vee[,,)( bb=b-
where p is the learning rate and E(2, c) is the energy function measuring the compatibility between the transformed noisy data point 2, and the given condition representing the real data distribution in this work. Equation 4 guides the sampling process and generates distribution-consiseee After that, the optimized 2 is obtained via Equation 3. However, directly measuring the distance between intermediate results z, with condition e is impractical due to the difficulty in finding a pre-trained network that provides meaningful guidance when the input is noisy.
samples.
To address this issue, we leverage the capability that the diffusion model can predict the noise added to ze, and thus predict a clean data point Zuje. as shown in Equation 5. Then, the new energy function Da (Zole, e) based on the predicted clean data point is constructed to approximate s(z,,
21
(2,,))
Zo1e
(5)
Vos
where aj represents the noise scale and da is the learned denoising network. Finally, we employ hierarchical prototypes P. and Pg conditions to construct our energy guidance in the following manner:
e - PV (DS Pc) + (z()e Pg)),
(6)
b'= - (D. p.) + (zo Pg)).
Unlike existing methods that exclusively optimize the final sampling result zo. our approach focuses on optimizing intermediate denoising steps within the sampling process. The detailed algorithm is shown in Appendix C. This novel strategy leads to substantial improvements in optimization results and will be further explored in Section 4.4.
4 Experiments
4.1 Experimental Setups
We assess the performance of DistDiff across six image classification datasets, encom- passing diverse tasks such as general object classification (Caltech- 101 [16), CIFAR 100-Subset [31], ImageNette [27]), fine-graine fine-grained classification (Cars [30]), textual classification (DTD (8]) and medical imaging (PathMNIST [68]). More details are provided in Appendix B.1.
Datasets
We conduct a comparative analysis between DistDiff and conventional image transformation methods, as well as diffusion-based expansion methods. Traditional image transfor- mation techniques considered the comparison comprise Autonugment [9], RandAugment [10], Random Erasing [82], GridMask [5], and interpolation-based techniques like MixUp [76] and CutMix
Compared Methods
5

---

[74]. For generative- based methods, we include the direct application of stable diffusion for data expansion, as well as the most recent ttate-of-the-eee method, Stable Diffusson 1.4 [50], LECF [22]. GIF-SD [78]. The implementation details of these techniques are provided in Appendix B.3 and B.4.
4.2 Implementation Details
In our experimental setup, we implement DistDiff based on Stable Diffusion 1.4 [50]. The images created by Stable Diffusion have resolution of 512 x 512 for all datasets. Throughout the diffusion process, we employ the DDIM [60] sampler for a 50-step latent diffusion, with hyper-paramene parameters for noise strength set at 0.5 and classifier free guidance scale 7.5. The in Equation 3 is 0.2 by default. We use a ResNet-50 [21] model trained from scratch on the original datasets our guidance model. We assign K 3 to each class when constructing group-level prototypes, the learning rate p is 10.0, and optimization step M is set to 20 unless specified otherwise. After expansion, we concatenate the original dataset with synthetic data to create expanded datasets. We then train the classification model from random initialization for 100 epochs using these expanded datasets. During model training, we process images through random cropping to 224 x 224 using random rotation and random horizontal flips. Our optimization strategy involves using the SGD optimizer with momentum of 0.9, and cosine decay with an initial learning rate of 0.1. All results are averaged over three runs with different random seeds. More implementation details can be found in the Appendix B.2.
4.3 Main Results
Comparison with Baathesis-Baseer Methods To investigate the effectiveness of methods for generating high-quality datasets for down- stream classification model training, we ini- tially compare our method, DistDiff, with exist- ing synthesis- wandeesis-based methods on Caltech- in terms of classification performance. Figure 3 highlights the superiority of our method over state-of-the-art techniques. Compared the original stable diffusion, DistDiff exhibits an av- erage improvement of 6.25%, illustrating that our method retains more distribution-aligned in- formation from the original datasets. Addition-
Calech=101
Cliut00-Subsett
$100
623
11,43
66:57
11.63
80
C0P 007
LEGF
Meboa
Manae
Figure 3: Our method outperforms state-of-ther the--art data expansion methods when trained on expanded datasets, underscoring the importance of high-quality generator training a classifier.
ally, GIF-SD [78], which uses pre-trained CLIP [46] model to enhance class-maintained information encourage batch-wise sample diversity, is also surpassed by our method by 5.46% in accuracy. This can be attributed to DistDiff guiding the generation process from the distribution level with hierarchical prototypes, providing better optimization signals.
and employs KL-divergence
We also evaluated DistDiff against LECF [22], which enhances language prompts and filters samples with low confidence. We use L.E enhanced to synthesize 5x synthetic datasets and filter with different thresholds. We assessed multiple Clip Filtering strengths LECF and found that LECF did not achieve better performance compared to the original Stable Diffusion. This is due to our SD baseline already generates high-quality samples, and the additional filtering post-process may lead to data loss. Besides, both LECF and GIF-SD use auxiliary models to filter/guide the diffusion generation results. However, there are two main strengths of our DistDiff compared to these methods. First, DistDiff generates images end-to end without requiring baser- postprocessing. Second, our method not sensitive to the classification performance of the auxiliary model, which is proved in the Appendix D.2. This suggests that DistDiff is simpler and more generalizable for data expansion tasks.
Comparison with Transformation- -assd Augmentation Methods In Table 1, we present comparison of our methods with widely adopted data augmentation techniques for Caltech- Caltech-10limage classification. Our DistDiff method surpasses transformation-bee based augmentation methods by intro- ducing a broader range of new content into images. Additionally, we demonstrate the compatibility of approach with transformation-base data augmentation methods, leading to further improvements.
Scaling in Number of Data We evaluate the scalability of our approach by assessing its advantages in classification model training across four datasets. We compare the performance of DistDiff with the original real dataset and strong augmentation method AutoAug [9] with varying numbers of

---

Table Comparison of transformation-base based augmentation methods on Caltech- Our approach. combined with default augmentation (crop, flip, and rotate), consistently outperforms existing advanced transform-bann based methods and can be further improved by combining these techniques.
Default
AutoAug
RandAug
Random Erasing
GridMask
MixUp
CutMix
Original DistDiff
66.71 83.38
74.34
74.07
74.22
73.88
78.64
70.13
82.93
83.21
83.05
83.48
81.06
85.27
Cottech-101
CHARI Se-berrt
Staribedlaar
DTD
mided
Duall
Dulh
Nanber eCGanatied Samples
Narbox of Geacandd Sarples
Naatber Gereonad Sarpla
Natiber of Gewentdd Samplei
Figure 4: Performance comparison across different scale data sizes. Our method demonstrates significant improvements in classification model performance in both low-data and large-scale data scenarios, outperforming the transformation method AutoAug and the synthesized method Stable Diffusion 1.4.
generated examples, as depicted in Figure 4. As the data expansion scale increases, the corresponding improvement in accuracy also enlarges. The accuracy on Caltech- 101 achieved with our 5x expansion surpasses even the 20x expanded dataset obtained by AutoAug and diffusion baseline. This indicates that DistDiff exhibits superior efficiency in data expansion compared to existing methods.
Versatility to Various Architectures We conduct an in-depth assessment of the expanded datasets generated by DistDiffacross four distinct backhones: ResNet-50 [21]. ResNeXt-50 [66]. WideResNNt- 50 [75], and MobileNetv2 [56]. These backbones are trained from seratch on 5x expanded Caltech- Maltech-11ddee 101 by DistDiff. The results presented in Table affirm that our innovative methodol- ogy is effective and versatile across spectrum of architectures.
Comparison with Stronger Classification Models As we know, data expansion is typically applied in scenarios with data scarcity. However, if we use models pre-trained on large-scale datasets, the performance on the original training set can be significantly enhanced. In such cases, does our expanded dataset still offer improvements? To validate our method, we fine-tuned a ResNet-50 model pre-trained on ImageNet-Ik [11] for ImageNette. Caltech-101, and StanfordCars, and a LAION [57] pre-trained CLIP- ViT-B22 [6] model for PathMNIST. As shown in Table 2, the model achieved a high accuracy of 99.4% on ImageNette, which is a subset of its pretrained datasets. Further data expansion resulted in slight decrease in performance. Similarly, on the general image dataset Caltech- 101, which shares significant overlap with ImageNet data, our method demonstrated only slight improvement. However, on the more challenging fine -eeiigraindd dataset StanfordCars, our method demonstrated obvious 3.56% accuracy improvement. For the medical image dataset PathMNIST, which exhibits significantly different distribution, using DistDiff for data expansion effectively boosted classification performance by 5.1. This highlights the importance of scaling up data when transferring pre- pre-trained models to downstream tasks that exhibit significant distribution shifts.
Table 2: Comparison of using stronger pre- pre-trained baseline models. On ImageNette [28], Caltech- 101
[16), and StanfordCars [30] datasets, we employ an ImageNet- 1k [11] pre-trained ResNet-50 [21] model. For the PathMNIST [68] dataset, fine-tune using the stronger CIIP-VVT-B/32 baseline.
Dataset
ImageNette
Caltech-101
StanfordCars
PathMNIST
Original
99.40
96.87
87.61
84.29
Expanded 5x by SD Expanded 5x by DistDiff
98.51 (.89
96.91 (+0.04)
90.19 (+2.58)
86.81 (+2.22)
99.30
100
97.00 (+0.13)
91.17 (+3.56)
89.47 (+5.18)
Qualitative Analysis In addition to the quantitative experiment results, we also gain more intuitive understanding of the diverse changes facilitated by our method through visualization the generated guidance approach
results. As shown in Figure 5, the images generated using our distribution- exhibit high fidelity and diverse synthetic changes, including object texture, background, and color contrast. More visualization results can be found in Appendix D.3.
7

---

Original
Visualization of the proposed method
Original
Visualization the proposed method
Figure 5: The visualization of synthetic samples generated by our method, showcasing high fidelity, diversity, and alignment with the original data distribution.
4.4 Ablation Study
Hierarchical Prototypes We delved further into how each component of DistDiff impacts its data expansion performance. As depicted in Table 4, utilizing both Pe and Pg contributes to enhancing the model's expansion performance, showcasing their ability to optimize the generated sample distribution at the class-level and group-level, respectively. Moreover, combining P. and Ps. results in a further performance improvement, validating the effectiveness of integrating representations from different hierarchical levels. Additionally, with the introduction of our approach, the Fréchet Inception Distance (FID) values notably decrease, indicating that our proposed FID effectively optimizes the model to generate samples more aligned with the real distribution, thereby reducing the domain gap between the generated dataset and the real dataset.
Table Performance comparison of models trained on original Caltech- 101 datasets and 5x expanded datasets by DistDiff.
Table 4: Comparison of accuracy and FID in expanding 1altech-101 by 5x, with and without hierarchical prototypes in DistDiff.
Backbone
Original
DistDiff
Pe
Ps
Accuracy
FID-3K 4
ResNet-50 [21]
66.71
83,09
76.57 + 0.35
72.56
ResNeXt-50 [66]
67.60
83.75
/
82.70 + 0.07
68.82
WideResNet-50 [75]
66.51
83.51
82.84 0.54
68.66
MobileNetV2 [56]
74.39
83.85
83.09 + 0.11
67.72
Augmentation within Diffusion In the application of energy guidance, perturbation is introduced at the M-th step, and these subsequent predicted data points are optimized. Theoretically, optimizing predictions different stages yields distinct effects. Our exploration focused on the optimization step M, and the experimental results are illustrated in Table 5. When M is small, indicating optimization a later stage (i.e., the refinement stage), the change in generated results already minimal, resulting in relatively consistent optimization outcomes. Conversely, when M increases, corresponding to optimization at an intermediate stage (i.e.. the semantic stage), the generated results are in the stage of forming semantics and exhibit significant changes. Hence, this stage plays crucial role in determining the final generated results. Furthermore, there decline performance during the early chaos stage (M=25), the data points in this initial phase are too chaotic to establish an optimal target for optimization. We observed that achieving higher data expansion performance is possible when optimized in the semantic stage, with optimal results obtained when M=200
Table 5: Comparison of opti- mization in different phases.
Table 6: Ablation of the num- ber K of Ps in DistDiff
Table 7: Results of introduc- ing more optimization steps.
M
Accuracy
K
Accuracy
Step
Accuracy
1
81.54 + 0.32
2
82.69 + 0.51
1
82.54 + 0.54
10
82.21 + 0.22
3
83.09 + 0.11
2
82.94 + 0.43
20
82.36 + 0.05
4
83.08 + 0.13
3
82.77 + 0.19
25
82.11 + 0.55
5
83.08 = 0.21
4
82.55
+ 0.30
8

---

More Optimization Steps Furthermore, natural idea arises regarding the potential improvement effectiveness through the optimization of more optimization steps. Therefore, we further explored increasing the number of steps in the semantic stages. As shown in Table 7, increasing the number of optimization steps in semantic stages enhances performance. However, further increases in optimization steps lead to a decline in performance. This can be attributed to excessive optimization strength in energy guidance, which causes data distortion.
Compared with Direct Guidance on Latent Point We evaluate transform guidance strategy against an alternative strategy that directly guides the latent data points while ignoring the residual transform preprocess, method found useful in previous works [24, 73J. We initially conducted grid search for this alternative strategy to find the optimal learning rate, P (i.c., [0.1, 1, 10, 20]), and guide step, M (i.e., [1, 10, 20)). We found the best result of 76.77% was achieved with p 10 and M = 10, which is only slightly better than the original Stable Diffusion but lags behind our DistDiff, which achieved 83.19% 19%. This indicates that applying the residual multiplicative transformation the latent feature offers more optimization potential.
Determination of Group-Level Prototype Number K The determination of the number K of group-level prototypes is crucial for accurately approximating the real data distribution. In Table 6, we compare the outcomes associated with varying numbers of prototypes. The results highlight that the optimal number of prototypes found K = We posit that an insufficient number of prototypes may impede the characterization of the real distribution, leading to diminished performance. Conversely, points, also resulting in suboptimal performance. Furthermore, we present a visualization analysis of group-Ievel prototypes in Figure 6. The visual representation demonstrates that an appropriate number of group-level prototypes can effectively cover the real distribution space, aligning with the underlying motivation of DistDiff.
excessive number of prototypes may lead overfitting of noisy sample
(a)
(b) K 2
(c) =
(d) K 4
Figure 6: The visualization of group-level prototypes alongside original sample features. Here the sample point and is group-level prototype. By selecting appropriate number K, these prototypes effectively span the feature space, providing an approximation of the real data distribution.
Computational Efficiency Our DistDiff is not only wainiig-free but also highly efficient in processing. As illustrated Figure 2, DistDiff introduces only few optimization steps in the original diffusion process. We analyze the time costs of our methods. Stable Diffusion generates per sample 12.65 seconds on average, while our DistDiff achieves the same 13.13 seconds, with the increased time costs being negligible. We can conclude that DistDiff achieves a notable improvement over stable diffusion models, with only a slight increase in computation costs.
Conclusion
This paper presents DistDiff, a distribution-aware data expansion method employing stable diffusion model for data expansion. The proposed method optimizes the diffusion process to align the synthe- sized data distribution with the real data distribution. Specifically, DistDiff constructs hierarchical prototypes to effectively represent the real data distribution and refines intermediate features within the sampling process using energy guidance. We evaluate our method through extensive experiments on six datasets, showcasing its superior performance over existing methods.
6 Acknowledgments
This work supported by the National Natural Science Foundation of China under Grant 62072271.

---

References
[1] Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprint arXiv:171.00434 2017.
[2] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304.08466 2023.
[3] Hritik Bansal and Aditya Grover. Leaving reality imagination: Robust classification via generated datasets. arXiv preprint 23ii:23002.02033, 2023.
[4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint (tfi::1809.119900 2018.
[5] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Gridmask data augmentation. arXiv preprint arXiv:2001. 044006 2020.
[6] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.
June 2023.
[7] Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image teattt--mmgeppee generation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3043-30044 2023.
[8] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference computer vision and pottern recognition, pp. 16006-663,,20 2014.
[9] Ekin D Cubuk, Barret Zoph. Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data, arXiv preprint arXiv: 09500, 2018.
[10] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. pp. 702-703, 2020.
[11] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. Icce, 2009.
[12] Terrance De and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout, 2017.
[13] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[14] Lisa Dunlap, Alyssa Umino, Han Zhang, Jiezhi Yang. Joseph E Gonzalez, and Trevor Darrell, Diversify your vision datasets with automatic diffusion- based augmentation. arXiv preprint arXiv: 16289, 2023.
[15] Lisa Dunlap, Alyssa Umino, Han Zhang, Jiezhi Yang. Joseph E Gonzalez, and Trevor Darrell, Diversify your vision datasets with automatic diffusion-based augmentation. Advances in Neural Information Processing Systems, 36. 2024.
[16] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178-178. JEEE, 2004.
[17] Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zao. Diverse data augmentation with diffusions for effective test-time prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2704-2714, 2714, 2023.
[18] Ian Goodfellow. Jean Pouget-Abade Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozain, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1)): 139-144, 2020.
[19] Sven Gowal, Sylvestre- Rebufli, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A Mann. Improving robustness using generated data, Advances in Neural Information Processing Systems. 34:4218 4233, 2021.
10

---

[20] Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R Venkatesh Babu. Deligan: Generative adversarial networks for diverse and limited data. In Proceedings the IEEE conference on computer vision and puttern recognition, pp. 166-174, 2017.
[21] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
[22] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr. Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? arXiv preprint arXi::2110005574 2022.
[23] Dan Hendrycks, Norman Mu, Ekin Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv: MIXiv::912.02781 2019.
[24] Jonathan Ho and Tim Salimans. Classifier free diffusion guidance. arXiv preprint arXiv:2207.12590 2022.
[25] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole. Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models. 2022.
[26] Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Mach. Learn. Res., 23(1), jan 2022.
[27] Jeremy Howard. A smaller subset of 10 easily classified classes from imagenet, and a little more french, 2020. URL https:/ggithub. comffastaifmmggenttt.
[28] Jeremy Howard. A smaller subset of 10 easily classified classes from imagenet, and a little more french, 2019. URL https:// /ttp://gggttt/tt//e www//attta/iiaaaaaaaoaee imtittaaiimmmneeettte..
[29] Khawar Islam, Muhammad Zaigham Zaheer, Arif Mahmood, and Karthik Nandakumar. Diffusemix: Label- pabel-preserving data augmentation with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp.
2024.
[30] Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting large-scale dataset of fine-grained cars. 2013.
[31] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
[32] Bohan Li, Xinghao Wang, Xiao Xu, Yutai Hou, Yunlong Feng, Feng Wang, and Wanxiang Che. Semantic- guided image augmentation with pre-trained models. arXiv preprint arXiv:2302.000070, 2023.
[33] Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Krcis, Sanja Fidler, and Antonio Torralba. Bigdataset- gan: Synthesizing imagenet with pixel-wise annotations. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 21330-21340, 2022.
[34] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified Eision-language understanding and generation. International Conference on Machine Learning, pp. 12888-12900 PMLR, 2022.
[35] Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Zhang. Yanfeng Wang, and Weidi Xie. Guiding text-to-image diffusion model towards grounded generation. arXiv preprint arXiv:2301.0522) 3(6)77 2023.
[36] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14611-1477, 2022.
[37] Simian Luo, Yigin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference, 2023.
[38] Siwei Lyu. Deepfake detection: Current challenges and next steps. pp. 1 6, 2020.
[39] Giovanni Mariani. Florian Scheidegger, Roxana Istrate, Costas Bekas, and Cristiano Malossi. Bagan: Data augmentation with balancing gan. arXiv preprint arXiv:
2018.
[40] Mare Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van learning: survey and performance evaluation on image classification. IEEE
De Weijer. Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513-555 2022.
11

---

[41] Kangfu Mei and Vishal Patel. Vidm: Video implicit diffusion models. Proceedings of the AAAI Conference on Artificial Intelligence, 378))911779 Jun. 2023.
[42] Tachong Moon, Moonseok Choi, Gayoung Lee, Jung- Ha, and Juho Lee. Fine-tuning diffusion models with limited data. In NeurlPS 2022 Workshop on Score-Based Methods, 2022.
[43] Ranjita Naik and Besmira Nushi. Social biases through the text-to-image generation lens. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pp. 786-808, 2023.
[44] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.1074) 2021.
[45] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11, 2023.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021.
[47] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint ar arXiv:2204.06/25, 1(2)3, 2022.
[48] Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models, Advances in neural information processing systems, 32, 2019.
[49] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
[50] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2022.
[51] Candace Ross, Boris Katz, and Andrei Barbu. Measuring social biases in grounded vision and language embeddings. arXiv preprint arXiv:2002. 08911, 2020.
[52] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject- driven generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500 -20000,, 2023.
[53] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings. Association for Computing Machinery, 2022.
[54] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479- 774994, 2022.
[55] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4713-444 4726, 2023. doi: 10. 1.110//PAAM..22222244446.
[56] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510 45200 2018
[57] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis. Mitchell Wortsman, et al. Laion-Sb: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 15:25278-25294, 2022.
[58] Jordan Shipard, Amold Wiliem, Kien Nguyen Thanh, Wei Xiang, and Clinton Fookes. Diversity is definitely needed: Improving model-agnostic shoo-shot classification via stable diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 769-778, 2023.
12

---

[59] Connor Shorten and Taghi M Khoshgoftaar. A survey image data augmentation for deep learning. Journal big data, 6(1):1-48, 2019.
[60] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:0010002502, 2020.
[61] Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effecttive data augmentation with diffusion models. arXiv preprint 2.02.0:.300,079444 2023.
[62] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, Steven Liu, William Berman, Yiyi Xu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models, 2023. StRpsttt::// httppp://///thtb...//h com/hugginggfacee diffacerrr.ffff......
[63] Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 1206-1213 1217, 2023.
[64] Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefci Hc, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annotations using diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[65] Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. Difffit: Unlocking transferability of large diffusion models via simple parameter-efficien fine-tuning. arXiv preprint arXiv:2304.006 06648, 2023.
[66] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1492-1500, 2017.
[67] Austin Xu, Mariya Vasileva, Achal Dave, and Arjun Seshadri. Handsoff: Labeled dataset generation with no additional human annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7991-8000, 2023.
[68] Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023.
[69] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023.
[70] Ling Yang. Jingwei Liu, Shenda Hong, Zhilong Zhang. Zhilin Huang, Zheming Cai, Wentao Zhang, and Bin Cui. Improving diffusion-based image synthesis with context prediction. Advances in Neural Information Processing Systems, 36, 2024.
[71] Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui. Mastcring -xt-tt-imagee diffusion: Recaptioning, planning, and generating with multimodal llms. arXiv preprint arXiv:2401. 17008 2024.
[72] Ling Yang. Zhilong Zhang, Zhaochen Yu, Jingwei Liu, Minkai Xu, Stefano Ermon, and Bin CUL. Cross- modal contextualized diffusion models for text-guided visual generation and editing. In The Twelfth International Conference on Learning Representations, 2024. URL httpp::/ net/forum? id=nFM86uF2xq.
[73] Jiwen Yu, Yinhuai Wang. Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training-free free energy- guided conditional diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23174-23184, 2023.
[74] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEFF/CVF international conference on computer vision, pp. 6023-6032, 2019.
[75] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv: 2016.
[76] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz Paz. mixup: Beyond empirical risk minimization. arXiv preprint 4rXiv:1710.09412. 2017.
13

---

[77] Xinchen Zhang, Ling Yang. Yagi Cai, Zhaochen Yu, Jiake Xie, Ye Tian, Minkai Xu, Yong Tang, Yujiu Yang, and Bin Cui. Realcompo: Dynamic equilibrium between realism and compositionality improves text-to-image diffusion models. arXiv preprint MxXxx:2402.12008 2024.
[78] Yifan Zhang, Daquan Zhou, Bryan Hooi, Kai Wang, and Jiashi Feng. Expanding small-scale datasets with guided imagination. arXiv preprint arXiv:2211.1333 2022.
[79] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10145-10155,
2021.
[80] Bo Zhao and Hakan Bilen. Synthesizing informative training samples with gan. arXiv preprint arXiv:2204.07511 2022.
[81] Long Zhao, Ting Liu, Xi Peng. and Dimitris Metaxas. Maximum- eanumpy adversarial data augmentation for improved generalization and robustness. Advances in Neural Information Processing Systems, 33: 14435-1447, 2020.
[82] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random crasing data augmentation. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 15001-13008 2020.
[83] Yongchao Zhou. Hshmat Sahak. and Jimmy Ba. Training thin air: Improve image classification with generated data, arXiv preprint arXtv:2305. 15316, 2023.
14

---

A Limitations and Societal Impacts
Limitations Although DistDiff achieves better FID scores and enhances classifier performance without the need for training, it may incur additional computation time, which accumulates as data scales up. Additionally, our method requires an extra guide model, increasing development costs. Integrating fast sampler [37] and lightweight guide model represents a promising direction for maximizing the effectiveness of diffusion- baseed data expansion methods in classifier training.
Societal Impacts Generative models [47, 50] can significantly reduce the costs associated with manual data collection and annotation. Our approach builds an efficient method that enhances the distributional consistency of these generative models for downstream tasks without requiring training, thereby improving performance on downstream classifiers. This capability can assist organizations and researchers with limited data resources in developing more effective machine learning models.
However, since generative models are pre-trained on vast, diverse vision-language datasets from the internet, these data may contain social biases and stereotypes [7, 43, 51 leading to discriminatory generated outputs. Therefore, integrating mechanisms Nevertheless, our method guides the generation process decisions based on task-specific hierarchical prototypes, fostering AI systems that better align with downstream task distributions and exhibit fewer biases.
detect and mitigate biases is crucial.
Another potential concern is the misuse of generated data, which could be exploited for malicious purposes such as deepfakes [38], resulting in misinformation dissemination and adverse societal impacts. Proper constraints the proliferation and correct usage of generative models crucial. This necessitates the establishment of relevant regulations and guidelines to ensure the responsible development and utilization of synthetic data models.
B
More Implementation Details
B.1
Datasets
Table provides the detailed statisties of six experimental datasets, including Caltech-101 [16], CIFAR 100-Subset [31]. StandardCars [30]. ImageNette [28]. DTD [8]. and PathMNIST(68). (00-bbttt and PathMNIST subsets that randomly sampled from the original trainset with 100 samples per class. Specifically, our evaluation encompasses datasets that span diverse scenarios, incorporating generic objects, fine-grained categories, and textural images. These diverse datasets enable a comprchensive assessment of both the robustness and effectiveness of our proposed methods. Additionally, we also provide the text template employed the conditional diffusion process in Table 9,
Table 8: Summary of our six experimental datasets.
NAME
CLASSES
SIZE (TRAIN TEST)
DESCRIPTION
CALTECH- 101
100
3000 6085
RECOGNITION OF GENERIC OBIECTS
CIFAR 100-SUBEET
100
10000 / 10000
RECOGNITION OF GENERIC OBJECTS FINE-GRAINED CLASSIFICATION OF CARS RECOGNITION OF GENERIC OBJECTS TEXTURE CLASSIFICATION RECOGNITION OF COLON PATHOLOGY IMAGE
STANDARDCARS
196
8144 / 8041
IMAGENETTE
10
9469 / 3925
DTD
47
3760 / 1880
PATHMNIST
9
900 7180
B.2 Our Method
We implement our methods using the PyTorch framework with Python 3.10.6. We utilize the diffuser [62] to implement our base diffusion models. For all datasets, we generate images with noise strength of 0.5, except for the medical image dataset PathMNIST, where we use a noise strength of 0.2. The guidance step M in our method is 20 by default, except for 10 for PathMNIST. The half- half-precision floating-point used in the generation process to reduce memory costs. The generation, training, and evaluation processes are conducted a single GeForce RTX 3090 GPU. We report
15

---

Table 9: Text templates for six experimental datasets.
NAME
TEMPLATE
CALTECH- 101
"A PHOTO OF [CLASS]."
CIFAR S00-SU
"A PHOTO OF A [CLASS]."
STANDARDCARS
"A PHOTO OF A [CLASS] CAR."
IMAGENETTE
"A PHOTO OF [CLASS]."
DTD
"[CLASSS] TEXTURE."
PATHMNIST
COLON PATHOLOGICAL IMAGE OF [CLASS]."
the best test accuracy averaged over three runs and calculate the FID metric using 3000 samples. In order to reduce generation time costs, we pre-compute and save the latent embeddings for all training samples. We use Stable Diffusion 1.4 as our base generation model". We follow previous data expansion methods by expanding each sample certain ratios. The sampling category distribution consistent with the original dataset category distribution.
B.3 Synthesis-Based Augmentation Contenders
Due to benchmark and training setting differences in existing works [78, 33), in order fairly compare existing methods with ours, we reproduced two statc-of-the-an generation- methods: GIF-SD [78] and LECF [2] on our benchmarks and base diffusion model. For GIF-SD, we used pretrained ViT-B32 model to facilitate its class-maintained optmization, and a batch size of facilitate its KL--divergence based diverse sampling. For LECF, we generated 200 prompts for each class prompt, and generated samples with sizes equal to our method's, filtering these samples with specified threshold using pretrained ResNet50-CLL CLII model. As shown Section 4.3, our methods surpass these contenders.
B.4 Transformation-Ban -aseed Augmentation Contenders
We outline the transformation-basse based augmentation methods compared in our experiments. The effectiveness of all these methods are evaluated the same datasets and with the same configuration (e.g., learning rate, epochs, batch size, etc.) as our method.
Default: We random crop, random horizontal flip, and random rotation with 15-degree angle as default augmentation strategics.
AutoAug [9]: We employ the AutoAugment function in PyTorch with the widely used ImageNet policy, which includes a diverse range of color and shape transformations, During training, one transformation strategy is randomly selected and applied.
RandAug [10]: Similar AutoAug, we randomly choose two operations from predefined policy list and apply them to the training samples.
Random Erasing [82]: We use the random erase function in PyTorch for Random Erasing. This function randomly selects a rectangular region within an image and erases its pixels with a 50% probability. The proportion of crased area relative to the input image ranges from 0.02 to 0.33, and the aspect ratio of the crased area ranges from 0.3 to 3.3.
GridMask [5]: GridMask is implemented with its official configuration. The probability of applying GridMask linearly increases from 0.8 as training epochs progress up to the 80th epoch, after which it remains constant until 100 epochs.
MixUp [76]: Synthetic interpolated images are generated within each data batch. We sample the interpolation strength from a beta distribution (beta = 1). The loss function is also adjusted accordingly.
CutMix [74]: Similar to MixUp, CutMix replaces specified regions original images with input from another image. The loss function is modified accordingly.
'We use the pretrained weights CCompWii/stableddiffuiiooooo from Hugging Face. https:// huggingface. co wttp:///wssaaaeliifftuuuiie
16

---

C
Pseudocode
We present the pseudocode for our algorithm in Algorithm 1, illustrating the hierarchical energy guidance within the diffusion sampling process.
Algorithm Optimization Process of our proposed DistDiff
Input: Hierarchical prototypes Pe and Pgj Latent data point žy which is encoded from the sample x; Optimization step M; Pre-defined parameters 3,; Perturbation constraint Pre-trained feature extractor 0; Denoising network 15.
for 1. = T 1,.,,.... .....,.0 do
~ (1, I) i if 0, else 6 =0 2; = (1+ 1(41) + 1441 + + 1) + v 8
if = .M then
Initialize ~ ((, 1), b, ~ (0.
2: ( +
201: = (z: Vl-a 2t, 1)) Voi
Dd Solt: Pc) = [10(zolt) Pell2
De Zoje: = [10(z0jt) - Pxl12
= Dg 20 Pc) ) + Dij (0 Pg)
update e', b' + arg min . (Equation 6)
update 21 +- 2 = -P,c(( (1 + =2))2 b))
end if end for Output: Z0.
D More Experimental Results
D.1 Model Performance
DistDiff is Robust to Guidance Model As mentioned in Section 3.2, employ extra feature extractor as our guidance model. To assess the impact of different guidance models, we compared two backbones: ResNet-50 [21] trained the original dataset from scratch (weak backbone) and CLIP [6]. strong backbone pre-trained on large datasets and fine-tuned on the original dataset. Table 10 presents the accuracy of the guidance models and the corresponding tuned classifier (ResNet-50). Our DistDiff method demonstrates robustness, showing negligible changes in accuracy (0.25%) across different guidance models. This highlights the robustness of approach. To ensure fairness in comparison without knowledge leakage from large pre-traines models, we default to using a randomly initialized ResNet-50 trained on the original dataset as guidance model.
Table 10: Comparison of guidance models Caltech- 101 dataset. We compared the accuracy of two guidance models on the original Caltech- 1011 dataset. Additionally, we evaluated the performance of downstream classifier trained the 5x expanded dataset using corresponding guide model.
Guide Model
Accuracy (%)
Guide Model
Downstream Classifier
Weak (Random initialized and trained ResNet) Strong (Pre-trained and finetuned CLIP)
66.71 + 0.47
82.94 + 0.43
92.24 + 0.25
83.19 + 0.69
Determination of Guidance Scale p. We compared different learning rates p. and the experimental results are shown Table 11. A learning rate that too low results insufficient optimization. Conversely, learning rate that is too high causes over-optimization, leading to image distortion. Both result in suboptimal performance. We used p 10, which achieves the best results, as the default learning rate.
17

---

Table 11: Comparison of different learning rate p.
p
0.1
1.0
10.0
20.0
Accuracy (%)
82.49 0.33
82.74 = 0.32 83.09 + 0.11 82.46 + 0.35
Comparison of Varying Gradient Weight. We compared different contribution levels of two hierarchical prototypes. Specifically, we scaled the gradient of Pa by certain coefficient, Ag. The results in Table 12 indicate that appropriate scaling weight can further enhance overall performance.
Table 12:
Comparison of different gradient weights Ag.
X
0.1
0.3
0.5
0.7
0.9
1.0
2.0
Accuracy (%)
82.61
82.79
83.14
83.12
83.38
83.09
82.73
More Efficiency Comparison The high efficiency of DistDiff stems from its direct guidance on the pre-trained model without the need retrain the diffusion models. We provide computational time comparison in Table 13, Compared to the stable diffusion baseline model, only introduce an extra 0.48 seconds of inference time per sample, which is only 4% of the original inference time. Our method incurs minimal additional time costs and generates samples directly, unlike LECF, which needs extra post-pri prosesronsssing to filter low-confidence samples. This makes LECF slower in generating the same number of samples. We are incorporating this cost analysis into the paper.
Table 13:
Inference Efficiency comparison with existing methods on Caltech-101 dataset. * denotes that the actual time required of LECF to derive one sample after filter post-processing Evaluation processes are conducted on a single GeForce RTX 3090 GPU.
Method
SD
LECF
GID-SD
Ours
Inference Time (s) 12.65
12.73 (17.20 *)
13.47
13.13
More Baseline Comparison We add two more baseline: DA-Fusion [61] and DiffuseMix [29], for comparison. As shown in Table 14. Our method outperforms both baselines. DA-Fusion modifies images while respecting semantic attributes but lacks fine-grained control and incurs additional time costs (around 14 minutes per class on 3090 GPU) Our training- frining-free method uses hierarchical guidance for better alignment with the real distribution. DiffuseMix, which uses bespoke prompts and fractal images for augmentation, treats all datasets equally and may not handle distribution shifts well. Our method shows superior performance compared to these approaches.
More Ablation Experiments We add more ablation experiments StanfordCars dataset. Our method shows more benefits with fine-grained datasets with greater class variance, as illustrated in Table 15. Additionally, having more groups is not always beneficial; excessive number of groups may cause prototypes to fit noise points or outliers, which could degrade performance.
DiftDiff is Effective Large-Scale Dataset In this work, follow previous work, such as GIF-SD and DA-Fusion, in conducting experiments on small datasets, as data augmentation strategies often necessary in data-scarcity scenarios. Besides, we also conducted experiments ImageNet with 0.2x expanding ratios, resulting in approximately 256K generated samples. As shown in Table 16, our method demonstrates improvements on large-scale datasets.
D.2 Further Analysis
Trade-Off Between Fidelity and Diversity The data expansion task requires both high fidelity and diversity for effective model training. However, this principle does not universally apply across all scenarios. We assessed the trade-off between high fidelity and diversity by adjusting the diffusion model's s strength in adding noise to original images. As the noise strength increases, the diversity of generated data enhances, accompanied by decrease in FID scores. The resulting accuracy and FID scores are presented in Figure 7, where we observed an inverse relationship between FID (fidelity metric) and accuracy across the Caltech- 101 and PathMNIST datasets. We found that introducing
18

---

Table 14: Comparison of accuracy in expanding Caltech- 101 by 5x.
Method
DA-Fusion
DiffuseMix
Ours
Accuracy (%)
79.14
82.33
83.09
Table 15: Prototypes comparison of accuracy in expanding StanfordCars by 2x. We trained ResNet50 with 448 x 448 resolution for 128 epochs.
Method
SD
Pe
+ Pg(K - 1)
+ Pg Py(K - 2) + Pp (K 3)
Pa + +P 4)
Accuracy 88.45
89.55
89.69
90.36
90.69
90.62
Table 16: Comparison of accuracy in expanding ImageNet by 0.2x. We trained ResNetl with
224 224 resolution for 90 epochs.
Method
Original
SD
Ours
Accuracy (s)
69.30
69.66
69.95
more new content on general datasets, such as Caltech- 101, can benefit model training by emphasizing the need for diversity. In contrast, on medical datasets with substantial distribution shifts, maintaining the original data distribution and minimizing disturbances proves to be more effective.
D.3 More Visualization Results
In this section, we provide visualization comparison between original stable diffusion (SD) and our DistDiff in Figure 8. Visualization results indicate that our method exhibits finer texture details, more diverse style variations, and stronger foreground--ackkrroudd contrasts, making our samples align real sample distributions. However, the main content of our images still shows minimal deviation from SD, making challenging for users to discern differences through visual observation alone. In Section 4.4, we provide quantitative analysis to validate that our method generates data with more consistent distributions, thereby enhancing performance downstream classification tasks.
In addition, we visualize more generated samples across six datasets in Figure 9 to demonstrate the effectiveness of DistDiff. These visualizations confirm that our approach can generate samples with more distribution-consistent patterns.
19

---

FTD PathMNIST
50
100
150
200
250
Callech
ParhMmist
Accuracy
Netse Strength Incresses
70
FID Caltech
Figure 7: Comparison with FID and accuracy varying noise strengths.
SD-14
DistDiff
SD-1.4
DistDiff
Figure 8: Comparison of visualizations between original Stable Diffusion 1.4 and our DistDiff.
20

---

Caltech-101
1 000-Su
StanfordCars
DTD
StanfordCars
PathMNIST
Figure Visualization of synthetic images produced by our method.
21

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract provides a concise summary of the main findings and contributions of the paper, while the introduction elaborates the problem statement and research objectives, thereby clarifying the contributions.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Appendix A, we expound upon the limitations of work conducted and provide brief discussion thereof.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
22

---

Answer: [Yes]
Justification: In Section and Appendix C, we provide detailed mathematical derivations for all the formulas appearing in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 4.1, we introduced the details of experimental setup and model training to ensure reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
23

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide open access the data and code, with sufficient instructions faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm aublic/guides/Codeee
iereinisssoPooli onPolicy icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nnpss wwtppcccccc////ccccccccccccc..lt wttpc//gg//eeeeeeecceoeeccsseees
ietmiisssonnn ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4.1, we introduced the details of experimental setup and model training, and conducted ablation experiments in Section 4.4 to elucidate the selection of hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars of our experimental results in Section 4.3 and 4.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: For the experiments, we furnished detailed specifications of the GPU models used along with their corresponding tasks. Furthermore, included specific information regarding the model training batch size and the number of training epochs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : htt////eerrp..p/ee httppulic//thiisuiii..eo
Answer: [Yes]
Justification: principles.
We have carefully reviewed the NeurIPS Code of Ethics and adhere to its
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative societal impacts of the work performed in Appendix A.
25

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., denloymemt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators original owners of assets, such code, data, models, used in the paper, are properly credited. Additionally, the license and terms of use associated with these assets explicitly mentioned and respected in accordance with ethical and legal standards.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
26

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi wwwrssssthhhdeee thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing research with human subjects.
Guidelines:
27

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "UQOB48UPHt", "reviewer": "Reviewer_ghiL", "summary": "This paper focuses on data augmentation or expansion by generating synthetic data from pre-trained large-scale diffusion models. To ground the samples from these large-scale diffusion models, the paper proposes an energy-based guidance approach where the energy function depends on hierarchical prototypes. In the paper, Hierarchical prototypes are essentially feature vectors that define the object classes. The hierarchical prototypes are obtained as follows - First, the features for each class are aggregated to obtain a class level representation. Further, the features of each class are clustered into K clusters to obtain sub-group representations within each class. \n\nThe paper shows clear improvement for classification tasks on many standard datasets, when classifiers are trained from scratch on the augmented dataset.", "questions": "Can the authors please address the questions 2 and 3?\n\nAdditionally, how is the distribution across classes chosen for the synthetic data generation process? Are all classes equally sampled? I couldn't find this information in the paper.", "limitations": "Yes, the authors have addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses a very important problem - How to perform effective data augmentation using synthetic data generation models. The approach does not require any training or fine-tuning of the diffusion models to adapt the generation to the required data distribution. The paper has detailed ablation studies for each design choices. Also, the experimental results suggest considerable improvement over the prior data expansion approaches.", "weaknesses": "1. There seems to be some confusion in the explanation. Section 3.3 (Transform Data Points) seems to suggest that the approach always starts with a sample from the dataset. However, the algorithm in the appendix does not include any sample from the dataset.\n\n2. Is the approach extendable to other supervised learning tasks like segmentation, detection, etc? It looks to me like the augmentation approach is tailored for classification tasks only, with the usage of class-specific hierarchical prototypes. Whereas, traditional augmentation approaches like random cropping, rotation, etc., are generic.\n\n3. One of the main contributions of the paper is the residual multiplicative transformation. The paper does not give a clear answer to \"why should I not adjust the latent $z_t$ directly to optimize for the energy function $g_t$?\" The paper shows empirical explanations in the ablation, but there is a lack of concrete reasoning as to why this approach works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude for the detailed and professional attention you have given to our work during the review process. We greatly appreciate your recognition of the **very important problem** our work addresses, as well as your acknowledgment of our **detailed ablation studies** and the **considerable improvements** made.\n\nBelow are our detailed responses to the weaknesses and questions:\n\n**Re: Weakness #1**  \nIn Section 3.3, we introduce our method starting with a sample $x$. However, in the algorithm presented in the appendix, we begin with the latent point  $z_T$, which omits the image encoding process from $x$ to $z_T$ for simplicity. Thank you for pointing this out. We recognize that this may cause confusion and will revise it in the final version.\n\n**Re: Weakness #2**  \nOur method effectively generates classification data and has the potential to be extended to detection and segmentation tasks. This extension requires incorporating more advanced foundational models, such as ControlNet [1], which can use layout maps or segmentation masks as conditions to control the spatial positioning of targets. An intuitive approach to extend our method might involve cropping features from relevant target regions, averaging them to obtain feature vectors, and then constructing prototypes to guide the generation process.\n\n\n**Re: Weakness #3**  \nThe residual multiplicative transformation applies channel-level transformation to the original latent point. Compared to directly optimizing the latent point, this transformation constrains the optimization space, preventing out-of-control guidance and making the optimization process easier.\n\n**Re: Question #1**  \nWe apologize for missing this detail. We follow previous data expansion methods by expanding each sample certain ratios. The sampling category distribution is consistent with the original dataset category distribution. We will add an description in the final version.\n\n[1] Adding Conditional Control to Text-to-Image Diffusion Models, ICCV2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xbLt9VVs3H", "reviewer": "Reviewer_4Bhc", "summary": "The authors present DistDiff, a training-free data expansion framework based on a distribution-aware diffusion model. DistDiff constructs hierarchical prototypes to approximate the real data distribution, optimizing latent data points within diffusion models through hierarchical energy guidance. The framework demonstrates its capability to generate distribution-consistent samples, significantly improving data expansion tasks. DistDiff consistently enhances accuracy across a diverse range of datasets compared to models trained solely on original data.", "questions": "1.\tIn Figure 6, it is unclear why the optimal number of prototypes is set at K=3, as the difference from the figure is not readily apparent and the quantitative metrics only differ by 0.01. Could this be further elaborated to justify the selection of K=3 as the optimal prototype count?\n\n2.\tIt is suggested that the captions be further elaborated to explain the entire pipeline process in detail. Currently, the process terms used in the caption do not correspond directly to those labeled in the figure, leading to potential confusion. A clearer alignment between the text and graphical elements would improve comprehension and the overall effectiveness of the figure.\n\n3.\tIt is recommended to revise Figure 1. While it depicts the complex process of DistDiff, it does not provide clear benefits. Including performance comparisons or showcasing significant differences in generated results would enhance the figure's utility and informative value.", "limitations": "As stated above.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors claim high efficiency for DistDiff.\n2. The framework effectively generates samples that align with the original distribution, markedly enhancing data augmentation tasks.\n3. DistDiff consistently improves accuracy across a wide variety of datasets when compared to models trained only on the original data.", "weaknesses": "1.\tThe manuscript introduces the concept of hierarchical prototypes but falls short in sufficiently explaining how these prototypes are selected or generated from the dataset. A more detailed description or examples of the prototype generation process would greatly enhance the reader’s understanding and bolster the credibility of the proposed method.\n\n2.\tThe authors claim high efficiency for DistDiff but do not provide supporting data. It is recommended to add a comparison table detailing DistDiff's computational time, resource usage, and scalability against other methods, to substantiate its efficiency claims.\n\n3.\tThe paper claims that \" these two scores reinforce each other and are indispensable mutually,\" in Table 4, the observed impact appears minimal without P_g. Additionally, the k value for P_g seems to have little effect on the results. Further explanation and clarification are needed to substantiate these claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort you invested in reviewing our paper. \nWe address the raised concerns as follows:\n\n**Re Weakness #1:**\n\nThank you for your valuable feedback. We appreciate your suggestion to provide a clearer explanation of the hierarchical prototypes.\nWe recognize that the manuscript would benefit from a more detailed description of the prototype generation process. In Lines 135-136 and Lines 139-141, we describe how prototypes are selected or generated from the dataset. Specifically, our method first extracts feature vectors using a pre-trained image feature extractor. Class-level prototypes are then obtained by averaging feature vectors within each class. To refine these prototypes further, we apply the agglomerative hierarchical clustering algorithm [1] to group samples from the same class into $K$ clusters. The group prototypes are computed by averaging feature vectors within each cluster.\n\nIn the final version of the manuscript, we will include a more comprehensive explanation of the prototype generation process and provide illustrative examples to clarify this methodology further.\n\n\n**Re Weakness #2:**\n\nThe high efficiency of DistDiff stems from its direct guidance on the pre-trained model without the need to retrain the diffusion models. We provide computational time in Lines 334-339 and resource usage in Line 617. Compared to the stable diffusion baseline model, we only introduce an extra 0.48 seconds of inference time per sample, which is only 4% of the original inference time.\n\n We included an efficiency analysis in _Table 1_. Our method incurs minimal additional time costs and generates samples directly, unlike LECF, which needs extra post-processing to filter low-confidence samples. This makes LECF slower in generating the same number of samples. We are incorporating this cost analysis into the paper.  \n| Method   | Inference Time (s)    |\n|----------|-----------------------|\n| SD       | 12.65                 |\n| LECF     | 12.73 (17.20 \\*)      |\n| GID-SD   | 13.47                 |\n| Ours     | 13.13                 |\n\n\n*Table 1:* Inference Efficiency comparison with existing methods on Caltech-101 dataset.  \\* denots that the actual time required of LECF to derive one sample after filter post-processing. Evaluation processes are conducted on a single GeForce RTX 3090 GPU. \n\n\n**Re Weakness #3 and Question #1:**\n\nAs shown in Table 4, both scores significantly improve performance, and further improvements can be achieved by combining them. However, we did not provide a clear analysis of their relationship, and the claim that these two scores are \"indispensable mutually\" lacks sufficient evidence and may not be appropriate. This is primarily due to the small class variance in the Caltech-101 dataset, where the differences between $p_c$ and $p_g$ are minimal. In contrast, when we applied hierarchical prototypes to a dataset with larger class variance, such as StanfordCars, the differences between hierarchical prototypes were more pronounced. This led to a more significant combined improvement, as shown in  _Table 2_.\n\nAdditionally, we selected $K=3$ as the optimal prototype since it shows the best quantitative performance, although changes in the $K$ value are not apparent in the Table 6 of the manuscript.  The influence of $K$ is more pronounced for fine-grained StanfordCars dataset, highlighting the importance of $p_g$, as shown in _Table 2_.\n\n| Method | Accuracy |\n|------------|----------|\n| Stable Diffusion Baseline | 88.45    |\n| $p_c$  | 89.55    |\n| $p_c + p_g$ $(K=1) $       | 89.69    |\n| $p_c + p_g$ $(K=2) $       | 90.36    |\n| $p_c + p_g$ $(K=3) $       | 90.69    |\n| $p_c + p_g$ $(K=4) $       | 90.62    |\n\n*Table 2:* Prototypes comparison of accuracy in expanding StanfordCars 2$\\times$. We trained ResNet50 with a 448 × 448 resolution for 128 epochs.\n\n**Re Question #2:**\n\nThank you for the valuable suggestion. We will revise the captions to provide a more detailed explanation of the entire pipeline process and ensure that the terms used in the captions align clearly with those labeled in the figure. This will help improve comprehension and the overall effectiveness of the figure.\n\n**Re Question #3:**\n\nWe appreciate your insightful recommendation. We will revise Figure 1 to include performance comparisons and highlight significant differences in the generated results. This will enhance the figure's utility and informative value, making the complex process of DistDiff clearer and more impactful.\n\n[1] Hierarchical clustering. Introduction to HPC with MPI for Data Science 2016.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pKWUyczVsE", "reviewer": "Reviewer_EsYJ", "summary": "This paper proposed a training-free data augmentation method based on the diffusion models. To alleviate the poison phenomenon of the diffusion model, which is that the distribution of generated images will deviate from the natural distribution, this paper proposed a simple yet effective method based on the prototypes to introduce additional constraints to alleviate the deviation. The experimental results show that the proposed method achieves the SOTA performance compared to the baselines in this paper.", "questions": "1. How to keep such a low time cost for the proposed method? The motivation for this question is that the author claims: \"Stable Diffusion generates per sample in 12.65 seconds on average, while our DistDiff achieves the same in 13.13 seconds.\" The weakness of the guidance method (i.e., the energy function guidance used in this paper) is that it will dramatically increase the time cost, especially for a stable diffusion model. The reason is that the guidance method needs to calculate the gradient of the diffusion model. Concretely, please see Eq. 6. Eq. 6 needs first to calculate $\\nabla_{e}D_{\\theta}^{c}(z_{0|t},p_{c})$. $\\nabla_{e}D_{\\theta}^{c}(z_{0|t},p_{c}) = \\frac{\\partial D_{\\theta}^{c}(z_{0|t},p_{c})}{\\partial z_{0|t}}\\frac{\\partial z_{0|t}}{\\partial e}$. By Eq. 5 and Algorithm 1, $\\frac{\\partial z_{0|t}}{\\partial e} = \\frac{\\partial z_{0|t}}{\\partial z_{t}}\\frac{\\partial z_{t}}{\\partial e}$. In this condition, $\\frac{\\partial z_{0|t}}{\\partial z_{t}}$ needs to calculate $\\frac{\\partial \\psi}{\\partial z_{t}}$ and $\\psi$ is the pre-trained diffusion model. Therefore, the overall process needs to calculate the gradient of the diffusion model at least 50 times, while the time cost only increases by less than 1 second, which is hard to understand. \n\n2. Could the author offer the ablation study for K=1 that only one group situation? The motivation for this question is that Table 6 shows that the increase of K has a limited influence on performance, which is confusing. In theory, following the storyline of this paper, more groups should increase the diversity of the generated images, which should have a positive influence. Based on the Table 6, it seems that group strategy is redundant.", "limitations": "No additional limitations, including societal impact, should be discussed. All my concerns are listed in Weaknesses and Questions. If the author could clarify these concerns, I'm willing to increase my score.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Leveraging the prototypes as the constraint to guide the generation process is novel. Directly using the diffusion model to enrich the dataset will cause the poison phenomenon that will harm the classifier's performance since the distribution of generated images deviates from the natural image distribution. In this case, this paper leverages the cluster-based method, i.e., the prototype technology is sound and interesting.\n\n2. This paper is easy to follow.\n\n3. The proposed method achieves the SOTA performance compared with various baselines.", "weaknesses": "1. The baselines lack the latest works, such as Brandon et al.[1] and Khawar et al [2]. For Brandon et al., although this method needs to fine-tune the pre-trained diffusion models, the time cost for fine-tuning them is very low and is close to the time cost reported in this paper. Under the closed computation cost, it should be considered the baseline.\n\n[1] Effective Data Augmentation With Diffusion Models. Brandon et al.. ICLR 2024.\n\n[2] DIFFUSEMIX: Label-Preserving Data Augmentation with Diffusion Models. Khawar et al. Arxiv:2405.14881.\n\n2. The datasets used in this paper may not be enough. Since this paper does not only focus on small datasets, ImageNet should be considered the dataset to test the performance of the proposed method, which is similar to DIFFUSEMIX [2]. ImageNette is insufficient to replace ImageNet since it only contains 10 classes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort you have dedicated to reviewing our paper. We value your feedback on our method, noting that the **prototype technology is sound and interesting** and that it is **easy to follow.** We also thank you for recognizing that our method **achieves SOTA performance**  compared to various baselines.\n\n\n\n**Re Weakness #1:**\n\nWe apologize for missing this comparison. We have now included the baseline results in _Table 1_ and will update the revised paper accordingly. Our method outperforms both baselines. DA-Fusion modifies images while respecting semantic attributes but lacks fine-grained control and incurs additional time costs (around 14 minutes per class on a 3090 GPU) . Our training-free method uses hierarchical guidance for better alignment with the real distribution. DiffuseMix, which uses bespoke prompts and fractal images for augmentation, treats all datasets equally and may not handle distribution shifts well. Our method shows superior performance compared to these approaches.\n\n| Method      | Accuracy |\n|-------------|----------|\n| DA-Fusion   | 79.14    |\n| DiffuseMix  | 82.33    |\n| Ours        | 83.09    |\n\n*Table 1:* Comparison of accuracy in expanding Caltech-101 by $5\\times$.\n\n**Re Weakness #2:**\n\nWe follow previous work, such as GIF-SD and DA-Fusion, in conducting experiments on small datasets, as data augmentation strategies are often necessary in data-scarcity scenarios. Taking your advice into account, we have included results on ImageNet with $0.2\\times$ expanding ratios, resulting in approximately 256K generated samples. As shown in Table 2, our method demonstrates improvements on large-scale datasets.\n\n| Method   | Accuracy |\n|----------|----------|\n| Original | 69.30 |\n| Ours     | 69.95 |\n\n*Table 2:* Comparison of accuracy in expanding ImageNet by $0.2\\times$. We trained ResNet18 with a $224 × 224$ resolution for 90 epochs.\n\n**Re Question #1:**\n\nYou might have some misunderstanding regarding our method's principles. The energy function guidance used in our paper does not dramatically increase the time cost. As illustrated in Lines 310-315 (_More Optimization Steps_) of the manuscript, our method only introduces two additional optimization steps, which means calculating the gradient **2 times rather than 50 times**. Therefore, our method **does not dramatically increase the time cost** as you suggested.\n\n\n\n**Re Question #2:**\n\nWhen $K=1$,  the group-level prototype becomes the class-level prototype, leading to results similar to those with class-level prototypes. In Table 6 of our manuscript, performance changes are not apparent, as the differences between $p_c$ and $p_g$ are minimal in the small class variance dataset Caltech-101.  Our method shows more benefits with fine-grained datasets with greater class variance, as illustrated in _Table 3_ with StanfordCars. Additionally, having more groups is not always beneficial; an excessive number of groups may cause prototypes to fit noise points or outliers, which could degrade performance.\n\n| Method | Accuracy |\n|------------|----------|\n| Stable Diffusion Baseline | 88.45    |\n| $p_c$  | 89.55    |\n| $p_c + p_g$ $(K=1) $       | 89.69    |\n| $p_c + p_g$ $(K=2) $       | 90.36    |\n| $p_c + p_g$ $(K=3) $       | 90.69    |\n| $p_c + p_g$ $(K=4) $       | 90.62    |\n\n*Table 3:* Prototypes comparison of accuracy in expanding StanfordCars by $2\\times$. We trained ResNet50 with a 448 × 448 resolution for 128 epochs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work is focussed on efficiently utilizing diffusion models for data augmentation with an aim to be useful primarily for visual classification tasks. The strength of the work lies in being able to utilize pre-trained diffusion models and the minimal additional latency required at runtime as well as the novel formulation of the algorithm. I recommend acceptance based on the contribution as well as the discussions happened during the rebuttal window.
Decision: Accept (poster)