Real-Fake: Effective Training Data Synthesis Through Distribution Matching
Authors: Jianhao Yuan
Jie Zhang
Shuyang Sun
Philip Torr
Bo Zhao
Venue: ICLR 2024 poster (2024)
Keywords: Training Data Synthesis
Distribution Matching
Data Efficiency
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Synthetic training data has gained prominence in numerous learning tasks and scenarios, offering advantages such as dataset augmentation, generalization evaluation, and privacy preservation. Despite these benefits, the efficiency of synthetic data generated by
current methodologies remains inferior when training advanced deep models exclusively, limiting its practical utility. To address this challenge, we analyze the principles underlying training data synthesis for supervised learning and elucidate a principled theoretical framework from the distribution-matching perspective that explicates the mechanisms governing synthesis efficacy. Through extensive experiments, we demonstrate the effectiveness of our synthetic data across diverse image classification tasks, both as a replacement for and augmentation to real datasets, while also benefits such as out-of-distribution generalization, privacy preservation, and scalability. Specifically, we achieve 70.9% top1 classification accuracy on ImageNet1K when training solely with synthetic data equivalent
to 1 × the original real data size, which increases to 76.0% when scaling up to 10 × synthetic data.

Published as a conference paper at ICLR 2024
REAL-FAKE: EFFECTIVE TRAINING DATA SYNTHESIS THROUGH DISTRIBUTION MATCHING
Jianhao Yuan Jie Zhang Shuyang Sun Philip Torr Bo Zhao
University of Oxford t ETH Zurich Beijjng Academy of Artificial Intelligence
ABSTRACT
Synthetic training data has gained prominence in numerous learning tasks and scenarios, offering advantages such as dataset augmentation, generalization evalua- tion, and privacy preservation. Despite these benefits, the efficiency of synthetic data generated by current methodologies remains inferior when training ad advanced deep models exclusively, limiting its practical utility. To address this challenge, we analyze the principles underlying training data synthesis for supervised learning and elucidate a principled theoretical framework from the distribution-matching perspective that explicates the mechanisms governing synthesis efficacy. Through extensive experiments, we demonstrate the effectiveness of our synthetic data across diverse image classification tasks, both as a replacement for and augmenta- tion to real datasets, while also benefits such as distr--distiiittion generalization, privacy preservation, and scalability. Specifically, cation accuracy on ImageNet1 K when training solely with synthetic data equivalent x the original real data size, which increases 76.0% when scaling up 10 x synthetic data
achieve 70.9% top1 classifi-
INTRODUCTION
Large-scale annotated datasets (Deng al., 2009; Lin et al., 2014) are essential in deep learning for image recognition, providing the comprehensive information that models need to effectively discover patterns, learn representations, and generate accurate predictions. However, manually collecting such datasets is time-consuming and labor-intensive, and may cause privacy concerns. Given these challenges, training data synthesis offers a promising alternative to traditional data collection for augmenting or replacing original real datasets.
Among numerous image synthesis strategies, deep generative models have gained significant attention, primarily due to their capacity to produce high-fidelity images. Farly studies (Besnier et al., 2020; Li al., 2022: Zhang et al., 2021: Zhao & Bilen, 2022) utilize Generative Adversarial Networks (GANs) to synthesize annotated training data for image classification and segmentation. Recently, more works have focused on synthesizing training data with the powerful diffusion models for self-supervised pre-training (Tian et al., 2023), transfer learning (He et al., 2022), domain generalization (Yuan et al., 2022; Bansal & Grover, 2023), and supervised image classification (Azizi et al., 2023; Sarıyıldiz et al., 2023; Lei et al., 2023). However, despite the extensive research, notable performance gap persists when comparing the performances of models trained on synthetic data to those trained on real data. A primary reason for this discrepancy is the misalignment between the synthetic and real data distributions, even the diffusion model is trained on web-scale datasets, as illustrated in Fig. ((a). While previous works have attempted such as prompt engineering (Sanyıldız et al., 2023; Lei et al., 2023) and the expensive inversion approaches (Zhao & Bilen, 2022; Zhou et al., 2023), these solutions are neither sufficient nor efficient. Furthermore, there is an absence of theoretical frameworks that can adequately explain and analyze the efficacy of synthetic training data in a principled way.
address this issue through heuristic-driven approaches
To further enhance the quality and utility of synthetic training data produced by deep generative models, we present a theoretical framework for training data synthesis from a citiiiuutio-mmatccin
'Code released at: https: //g/ thub. com/ BAAI I 1 I-DAAI/TTTTTT innin - Synthesis Corre- sponding to Bo Zhao [bozhaonan in@ggral com)

---

Published as a conference paper at ICLR 2024
1W11
Synthetir Basslin Syntheck
SkSketch
Fax Cata
Dun (Sperided
Ours (hyrthetcc
Oura
Dum Gontredt
W-1K
IN-1
IN-
(a) Distribution Visualization
(b) Baseline Comparison
(c) Scale-up Performance
Figure I: Left: Visualization of the synthetic and real ImageNet data distribution using the first two principal components of features extracted by the CLIP image encoder. Our synthetic data better aligns with real data distribution than the baseline (vanilla Stable Diffusion). Middle: Our synthetic data achieves better performance compared with the baseline, and can effectively augment real data across all datasets. Right: Scaling up synthetic training data can improve the image classification performances in both in-distribution and out-of- cutt-f-distribution (OOD) tasks, even outperform training with real data in OOD tasks.
perspective. Specifically, starting from the first principle of supervised learning (Sutskever, 2023; Cunningham et al., 2008), we recast the training data synthesis as a distribution matching problem. This emphasizes two primary principles of synthetic data: (1) The distribution discrepancy between target and synthetic data, and (2) The cardinality of the training set.
Building this foundation, implement the framework on the oate--of-the-ar text-to-image diffusion model, Stable Diffusion (Rombach et al., 2022), to undertake a careful analysis and refinement of the training objectives, condition generation, and prior initialization achieve better alignment between synthetic and target data distributions. We empirically validate our theoretical framework and synthesis method across diverse benchmarks, covering various scenarios: (1) training exclusively with synthetic data, (2) augmenting the real training data, and (3) evaluating the scaling law between synthetic data and performance. In particular, for ImageNetlk classification using ResNet50 (He et al., 2016), training solely with synthetic data equivalent to x the original real data size yielded 70.9% Top1 classification accuracy, which increases to 76.0% when using x synthetic data. Additionally, we explore (1) mut-of-distribution (OOD) generalization and (2) privacy preservation when learning with synthetic data, and present promising results. Beyond advancing the state of the art, our findings offer insights into potential strategies for refining the training data synthesis pipeline. Our primary contributions are as follows:
Introducing principled distribution matching framework for training data synthesis, em- phasizing two foundational aspects that drive the effectiveness of synthetic data.
Employing the state-of-the ofrtr--arttttttt ext-tto-maae diffusion model, with a comprehensive analysis and refinement of its components, to design an effective training data synthesis strategy.
Advancing the state-of-the- the-art in training data synthesis for image classification tasks, while demonstrating the advantages in OOD generalization and privacy preservation.
2 BACKGROUND
2.1 TRAINING DATA SYNTHESIS
Synthesizing informative training samples remains a compelling yet challenging area of research. One line of work focuses synthesis through pre-trained deep generative models. Early attempts (Zhang et al., 2021; Li et al., 2022: Zhao Bilen, 2022) explore Generative Adversarial Networks (GANs) (Goodfellow et al., 2014; Brock et al., 2018 8) for informative and annotated training sample synthesis. Recently, diffusion models have gained more attention. He et al. (2022); Tian et al. (2023) employ synthetic data from text-com conditonit diffusion models for self-supervised pre-traing pre-training and few/zero-shot learning, highlighting the transfer learning capacity of synthetic training data. Also, Yuan et al. (2022); Bansal & Grover (2023); Vendrow et al. (2023) demonstrate the utility of synthetic
2

---

Published as a conference paper at ICLR 2024
data in uut-of-distributie settings by augmenting and diversifying the training dataset. Additionally, Sariyidıiz et al. (2023); Lei et al. (2023); Azizi et al. (2023) utilize Stable Diffusion (Rombach et al., 2022) and Imagen (Saharia et al., 2022) with prompt engineering to synthesize dlass-conditioned samples, showcasing the potential of using synthetic data solely for training classification models.
Despite the promising results in various tasks, a noticeable performance gap remains between models trained on real and synthetic datasets. This difference can be attributed to the misalignment between the distributions of the synthetic training data and the target downstream data. Existing research (Sarryıldız et al., 2023; Lei et al., 2023) employs prompt engineering to bridge the domain gap, which insufficient. Zhou al. (2023) implement diffusion inversion to obtain synthetic images close to real ones, which is expensive and unscalable to large datasets. To understand the mechanisms underlying the efficacy of synthetic data, we aim to find the theoretical principle and further tackle the challenges associated with suboptimal training data synthetic.
Another line of work focuses on synthesizing informative training samples by distilling the large real dataset into smaller synthetic one, i.e., dataset distillation. These methods optimize synthetic images (pixels) through the minimization of meta-loss (Wang et al., 2018), gradient matching loss (Zhao et al., 2021), trajectory matching loss (Cazenavette et al., 2022) and distribution matching loss (Zhao & Bilen, 2023; Zhao et al., 2023). In particular, distribution matching approaches have gained prominence due to their model-agnostic nature. However, the intricate and expensive optimization processes of dataset distillation often pose challenges in scalability. Despite these challenges, the principled approaches of dataset distillation have proven effective and inspired our work.
2.2 DIFFUSION PROBARILISTIC MODELS
Diffusion Probabilistic Models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Nichol & Dhariwal, 2021) are latent variable models that learn a data distribution g(20)) by reversing a gradual noising process. They define a forward diffusion Markov process q (œ1.1 zu) = II (x, xx-1) that uses a handcrafted Gaussian transition kernel, q (x, | xx-1) = N (x,; -I 8.I). with a noise schedule 3, E (0, 1), to transform the data distribution into a known prior Gaussian distribution. Then, a reverse Markovian process pe (mo:r) = (xTr I[-= Pe (at-1 2x) is learned to gradually remove noise added in the forward process, with Gaussian transitions parameterized by neural network: Po (xx-1 x2) = N (x Po (Ix, t). Eo (x,, t)). The training objective of the diffusion model is to minimize the Kl--divegeence between the joint distributions of the forward and backward processes. This can be further simplified to optimizing the standard variational bound on the negative log-likelihood of the model (Yang et al., 2022) (detailed proof can be found in Apx. A).
(1) In practice, by marginalizing over the intermediate sampling steps, an analytic form of the forward process can be obtained: q (ær wo) = M (ax; va, (1 oi I)) where a == 3. and o; IT ag. By further setting the variance in the reverse process to non-learnable constant o, and choosing specific parameterization of co to predict the added noise c, the overall training objective becomes:
KL [g (Iv. .1. ,T) Iipe (Iu. -1. xr 2 E [- log (zx))
B2 20 a.)
|e-cco co (Vā + vI vI- |1
(2)
where 22 V0,20 + vT 6 is derived from the reparameterization of the marginal distribution q (xx zo). This objective (Ho al., 2020) aligns with the denoising score matching loss of Score- based Generative Models (SGM) (Song & Ermon, 2019) when appropriately reweighted.
Leveraging the theoretical foundation, in this study, we focus on the state-of-the-art Latent Diffusion Model (LDM), Stable Diffusion (Rombach et al., 2022), to provide an alternative interpretation of its training objectives and generative sampling process from distribution matching perspective for training data synthesis.
3 TRAINING DATA SYNTHESIS: A DISTRIBUTION MATCHING PERSPECTIVE
The goal of training data synthesis is to generate synthetic data from the target distribution = q(x, u) of data and annotation y. In supervised training (Sutskever, 2023; Cunningham et al., 2008),
3

---

Published as a conference paper at ICLR 2024
the difference between training and testing error over the whole target distribution D is bounded by the inverse of the square root of the sampled training set cardinality |S|?. Please refer to Apx. B for the detailed proof.
log [F
+ log 1/6
Pig-DiaI
(J) - Trains(s) S
AS > 1
(3)
[S]
For training data synthesis given fixed model space F (i.e. fixed model structure with finite parameters), a salient takeaway from Eq. (3) is the identification of two pivotal factors: (1) Training and testing data distribution discrepancy (2) Cardinality of training set. This formalizes the first principle of training data synthesis: With infinite training samples from target distribution, the testing error will converge to the minimized training error.
However, direct sampling from data distribution can be intractable, we instead learn a generative model pe, parameterized with 0, that synthesize data following the same distribution, i.c., pe(x, 3) = (a, y) (Bishop, 2006). This effectively converts the problem of informative training data synthesis into distribution matching problem. We can further reframe such distribution matching problems as (0)()))((()) Po(y) [y) pp (2: (Li et al., 2017). This allows us to separate the problem of matching a joint fata-annotation distribution as two sub-problems: (1) data distribution matching ((x) = pa (z); (2) conditioned class likelihood matching ()y(<r) po po(y)æ), le). Under the classífication protocol, the former ensures in-distribution data synthesis, and the latter ensures a robust decision
boundary between classes. Overall, the objective of training data synthesis for supervised learning can be framed as the following optimization:
S
arg argmin (D(g())) + D(a(vla)) XlS1) S-pol#.y)
where St denotes the optimal synthetic data sampled from the learned distribution po (2. v). D):.)) is distance measure between two distributions. The regularization term X|Sk with À R encourages a larger training set. Fortunately, both distribution matching problems can be solved with deep generative models. In particular, with the diffusion model, the data distribution is learned through a denoising process, and the class likelihood can be modeled through classifier or classifier-free guidance (Dhariwal & Nichol, 2021; Ho Salimans, 2022).
Based on this theoretical framework, we perform an analysis of each component of the diffusion model in the context of distribution matching and propose potential improvements. Specifically, we setcchin--cetriic synthesis framework tailored for training data synthesis,
introduce distribution including three aspects of (1) feature distribution alignment (2) conditioned visual guidance (3) latent prior initialization.
3.1 DISTRIBUTION MATCHING WITH MAXIMUM MEAN DISCREPANCY
Firstly, quantify the distribution discrepancy between target and synthetic data. Although the objective of KL-divergence minimization in the diffusion model has implicitly provided an upper bound for the data distribution matching (Yang et al., 2022), it is observed be loose (Kingma et al., 2021) due to the gap between the bound and the negative lig-likelioood Eq. (1). More importantly, the discrete nature of empirical distribution (i.e. data samples) the context of training data synthesis further makes it a suboptimal measure. Instead, we measure this discrepancy with the Maximum Mean Discrepancy (MMD) (Gretton et al., 2008), which alleviates the problem of potentially biased discrete distribution as a sample test and is widely and successfully applied in dataset distillation Zhao Bilen (2023) because of its non-parametric nature and simplicity. At the end, we find a mathematical equivalence between the training objective of diffusion model and the minimization of the MMD upper bound under certain assumptions, which allows us to further relax the variational bound and better align the data distribution in the feature space.
Consider a (real) target dataset R = {(k (1) ...., (Kiz), MIR) } and a synthetic dataset S {81,,))) (+1)), s) Our objective is to minimize the MMD between target data distribution
'The probability that the error is lower than bound, which explicitly depends on the cardinality of the training set, is guaranteed to be higher than 1 6, where 8 is small.
4

---

Published as a conference paper at ICLR 2024
(() and the synthetic data distribution po(x) represented by some feature extractor ve F.
MMD[F, sup 1 x
(Byy [R)) Ep (v)())),
(5)
where de represents a function residing within a unit ball in the universal Reproducing Kernel Hilbert Space 7 (RKHS) (Hilbert, 1904). By following the reproductive properties of RKHS and empirically estimating the expectations for all distributions, we can simplify Eq. (5) to Eq. (6):
- Σ vo (Ka)
MM
vo (5)) 1
(6)
S
In Latent Diffusion Model, t can be conceptualized as the Variational Autoencoder (VAE) with the latent embedding serving as the feature map for MMD computation. The distributions of target and synthetic data can be effectively approximated by the initial moise-free latent embedding 2g and the predicted denoised latent embedding De (3z,,) t). Assuming that MMD is computed within the same batch, where |R| = [S] = V, our objective can be further refined as || Σ (xc ag (æt, l) Following Ho et al. (2020), choosing the parameterization with Eg predictor of added noise e from I2 denoising, allows us to frame our distribution matching objective (MMD) as Eq. (7):
W
W
LMMD =
Σ (E
Eo (wt, ()) IP S
Σ (I Eo (xt, ()) 1i L
W
W
Since the mean and norm operations are both convex functions on the left-hand side, by applying Jensen's inequality, we obtain an upper bound on the original distribution matching objective under equal weighting to each time step. Note that, this upper bound expression resembles the diffusion model training loss Eq. (2). This indicates when training the diffusion model, we implicitly optimize an upper bound of the MMD between synthetic and real data distribution. Moreover, this allows to directly optimize the objective in Eq. (7) to mitigate the losseness in the variational bound. We use this objective to augment the original diffusion model loss during finetuning, which could ensure more aligned feature distribution under the MMD measure.
3.2 CONDITIONED GENERATION VIA TEXT- VISION GUIDANCE
Beyond the general feature-level distribution matching, i.e., ((x) = pe(x), a pivotal aspect of training data synthesis is to ensure congruent conditional class distribution with well-defined decision boundaries, i.c., 7(y| a>) = pe (y|z)) x)) Classifier( free) guidance (Dhariwal & Nichol, 2021; Ho & Salimans, 2022) plays crucial role in the conditioned sampling process in the diffusion model. Under SGM framework, to match the conditioned class likelihood, we can equivalently match the score function of each distribution, i.e. Va, log q(v) aa: Va,, log po (y(y)ar), az), which is estimated through the noise prediction co (a:)) by reformulating the conditional score function as Eq. (8):
1
VR, log po (y x.) we log pe (x; y) Va, log pe (z:) VI ae
(Eo (It, co (xx)) (8)
Most works of training data synthesis align the conditioned distribution via text-prompt engineering with class-level description (He et al., 2022), instance-level description (Lei et al., 2023), and lexical definition (Sarryıldiz al., 2023). Following (Lei et al., 2023), we incorporate the class name with the BLIP2 (Li et al., 2023) caption of each instance as the text prompt. Moreover, while text condition offers certain adaptability, ignores intrinsic visual information, including both low-level ones, e.g.. exposure and saturation, and high-level ones, e.g. co-occurrence of objects and scenes. To address this, we adopt a more direct prompting strategy by conditioning on image features. In particular, we extract image features encoded with the CLIPP (Radford et al., 2021) image encoder, compute the mean of image embeddings for randomly sampled images of a class, and estimate the intra-class feature distribution, i.e. the mean feature. This is then concatenated with the text embeddings for jointly finetuning the diffusion model using LoRA (Hu et al., 2021), 1). thus injecting the extra conditional control into the cross- atts-att layer of the denoising UNet (Ronneberger et al., 2015). The resultant multi-modal condition (embedding) takes form in "photo of [classname], [Tmage Caption], [Intra-class Vi sual Guidance]
5

---

Published as a conference paper at ICLR 2024
3.3 LATENT PRIOR INITIALIZATION
Another key factor ensuring the data distribution matching of synthetic data is the latent prior po (œT) in the sampling process of LDM. It acts as the informative guide for the reverse diffusion process, governed by Langevin dynamics (Song & Ermon, 2019; Parisi, 1981), as shown in Eq. (9):
- at + V20zz Cr2 VVV my log Po (a: (rr) 24 N (O,I),
where po [ ( represents the conditional distribution of æ; given the latent prior Zy, and the corresponding score function Va, log (ogpo(x,)ar) guides the Langevin dynamics towards regions of higher probability in the data distribution, ensuring the synthetic samples align closely with the target distribution. While diffusion models often employ Gaussian distribution as the initial prior 2T ~ 9 (Ir) := 'N (ær; 0, I), recent studies using latent inversion (Zhou et al., 2023; Lian et al., 2023) and learning-basee -based prior rendering (Liao et al., 2023) have highlighted the advantages of informative non-Gaussiian latent priors with better sampling speed and synthesis quality. However, obtaining such an informative prior can be expensive due to intensive computation or need for external architecture. Instead, following (Meng et al., 2021), we leverage the VAE encoder to obtain the latent code of specific real samples, which is extremely cheap. This also provides an informative latent prior initialization closely aligned with the target distribution, resulting in better synthetic samples.
4 EXPERIMENTS
Settings. We empirically evaluate the proposed distribution hattolttiiiommmtiinnn. matching framework and assess the utility of synthetic training data. We explore the application of synthetic data in various supervised image classification scenarios: (1) Replacing the real training set (Sec. 4.1). (2) Augmenting the real training set (Sec. 4.2), and (3) Evaluating the scaling law of synthetic training data (Sec. 4.3). We aim to validate two main factors identified in Eq. (4): better alignment between target and synthetic data (in scenarios and 2) and the advantages of larger training set cardinality (in scenario 3). Then, we further explore the (4) diitrii-ditiiiiii generalization (Sec. 4.4) and (5) Privacy-presee preservation (Sec. 4.5) of synthetic training data. For all experiments, we finetune Stable Diffusion 1.5 (SDv1.5) (Rombach et al., 2022) with LoRA.
Datasets. We conduct benchmark experiments with ResNet50 (He et al., 2016) across three Ima- geNet datasets: ImageNette (IN-10) (Howard, 2019), ImageNet100 (IN-100) (Tian et al., 2020), and ImageNet1 K (IN-1K) (Deng et al., 2009). Beyond these, we also experiment with several fine-grained image classification datasets, CUB (Wah et al., 201 1), Cars (Krause et al., 2013), PET (Parkhi et al., 2012), and satellite images, AT (Helber et al., 2018).
More details of dataset specifications (Apx. D), data synthesis (Apx. E), model training (Apx. F), and experiment setting (Apx. G) are provided in the Appendix.
4.1 IMAGE CLASSIFICATION WITH SYNTHETIC DATA ONLY
We begin by assessing the informativeness of synthetic training data as a replacement for real training data on image classification tasks. To replace the real training set, synthesize training samples with the same number as the real training set in each dataset.
As shown in Tab. 1, in comparison real data, our synthetic data reduces the performance gap to less than 3% for small-scale dataset IN-10. In the context of the more challenging large-scale dataset, IN-IK, the performance differential remains under 10%. Regarding training data synthesis methods, our method outperforms all tfee--ofthhe-ee art techniques across all benchmarks. It is crucial to highlight that our synthetic data exhibits improvements of 16.8% and 28.0% IN- 1K top-1 accuracy. compared to CiP (Lei et al., 2023) and Fakelt (Sanyıldız et al., 2023) respectively, given the same generative model backbone. In fine-grained classification tasks, our method demonstrates a more significant improvement compared to the -tate-of-the-art method (Fakelt), which can be attributed to the need for a more aligned decision boundary.
Ablation Study, We next perform a more comprehensive ablation study to evaluate the efficacy of our proposed enhancements: distribution matching objective (Sec. 3.1., conditioned visual guidance (Sec. 3.2), and latent prior initialization (Sec. 3.3). Due to computational cost, we conducted the

---

Published as a conference paper at ICLR 2024
Table I: Synthetic Image Classification Performance Top-1 accuracies of ResNet50 reported on seven datasets. The upper part indicates training with synthetic data only, and the lower part indicates joint training with combined real and synthetic data.
Model IN-10 IN-100
IN-IK
CUB
Cars
PET
EuroSAT
Training without real data
BigGAN (Brock et al., 2018)
BigGAN VQ-VAE
42.7
VQ-VAE-2 (Razavi et al., 2019)
54.8
CDM (Ho et al., 2022)
CDM
63.0
Fakelt (Saryyıldiz al., 2023)
SDv1.5
42.9
33.7
47.1
75.9
94.0
Imagen (Azizi al., 2023)
Imagen
69.2
CiP (Lei et al., 2023)
SDv1.5
79.4
62.4
54.1
OURS
SDv1.5
90.5
80.0
70.9
64.3
81.8
89.2
94.6
A with the previous state-of-the-art
+10.1
+17.6
+1.7
+300.6
434.7
+13.3
+0.6
Training with real data
Baseline real data only
93.0
86.3
79.6
81.5
89.5
93.2
97.6
OURS real data A with the real data
SDv1.5
95.1
88.2
79.9
83.5
90.3
94.0
98.9
+2.1
+1.9
+0.3
+2.0
+0.8
+0.8
+1.3
ablation study on the IN-10 and IN-100 datasets. As illustrated in Tab. 2, every proposed module contributes remarkable performance improvement. The combination of three modules achieves the best results, which outperform the baseline by 10.7% and 14.0% on IN-10 and IN-100, respectively. Further analysis and details provided in Apx. G.5.
Table 2: Ablation study on proposed improvements with IN-10 and IN-100 Top-1 accuracy. Ticks under each column represent the implementation of the corresponding module in data synthesis. Finetune indicates whether finetune Stable Diffusion on target dataset with the augmented distribution matching loss or original diffusion loss Ho al. (2020).
Latent Prior Sec. 3.3
Visual Guidance Sec. 3.2
Distribution Matching Sec. 3.1
Finetune
ImageNette
ImageNet100
79.8
66.0
80.8
73.3
82.3
74.0
81.2
73.8
82.9
75.1
88.0
76.3
88.7
78.3
89.5
79.3
88.9
79.8
90.5
80.0
4.2 AUGMENTING REAL DATA WITH SYNTHETIC DATA
We study whether the synthetic data can serve as dataset augmentation. We compare the models trained with only real data and those trained with both real and synthetic data. As shown lower part of Tab. 1, we observe improvements across all benchmarks when combining the synthetic and real data. Especially, our synthetic data boosts the performances by 2. and 1.9% on IN-10 and IN-106 datasets respectively. This validates that our synthetic data align well with the real data distribution.
4.3 SCALING UP SYNTHETIC TRAINING DATA
Besides data distribution aligmment, training set cardi- nality is another key factor influencing the utility of synthetic data identified Eq. (4). Fortunately, it is easy to synthesize more data for deep generative model. In the experiment, we scale up the synthetic dataset and train the image classifiers using increasing amounts of synthetic data, ranging from X to 10x the size of the original real dataset. As shown in Fig. 2, solely scaling
Table 3: Scaling up synthetic ImageNet- I
Synthetic real data size
Real data?
Top-1 1
Top-5
x (1,3M)
70.9
89.9
x2 (2.6M)
72.9
91.1
x5 (6.4M)
74.5
92.1
(13M)
76.0
93.1
x1 (1.3M)
79.6
94.6
7

---

Published as a conference paper at ICLR 2024
Synthetic
Real Data
Accuracy
T0p3.
07
Synthetic
Real Data
Dataset Scale Muttiplerr
Dataset Multipller
(a) Imagenette
(b) Imagenet100
Figure 2: Effect of Scaling Up Synthetic Dataset Top-1 image classification performance with synthetic data only (indicated by blue solid curve) increases with synthetic dataset size, eventually outperforming real data (indicated by red dash line). The horizontal axis represents the amount of synthetic data used as multiples
original real dataset size.
up the synthetic dataset enables the image classification performance to surpass that of models trained on real data, even without trained on it. For the small-scale dataset IN-10, the threshold scale of synthetic data to outperform the real data is only 2.5x the size the real data. While, the challenge greater for larger-scale dataset IN- 100. To surpass the real data efficacy, synthetic data with 6.5x size is needed on IN-100. As shown in Tab. 3, we also scale up synthetic training images for the large dataset IN-IK, which has 1.3M real samples. The results show that the trend of increasing image classification accuracy along with the increasing synthetic data, still holds, where we achieve 72.82% and 76.00% with 2 and 10 x ImageNet- KK size, respectively. By scaling up synthetic data, reduce the gap between real and synthetic data down to 3.6% in top-1 accuracy and more encouraging 1.5% in top-5 accuracy.
4.4 GENERALIZATION TO OUT-OF-DISTRIBUTEE DATA
We also investigate the Out-of-distribution (OOD) generalization performance on four OOD variants of IetgeNet: (1) ImageNet-v2 (IN-v2) (Recht al., 2019) (2) ImageNet-Sketch (IN-Sketch) (Wang
et al., 2019) (3) ImageNet-R (IN-R) (Hendrycks et al., 2021a) a) (4) ImageNet-A (IN-A) (Hendrycks al., 2021b). We test the ResNet50 (He al., 2016) trained with in-distribution real data (i.e.
ImageNet- IK training set) or synthetic data (i.e. the generative model is only tuned on ImageNet-IK training set like above experiments) on the OOD test sets.
Table 4: OOD Generalization Performance. xk indicates training solely with the synthetic dataset of scale l times that of real dataset. Top-1 and classification accuracies are reported.
ImageNet-v2 (Recht 2019)
ImageNet-Sketh (Wang et al., 2019)
ImugeNet-R
ImageNet-A
(Hendrycks al., 2021a)
(Hendrycks al., 2021b)
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Training without real data
Fakelt Sanyyldzz (2023)
43.0
35.2
26.3
45.3
15.1
CiP Lei al. (2023)
53.8
80.5
18.5
35.5
33.6
51.1
5.2
21.7
OURS
67.2
88.0
21.9
38.2
35.4
51.9
4.5
23.5
OURS
25.2
36.1
52.5
OURS x5
71.1
90.6
27.8
46.2
39.7
56.3
9.4
34.9
OURS x10
73.0
91.4
29.2
48.1
41.0
57.5
11.3
39.4
Training with real data
Buseline real data only OURS real data
74.7
28.1
45.8
54.1
34.7
75.7
92.7
29.0
16.8
10.5
55.9
9.0
36.1
As illustrated in Tab. 4, we observe that when training with 1x synthetic data only, our method achieves the best generalization performance across three out of four benchmarks, outperforming previous synthesis strategies. When jointly training with real data, our synthetic data further boosts the OOD generalization performance of real data. More importantly, when we scale up the synthetic data, its OOD generalization performance exceeds that of real data, e.g. ImageNet-Sketch, ImageNet-R and ImageNet-Ar even before achieving comparable performance on the in-distribution test set. This further highlights the promising utility of synthetic training data in enhancing OOD generalization.
8

---

Published as a conference paper at ICLR 2024
10a
109
True Positive Rate
10"1
True Positive Rate
10-1
10
10
10"
Pieato, daninn)
10
Syndhers, (aniin))
Priutte, (ontine, Sand
Santheric func) asc
sur 0.619
10"4
(olhine
10
air-0..1 0.51
Prlctte (ottine, food
Syntheric, (offine, fondi
10-5
10-5
10
10
10-1
10
10 False Positive Rate
10-1
False Positive Rate
Figure 3: Membership Inference Attack (MIA) Performance with LiRA. LiRA achieves a TPR of 0.001 at low FPR 0.1% when applied synthetic data, while the results for private data 0.01% %, which indicates training with synthetic data is much more Privacy-preserving
synthetic
top1
tep2
top3
synthetic
top1
top2
top3
gar barbage truck chaiin SaW
french horn church
Figure 4: Visualization synthetic and retrieved real data with SSCD. The synthesized data
does not exhibit evident copying or memorization.
4.5 PRIVACY ANALYSIS
Synthetic training data is promising solution prrsevy-res learning. In this subsection, we examine the privacy-preserving nature of our synthetic data from the perspectives of both privacy attacks and visual similarity. Further experiment details are provided in Apx. G.4.
Membership Inference Attack. We implement the membership inference attack that enables an adversary to query a trained model and determine whether specific example is part of the model's training set. Specifically, we follow the tf-th-rftt Likelihood Ratio Attack (LiRA) Carlini al. (2022) and report the MIA results for two training approaches: the privacy training dataset and synthetic data, in the low palse-positive rate regime. To account for training duration, we conduct 10 sampling iterations. Concretely, we initially divide the privacy data in IN-10 into two halves: member data and mommeemerr data. We only employ LoRA to finetune the diffusion model the member data, then generate synthetic data of equal size. Subsequently, we train a ResNet50 on synthetic data. As depicted in Fig. 3, regardless of online attack, offfine attack, or fixed variance scenarios, the model trained on synthetic data demonstrates superior defense against MIA.
Visual Similarity. We follow the previous works (Zhang et al., 2023; Somepalli et al., 2023) and employ the Self-Supervised Content Duplication (SSCD) (Pizzi et al., 2022) method for content plagiarism detection. We use the ResNeXt101 model trained on the DISC dataset (Douze et al., 2021). For a given query image and all reference images, infer through the detection model to obtain a 1024-dimensional feature vector representation for each image. By computing the inner product between the query feature and each reference feature, obtain the similarity between them. Subsequently, we select the the top five reference features with the highest similarity to the query feature. Then, we rank the similarity and present the top 3 images with their corresponding real data in Fig. 4. From the figure, we observe that there are no apparent issues of copy or memorization (Carlini al., 2023). At least visually, these synthesized images should provide sufficient privacy protection.
CONCLUSION
In this work, we propose principled theoretical framework for training data synthesis from iesributtionmmathhinn perspective. Based on this, we empirically push the limit of synthetic training data by advancing the state-of-N the-art performances also demonstrate promising benefits of improving the OOD generalization and privacy preservation performances by training models with synthetic data.
diverse image classification benchmarks. We

---

Published as a conference paper at ICLR 2024
Acknowledgement. This work is funded by National Key R&D Program of China (20) and NSFC-62306046.
REFERENCES
Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American mathematical society, 68(3)3337-404, 1950. 16
Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304.08466, 2023. 1.
Hritik Bansal and Aditya Grover. Leaving reality imagination: Robust classification via generated datasets. arXiv preprint arXiv:2302.02250 2023. 1,2
Victor Besnier, Himalaya Jain, Andrei Bursuc, Matthieu Cord, and Patrick Pérez. This dataset does not exist: training models from generated images. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2020.
Christopher M. Bishop. Puttern Recognition and Machine Learning. Springer, New York, NY, 2006.
4
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018. 2, 17
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897-1914. IEEE, 2022. 9,
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253-5270, 2023. 9
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2022. 3
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020. 19
Pádraig Cunningham, Matthieu Cord, and Sarah Jane Delany. Supervised Learning, pp. 21-49. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008. ISBN 978-3-540-75171-7. doi: 10.1007/ 75111-7 2. 2, 3, 15
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10.1 1LL.0C/CV..20990506666
1, 6, 16, 17,20
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8799 8794, 2021. 4,
Matthijs Douze, Giorgos Tolias, Ed Pizzi, Zoë Papakipos, Lowik Chanussot, Filip Radenovic, Tomas Jenicek, Maxim Maximov, Laura Leal-Taixé, Ismail Elezi, al. The 2021 image similarity dataset and challenge. arXiv preprint (tth:22200.96772. 2021. 9, 19
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Arthur Gretton, Karsten Borgwardt, Malte J Rasch, Bernhard Scholkopf, and Alexander Smola. A kernel method for the two-sample problem. arXiv preprint arXiv 2008.
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016. 2, 6, 8, 17
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? arXiv preprint arXiv:2210.07574, 2022. 1, 2.5
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Introducing eurosat: novel dataset and deep learning benchmark for land use and land cover classification. In IGARSS 2018-2011 IEEE International Geoscience and Remote Sensing Symposium, pp. 204-207. IEEE, 2018. 6, 16, 17
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distributinn generalization. ICCV, 2021a. 8, 18, 21
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021b. 18, 21
David Von Hilbert. Grundzüge einer allgemeinen theorie der linearen integralgleichungen, von david hilbert. 1904. 5
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. 2022. 4,
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020. 3, 5, 7, 17, 19
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1):2249-22) 2281, 2022. 17
Wassily Hoeffding. Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding, pp. 409-426, 1994. 15
Jeremy Howard. Imagewang, 2019. URL https: htt.l//tpp//gggtaaa.a.c///aaaaap 6, 16, 17
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 2021. 5, 17
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-21707, 2021.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. object representations for fine-grained categorization. In Proceedings of the IEEE International Conference Computer Vision (ICCV)
Workshops, June 2013. 6, 16, 17, 20
Shiye Lei, Hao Chen, Sen Zhang, Bo Zhao, and Dacheng Tao. Image captions are natural prompts for text-to-image models. arXiv preprint arXiv:2307.08522 2023. 3, 5, 6, 7, 17
Chongxuan Li, Taufik Xu, Jun Zhu, and Bo Zhang. Triple generative adversarial nets. Advances in neural information processing systems, 30, 2017. 4
Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Kreis, Sanja Fidler, and Antonio Torralba. Big- datasetgan: Synthesizing imagenet with pixel-wise annotations. Proceedings of the IEFF/CV Conference Computer Vision and Pattern Recognition, pp. 21330-21340, 2022. 1.2
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman -mage pre- training with frozen image encoders and large language models. arXiv preprint 3X1.:23001.1599 2023. 5, 16
11

---

Published as a conference paper at ICLR 2024
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. LIm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305.13655, 2023. 6
Liao, Songwei Ge, Yiran Xu, Yao-Chih Lee, Badour AlBahar, and Jia-Bin Huang. Text-driven visual synthesis with latent diffusion prior. arXiv preprint WxXiv2332.088510 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Ziwei Liu,, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015. 20
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint NrXiv:2/08.01073, 2021. 6
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018. 19
G. Parisi. Correlation functions and computer simulations. Nuclear Physics B, 180(3):378-384 1981. ISSN 0550-3213. doi:
6
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012. 6, 16, 17
Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthijs Douze. A self- supervised descriptor for image copy detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14532-14542, 2022. 9,
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8763. PMLR, 2021. 5
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019. 7,
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? International conference on machine learning, pp. 5389-5400. PMLR, 2019. 8, 18
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10699) 2022. 2, 3, 6, 16
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Cossuter-Assisted Intervention-MICCO 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234-241. Springer, 2015. 5
Chitwan Saharia, William Chan. Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022. 3
Mert Bülent Sanyıldız, Karteek Alahari, Diane Larlus, and Yannis Kalantidis, Fake it till you make it: Learning transferable representations from synthetic imagenet clones. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattrn Recognition (CVPR), pp. 801 8011-1 June 2023. 5, 17
12

---

Published as a conference paper at ICLR 2024
Jascha
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning. pp. 2256-2265. PMLR, 2015. 3
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6048 6088-6058, 2023. 9, 18
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019. 3,
Ilya Sutskever. An observation generalization, 2023. 2, 3, 15
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 776-794. Springer, 2020. 6, 16, 17
Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-ime image models make strong visual representation learners. arXiv preprint arXiv:2306.0099. 00984, 2023. 1, 2
Joshua Vendrow, Saachi Jain, Logan Engstrom, and Aleksander Madry. Dataset interfaces: Diagnos- ing model failures using controllable counterfactual generation. arXiv preprint arXiv:2302.07865 2023. 2
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Caltech-
(cub-200- 2011). Technical Report CNS-TR-201 1-001, California Institute of Technology, 2011. 6, 16, 17, 20
Haohan Wang. songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pp. 19506-10516 2019. 8, 18
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint
2018. 3
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXhv:2110.00476, 2021. 17
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint arXiv:2209.00796, 2022. 3,4, 4, 15
Jianhao Yuan, Francesco Pinto, Adam Davies, Aarushi Gupta, and Philip Torr. Not just pretty pictures: Text-to- Text-to-image generators enable interpretable interventions for robust representations. arXiv preprint 22. 1237, 2022. 1,2
Jie Zhang, Xiaohua Qi, and Bo Zhao. Federated generative learning with foundation models. arXiv preprint X31.:2336666644 2023.
18
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, fran-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10145-10155, 2021. 1,2
Bo Zhao and Hakan Bilen. Synthesizing informative training samples with GAN, NeurIPS 2022 Workshops SymtheticData4MLL, 2022. 1,2
Zhao and Hakan Bilen. Dataset condensation with distribution matching. /EFF/CVF Winter Conference Applications of Computer Vision (WACV), 2023. 3, 4
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference on Learning Representations, 2021. URL https: attpps://ppnnee //ppereevee.. net/ pee//forue forum d=mSAKkL.Y
13

---

Published as a conference paper at ICLR 2024
Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2023. 3
Recognition, pp.
Yongchao Zhou, Hshmat Sahak, and Jimmy Ba. Training on thin air: Improve image classification with generated data, arXiv preprint arXiv:2305. 2023. 1, 3,
14

---

Published as a conference paper at ICLR 2024
A MINIMIZATION OF KL-DIVERGENCE OBJECTIVE IN DIFFUSION PROBABILISTIC MODEL
Consider the training objective of the diffusion model, minimization of KL-divergence between the joint distributions of forward and backward process and po over all time steps:
KL (q (Zo: 21. 2T) Ioo (To. xT))
(10)
By definitio of KL-divergence and the property of Markovian transition, we can further decompose the expression in time (Yang et al., 2022):
T
Po (0t-1 (xx)
KL (gllpe) = Egg log (xT)
Σ
log
+ const
(11)
(xx x(-1)
Using Jensen's is inequality, we can derive:
KT. (q (Fo, $1, .T) Ippo (æo, 21,'" ,T)) 2 log (æo) + const
(12)
The objective in training is to maximize the variational lower bound (VLB) of the log-likelihood of data zo, which equivalent minimizing the negative VLB as Eq. (1).
B TRAINING DATA SYNTHESIS FOR SUPERVISED LEARNING
Consider the following formulation of supervised learning (Sutskever, 2023; Cunningham et al., 2008):
log F + log 1/6 vf F > 11-8
PIS-DISI Test - Trains (f) <
(13)
S
where D denotes the distribution of the whole target dataset and S is training subset of D. F represents all potential functional space, which can be represented by neural network with fixed structure, where each unique combination of parameter values corresponds to unique f. The training and test errors are defined as Trains()) І E(x,))es (X,)) Y.) and Tes)
DI Exx.)) L (f (X,) Y)), respectively. Mathematically, it is evident that when the training loss minimal and the training set cardinality significantly exceeds the model parameters, the test loss will also be notably low.
We now prove this proposition. Consider the formulation of test error:
Pra-Dial Test (f) Trains(f) 2 & for some F <
Σ
Pia [Test (ʃ) Trains 2
JEF
= Σ Pr
Σ (L.
E[4 2
S
JEF
I 1 exp (-2(S(
(14)
The first step is a straightforward extension the original formulation. The second step em- ploys Hoeffding's inequality (Hoeffding, 1994). Given a series of independent random variables X1, X2,X2...., Xn bounded by a; < X; < b;, Hoeffding's 's inequality states:
2/2
Pr Σ X. E
X
2 t
S exp
(15)
Σ (b. a;)
It bounds the probability that the sum of independent bounded random variables deviates from its expected value beyond a set limit. Consider each L; E[L] as a bounded random variable, capturing the deviation of specific data point's loss from its expected value. By assuming bounded loss function L,, which is valid as we can project any unbounded loss function monotonically to a bounded range of (0, 1] through sigmoid function, obtain the upper bound the probability of
15

---

Published as a conference paper at ICLR 2024
this disparity surpassing threshold t, culminating in the expression exp the convergence of test error to zero when the size of the training dataset goes to infinite:
exp (-2|S|t²). This implies
lim
Pr
sup /EF
(Tfstp(f) - Traing (f)) > =0
(16)
1S|->00
which eventually validates statement on the implication of training data synthesis.
C
MAXIMUM MEAN DISCREPANCY (MMD) IN RKHS
The Maximum Mean Discrepancy (MMD) in a Reproducing Kernel Hilbert Space (RKHS) (Aron- szajn, 1950) H provides metric to measure the difference between two probability distributions.
MMD[F, p. = sup 1l $1
(Eg [V)( Ep (s))))
(17)
MMD[F.P.R p. = sup
(UT - Ms,, v0)
(18)
S1
MMD* pp
sup [volln,
(NT us, Do²H
(19)
[MD2 = 11NT -ust-
(20)
In Eq. (17), the MMD between distributions p and q is formulated as the supremum of the difference of expectations under these distributions, constrained by the norm of the function vh in the RKHS. This difference can be further expanded using the inner product in the RKHS, leading to an expression in terms of the mean embeddings of the distributions, shown in Eq. (18). Squaring the MMD gives us a more interpretable metric, which is the squared distance between the mean embeddings of the two distributions in the RKHS, depicted in Eq. (19) and Eq. (20). This leads to the derivation from Eq. (5) to Eq. (6) in Sec. 3.1.
D DATASET DETAILS
We adopt seven datasets total across different settings: ImageNette (IN-10) (Howard, 2019), ImageNet100 (IN-100) (Tian et al., 2020), and ImageNetIK (IN-IK) (Deng et al., 2009), CUB (Wah et al., 2011), 1), Cars (Krause et al., 2013), PET (Parkhi et al., 2012), EuroSAT (Helber al., 2018). The detailed dataset statistics specified in Tab. 5.
Table 5: Detailed Dataset Statistics.
IN-10
IN-100
IN-IK
CUB
Cars
PETS
EuroSAT
Datasel Size
14197
141971
1419712
11788
16,185
73499
27030
Training Data
12811
Test Data Size
1000
10000
100000
5794
8041
3669
$400
Classes
196
Domain
Natural Image
Natural Image
Natural Image
Fine grain Bird
Fine grain Cars
Pet Breed
Satellite Image
E TRAINING DATA SYNTHESIS DETAILS
We use Stable Diffusion v1.5 (Rombach et al., 2022) across all benchmarks. Unless specified (i.c. we specifically scale up synthetic data in Sec. 4.3), we generate an equal amount of synthetic data samples as in the corresponding real dataset for both replacing and augmenting real training data. The stable diffusion generation parameters are specified in Tab. We use text prompt mentioned in (Intra-class Visual Guidance] ] for ImageNet dataset (IN-10, IN-100, IN-1K); for the rest four datasets, do not the image caption generated with BLIP2 (Li et al., 2023). Also, all generations use a uniform set of negative prompts "distorted, unrealistic, blurry, out of frame
Sec. 3.2, of the form "photo of [classname], [Image Caption],
16

---

Published as a conference paper at ICLR 2024
Table Hyperparameters used in training data synthesis.
Model
Sampling steps
Scheduler UniPC
Guidance scale
Img strength
Image size 512 x 512
Stable Diffusion vI.5
30
2.0
0.75
For fine-tuning the Stable Diffusion, we use Low-Rank Adaptation (LoRA) (Hu al., 2021) with the MMD augmented objective Sec. 3.1 and inject the visual guidance Sec. 3.2. The fine-tuning hyperparameters used are specified in Tab. We use the loss function mentioned in Sec. 3.1, where
Table 7: Hyperparameters used in LoRA fine-tuning
Rank
Epoch Leanring Rate 300
Batch Size
SNR-7
Image size
Stable Diffusion v1.5
le-4
8 x
0.75
512 x 512
we augment the MMD distribution matching loss with the simple diffusion model training objective Ho al. (2020) as in Eq. (21), we choose 7 = 0.05.
W
Loverall IiSimpte + TLMMD =
1
Σ
(1 co (2., 1i + +oll
1
(E
60 (It, t)) 11
W
W
(21)
F DOWNSTREAM IMAGE CLASSIFICATION TRAINING DETAILS
We adopt ResNet-50 (He et al., 2016) through all benchmarks in the image classification task. For all three ImageNet datasets, we train the classifier model from scratch. Follow the training recipe of (Lei et al., 2023) and (Wightman et al., 2021), I), the training hyperparameters are specified in Tab.
G EXPERIMENT SETTING DETAILS AND MORE RESULTS
G.1 IMAGE CLASSIFICATION WITH SYNTHETIC DATA ONLY
We compare with the baselines of previous works on synthetic ImageNet data for supervised image classification. All baselines synthesize equal amounts of data as real datasets. The accuracies reported from the original baseline papers. except for Fakelt (Sarryıldız al., 2023) in CUB (Wah et al., 2011), Cars (Krause et al., 2013), PET (Parkhi et al., 2012), and EuroSAT (Helber et al., 2018). The number reported based on our implementation of the method with the base prompt photo/ /imaee of [c] [classname]]
Note that there is a difference the generation resolution among different methods. particular BigGAN-deep (Brock et al., 8), VQ- VAE-2 (Razavi et al., 2019), and CDM (Ho et al., 2022), as reported in (Azizi et al., 2023), use image resolution of 256 x 256. CiP (Lei et al., 2023), Fakelt (Sarryıldız et al., 2023), and our method uses image resolution of 512 x 512. Imagen (Azizi et al., 2023) uses image resolution of 1024 x 1024. Even though we resize all samples in downstream training 224 x 224, the generation resolution still has influences reported in (Azizi al., 2023) potentially due to the generation details and quality. Thus, comparing our method and nta-Stable Diffusion baselines can potentially be unfair comparison. However, main focus in training data synthesis is in comparison with real data and the comparison between ours and other Stable Diffusion-basse method show our framework leads to improved synthetic data informativeness to close the gap between real and synthetic data.
G.2 AUGMENTING REAL DATA WITH SYNTHETIC DATA
For CUB (Wah et al., 2011), Cars (Krause et al., 2013), and PET (Parkhi et al., 2012) datasets, we use pre-trained ResNet-50 due to the scarcity of data to train decent ResNet-50 classifier. For EuroSAT (Helber et al., 2018), 8), ImageNette (IN-10) (Howard, 2019), ImageNet100 (IN-100) (Tian et al.. 2020), and ImageNet1 K (IN-1K) (Deng et al., 2009) train the model from scratch. We augment the real data with an equal amount synthetic data, which effectively raises the amount of training by two.
17

---

Published as a conference paper at ICLR 2024
Table 8: Training Hyperparameters of Different Dataset.
IN-10
IN-100
IN-IK
CUB
CARS
PETS
EuroSAT
Train Res
Test
> 224
224 224
224
448
448
448
>
448 448
Epochs
200
200
200
200
200
200
Batch size
128 8
128 x
512 8
128
128 x 8
128
128
Optimizer
SGD
SGD
LAMB
SGD
SGD
SGD
SGD
I.R
0.1
0.1
Se-3
0.1
0.1
0.1
LR decay
multistep
multistep
cosine
multistep
multistep
multistep
multistep
decay
0.2
0.2
0.2
0.2
0.2
0.2
decay epochs
50/100/150
50/100/150
50/100/100
50/100/150
50/100/150 5e-4
50/100/150
Weight decay
5e.4
5e-4
0.02
Se-4
Se.4
Se-4
Warmup epochs
Label smoothing
Dropout
Stoch. Depth
0,05
H. flip
Rand Augment
X
X
7/0.5
X
X
X
Mixup alpha
X
X
X
A
Cutmix alpha
CE loss BCE loss
Mixed precision
G.3 GENERALIZATION TO
DATA
We train the image classifier on synthetic in-distribution synthetic and real ImageNet-IK and then test four Distrritiitiito test sets: (I) ImageNet-v2 (IN-v2) (Recht et al., 2019), which is different ImageNet test set collected on purpose to eliminate the effect of adaptive overfitting for original ImageNet test set; ImageNet-Shee Sketch (IN-Sketch) (Wang al., 2019), which contains sketch of ImageNet classes with black and white scheme only; ImageNet-R (IN-R) (Hendrycks et al., 2021a), which contains renditions of ImageNet classes such as art, cartoon, graphies, etc.; and ImageNet-A (IN-A) (Hendryeks et al., 2021b), which contain natural adversarial examples for ResNet models trained with ImageNet- 1K training set. In particular, ImageNet-A and ImageNet-R only contain 200 classes subset of ImageNet- I,,, thus only consider the output logits of the image classifier over these sub-classes.
G.4 PRIVACY ANALYSIS
Membership Attack In this study, we implement a membership inference attack, which allows an adversary to determine whether specific example belongs the training set of trained model. To achieve this, we adopt the Likelihood Ratio Attack (LiRA) as the state-of-the-arr method, as proposed by Carlini al. Carlini al. (2022). Our objective attack by conducting experiments on two different training approaches: the privacy training dataset and synthetic data. Specifically, we focus on the low false-positive rate regime, where the attack expected to be more effective. To begin, we train a total of 16 shadow models using random samples from the original IN-10 (ImageNette) dataset. Half these models are trained on target point (x, y), while the other half are not. This division allows to create both IN and OUT models for the target point (x, y). By fitting two Gaussian distributions to the confidences the and OUT models, measured in logit scale, are able capture the characteristics of these models. Next, query the confidence of the target model on the target point (x, y). Using a parametric Likelihood-ratio test, we compare this confidence value with the distributions obtained from the IN and OUT models. Based on this comparison, we can infer whether the target point (x. y) is likely to be member of the training
evaluate the performance of this
not. In order to train the shadow models, we utilize the ResNet50 architecture. We always begin by finetuning the diffusion model using the sampled data. Once the finetuning is complete, we generate synthetic data of equal size the original dataset. Subsequently, we train the shadow models using this synthetic data. As result, we have total of 16 finetuned diffusion models and ResNet50 models for our experiments. By implementing the membership inference attack and both the privacy training dataset and synthetic data, we aim to provide
evaluating its performance insights into the vulnerability of trained models to such attacks. The results of our experiments shed light on the effectiveness of the Likelihood Ratio Attack in the low false-positive rate regime and contribute to the ongoing research on model privacy and security.
Visual Appearance In this study, we build upon the prior research conducted by Zhang et al. (Zhang al., 2023) and Somepalli et al. (Somepalli et al., 2023) and utilize the Self-Supervised Content
18

---

Published as a conference paper at ICLR 2024
Duplication (SSCD) method proposed by Pizzi et al. (Pizzi al., 2022) for content plagiarism detection. SSCD is a self-supervised approach that is based on SimCLR (Chen et al., 2020) and incorporates InfoNCE loss (Oord et al., 2018), entropy regularization of latent space representation, and various data augmentation techniques. For the experiment, we use the ResNeXt101 model trained the DISC dataset (Douze al., 2021) provided by the official https: / thub. com/ baacbookresaacch/eeeee
copy Seopy-deteet ect ionitoo//ttie streeeee///ee/eeeeeee / /maaiii in as our detection model. To train our model, we employ the ResNeXt101 architecture and train it on the DISC dataset (Douze et al., 2021). We first extract 9024-dimensional feature vectors for both the query image and all reference images using the trained detection model. These feature vectors are then normalized using L2 normalization. By computing the inner product between the query feature vector and each reference feature vector, we obtain similarity score, which indicates the degree of similarity between the query and reference images. Based on these similarity scores, we select the top five reference features that exhibit the highest similarity with the query feature. These selected features allow us to retrieve the corresponding images from the dataset. To further evaluate the similarity between the query and reference images, we rank the similarity scores and present the top 3 images along with their corresponding real data.
G.5 FURTHER ABLATION STUDY ANALYSIS
In Sec. 4.1, conducted ablation study on three key techniques: (1) Latent Prior, (2) Visual Guidance, and (3) Distribution Matching with MMD loss. Note that, Visual Guidance and Distribution Matching require finetuning on Stable Diffusion with either simple diffusion model loss Ho al. (2020) if Distribution Matching not adopted or otherwise augmented distribution matching loss. Latent Prior can be adopted with or without finetuning on Stable Diffusion. This section provides more detailed analysis of the effectiveness of each module.
Lafent Prior The application of Latent Prior demonstrates significant performance improvements. Specifically, implementing Latent Prior results in top-1 accuracy enhancements of 8.2% and 10.3% on ImageNette and ImageNet-100 100, respectively, when compared the baseline. This underscores the efficacy of Latent Prior in our framework. However, even without Latent Prior, Visual Guidance and Distribution Matching Loss also contribute to performance enhancements. Without Latent Prior, these methods lead to improvements of 1.1% and 1.8% on ImageNet- 10 and ImageNet-100, respectively. When integrated with Latent Prior, the improvement further increases to 2.5% and 3.7% on these datasets.
Visual Guidance For Visual Guidance, we employ the "[intra-class Visual Guidance]" token in addition to the base prompt template "photo of [classname], [Image Caption]". We achieve substantial performance gains with the inclusion of Visual Guidance, leading to improvements of 1.5% and 0.7% on ImageNet-10,, and 0.6% and 0.2% ImageNet-100. respectively, when combined with the Latent Prior. This demonstrate the benefit of having detailed image features along with textual description for informative training data synthesis.
Distribution Matching Loss Motivated by the known limitations of diffusion loss's looseness, as discussed in See. 3. we further include MMD loss have a tigher bound over the distribution discrepancy between real and synthetic data. The results demonstrate that the addition of MMD loss brings about consistent improvements. Specifically, on top the vanilla diffusion loss, the MMD loss contributes an increase of 0.4%/0.5% on ImageNette and ImageNet- 100. When combined with Visual Guidance, these improvements are 0.6% %, and with both Latent Prior and Visual Guidance, they are 1.0%/0.7%, respectively.
G.6 MULTI-SEED EXPERIMENT
To examine the stability of our proposed method, we deliver three-seed multi-run experiment on the ImageNet (IN- 10, 100, IN-1K) dataset, following the same training recipe as in Tab. 6. As demonstrated in Tab. 9, we observe that for small-scale dataset (IN-10), the variability is larger com- pared to larger datasets 100, IN- but for all settings, variation remains within an acceptable range and maintains substantial margin against the baseline method.
19

---

Published as a conference paper at ICLR 2024
Table 9: Multi-Seed Experiment Results. Top-1 Classification Accuracy across various datasets (IN- 10, IN-100, (N-1k) over multiple runs. Average shows the mean and standard deviation of results of multiple runs.
IN-10
IN-100
IN-IK
Run
90,5
80.0
70,9
Run 2
91.2
79.9
70.6
Run
90.7
80.2
70.9
Average
90.8 + 0.29
80.0 + 0.12
70.8 + 0.14
G.7 MORE COMPLEX HUMAN FACE DATASET
To further explore the applicability of our proposed framework, we conduct experiments on more complex tasks involving human faces. The challenges involved with the task the human face usually come with more fine-grained facial attributes and natural demographic bias with spurious correlation in data distribution. Thus, we leverage the CelebA Liu et al. (2015) dataset under general facial attribution classification task. Consistent with other experiments, we synthesize training data with Stable Diffusion 1.5 and train ResNet50 from scratch for 30 epochs.
Facial Attribute Classification One of the primary tasks related to human face recognition is attribute classification. We conduct this the CelebA dataset, with three different attributes for prediction. For synthesizing training data, the "A [Attribute Name] person' serves as the base prompt template. We select subset of CelebA dataset and formulate three binary classifications of Smiling, Attractive, and Heavy Makeup. As shown in Tab. 10, for concert facial attribute like
Table 10: CelebA Facial Attribute Classification Result.
Smiling
Attractive
Heavy Makeup
Real Data
89.65
86.22
85.60
Baseline
76.90 86.93
65.11 74.74
73.35
OURS
82.55
Smiling and Heavy Makeup, we observe synthetic data from our framework can represent facial attribute levels close to the training data. For more subtle concepts like Allructive, the distribution shift between synthetic data and real data is larger, leads to slightly less optimal results. In all cases, our synthetic data outperform the baseline by a considerable margin. This validates the capability of framework to adapt to fine-graines facial features.
Future work in More Complex Face Recognition While we primarily explore the utility of synthetic data in capturing general facial features, delving into the domain of individual human face recognition presents a considerably more challenging task. This complexity arises from the extreme scarcity of individual-specin Recognizing the significance and complexity of this area, we identify it as an important avenue for future work.
data and the heightened need for privacy preservation in such contexts.
H FURTHER DISCUSSION
H.1 SIGNIFICANT OF CLASS-LIKELLHHOO MATCHING IN FINE-GRAINED CLASSIFICATION
In Sec. 4.1, we observe more significant advantage of our synthetic data over fine-grain classification tasks (i.e. CUB (Wah et al., 2011), Cars (Krause et al., 2013)) than natural image classification (i.e ImageNet (Deng et al., 2009)) in comparison to baseline. This is due to the requirement of more detailed decision boundaries among classes. This highlights the importance of the class-likelihodd alignment q(y(x) pe (yllzz)) This also indicates solely increasing in the capacity of the generative model synthesizing high-quality images insufficient to handle hard image classification without explicit alignment between the joint distribution between image and annotation.
20

---

Published as a conference paper at ICLR 2024
H.2 BETTER OUT-OF-DISTRIBUTION GENERALIZATION PERFORMANCE
In the assessment of OOD generalization performance Sec. 4,4, we find that with scaling-up ImageNet- IK performance, although the in-distribution performance of training with synthetic data is still inferior to real data, the OOD generalization performance already outperforms the real counterpart an earlier breaking point (i.e. in ImageNet-R (IN-R) (Hendrycks et al., 2021a) and ImageNet-A (IN-A) (Hendryeks et al., 2021b)). While we synthesize training data to reproduce the target data distribution, the superior OOD generalization for models trained on synthetic data indicates a greater potential for synthetic data in strengthing the OOD generalization performance.
21


Reviews:
[{"review_id": "MuzivOibHp", "reviewer": "Reviewer_WEFi", "summary": "The paper proposes a novel diffusion-based approach for generation of synthetic training data. The key innovations of the method are three-fold. Firstly, an MMD-based distribution matching objective between the synthetic and target datasets is employed as fine-tuning step for the diffusion model. Second, visual guidance in the form of intra-class mean features from CLIP plus image captions and class names are used to condition the diffusion model. Lastly, a VAE is employed to obtain the latent codes of real samples for initializing the latent prior of the diffusion model. Experimental evaluations validate the efficacy of the proposed approach across a range of settings and datasets.", "questions": "Why are so many of the cells in table 1 empty?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**Clarity.** The writing, figures, and tables are exceptionally clear.\n\n**Originality.** The paper contains three innovations: feature distribution alignment via MMD, conditioned visual guidance, and VAE-based latent prior initialization. While not algorithmically revolutionary, these innovations are well-motivated and highly effective. \n\n**Quality.** The paper provides interesting theoretical justifications for the proposed innovations and a thorough evaluation of the proposed method, including adequate comparisons to the SoTA and a carefully conducted ablation study.\n\n**Significance.** The proposed approach outperforms the SoTA by huge margins (10%, 17%, 30%, etc.) across a range of datasets for both in and out of distribution data.", "weaknesses": "Details of privacy analysis are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x31k50f2Wt", "reviewer": "Reviewer_8NDL", "summary": "This paper studies the problem of synthesizing data which better matches the distribution of real data, and further improve the performance of model training. It comes up with a framework containing two parts of emphasizing the synthetic data distribution. Based on the experimental and theoretical analysis, the performance improves by a significant number on different tasks.", "questions": "Please refer to the previous sections", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The study problem is interesting. As the synthetic data is easier to obtain and the real data often comes with privacy concerns, it is useful to understand how synthetic data can contribute, as well as the training process;\n2. The proposed framework is sound and easy to understand;\n3. The experiments are extensive, and different types of tasks are considered.", "weaknesses": "1. It seems there is only 1 run on all the experiments. It is beneficial to include more than 1 repeats for the experiments to exclude the confounders;\n2. The experiments are extensive, and I'd also like to see how the proposed framework work on more complex tasks, for example tasks on human faces;\n3. Some figures can be improved, for example, Fig. 3, the text overlaps with the curves;", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "614G38pyQf", "reviewer": "Reviewer_BpUK", "summary": "This paper proposes a principled theoretical framework for training data synthesis from a distribution matching perspective. The key idea is to frame training data synthesis as matching the joint distribution of data and labels between real and synthetic datasets. This involves two aspects: (1) matching the marginal data distribution, and (2) matching the conditional label distribution given data.\n\nBased on this framework, the authors improve upon the state-of-the-art Stable Diffusion model for image synthesis through three strategies: (1) adding an MMD loss to better align feature distributions, (2) incorporating visual cues along with text prompts for conditioned sampling, and (3) using a informative prior for latent diffusion sampling.\n\nExperiments demonstrate improved image classification performance using the proposed synthetic data, outperforming previous methods. Advantages are shown in replacing real data, augmenting real data, scaling up synthetic data, and for OOD generalization and privacy preservation.", "questions": "see weakness", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* The paper offers a theoretical perspective on training data synthesis, formalizing the objective as distribution matching between real and synthetic datasets. The framework clearly decomposes the overall problem into marginal data distribution matching and conditional label distribution matching. This exposes the different requirements.\n* The proposed improvements are well-motivated based on aligning with the principles of the theoretical framework. \n* Thorough experiments validate the utility of the proposed synthetic data on diverse image classification benchmarks. Gains are shown over state-of-the-art baselines. The work highlights promising advantages of synthetic data for OOD generalization and privacy preservation.", "weaknesses": "* The improvment over Imagen seems limited. The comparisons between the proposed method and Imagen are limited to ImageNet-1K, making the superiority unclear on other datasets. \n\n* This work utilizes the prompt in format of \"photo of [classname], [Image Caption], [Intra-class Visual Guidance]\". Are there any other comparative methods that also use the same \"[Image Caption]\"? Whether the improvment are mainly from \"[Image Caption]\" instead of \"[Intra-class Visual Guidance]\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3OUpbqV6EG", "reviewer": "Reviewer_wBAQ", "summary": "The authors introduce a distribution matching (MMD) framework and implements it within the Stable Diffusion to improve data synthesis. In particular, feature distribution alignment, conditional visual guidance and latent prior initialization are introduced to finetune the Stable Diffusion.  Experiments demonstrate the effectiveness of their synthetic data across image classification, OOD generalization and privacy preservation tasks.", "questions": "See the weakness above. Other questions:\n\n1. For the finetune only in Table 2, does it mean finetuning the Stable Diffusion with original losses on the target datasets? \n2. For the latent prior only in Table 2, does it mean simply adopting the proposed prior for the pretrained Stable Diffusion to synthesiz the target images?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "1. The paper is easy to follow.\n2. Experiments on various tasks are provided and the results appear promising.", "weaknesses": "1. The overall idea of Maximum Mean Discrepancy (MMD) for learning data distribution is not novel, which has been explored before, like [1].   \n2. The derived MMD loss based on diffusion model (Eq. (7)) is the same as the simplified diffusion model loss (Ho et al. 2020). This means the proposed loss does not bring anything new for the current diffusion model. \n3. The effectiveness of the proposed MMD loss (Eq. (7)) is doubtful. From Table 2, the performance gain brought by MMD is not significant compared to finetuning only. \n4. The promising performance gains are significantly attributed to the latent prior (Table 2). Given that the latent prior improvement is not introduced by this work but has been already explored by the literature (Section 3.3), the contribution of this work is marginal.\n\nMinor:\n1. $\\epsilon_0$ is not defined in Eq. (7).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)