The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better
Authors: Scott Geng
Cheng-Yu Hsieh
Vivek Ramanujan
Matthew Wallingford
Chun-Liang Li
Pang Wei Koh
Ranjay Krishna
Venue: NeurIPS 2024 poster (2024)
Keywords: Synthetic training data
task adaptation
data-centric machine learning
Primary area: machine_vision
TL;DR: Our paper asks: does training on synthetic images from a generative model provide any gain beyond training on the upstream data used to train the generative model?

Abstract: Generative text-to-image models enable us to synthesize unlimited amounts of images in a controllable manner, spurring many recent efforts to train vision models with synthetic data. However, every synthetic image ultimately originates from the upstream data used to train the generator. Does the intermediate generator provide additional information over directly training on relevant parts of the upstream data? 
Grounding this question in the setting of image classification, we compare finetuning on task-relevant, targeted synthetic data generated by Stable Diffusion---a generative model trained on the LAION-2B dataset---against finetuning on targeted real images retrieved directly from LAION-2B. We show that while synthetic data can benefit some downstream tasks, it is universally matched or outperformed by real data from the simple retrieval baseline. Our analysis suggests that this underperformance is partially due to generator artifacts and inaccurate task-relevant visual details in the synthetic images. Overall, we argue that targeted retrieval is a critical baseline to consider when training with synthetic data---a baseline that current methods do not yet surpass. We release code, data, and models at [https://github.com/scottgeng00/unmet-promise/](https://github.com/scottgeng00/unmet-promise).

The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better
Scott Geng
Cheng-Yu Yu Hsieh
Vivek Ramanujan*
Matthew Wallingford* Ranjay Krishna* 4
Chun-Liang Li
Pang Wei Koh #
University of Washington
Allen Institute for Al
sgengêcs washington, edu
Abstract
Generative text-to-image models enable us to synthesize unlimited amounts of images in controllable manner, spurring many recent efforts train vision models with synthetic data. However, every synthetic image ultimately originates from the upstream data used train the generator, Does the intermediate generator provide additional information over directly training on relevant parts of the upstream data? Grounding this question in the setting of image classification, tuning on task-relevant, targeted synthetic data generated by Stable Diffusion generative model trained on the LAION-2E dataset against finetuning on targeted real images retrieved directly from LAION-28 2B. We show that while synthetic data can benefit some downstream tasks, it universally matched or outperformed by real data from the simple retrieval baseline. Our analysis suggests that this underperformance is partially due to generator artifacts and inaccurate task-relevant visual details in the synthetic images. Overall, argue that targeted retrieval is a critical baseline to consider when training with synthetic data- bascline that current methods do not yet surpass. We release code, data, and models at https: //githbb com/ wwom/sootteegg/tt /unmetutttteee promiise............
compare fine-
1 Introduction
The success of modern machine learning systems fundamentally relies on the quantity [33, 10, 28, 57, 51, 65), quality [20, 72, 45, 44, 36), and distribution [17, 22, 62, 70, 9] of the data they trained on. However, acquiring large amounts of quality data remains challenging, due to the sheer cost of data collection and annotation. As demand for training data continues rise, the field is actively exploring approaches to automatically curate data at scale [20, 2, 18]. One burgeoning approach is to source synthetic training data from conditional generative models. Generative models enable data to be tailored to specific requirements and generated at scale, presenting promising alternative to the challenges of real data curation. Recent work highlights this potential: for example, in natural language processing (NLP), researchers prompt strong proprictary language models to cheaply synthesize large-scale datasets for instruction tuning [68, 29, 61).
Analogously, in computer vision- the focus of our research-many recent works train models on synthetic images from modern text-to-image generators, aiming to achieve state-of-the-art visual recognition performance [56, 63, 3, 23, 24]. For example, SynCLR [62] cleverly prompts Stable Diffusion for synthetic images tailored pre-specified downstream image recognition domains; they find that a CLIP-like model trained from scratch on the resulting targeted synthetic images can outperform CLIP trained on LAION-2B, significantly larger untargeted dataset of real images. This result quite surprising. Stable Diffusion is also trained on LAION-2B, so by the data processing inequality, the synthetic images it generates cannot contain any additional information about LAION-
"Equal advising.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

some target Real (General) Image-Text Dataset
Figure I: Given an upstream dataset of general real image-text pairs, we aim to curate a largeted dataset to train a learner task. We can either (1) retrieve targeted real images directly from the upstream dataset, or we can (2) first train intermediate genera- tive model and then synthesize tar- geted synthetic images. By compar- ing these two approaches, our paper seeks measure what value training on generated synthetic data adds.
Targeted Real Data
Learner
Retrieve
shoppy dogg
Train
Synthesize
0_0
HE
Targeted
Generative Model
Synthetic Data
Learner
2B over the original real LAION-2B images. Yet, training on these derivative synthetic images appears to outperform training directly on LAION-2B. How do we make sense of these additional gains? Does the generative model truly add useful information on top of its pretraining data?
In this paper, we argue that the performance gained from training on generated synthetic images needs to be contextualized against a critical baseline often missing in prior work: training on real images subselected from the generative model's pretraining data. In particular, prior work has often compared task-targe task-targeted synthetic images to general, untargeted real images (c.g., full LAION-2BI 2B), thereby entangling the effects of training on synthetic versus real images with the effects of targeted versus general data collection. However, these variables are not intrinsically conflated. Any generative model we use to synthesize images fundamentally derives from its upstream training data. Instead of using that upstream data to train an intermediate generative model and synthesize targeted synthetic images, we can alternatively seek directly identify targeted real images from the upstream source through retrieval (Figure 1). By comparing synthetic training data against this retrieval baseline, we isolate the value added by the generative model.
We formalize our study under the ubiquitous problem of task adaptation, where we seek to curate task- targeted images to finetune a pretrained vision model. We empirically compare training on targeted synthetic images generated from Stable Diffusion 1.5-a text-to-iman model trained on the upstream LAION-2B againss training on targeted real images retrieved from LAION-2B itself. We perform hundreds of experiment runs across an order of magnitude of data scales on five visual recognition tasks (ImageNet [14], Describable Textures (DTD) [12], FGVC-Aircraft [41], Stanford- Cars [35], and Oxford Flowers 102 [46]) where training on synthetic data has shown promise [62, 23).
Together, we find that training on targeted real data retrieved from generative model's upstream training dataset outperforms training on synthetic data from the generative model. For example, while training on targeted synthetic images can improve downstream accuracy by up to 7.1% (absolute) on its best-case benchmark -iirrrri,,,ff), training on targeted real images helps even further, boosting accuracy by massive 17.7%. On other benchmarks, such as ImageNet, we find that training on synthetic images can sometimes hurt performance even when training on real data improves it. We further show that these findings hold across several different versions of Stable Diffusion, as well as when we train on a mix of synthetic and real data. Our analysis suggests that the consistent underperformance of models trained on synthetic images is partially due low-level generator artifacts in the synthetic images (e.g., blurs), and partially because synthetic images may distort high-level class-specific visual details that real images preserve.
Overall, we conclude that retrieval is a critical baseline to consider when evaluating the true added utility of generated synthetic training data. Our goal is not make normative claims about whether synthetic data will ever surpass this standard, but to contribute a simple baseline to aim for, and clean set of experiments explicitly measure progress towards surpassing it. For instance, by conceptualizing retrieval from generator's 's training data as a strong alternative to synthesizing data, a natural future direction for improving synthetic training data is to synthesize image compositions that are explicitly absent from the generator's 's upstream training set. Images generated in this manner may offer unique value beyond what can be retrieved from the training data. Finally, in settings where the upstream dataset of generator is unavailable altogether (e.g.. due to privacy concerns, due to proprietary data, or due to download bandwidth restrictions), the retrieval baseline is unrealizable by assumption; synthetic data therefore retains strong utility for distilling knowledge from generative
2

---

models and for privacy preservation. We release all code, models, and over 1TB of generated images to guide future work (https://gi com/ swwttgengooo /unnttt///mttttooeeoooee promise)..
2 Related Work
Learning from synthetic data. Synthetic data has been widely explored in the context of many machine learning problems [68, 29, 25, 32, 61, 58, 5, 40J. For example, in NLP, synthetic data generated from strong large language models [47, 11] has been used to distill foolloii behavior [68] and task-specific knowledge [30] into smaller models. In computer vision, prior works have sought to use synthetic data to improve the state-of-the -o--th---tttttttt across breadth of visual tasks, such as object detection [48, 31], semantic segmentation [54, 52, 8J, and optical flow [15]. Traditionally, this synthetic training data has been sourced from expert-crafted simulation and rendering pipelines [49, 15, 52, 8, 54, 31J. Recent advances in text-to-image synthesis via diffusion models [59, 27, 53] are changing this paradigm, inspiring a new wave of works that seek to train visual models on synthetic data algorithmically sampled from large-scale generative models [24, 63, 56, 23, 74, 37]. Recent works have also sought algorithmically sample synthetic data from diffusion models trained on data-scarce domains [55]. This structural shift in the source of synthetic images from supervision from upstream data raises a critical question: does the intermediate step of training a generator and sampling synthetic data provide any added useful information over simply training on relevant parts of the upstream data directly? Our work formalizes and empirically grounds this question, contributing experiments and baselines to rigorously measure the benefits of training on modern data-derived synthetic data.
programmatic simulation to a learned generator that itself derives
Adapting pretrained vision models. Large-scale pretrained image models such as CLIP [50, 10] offer transferable visual features that benefit a wide range of downstream vision tasks. It is now common practice to pretrained models
starting point when deploying downstream task- specific models instead of training them from scratch [69, 66]. From an algorithmic perspective, many methods have been proposed to adapt CLIP models to downstream tasks, each with varying trade-offs [71, 73, 42, 21, 4, 69]. We choose to study simple full-finetuning, centering our work on the data adapt as opposed to the algorithm. In particular, the quality and relevance of adaptation data has a crucial impact on downstream task performance; distribution shifts at inference time can significantly hurt performance [34]. Acquiring task-targeted data thus remains an active area of research [39, 67, 25J. Most related to our work is [39, 67]. who also employ retrieval as a technique for collecting task targeted-data. Our work builds upon these methods construct baselines for systematically measuring the true added utility of model-genente geneeatedd synthetic training images.
Retrieval as a baseline for synthetic data. Recent studies [74, 7] similarly use retrieval from generative model training data as a baseline for synthetic training images. For example, [74] adapts Stable Diffusion with real medical images and then generates synthetic medical images to train a ResNet from scratch; the resulting ResNet outperforms a ResNet trained on LetONeretrieved medical images. Consistent with our work, [74, 7] find that in the low-data regime of general image recognition tasks (e-g., subsampled ImageNet- I STL,10),, augmenting real datasets with LAION- retrieved images outperforms augmenting with Stable Diffusion generated images when training ResNets from scratch. [7] further finds that this performance gap in the low-data regime persists across many synthetic image generation methods, including when they finetune Stable Diffusion with task-specific real data. In the medium- data regime (e.g. full ImageNet-Ik, Ik), [7] finds augmenting with synthetic data matches but does not outperform augmenting with retrieved data. Our work generalizes the shared motivation of measuring the utility of synthetic training images to the modern data-rich regime and wide range of visual classification tasks.
3 Problem Setting and Method
Given a large dataset D of general real image-text pairs and a downstream visual classification task specified set of text class names C, we aim to algorithmically curate a targeted adaptation dataset Dc of images 2; and one-hot class labels yi to finetune and improve a pretrained vision model's performance the downstream task. We compare two high-level approaches for sourcing this targeted data, shown in Figure 1: (1) we retrieve targeted real images directly from D, forming a targeted retrieved dataset D Di (retrieved)
C. D. Alternatively, (2) we generate targeted synthetic images
3

---

by prompting an intermediate text-to-image generative model trained on D, forming a targeted synthetic dataset D (syathetic) We detail each approach below.
3.1 Sourcing data by generating synthetic images
We follow SynCLR [62], representative method for curating synthetic training data from off-the- shelf text-to-image models. Given the set of visual class names C, we first synthesize a large corpus of corresponding image captions by prompting a large language model (details in Appendix C.I). For example, if the class name с E C' is "rose," then generated caption might be "a close-up of pink rose in bloom." We then use these captions as input for text-to-image generator G trained on the upstream data D. yielding large label ys based on the class name used to synthesize its caption. These synthetic images and labels { } form our curated dataset D (symthetie)
of synthesized images T.. Each image is assigned class
3.2 Sourcing data by retrieving real images
Rather than querying a generator trained on an upstream dataset D, we can directly train parts of D itself by retrieving relevant data. D consists of image-text pairs (zt, t.). To retrieve relevant pairs, we consider two strategies. We additionally deduplicate all retrieved images with respect to our evaluation datasets following [20] to minimize test set leakage. We apply NSFW filtering [57].
Strategy 1: hard substring matching. Inspired by [67], we retrieve the set D (retrieved) of all images T2 whose corresponding caption 1; contains least one target class name E C as substring:
De (reirivee)) [(xi v.): (xi, 44) D such that some class E C is substring of 4;)
Here, label is assigned based on the class contained in 4. If an image-text pair (æz 44) E 'D has text 4 containing multiple class names c, c' E C, then we simply retrieve æ; multiple times and assign each instance a different label, once for each unique matched class name
Strategy 2: semantic k-NN retrieval. Hard substring matching is simple and effective when the target visual concepts C are concrete entities that are likely to be described in text captions (e.g.. c c= "fire lily"), but may be less effective when the concepts are abstract (c.g., c = "lined texture"). Thus, also consider semantic (soft) retrieval via CLIP image-text embedding space similarity We convert each target class name e E C. into a set of natural language search queries Qe based on the templates from the original CLIP paper [50]. For each query Oe Qc. we use approximate k-NN search [16] to retrieve the set S., of k-nearest image-text pairs (I, (4) E D by CLIP similarity between the query ge and either the image æ; the text b;:
Sqc = far top-k CLIP(II(, gc) U farg top-k lorg (xnAAi)D
(I.A)ED
We assign each image-text text pair (I,, t4) Sq. class label ys based on the class name in query dc- We form the targeted dataset D (reutiveed) by unioning over all queries E Qa and all classes E C:
Di (letrovee)
[1r vi): (xx, W:) E Sg.
eEC q.EQ.
3.3 Additional data filtering and postprocessing
Data filtering has been shown to improve training performance for both real and synthetic data and widely used [20, 24]. We filter both our synthetic and retrieved datasets following current best filtering practices for synthetic image data [24]. Given curated dataset De. we compute the CLIP similarity of each v; E De with text corresponding to assigned label yi (e.g., "a photo of (class name]") )"), constructed using the CLIP zero-shot classification templates [50]. When there are multiple templates for given class, we aggregate by taking the maximum similarity across templates. We
We perform semantic retrieval using precomputed LAION-2B embeddings from OpenAl CLIP VIT. L/14 [50], the same model Stable Diffusion uses embed text prompts during generation.

---

ImageNet-1K
DTD
FGVC-Alrcraft
StanfordCars
Flowers102
LP Accuracy (%) 75 Accuracy (%)
73
64
70
92
85
72-
62
60
90
80
71
00
60
75
70-
06
58
69
30
84
70
256
102420484096
16
64
128
256
512
32
64
128
256
512
82
16
128
65
64
120
256
90
90
81
96
80
94
70
79
92
96
256 512
102420484096
16 64
128
256
512
16
32
64
128
256.522
90
16
32
64
128
256
16
64
128
256
Lcale Adaptation Dataset Size (K)
Off-the-sheif CLIP
Finetuned Retrieved Data
Fintuned on Synthetic Data
We adapt pretrained CLIP image encoder (dashed purple line) to different downstream image classification tasks, using either (a) targeted synthetic data (orange triangles) generated from Stable Diffusion model trained LAION-2B using (b) targeted real data (blue circles) directly retrieved from LAION-2B. We measure performance via downstream zero-shot (ZS) and linear probing (L.P) accuracy, aggregating results deviation). Overall, while adapting CLIP with targeted synthetic data can sometimes improve performance
Figure 2:
least seeds (error bars indicate + standard
off-the-shelf model, synthetic data universally outperformed targeted real data. This gap persists even when we scale the sample size of the synthetic adaptation dataset beyond the maximum amount of (finite) targeted real data considered (gray shaded regions).
matched by
keep the top 30% of images per class by aggregate similarity. Intuitively. filtering helps remove generated and retrieved images with class-misaligned content. For example, an image labeled "dog" without any dogs present (i.e., due retrieval or generation errors) would receive a lower CL.IP similarity score and likely be discarded. See Appendix C.2 for further discussion.
Our synthetic adaptation datasets (synthetic) class- Mlss-balanced by construction (i.e., uniformly
generate images for each class). We further postprocess the retrieved adaptation datasets D (retrieved)) to improve class balancing by manually fixing a global threshold M and truncating the dataset such that each class label occurs at most M times.
Main Experiments
seek to measure the utility of learning from model-generated synthetic images. Grounding this question empirically, our experiments compare finetuning a pretrained CLIP model on (1) targeted
synthetic images DC (xymthetiic to (2) finetuning on targeted retrieved real images D (retrieved)
Benchmarks. We focus on five downstream tasks where synthetic data has shown promise compared to similar scale untargeted real data [62]. We select (a) ImageNet- IK [14] and Describable Textures (DTD) [12] to evaluate recognition performance on broad categories and (b) FGVC-Aircraft [41], StanfordCars [35], and Oxford Flowers 102 [46] evaluate performance in fine-grained settings. We use standard pre-defined train, test, and validation splits when available, and otherwise randomly subset the training set create missing train-valle vii--valdation splits (details Appendix D.2).
Finetuning data curation. For each downstream benchmark, we first curate an adaptation dataset De (Section 3) by either (1) generating synthetic images with Stable Diffusion 1.5 [53], trained the LAION-2B dataset [57], or (2) retrieving real images directly from LAION-2B. We treat the choice between substring-bar based and semantic retrieval strategies hyperparameter, using downstream validation set accuracy to determine the best choice for each benchmark. Hyperparameters for retrieval detailed in Appendix D.I.
Model adaptation and evaluation. adapt a LAION-2B pretrained CLIP VIT-B/16 [10] image encoder by finetuning the curated adaptation dataset De with cross-entropy classification loss for
5

---

pre number of epochs. To elucidate the scaling trends of synthetic and retrieved data, we finetune across an order of magnitude different adaptation dataset scales, subsampled from the full targeted adaptation dataset De. We report zero-shot (ZS) and linear probing (L.P) test set accuracy, using the benchmark train set to train the I.P head. For both I.P and 7S evaluation, we use the validation set to identify the best epoch and finetuning hyperparameters. For each data scale, we aggregate accuracy across the results of at least three random seeds, and report the standard deviation due to seed randomness. Please refer to Appendix D.3 for further evaluation details, and Appendix D.4 for further training and hyperparameter details.
4.1 Main results: synthetic training data lags behind a baseline of retrieved real images
We present our main zero-shot and linear probing scaling results in Figure 2.
At equivalent data scales, finetuning with model-generated synthetic images can help, but is universally matched or outperformed by finetuning directly with images from the generator's 's training data. Consistent with prior research [62]. we find that training with targeted synthetic data can improve an unadapted model. For example, on FGVC-Aireraft the setting where previous works have found strongest
generated images improved downstream linear probing accuracy by an average of 3.8 percentage points over an off-the-shelf CLIP model (64.9% > 68.7%); on DTD, training with 110K synthetic images improves zero-shot accuracy by 3.3 points (56.3% + 59.6%).
finetuning with 139K Stable-Diffusie
However, the gains from training on synthetic data are consistently matched or surpassed by training on retrieved real data. For instance, on FGVC-Aircraft, finetuning with an equivalent 139K LAION- 2B retrieved images boosts performance by a massive 17.8 points (64.9% + 82. 32).% Moreover, adapting with retrieved data can improve performance even when synthetic data does not (e.g., on ImageNet and Flowers102 cero accuracy.) Finally, adapting with synthetic data can sometimes even hurt performance (ImageNet, StanfordCars, Flowers 102 zero-shot), while targeted retrieved data improves or least does not hurt performance across all settings considered. Given equal amounts of targeted retrieved and synthetic data, retrieved data is the clear winner.
Synthetic data can sometimes decrease the gap with retrieved data given additional scale, but remains behind. The amount of data we can retrieve is fundamentally limited by the finite upstream data pool. For example, even after searching all 2 billion LAION-2B samples for images containing an FGVC-Airerale class name in the caption, substring-based retrieval returned only 139K targeted images post-filterng
contrast, it is straightforward to create ever-larger synthetic datasets by simply generating more data.
Scaling the synthetic adaptation dataset size beyond the amount of retrieved data considered (illus- trated in the gray-shaded regions of Figure 2), we find that increasing the amount of targeted synthetic data does not always improve performance. For example, on DTD, synthetic data exhibits U-shaped scaling, with performance positively scaling up to 1 10K synthetic training images, after which performance declines. On ImageNet, Flowers 102, and StanfordCars, increasing the synthetic dataset size consistently hurts zero-shot accuracy and has minimal impact on linear probing performance.
On Aircraft, scaling helps; there is log-lineaa relationship between the size of the synthetic adaptation dataset and downstream linear probing accuracy (e.g., scaling from 139K- - 250K synthetic images improves linear probing accuracy from 68.7% + 70.7%). However, synthetic data still lags behind retrieved data: matching the performance of a mere 15K retrieved aircraft images requires scaling the synthetic dataset to 500K images, reflecting ~33 difference in dataset size and required finetuning compute. Naively extrapolating this ratio, matching the performance of the full 139K retrieved adaptation dataset would require nearly 5M synthetic images after top 30% filtering. We note, however, that synthetic data is unlikely to truly scale infinitely, as synthetic data fundamentally derives from the (finite) training set of our generative model. Still, the performance of synthetic data is likely unsaturated at the 500K scale (i.e., accuracy is still trending up); due to compute limitations, studying whether further scaling can outperform retrieved data is left for future work.
Synthetic data can improve model's task representation without significantly improving the model's task performance. Broadly speaking, zero-shot task accuracy measures a model's ability to directly solve the downstream task, whereas linear probing accuracy measures the quality of the model's learned task- releveellant representation. We find that even when training on synthetic data improves the model's 's representation (i.e., downstream linear probing accuracy), it may not

---

significantly improve the model's zero-shot accuracy. In contrast. when training on retrieved data improves the model's representation, zero-shot accuracy also exhibits positive scaling. For example. CLIP adapted with either 15K retrieved images or 500K synthetic images both achieve similar linear probing aceuracy (~ 72%), yet the model adapted with synthetic data achieves a much worse zero- shot accuracy (28.9% versus 39.5%). We discuss possible reasons for this qualitative discrepancy in model behavior our analyses below (Section 5.1).
Analysis
In this section, we explore two questions to better understand our main results. First, what factors drive the underperformance synthetie data? Second, do our findings hold under variations of our experimental setup? We focus analysis experiments ImageNet, to understand general image recognition performance, and FGVC-Aireraft, the sole benchmark where synthetic data exhibited strong positive log- linear scaling. Additional analysis experiments presented in Appendix B.
5.1 Why does synthetic data lag retrieved real data?
Qualitative visualizations. We visualize a random selection images from our curated synthetic and retrieved adaptation datasets in Figure 3. Compared to retrieved real images, we observe that the synthetic images (1) contain low-level generator artifacts, and (2) differ in visual content distribution, both in terms of semantic details and overall image composition. For example, although the synthetic FGVC-Aircraft adaptation images (top two rows of Figure 3) are recognizable as airplanes, the visual content often contains incorrect class-relevan semantic details: correctly-depicted "Airbus A320" should have one engine per wing and two sets of wheels at its rear, yet our synthetic images often exhibit incorrect engine or wheel configurations. This qualitative discrepancy in visual detail precision may partially explain why training on synthetic data does not improve task zero-shot accuracy; synthetic images do not retain enough dlass-accurate details directly teach the model the downstream task. In contrast, training on synthetic images can improve linear probing accuracy, because the synthetic images still broadly look like aireraft and thus may help align the model's representation to the downstream domain.
Test Set Image:
Retrieved Real Images
Generated Synthetic Images
"Airbus A320"
"Cessna 172"
"Flute"
"Tabby Cat"
Figure 3: We visualize retrieved real images and synthetic images from our targeted adaptation datasets for FGVC-Aireraft (top two rows) and ImageNet- IK (bottom two rows), alongside ground truth images (left column) for reference. Compared retrieved images, synthetic images often (1) contain generator artifacts (e.g., the blur on the edges of the "Cessna 172", the eyes and mouth of the "Tabby Cat") and also (2) distort class-relevant visual content, such as the engine configuration of a true "Airbus A320" (i.e., exactly one engine per wing) and the entire visual appearance of "Flute". We hypothesize that both factors contribute to synthetic training data's underperformance versus real training data.

---

Synthetically perturbing retrieved real images. To disentangle the effect of low-level generator artifacts and visual content differences between synthetic and retrieved real images on downstream model performance, we trained on "hybrid" images that have similar semantic visual content as our retrieved real images but contain generator artifacts like our synthetic images. Following SDEdit [43], we use Stable Diffusion to synthetically perturb our retrieved images to introduce model-specific artifacts present the synthetic images Stable Diffusion generates. Given a noise strength parameter [0, and retrieved image To, SDEdit adds Gaussian noise to 2o according to timestep & of Stable Diffusion's s time-dependent forward process. We then denoise the noisy image using the same reverse diffusion process as in eex-tto-image generation, yielding perturbed image x(() that looks semantically like while also containing Stable Diffusion-speciffi artifacts. Increasing increases the amount of Gaussian noise added to To, thereby increasing the severity of visual artifacts introduced in the resulting (x1)) (see Appendix E. for further details). In pseudocode,
htw.://fusssoooonnonoeesaaaaaala ottp:/ffuuuuooodddddoooooe////o 7): 7).
Starting from the full targeted retrieved adaptation datasets DC (nerroved) for FGVC-Aireraft and Ima- geNet, we use SDEdit to introduce generator artifacts into the retrieved real images over a range of y values. We visualize the resulting perturbed images in Figure 4, and plot the results of training these perturbed images across in Figure 5.
Retrieved
Synthetically Perturbed Images
Real Image
y 0.2
y= 0:4.4
y 0.6
y 0.8
"Airbus
Cat" 4220
"Tabby
Figure 4: We use Stable Diffusion to synthetically perturb real images according to a noise strength parameter 7 [0, 1]. where larger y increases the severity of generator-specife artifacts added by the perturbation. When 2 0.6, the introduced artifacts can be strong enough to damage task-relevant visual details for finegrained tasks like FGVC-Airerat (e.g., the airplane's engine and rear wheels). For broad tasks like ImageNet, artifacts have a lesser impact on class-relevant details; the "Tabby Cat" is recognizable as a cat even after perturbing with high y.
72
ImageNet-1K
83
ImageNut-11
Shot Accuracy (%
FGVC-Aircraft
Linerr Probe Accuracy C
80
FGVC-Aircraft
Zero Shot Accuracy
02
85
70
02
30
0.5
0.2
0.4
0.6
0.8
0.0
0.2
0.6
0.6
0.8
3.0
0.2
0.4
0.8
0.8
0.0
0.2
0.4
0.6
0.8
Porturbation Noise Strength (v)
Perturbation Noise Strength (v)
Perturbation Noise Strangth (v)
Perturbation Noise Strength (v)
Off-the-dhhllf CLIP
Finetuned on Reba veved Data ... Anstuned on Synthetic Data
Finebured on Syrthutically- Parturbee Ratrieved Data
Figure 5: We finetune pretrained CLIP model (dashed purple line) on retrieved real images that synthetically perturbed (green circles) with Stable Diffusion to introduce generator artifacts. The perturbation strength is controlled by a parameter 7 E [0, 1] where larger 7 introduces stronger arti- strong enough damage class-relevant details.
facts; within the gray-shaded region, the artifacts Our results suggest that generator artifacts do contribute to synthetic data's underperformance any artifact level causes performance drop below training retrieved images (dashed blue line). Moreover, differences in visual content between synthetic and retrieved images also matter; even with relatively strong perturbations (7 0.5), training on artifact-affliced perturbed images that retain the semantic content of retrieved images outperforms training on synthetic images (dashed orange line).

---

Our results suggest three takeaways. First, generator artifacts indeed contribute to the underperfor- mance of synthetic training images, especially for fine-graines classification tasks. On PGVC-Aircraft any amount of added generator artifacts drops downstream accuracy. Second, the impact of artifacts is relatively lower for broad classification domains such as ImageNet, where downstream performance is not significantly impacted until we perturb with relatively strong noise strength of y 0.5. Finally, visual content differences between synthetic and retrieved images also play a key role in the performance gap between synthetic and retrieved training data. When we perturb images with strength y = 0.5, the resulting images are heavily afflicted with artifacts, but still retain the important class-relevant details of retrieved real images, such correct airplane engine configurations. Training on y = 0.5 images significantly outperforms training on synthetic images. Intriguingly, training on aireraft images perturbed beyond the point where class-relevant visual details are damaged (1 2 0.6) still outperforms synthetic data; we speculate that this is because these heavily perturhed images still retain the overall image composition of retrieved images.
5.2 Synthesizing data via another generative model
For our main scaling experiments, we generate synthetic image datasets using Stable Diffusion 1.5 to maintain consistency with prior work [62, 23]. To what degree does our choice of generative model impact our findings? At the time of our study, Stable Diffusion 1.x models are the only modern text- to- to-image models with open source training data available to retrieve from. Therefore, we focus our study here on Stable Diffusion (SD) versions 1.1, 1.3, and 1.5. Starting from SD v1.1, which is trained on the full LAION-20 BB dataset, SI) v1.3 and v1.5 are derived by further finetuning on high-quality subsets of LAION-2B. This additional finetuning improves image generation fidelity [53], but may lead to the model forgetting parts of the LAION-2B distribution [19]. Following main experiment setup (Section 4), we use SD vI.1 and SD v1.3 to generate various-sized targeted synthetic adaptation datasets for ImageNet and FGrC-Aircraft. Results are plotted in Figure 6. Overall, while training with synthetic data from different generative models yields varying performance, synthetic data from all generative models considered consistently fall short of retrieval.
ImageNet-1K
ImageNet-1K
FGVC-Aircraft
FGVC-Aircraft
Accuracy (%)
73
Linear Probe Accuracy (%)
83
Zaro Shot Accuracy (9)
70-
50
72
85
60-
72-
50-
70
80
80
69
30
68
25.6
78
1024
20
32
256
60
36
32
128
512
Adaptation Dataset Size (K)
Adaptation
Adaptation
Adaptation
Otf-the CUP
LAIDN-08 Retrieved Data
Synthetic Data
+ Synthetic v1.11 + Synthetic Data (SD v1.3)
Figure 6: We ablate our choice of generative model used to synthesize images in our main experiments, comparing training on synthetic images from Stable Diffusion 1.5 (SD v1.5) to synthetic images from SD v1.1 and SD v1.3. Across multiple Stable Diffusion models-the only modern text-to-image generators with open-source training data available- with images directly retrieved from the generative models' training dataset (LAION-2H) outperforms training with generated images.
5.3 Mixing synthetic and retrieved data
On FivC-Arrraat,, finetuning CLIP with either synthetic or retrieved data alone consistently improves downstream task accuracy. The gains from retrieved data are stronger than the gains from synthetic data across all data scales; however, synthetic data may improve CLIP in ways that is complementary to retrieved data, and thus present orthogonal value. To test this possibility, we measure whether training on a mix of synthetic and retrieved Aircraft adaptation data significantly outperforms training with either alone. Starting from our largest retrieved adaptation dataset D (retrieved)) (139K images), we progressively add in increasing amounts of synthetic images from our synthetic adaptation dataset DÍ (syyathetic) and finetune pretrained CLIP model with the resulting mix. We plot results in Figure 7. We find that training on the mixed images outperforms training on synthetic images alone; however, training on a mix significantly drops performance compared to using retrieved data alone.

---

FGVC-Alrcraft
FGVC-AIrcraft
We finetune pretrained CLIP model (dashed purple line) on dataset comprising mix of targeted synthetic and retrieved real images (green squares). The mix con- structed by adding varying amounts of synthetic images to our full retrieved adaptation dataset. Training on the mix outperforms training on synthetic images alone (orange friangles), but is strictly worse than training on re- trieved images alone (blue circles).
Figure 7:
70
Linear Probe Accuracy (%6
85
Shot Accuracy
60
80
50
75
40
70-
30
65-
20
120
1024
60-
1024 Adaptation Dataset Size (K)
256
Adaptation Dataset Size (K)
Off-te-sheelf CUP
Finetuned Synthetic Data Only
Finetuned Retriaved Data Only
Anetuned on Retrieved Synthetic Mix
Discussion
Our work sought to answer key question: given that all model-generated synthetic images derive from the generator's upstream training data, does training synthetic images provide value over training on the parts of the upstream real data directly? We contribute set rigorous experiments to ground this question empirically, and discover that training on upstream real images collected via our simple retrieval baseline significantly outperforms training on synthetic images, Our initial question answered negatively. We therefore argue that retrieval a critical baseline surpass in order to show value from synthetic training data, and encourage comparison against it in future research.
Importantly, do not seek to make normative claims about whether training with synthetic images will ever surpass this haseline-future work may unlock gains that we have not yet found. As first step, we contribute analyses of why synthetic training images underperform upstream real images, finding that both generator artifacts and semantic errors within synthetic images are key areas for future improvement. Furthermore, given that image retrieval is strong alternative image synthesis, a natural next step is to generate image compositions that are explicitly rare or absent from the generator's upstream training dataset; we are optimistic that synthesizing these "missing" images may offer unique value beyond what is present in the existing upstream real images. Such an approach leverages the compositional generalization abilities of the generator, which recent research promisingly suggests may be stronger than the compositionality of discriminative model trained on the same upstream data [38, 13].
Finally, our findings assume access to the generative model's upstream training data, an assumption that may always hold. The upstream pool may be proprietary or strictly regulated due to privacy concerns. In such settings, training directly on the upstream data is impossible; synthetic data from a generative model trained on this unavailable upstream data remains an exciting alternative to acquire otherwise inaccessible information.
Limitations. As an empirical study, our compute budget limits the number of experimental varia- tions consider. Our results derived from adapting CLIP models with standard full finetuning; we conjecture that our findings generalize to other large-scale pretrained backbones and adaptation methods as well, but Stable Diffussin is the only text-to-image model with publicly available training data to retrieve from (i.e. LAION-2B); we do not study other generators trained on other data pools. Finally, we focus model accuracy, leaving comparison of model robustness and fairness from training on synthetic versus real data future work.
not able to test this empirically. Moreover, at the time of our work,
Acknowledgments and Disclosure of Funding
We graciously thank (in alphabetical order) Eric Frankel, Jacqueline He, Athena Tsu, and Rui Xin for their helpful comments and feedback. SG is supported by the NSF GRFP. PWK is supported by the Singapore National Research Foundation and the National AI Group in the Singapore Ministry of Digital Development and Innovation under the AI Visiting Professorship Programme (award number 10240002--
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[2] Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, al. A survey data selection for language models. arXiv preprint urXiv:2402.1682) 2024.
[3] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304.08466. 2023.
[4] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203. 17274, 2022.
[5] Manel Baradad Jurjo, Jonas Wulff, Tongzhou Wang. Phillip Isola, and Antonio Torralba. Learning to by looking at noise. Advances in Neural Information Processing Systems, 34: 2556-2560 0066-25592200 2021.
[6] Romain Beaumont. Clip retrieval: Easily compute clip embeddings and build a clip retrieval system with them. https: thub
httpa////gggaabb..arroo///////// c04 clip-retrieval, 2022.
[7] Max F Burg, Florian Wenzel. Dominik Zietlow, Max Horn, Osama Makansi, Francesco Lo- catello, and Chris Russell. Image retrieval outperforms diffusion models on data augmentation. Transactions on Machine Learning Research, 2023.
[8] Yuhua Chen, Wen Li, Xiaoran Chen, and Lue Van Gool. Learning semantic segmentation from synthetic data: A geometrically guided input-output adaptation approach. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1841-1850, 2019.
[9] Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi. Matteo Pagliardini. Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, et al. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079 2023.
[10] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2818-2829, 2023.
[11] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):
1-113, 2023.
[12] M. Cimpoi, S. Maji, Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
[13] Kevin Clark and Priyank Jaini. Text-to-image diffusion models zero shot classifiers. Ad- vances in Neural Information Processing Systems, 36, 2024.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Ieee, 2009.
[15] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages
2015.
[16] Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024.
11

---

[17] Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). In International Conference on Machine Learning, pages 2216-6234 PMLR, 2022.
[18] Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309. 17425, 2023.
[19] Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 1(4):128-133, 1999.
[20] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.
[21] Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19338-19347, 2023.
[22] Suchin Gururangan, Ana Marasovié, Swabha Swayamdipta, Kyle Lo. Iz Beltagy, Doug Downey, and Noah A Smith. Don't 't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004. 10964 2020.
[23] Hasan Abed AI Kader Hammoud, Hani Itani, Fabio Pizzati, Philip Torr, Adel Bibi, and Bernard Ghanem Synthclip: Are we ready for a fully synthetic clip training? arXiv preprins arXiv:2402.01832 2024.
[24] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? arXiv preprint TrXii:2210.0757.. 2022.
[25] Xuanli He, Islam Nassar, Jamie Kiros, Gholamreza Haffari, and Mohammad Norouzi. Generate, annotate, and learn: Nip with synthetic text. Transactions of the Association for Computational Linguisties, 10:826-842 842, 2022.
[26] Jonathan Ho and Tim Salimans. 22..:22207.1259.. 2022.
Classifier-free diffusion guidance.
arXiv preprint
[27] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:684006851, 2020.
[28] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 2022.
[29] Or Honovich, Thomas Scialom, Omer Levy, and Timo Schiek. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:22/2.09689) 2022.
[30] Cheng- Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen- Yu Lee, and Tomas Pfister. Distilling step-by-step! outperform- ing larger language models with less training data and smaller model sizes. arXiv preprint vrXiv:2300.02201. 2023.
[31] Matthew
Charles Barto, Rounak Mehta, Sharath Nittur Sridhar, Karl Rosaen, and Ram Vasudevan. Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks? arXiv preprint arXiv:1610.0133
2016.
[32] Jachun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, and Yejin Choi. Impossible distillation: from low-quality model to high-quality dataset & model for summarization and paraphrasing. arXiv preprint 2.0...663..6635 2023.
[33] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXh::001..0861,, 2020.
12

---

[34] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, al. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning. pages 5637-5664. PMLR, 2021.
[35] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[36] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXivv2107.064999 2021.
[37] Shiye Lei, Hao Chen, Sen Zhang, Bo Zhao, and Dacheng Tao. Image captions are natural prompts for text-to-image models. arXiv preprint arXiv:2307.08526 2023.
[38] Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly zero-shot classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2206-2217, 2023.
[39] Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jae Lee, and Chunyuan Li. Learning customized visual models with retrieval- -aageeeedd knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15148-15158 2023.
[40] Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang. Denny Zhou, et al. Best practices and lessons learned on synthetic data for language models. arXiv preprint arXiv:2404.07503, 2024.
[41] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv: 2013.
[42] Chengzhi Mao, Scott Geng, Junfeng Yang. Xin Wang, and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models. arXiv preprint arXiv:2212.07011 2022.
[43] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108. 01073, 2021.
[44] Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. Advances in Neural Information Processing Systems, 35:21455 21469,, 2022.
[45] Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt. Im- proving multimodal datasets with image captioning. Advances in Neural Information Processing Systems, 36, 2024.
[46] Maria- Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision. graphics & image processing, pages 722-729. IEEE, 2008.
[47] OpenAl. Chatgpt. 2022.
[48] Xingchao Peng, Baochen Sun, Karim Ali, and Kate Saenko. Learning deep object detectors from 3d models. In Proceedings of the IEEE international conference on computer vision,
pages 1278-1286, 2015.
[49] Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:/710.0692 2017.
[50] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PMLR, 2021.
13

---

[51] Vivek Ramanujan, Thao Nguyen, Sewoong Oh, Ali Farhadi, and Ludwig Schmidt. On the connection between pre-training data diversity and fine-tuning robustness. Advances Neural Information Processing Systems, 36, 2024.
[52] Stephan R Richter, Vibhav Vincet, Stefan Roth, and Vladien Koltun. Playing for data: Ground truth from computer games. In Computer Vision- ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 102-118. Springer, 2016.
[53] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[54] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 3234-3243, 2016.
[55] Daniel G Saragih, Atsuhiro Hibi, and Pascal N Tyrrell. Using diffusion models to generate synthetic labeled data for medical image segmentation. International Journal of Computer Assisted Radiology and Surgery, pages 1-11, 2024.
[56] Mert Bülent Sarryildzz Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learning transferable representations from synthetic imagenet clones. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8011-8021, 2023.
[57] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25294 2022.
[58] David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint 172 2017.
[59] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[60] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500,, 2020.
[61] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Perey Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction- following model. Stanford Center for Research on Foundation Models. https://crfim. stanford. eduz2023/03/13lllpaecd huml, 3(6)77 2023.
[62] Yonglong Tian, Lijie Fan, Kaifeng Chen, Dina Katabi, Dilip Krishnan, and Phillip Isola. Learning vision from models rivals learning vision from data. arXiv preprint arXiv:2312.17742. 2023.
[63] Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. Advances in Neural Information Processing Systems, 36, 2024.
[64] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 2tt.:::300.0928880 2023.
14

---

[65] Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philipp HS Torr, Adel Bibi, Samuel Albanie, and Matthias Bethge. No" zero-shot" without exponential data: Pretraining con- cept frequency determines multimodal model performance. arXiv preprini urXiv:2404.044 2024.
[66] Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, and Oncel Tuzel. Mobileclip: Fast image-text models through multi-modal reinforced training. arXiv preprint arXiv:231..17444 2023.
[67] Matthew Wallingford, Vivek Ramanujan, Alex Fang, Aditya Kusupati, Roozbeh Mottahhi, Aniruddha Kembhavi, Ludwig Schmidt, and Ali Farhadi. Neural priming for sample-efficient adaptation. Advances in Neural Information Processing Systems, 36, 2024.
[68] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self self-generated instruc- tions. arXiv preprint arXiv:2212. 105660 2022.
[69] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong. et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7959-799 0001-9911222 2022.
[70] Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. arXiv preprini 2.XX::2309.1677 2023.
[71] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training- clip-adapter for better vision-language modeling. arXiv preprint arXiv:21/1.0392. 2021.
[72] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
[73] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9))3337-2338, 2022.
[74] Yongchao Zhou, Hshmat Sahak, and Jimmy Ba. Using synthetic data for data augmentation to improve classification accuracy. 2023.
15

---

A Broader Impacts
We release all models and synthetic data from our work benefit future research. All assets released are meant to be scientific research artifacts. Moreover, all models released are task-specific classification models, limiting potential for misuse. Nonetheless, we not encourage the use or deployment of our models in practice.
B Additional Findings
We further validate the main findings of our paper by (1) providing further analysis of the synthetic training data and (2) considering two additional variations upon our experimental setup: applying more aggressive decontamination, and generating synthetic images with alternative prompting strategies.
B.1 Additional data analysis: CLIP score distributions
Do the distributions of CLIP similarity scores in the final synthetic and retrieved training datasets significantly differ? Since we filter the synthetic and retrieved data independently of each other to maintain a clean experimetnal setup (i.e., we do not use any information from the retrieved images to inform the selection of synthetic data, and vice versa), there may be differences in CLIP similarity that potentially explain gaps in downstream training performance.
We histogram the CLIP similarity score distributions of the resulting filtered synthetic and retrieved training data in Figure 8. Overall, despite setting the filter score threshold for synthetic and retrieved data independently,
find that the distribution of post-filtering synthetic image CLIP scores is right-shifted compared to the distribution post-filtering retrieved image CLIP scores. In other words, synthetic images have comparable average; CL.IP judges synthetic data to be higher quality on average. CLIP score differences alone do not explain the lagging training performance of synthetic data.
even higher CLIP scores than retrieved images
Further corroborating this finding with a case study, we observed that CLIP assigns the synthetic "flute" images in Figure 3 relatively high similarity scores of 0.285, 0.265 and 0.263, despite the fact they they are obviously wrong to the human eye. For reference, a CLIP score of 0.249 reflects the top 30% of all retrieved 'flute' images. Overall, while the current best practice of using CLIP score to filter synthetic data does improve performance, CLIP score remains limited. Future work may explore synthetic data filtering methods that do not depend on CLIP.
ImageNat-1K
FGVC-Alrcraft
StanfordCars
Flowers102
20
Density
10
10
10
0.4
0.3
Clip
Clip Targeted Retrieved Data
Clip
Clip
Clip
Targeted Synthetic Data
Figure 8: We visualize the distribution of CLIP scores in the filtered retrieved and filtered synthetic targeted adaptation datasets. Overall, CLIP assigns synthetic data higher scores on average, despite its lower training performance.
B.2 Additional experiment: further decontaminating retrieved data for benchmark train sets
For our main experiments, we decontaminated all retrieved data for the downstream benchmark test sets avoid test leakage. However, retrieving from LAION-2 -2B may also result in images from downstream benchmark training sets being included in the retrieved data. While this contamina- tion will also affect synthetic data, as the generator trained on the same data we retrieve from, the potential contamination is arguably more explicit when we retrieve directly. We thus further decontaminated all retrieved data for the benchmark train sets following [20]. We report the amount data removed by this additional decontamination step in Table and plot the results of training
16

---

wwice-decontamiinted retrieved data in Figure 9. Overall, while some retrieved images were indeed similar train set data (an average 1.9% of retrieved data was removed), discarding them minimally impacted performance. Our main findings remain unchanged.
ImageNet-IK 145370 (5.6%)
DTD
FGVC-Aireraft
StanfordCars
Flowers102 342 (0.1%)
0 (0.0%%
328 (0.2%)
6647 (3.5%)
Table We report the number of images removed from our original retrieved datasets (which are test set decontaminated) after further decontaminating with respect to the downstream benchmark train set. We report the percentage of the retrieved dataset removed in parentheses.
ImageNet-1K
DTD
FGVC-Aircraft
StanfordCars
Flowers102
45 Accuracy (%)
60
50
40
20
258
LP Accuracy (%)
83
87
90
86
98
85
80
96
80
84
94
70
79
63
92-
78
82
90
256 1024 2048
64 128
32
256
64
256
16
32
Lcale Adaptation Dataset Size
Offithe
Retrieved (Criginal. Test Decontaminated)
Retrivved (Train+ Decontmminedd
Figure 9: We further decontaminate original retrieved data (already test set decontaminated) for the benchmark train set and finetune off-the-shelf CLIP on the train+test decontaminated retrieved data. Overall, train set decontamination has little performance impact; synthetic data remains behind. Note, no train set contamination was found for DTD.
B.3 Additional experiment: generating synthetic data with alternative prompts
In our main experiments, we generate synthetic data via LLM-generated image captions following the SOTA method of SynCLR [62]. SynCLR shows that prompting for synthetic images with LLM captions outperforms many alternative prompting strategies at the hundred-million scale, such as prompting with alt-text from LAION images. To ensure that we utilize the most performant synthetic data generation method available, we sought to validate this finding in our experimental setup. We further compare against the strategy suggested [37], which involves prompting with BLIP-2 generated captions of real images.
Starting from the unfiltered retrieved training data, we used BLIP-2 to caption each image and construct prompts of the form "a photo of classname, BLIP-2 image caption" following [37]. We then performed top-30% CLIP filtering on the resulting synthetic images. We compare the performance of training with filtered synthetic images generated with three prompting distinct strategies to our filtered retrieved data in Figure Specifically, we compare training with filtered synthetic images generated from (1) our original LLM prompts, (2) BLIP-2 captions, and (3) LAION alt-text from retrieved data. Overall, among the three generation strategies, our original L.LM prompts perform best on ImageNet, and perform comparably to BLIP-2 captions and LAION alt-text Aircraft.
17

---

ImageNet-1K
ImageNet-1K
FGVC-Aircraft
FGVC-Aircraft
Linear Probe Accuracy (%g
70
Zero Shot Accuracy
Accuracy
60
73
81
50
25
40
70
69
79
83
20
80
256
1024
2048 4096
256
1024
2008
0096
120
32
120 256 512
Adaptation Dataset
Adaptation Dataset
Adaptation Dataset
Adaptation Dataset Size
CLP Baselin
+ Helr eved Date
Synchwilc Dass (Drignal LLM Cant ums)
Syntheiic Data (LAKIN Captions)
Synthetic Data (8L172 Caplions)
Figure 10: We compare the training performance of targeted synthetic images generated via (1) our original LLM LLM-guided prompts, (2) LAION alt-text from retrieved data, or (3) BLIP-2 captions of retrieved data. Our original LLM prompts yield the best synthetic data for ImageNet; all generation stratgies perform comparably on Aircraft. Regardless of the strategy chosen, synthetic data lags retrieved data.
C Details in Methodology
C.1 Sourcing data by generating synthetic images
Given set of visual class names C from target task, first synthesize large corpus of image captions for each class name by prompting a large language model (we use Llama-2 7B [64]). For each concept name C, following [62]. For the sake of completeness, we detail the prompts here:
three type of prompts to convert into an image caption
1. c + caption. We prompt the language model (LM) to directly translate the class name into caption using a prompt with 3 few-shot in-context examples.
c, background + caption. is randomly sampled from a set that is predetermined based on the domain of C. For example, contains a list of flower names, then possible background attributes might include "garden," "meadow," or "forest." These background attributes automatically generated by prompting strong instruction-tune language model such as GPT-4 [1] with the class names C. We provide the LM with 3 in-context examples of c, background
prompt the LM with an additional background attribute that
caption mappings.
c, relation + caption. We prompt with an additional spatial relationship attribute that is sampled from a domain-invariant set of relationships, such "next to," "below," "besides," etc. We provide 3 in-context examples c, relation + caption mappings.
Each of these captions are directly used text input to Stable Diffusion 1.5 to produce our targeted synthetic dataset D (symthtiic When sampling from Stable Diffusion, we denoise for 50 DDIM [60] steps starting from Gaussian noise, using classifier-free guidance [26] scale of 2.0. We choose Stable Diffusion our generator because (1) it is pretrained on an open-source dataset, LAION-2B, and (2) to better maintain consistency with recent work [62, 63, 56, 23].
Generating the images can be computationally expensive; every IM synthetic images gen- erated (pre-filtering) requires around 12 hours of generation on 64 NVIDIA A100 GPUs. To lower the barrier for future research, will release our generated synthetic images at https: / / thub. com/ con//sootgeeggO0 wwtp000/ummmttppee unmet-promise.
C.2 Filtering synthetic and retrieved data
We rank and filter synthetic data independently, using per-class threshold to identify and keep the top 30% of images for each class and for each dataset. We adopt CL.P as our synthetic image filter as it is the current best filtering method we aware of [24].
18

---

D Details in Experimental Setup
D.1 Retrieval hyperparameters
We perform k-NN retrieval with Ao = 2000 for every downstream benchmark except ImageNet- where & 500. We picked these values of k with rough target of retrieving 10K images per class. In particular, the original CLIP paper [50] has a different set of class template strings for each benchmark; query sets Qe for each benchmark differently sized, and values of l vary to reflect that.
For class balancing, we set M 10000. We do not tune k or M in our experiments.
Choosing based downstream validation accuracy, we use our substring retrieval strategy for FGVC-Aireraft and Flowers we use our semantic retrieval strategy for ImageNet-IK. DTD, and StanfordCars.
We use precomputed k-NN search indicies from LAION-2B [57] to query against OpenAl CLIP ViT-L/14 image embeddings. No search indicies are available for querying against text embed- dings; we construct our own using FAISS [16], using the configuration OPQ256 768, 168,IVF13101 HNSW32, PQ256x8. Computing 2 billion OpenAl CLIP ViT-L/14 text embeddings for the captions in LAION-2B took approximately 2 hours on 100 GPUs of varying capacity. Computing the search index from the embeddings took approximately 12 hours on 128 CPU cores.
D.2 Details for downstream benchmarks
We use the standard pre-defined train-test-validatio splits for FGVC-Aircrafi, DTD, and Flowers- 102. Standard validation splits are not available for StanfordCars and ImageNet-IK, We construct train-validation split for StanfordCars by randomly splitting 80% and 20% of the pre-defined training set (respectively) using torch. utils. data. rawddm_ndmmmss spplsssppttttt_ with random seed 42. We construct validation set for ImageNet-Ie by randomly subsampling 50K images from the pre-defined training set using torch. utils data. handom_splii with random seed 42.
D.3 Details for model evaluation
When performing zero-shot evaluation, the finetuned model W CLIP is directly used to inference the downstream test set without any additional tuning. When performing linear probing evaluation, replace W with randomly initialized linear head W', freeze the CLIP encoder, and train W with standard cross entropy classification loss on the downstream benchmark train set; we perform inference with W CLIP.
D.4 Details for model adaptation
Finetuning details. To finetune CLIP for specific downstream image classification task, first initialize a linear readout head W using the weights from the text-based zero-shot CLIP model [10]. Concretely, we initialize W using the CLIP text embeddings of the class names for the desired downstream task. We then append the classification head W on top of CLIP's vision encoder, and train end-to-end using standard cross entropy classification loss against one-hot labels.
We could alternatively choose to finetune CLIP with a contrastive objective, where each positive pair is synthetic retrieved image alongside its corresponding caption. However, find that cross entropy finetuning performs better across the board, so we use cross entropy finetuning for all experiments in our paper.
full adaptation dataset scale sweep for a single benchmark and fixed set of hyperparameters takes approximately 24-36 hours 2 NVIDIA A40 GPUs.
Random seed. For our main experiments, generator ablations, and data mixing experiments report results aggregated across at least three random seeds. The random seed is used to (1) seed the training algorithm, and (2) controls adaptation dataset subsampling.
Hyperparameter details. We start with relatively standard hyperparameters from prior work [69], and initially tune them in our setting by finetuning CLIP small-scale dataset of retrieved or
19

---

synthetic images from each downstream benchmark and grid grid-sweeping by learning rate and batch size. From the hyperparameters we tried this scale, we find the following work best for both synthetic and retrieved images across all downstream benchmarks:
Batch size: 512
Warmup steps: 500
LR schedule: Cosine decay
L2 weight decay: 0.1
We find that models are sensitive to learning rate; there is no one optimal learning rate across all settings. Thus, for our full-scale experiments, we sweep learning rate across (5c-4, le-5, 1c-6), and select the best learning rate for each downstream benchmark based on validation set accuracy.
We train with an A optimizer, using 31 = 0.9, f2 0.95.
On all benchmarks except ImageNet, we finetune for a fixed 30 epochs. On ImageNet, we train for a fixed 10 epochs to compute, we found that validation set accuracy plateaued early
E
Details in Analysis Experiments
E.1
Synthetically perturbing retrieved images
We SDEdit [43] to synthetically perturb the retrieved real images by adding Gaussian noise and then denoising with Stable Diffusion 1.5. We study the perturbation across 10 different choices the noise scale (i.e. perturbation strength) parameter 7 E {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8. 0.9, 1.0). All other hyperparameters for the perturbation (e.g. guidance scale, noise scheduler) are fixed to the same hyperparameters used to generate synthetic images from scratch (detailed in Appendix C.I) When using SDEdit to perturb a retrieved real image, we additionaly require text prompt to guide the denoising. We find that using an empty string for the prompt results incoherent images. We thus use the string "a photo of {classname)" where (classname) is the label name assigned to the retrieved image being perturbed. Thus, even noise scale y 1.0 (i.c., beginning the denoising at the endpoint of Stable Diffusion's 's forward diffusion process ), we would not expect model performance from training on perturbed retrieved images to be identical model performance from training on our synthetically generated images; our synthetically generated images are synthesized using LLM-written prompts that contain richer information.
Details in Licensing Information
F.1 Benchmarks
ImageNet-1 1 released under a custom license that specifies non-commercial research use only. Details can be found at https: //wwww imageenet. org/ download, php. Licensing information is unavailable for DTD, Flowers 102, FGVC-Aircratt, and StanfordCars; all four datasets are products of academic research and are publicly available online for download.
F.2 Models
Stable Diffusion 1.1, 1.3, and 1.5 are all released under a CreativeML OpenRAIL M license. Open- CLIP models and OpenAl CLIP are released under MIT License.
F.3 Data
LAION-2B metadata, precomputed embeddings, and k-NN search indices released under CC- B--4.0 licensing. As a web-scraped dataset, all images pointed to by the URLs retain their original licenses.
'Even at y 1.0, which maps input images to the endpoint of the forward diffusion process, Stable Diffusion's 's noise schedule does not convert the input image into pure isotropic Gaussian noise. This is due to the noise schedule parameters.
20

---

F.4 Software
We build off the open source code of [53, 39, 6, 16, 69]. FAISS [16] and clip-retrieval [6] are released under MIT license. SynCLR code [62] is released under Apache 2.0. Stable Diffusion code [53] released under CreativeML Open RAIL-M. WiSE-FT (the codebase we build off of for CLIP finetuning) is released under an MIT license.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our paper seeks to empirically measure what value training on synthetic images from generative model adds over training on the generative model's upstream training data directly. We present our baseline methods in Section 3 and our experimental setup empirically grounding this question in Section Our results supporting our main claim that synthetic training images are significantly outperformed by upstream real images are illustrated in Figure 2. The additional analysis discussed in the introduction and abstract is supported by Section 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We describe limitations throughout the paper, such as the fact that we assume access to the generative model's 's upstream training data. We further dedicate a section to limitations in the Discussion section of our main text (Section 6).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
22

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We detail our data curation methods in Section and our full experimental setup for measuring the added value of synthetic data at the beginning of Section 4. We additionally provide full hyperparameters and implementation details of data curation and model training in Appendix C and Appendix D. We have released all code necessary to replicate our study at https: //github.bbb/.t/// wwt.cooo/oooooo het///cootgggggoo granno0//mmet....memme ppromise, and are currently uploading trained models and generated synthetic images from our experiments for release to mitigate computational cost of reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
23

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have released all code used in https: /t:::://ittttt http://gthu...ommsstttgeeee com/scottgengo0 tettgggnngoouneeetttto ing all generated synthetic images used in our experiments for release. To prioritize safety (further discussed in safeguards below), we do not directly distribute the raw images we retrieved from LAION-2B web-scraped dataset);
study and are currently upload-
instead released all code, and we are currently uploading all other assets (such as text K-NN search indices) required to reproduce our data from publicly available assets.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips cc/
aublic/guidss http:/gideeeooeeeuiiiieeeeeeeeee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips. cc/ccc/ppoooo hts///ulic/gggee wwwpc//gg/ee//eeeeeeeeooieeeeeee ieloddSuumisssiooPeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We overview experimental setup in Section 4 and provide full training and test details (data splits, choice hyperparameters, optimizer) in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
24

---

7. Experiment Statistical Significance
Question: Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For all of main results, report I-sigma emror bars computed by aggregat- ing model accuracy across at least random experimental seed initializations, as described in the experimental setup of Section 4. We reiterate the meaning of our error bars in the caption of our main results figure (Figure 2).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
We detail compute resources and time required for generating images in Appendix C.I, for building retrieval search indices Appendix D.1, and for training models in Appendix D.4.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eurriga meeurrps.ce ips.ii...c... wttppppppcppppppppppppppppppp ltwp://eehhitsleeeeieleeeeeeee ines?
Answer: [Yes]
25

---

Justification: We document all details necessary to safely reproduce our experiments, as referenced in the other questions. We take steps to introduce safeguards by filtering all of our web-scraped data with NSFW filtering. Additionally, we take care to not use the full version of LAION-5B, which potentially contains CSAM and has been deprecated.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper is largely foundational. We describe broader impacts associated with our released assets in Appendix A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Our empirical study centers on LAION-2B, web-scraped dataset that may contain NSFW and other sensitive content. We mitigate risk by filtering all raw data we download from LAION-2B with NSFW filtering, described in Section 3.2. Due to potential risk for misuse, we will not directly release our raw retrieved images. We instead ensure reproducibility by releasing code and j-NN search indices for the pipeline we used to perform the retrieval from the LAION-2B URL metadata, which is hosted publicly online.
Guidelines:
The answer NA means that the paper poses such risks.
26

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Full licensing information for all software, datasets, and existing models used in our work are provided in Appendix F.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our work produces three classes of assets that we are currently uploading for release. (1) We will release cetheeicalll-generatteee
images and document full details of image generation in Appendix C. (2) We will release CL.IP model checkpoints trained on task-relevant synthetic and retrieved data. Full details of the training setup that created these assets are documented in Section 4 and Appendix D.4. (3) We will release k-NN search indices for querying against the text captions of LAION-20 documented in Appendix D.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
27

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "DNWzVI92NE", "reviewer": "Reviewer_ZHX8", "summary": "This paper evaluates the performance of training machine learning models on synthetic images generated by the Stable Diffusion generative model compared to using real images directly retrieved from the LAION-2B dataset, which was used to train the generative model. The authors argue that while synthetic images can benefit some downstream tasks, they are generally outperformed by directly using targeted real images. The study involves extensive empirical testing across several visual recognition tasks, showing that real data often provides better accuracy and fewer artifacts than synthetic alternatives.", "questions": "There is a main confuse that the key is \"retrieved data - synthetic data\" or \"high quality data - low quality data\" in this field, and the contribution to use retrieved real data as baseline should be more clear and specific. The inspiration this work provides for future work needs to be further clarified by the author, which is also the key to my rating.", "limitations": "The conclusions and exploration presented in this work is solid, but the fact that the original training data is of higher quality than the data generated by the generator is not a valuable conclusion.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "(1) This paper innovatively proposes the impact of synthetic data and the corresponding training pairs on downstream tasks, providing a comprehensive analysis by comparing the performance of models fine-tuned on synthetic and real data across multiple tasks and scales. It recommends using true data from the same prompt domain as a baseline for downstream tasks.\n\n(2) The experimental design includes a robust setup, featuring a variety of benchmarks that support the derived conclusions. The authors test their hypotheses across multiple data sizes and report the results with detailed statistical data.\n\n(3) Discussions on the limitations of synthetic data and the potential for real data to yield better training outcomes have a direct impact on the development approaches of future artificial intelligence models, especially in fields where data quality is critical.", "weaknesses": "(1) The author aims to emphasize the importance of the original training data as an evaluation baseline, but has not yet compared the downstream fine-tuning results of the real-and-synthetic data of non-training data.  And not clear about the advantage of using upstream real data as baseline but not the othe dataset.  \n\n(2) A lack of quantitative indicators to evaluate the quality of synthetic images, the authors only use CLIP for screening. For the generated image data, no more robust quantitative indicators are used to screen and eliminate images with low quality or obvious visual semantic noise. Because in most downstream methods based on synthetic data, generated images are usually screened by a lot of manual and computational methods to improve the fine-tuning quality. \n\n(3) Due to privacy restrictions, model open source and other issues, it is difficult to retrieve training data to compare with synthetic data, which may limit the future application of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! We are glad you found our experimental setup “robust” and our discussion to “have a direct impact on the development” of future models. We address each of your points below.\n\n**Q1 (Why retrieval baseline?):** Why emphasize the importance of retrieving data from the generative model’s training data as an evaluation baseline for synthetic data? Why adopt the retrieval baseline over simply comparing synthetic data to high-quality data outside the generative model’s training set?\n\n**A1:** Great question! Our research goal is not just to study whether synthetic training data can improve model performance (which as you noted, would not require the retrieval baseline). Rather, we seek to understand where performance gains from synthetic data come from, and to contribute a principled way of measuring whether model-generated synthetic data can surpass the real data it derives from (i.e., the data used to train the generator). To study this question empirically, we propose the retrieval baseline. This baseline enables us to disentangle whether gains from synthetic data are due to (a) the fact that synthetic images are implicitly subsampled from the generator’s huge real training set, or (b) due to the generator truly adding useful information beyond its training data. If we compare synthetic data to high-quality data outside the generator’s training set, then these two factors cannot be disentangled. For example, even if a synthetic data method outperforms high-quality outside data, the gains from synthetic data may simply be due to the generative model seeing higher-quality and larger-scale data during pretraining. In contrast, our baseline of retrieving real data from the generator’s training set controls for the effect of information from the upstream real data. Retrieval also controls for the effects of targeted data sampling (i.e., synthetic data is often targeted to specific tasks), which has been conflated with synthetic data in prior work [57].\n\nWe believe the research questions we pose—how to measure if/when synthetic data surpasses the real data it was trained on—are timely to tackle. There is surging interest in building SOTA vision models with synthetic data [3,21,51,57,58], and many works now show strong gains. We find that such gains—while exciting—still fall short of our retrieval baseline, suggesting that today’s synthetic data methods do not provide significant information beyond the generator’s training data. Hence, we hope our baseline will set a strong and simple target for the field going forward. We further discuss how our paper can impact future work in response to your **Q4** below. We will clarify all points in our final paper. Thank you for helping us strengthen our work!\n\n**Q2 (Insufficient filtering):** The CLIP score used in the paper to filter synthetic images is not sufficiently sophisticated and does not reflect current best filtering practices.\n\n**A2:** Thanks for the question! This point may be mistaken. While we agree that improving data filtering is an exciting future direction for improving the utility of synthetic training data, filtering based on CLIP score remains state-of-the-art [22]. In fact, many recent representative works do not apply any quality filters whatsoever to generated synthetic training images [3,21,51,57,58]. Our work uses CLIP filtering to optimize the performance of synthetic data, which [22] finds improves training performance over no filtering. Our study corroborates [22]’s finding; for example, applying CLIP filtering to synthetic training data improved ImageNet zero-shot and LP accuracy by 1.03 and 0.91 points (respectively) over no filtering at 4M data scale. \n\n**Q3 (Retrieval is impractical):** Retrieval from a generator’s training data is often not practical due to privacy concerns or closed-source data, which may limit the work’s applicability.\n\n**A3:** Good point! We make a similar note and discuss its implications for future research in the introduction and discussion of our paper (L70-73, L312-315). While privacy concerns and closed data are practical reasons to use synthetic data over retrieved data, these application constraints are orthogonal to our research question. Specifically, our goal in proposing the retrieval baseline is not to prescribe a general method for maximizing downstream task accuracy (which, as you noted, would require consideration of downstream constraints), but rather to advance our understanding of what value synthetic data provides beyond the training set of its generator.\n\n**Q4:** How can this research and the proposed retrieval baseline inspire future work?\n\n**A4:** We believe our work can inspire future research in two primary ways. First, comparison against our retrieval baseline provides a principled target for future synthetic data research to aim for. If any synthetic data method outperforms training on data retrieved from the generative model’s training data, then that is strong evidence that the generative model has added (e.g., through its inductive bias, through its compositional abilities) additional useful information on top of the data that it was originally trained on. In contrast, as discussed in **A1**, if a synthetic data method outperforms some other source of real data outside of the generative model’s training set, then we cannot draw a similar conclusion.\n\nSecond, by demonstrating that retrieving real data from the generator’s training set currently surpasses synthetic data, we hope to sharpen the field’s intuition of when existing synthetic data methods can be useful in the present day. Many recent works position synthetic data as a potential drop-in replacement for real data, even when the retrieval baseline is possible. Our findings do not corroborate such positioning; instead, existing synthetic data methods are most promising when downstream constraints (e.g., privacy concerns) prevent the retrieval baseline from being realized.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rr6HQAxtVa", "reviewer": "Reviewer_rgD8", "summary": "The paper tries to answer the question of whether the progress of pretraining classification backbones with images obtained from generative models is due to the advances in generative image modeling or from the fact that these are implicitly sampled from huge image collections. To answer this question, the paper proposes a simple baseline consisting of querying the original databases on which the generative models are trained, finding nearest neighbors for the task at hand, and training on these neighbors instead of the generative samples. The paper shows that this simple baseline outperforms naively sampling from the image models.\nFurthermore, the paper analyzes why the generated data underperforms real data, concluding that the downgraded performance is due (at least in part) to a lack of fine-grained detail (e.g. in the case of the FGVC dataset, the generated ) as well as artifacts in the generated images that introduce a domain gap with respect to real images.", "questions": "See weakness", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The question the paper tries to answer is highly relevant to the vision community (and also for other communities such as language), and pinpoints a clear deficiency of the baselines of papers tackling synthetic data generation with pre-trained models. It is highly positive for the community that the paper raises this concern and checks the actual performance of the proposed baseline when compared against simple ways of training for classification using synthetic images.\n\n- The analysis of what causes the models trained with synthetic images to fail is interesting.", "weaknesses": "- The main weakness of the paper is that it points out a deficiency of other papers, but its technical contribution is limited (i.e., nearest-neighbor retrieval). Although the question studied is really relevant to the community and the baseline pinpointed should always be considered in papers, the technical contributions of the paper seem limited.\n\n- Sentences 33-35 are (at minimum) a bit ambiguous due to wording (the \"over\" in L34 is ambiguous), or they are wrong. By the data processing inequality, the generated samples cannot contain any additional information about the images in LAION-2B that is not contained in the images of LAION-2B themselves. However, the generated images can contain additional information that is not present in the original LAION-2B images, and that is useful for the downstream classification task. In fact, the images generated by the generative neural network also contain information about the neural network architecture and the training algorithm. If this was not the case, the mutual information between the generated images and the training algorithm would be 0, and thus synthetic and real images would be indistinguishable.\nThe *promise* of using generated images is precisely that training on synthetic images adds crucial extra information to the information present on LAION-2B about regularities of the world that are embedded in the implicit biases of the neural network and its training procedure. The implicit biases of neural networks add extra information about the composability of concepts (e.g. \"a cow on the beach\" is the same as \"a horse on the beach\" with the cow replaced by a horse, although there are no cows in the beach during training), stability with respect to small perturbations in both text and image space (\"a cow in the beach\" and \"a cow in an island\" are roughly the same) and other useful regularities of the world. I strongly encourage the authors to rephrase this sentence and explain that the gains in [57] are well motivated by this.\n\n- Although interesting, the analysis in Section 5 is limited, and other SOTA image models should be tested. Only using StableDiffusion is well motivated in the experimental section (as it is required to have access to the training dataset), but in the analysis section, stronger public or proprietary models should be analyzed to see if they also suffer from the same deficiencies.\n\n- Similar baselines have already been used in the past, and they are not correctly cited. StableRep [57] implicitly has the baseline proposed in the paper, as it compares against using the complete set of real data used for training the generative model (i.e. all the neighbors), showing that it outperforms it. Similarly, the preliminary work of [1] also compares against using the entire training dataset, showing that in this preliminary case, the generative models do not outperform the real datasets. \n\n- The comparison is just against a simple way to generate synthetic images, although several works have proposed more sophisticated ways of sampling that target the shortcomings studied in Section 5. For example, [2] trains on a particular modality with scarce data, instead of using a large-scale pretrained model (which may not have fine-grained details about all the classes). As seen in Figure 2, training with generative samples underperforms over directly using CLIP, while works like [57] outperform it.\n\n- It would be good to study how close the retrieved samples are to the training splits of the datasets studied. Are the training samples of these datasets included in LAION-2B? If this is the case, and these are properly retrieved, it is expected that the real images are strictly upper bound (as they are perfect samples of the real distribution, and there won't be any distributional shift between training and testing).\n\nMinor comments:\n- L176 and Figure 2: The use of the term \"Zero-shot\" performance to refer to the text-query classification performance is a bit confusing, as the tested models have been finetuned with N-shots. If I understand correctly, this corresponds to applying the zero-shot classification technique of CLIP after finetuning on N samples for each class, but as the models have been finetuned with N-shots.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review! We are glad you found our research question “highly relevant” for both vision and NLP.\n\nBefore answering your points directly, we’d like to clarify a possible miscommunication. A key point in the review is that our proposed retrieval baseline already implicitly exists in other work. For example, SynCLR [57] compares synthetic images against the generative model’s full training set (i.e. retrieve everything) and finds that synthetic data performs better.\n\nWhile such results are exciting, this existing baseline—comparing synthetic data to the generator’s full train set—is distinct from our proposed baseline, and is insufficient for measuring the true added value from the generative model itself. When we sample synthetic training data, we often sample data that is targeted to tasks we want our learner to perform well. Works like [57] use this idea to generate synthetic data that beats the full real dataset for specific tasks. However, by comparing to the full real data, a task-agnostic distribution of images, [57] and other works [21] implicitly conflate the effects of training on synthetic versus real data with the effects of targeted data sampling. We aim to disentangle these factors and answer: are observed gains from synthetic data truly due to added information from the generator? Or is it since the synthetic data is implicitly sampled in a targeted manner from large upstream real datasets? To study this, we construct a retrieval baseline that explicitly controls for targeting (L39-45), thus overcoming a key limitation of prior work.\n\nOverall, we are excited by the same possibility you highlighted—that synthetic data can transfer useful information from the generator's implicit biases. However, to ensure our field progresses toward this goal, we must carefully measure the current state of synthetic data methods. We believe our work is a crucial step in this direction.\n\n**Q1:** “The main weakness of the paper is that… Although the question studied is really relevant to the community… the technical contributions of the paper seem limited.”\n\n**A1:** Our goal is not to innovate new methods; rather, we seek to introduce a conceptual baseline that helps our field better understand the true utility of synthetic training images. Our proposed baseline can be implemented with existing image retrieval techniques; we aim to keep our baseline simple to facilitate comparison against it.\n\n**Q2:** Unclear wording of data processing inequality sentence.\n\n**A2:** Thanks! We will revise to discuss that SynCLR [57] is indeed motivated. We agree that synthetic images can contain useful information from the generator’s training process. However, it is unclear if [57]'s gains are truly due to this added information—[57]'s gains may also be from targeting the synthetic data to the evaluation tasks. Our retrieval baseline controls for the effect of targeting, allowing us to better measure whether the observed gains from synthetic data are due to the generator’s added information.\n\n**Q3:** Why only analyze Stable Diffusion (SD), why not analyze closed-source models to see if they suffer similar limitations?\n\n**A3:** We analyze SD so we can contextualize its analysis with its performance relative to our retrieval baseline, which is only possible with open data. In contrast, if we find some closed-source generative model G yields higher quality images than SD, it would be unclear whether these gains in image quality reflect improvements in the information added by G to the synthetic images, or whether G was simply trained on a higher quality dataset than SD. We also aim to keep consistency with recent work [21,51,57] that all use SD as the generator. We will revise to clarify these points.\n\n**Q4:** The proposed baseline exists in prior works, which compare synthetic data against the generator’s full training set.\n\n**A4:** Please see our discussion above for a detailed response. Briefly, SynCLR [57] does not control for the effects of data targeting, and GenRep [1] samples data from unconditional GANs, whereas modern synthetic data is sampled in a targeted manner. Our retrieval baseline which controls for targeting is now necessary to understand synthetic data gains in the modern regime. StableRep [3] only compares against a small random subset of the full train data. We will clarify in our final paper. Thank you!\n\n**Q5:** The comparison is against a simple way to generate synthetic training images, but works like SynCLR [57] propose more advanced ways that can outperform CLIP.\n\n**A5:** To clarify, our paper exactly adopts [57]’s synthetic data method. [57] only significantly beats CLIP on the Aircraft task, which our results corroborate. Sampling data from generators trained on data-scarce domains as you cited [2] is an exciting idea that we will discuss in related work; we focus our study on sampling data by prompting web-pretrained models like Stable Diffusion to be consistent with other work [21,51,57].\n\n**Q6:** The high performance of retrieved data may be due to training set contamination.\n\n**A6:** Good point! The generator is trained on the same data we retrieve from, so in theory synthetic data can also have benchmark train set contamination. We agree this contamination is more direct when we retrieve data; we thus further decontaminated all retrieved data for the train sets following [18]. We report the amount of removed data in Table R1 and plot the results of training on decontaminated retrieved data in Figure R1 of the rebuttal PDF. Overall, while some retrieved images were indeed similar to train set data (an average 1.9% of retrieved data was removed), discarding them minimally impacted performance.\n\n**Q7:** Confusing zero-shot terminology.\n\n**A7:** Thanks! We will clarify in our paper: our zero-shot models are not trained on downstream benchmark data. We finetune CLIP on retrieved or synthetic data only and evaluate the resulting model as-is on the test set.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZrJ635VfPZ", "reviewer": "Reviewer_Etje", "summary": "There is a growing interest in training vision models using synthetic data. This paper explores the effectiveness of synthetic data compared to real images directly retrieved from image generator's training sets like LAION-2B. The experimental results indicate that, while synthetic data can be beneficial for some tasks, real data often matches or surpasses its performance. The paper suggests that these results are due to high-frequency generator artifacts and inaccuracies in task-relevant visual details present in synthetic data.", "questions": "Please address the weaknesses.", "limitations": "Limitations have been discussed in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper proposes a new baseline for training with synthetic data, which is novel and interesting.\n* This paper presents extensive experiments, such as different datasets and data scales.\n* The experiment of synthetically perturbed images is novel.", "weaknesses": "The proposed baseline using retrieved images is novel and inspiring. However, some experimental settings may limit the results of training on synthetic images, leading to unfair comparisons and potentially misleading conclusions.\n\n- Settings regarding image quality:\n  - Data filtering: The paper keeps images with top CLIP similarity as the training set (Section 3.3). Are synthetic and retrieved training data ranked and selected separately? What is the distribution of CLIP similarity in the final synthetic and retrieved training datasets?\n  - Synthetic images with generation artifacts: As shown in Figure 3, synthetic 'flute' images have obvious generation artifacts. These images should be filtered for a fair comparison. What is the CLIP similarity of these images? What are the model results when training on a 'clean' dataset without these images?\n\n- Settings regarding data distribution and variation:\n  - Prompts: The current prompts for generating images may provide less information and variation than retrieved real data, influencing accuracy. How about sourcing the generation model using captions from retrieved data or other real images, as suggested in [1]?\n  - Data scaling: Although synthetic images can be scaled efficiently, they tend to be similar when prompts are fixed. Do retrieved images have more variation with larger data scales than synthetic images? How do the results change if synthetic images in the scaling experiment are sourced using captions from retrieved images?\n\n[1] Image Captions are Natural Prompts for Text-to-Image Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! We are excited that you found our proposed baseline for measuring the utility of synthetic data to be “novel and inspiring.” \n\nYour feedback prompted us to perform additional experiments to further validate our paper’s findings, and we believe the new results corroborate our paper’s main finding that state-of-the-art synthetic data training methods lag our proposed retrieval baseline. Thank you for helping us strengthen our paper! We will include all new results in the next version of our paper, and describe the details below to address each of your points.\n\n\n**Q1 (CLIP score distributions post-filtering):** Do the distributions of CLIP similarity scores in the final synthetic and retrieved training datasets significantly differ? Are retrieved and synthetic images ranked and filtered separately, or together?\n\n**A1:** Great question! Significant differences in CLIP similarity could potentially explain gaps in downstream training performance and is worth studying. In our paper, **we ranked and filtered synthetic and retrieved training data separately** (i.e., to keep the experimental design as clean as possible, we do not use any information from the retrieved images to inform the selection of synthetic data, and vice versa). We use per-class score thresholds for filtering, which we found empirically beneficial for both synthetic and retrieved data. We histogram the CLIP similarity score distributions of the resulting filtered training data in Figure R2 of the rebuttal PDF. Overall, despite setting the filter score threshold for synthetic and retrieved data independently, we find that the distribution of post-filtering synthetic image CLIP scores is right-shifted compared to the distribution of post-filtering retrieved image CLIP scores. In other words, **synthetic images have comparable or even higher CLIP scores than retrieved images on average**; CLIP judges synthetic data to be higher quality on average. Thus, CLIP score differences alone do not explain the lagging training performance of synthetic data.\n\n**Q2 (Filtering out images with artifacts):** As shown in Figure 3, synthetic ‘flute’ images have obvious generation artifacts. These images should be filtered for fair comparison.\n\n**A2:** To clarify, the artifact-afflicted images shown in Figure 3 are *post-filtering*. Even though the ‘flute’ images of Figure 3 are obviously wrong to humans, CLIP assigns them relatively high scores of $0.285, 0.265$, and $0.263$. For reference, a CLIP score of $0.249$ reflects the top $30$% of all retrieved ‘flute’ images. Furthermore, we found obviously wrong and artifact-ridden synthetic ‘flute’ images that have even higher CLIP scores $> 0.3$, placing them in the top $5$% of all retrieved ‘flute’ data.\n\nOur paper adopts CLIP filtering since it is the current best synthetic image filter [22] we are aware of; in fact, many recent works forgo data filtering altogether [3,21,51,57,58] despite its positive impact on training performance [22] (our study corroborates this gain; CLIP filtering synthetic data improved ImageNet zero-shot accuracy by 1.03% and LP by 0.91% over no filtering at 4M scale). We aim to ground our argument in the current state of synthetic training data research, so we do not explicitly innovate new data filtering methods beyond CLIP filtering. Nonetheless, these findings spurred by your feedback makes it apparent that CLIP filtering is limited. We will include a more detailed discussion of this point in our final paper to motivate future filtering methods.\n\n\n**Q3 (Alternative prompts for synthetic data generation):** Synthetic images may suffer in diversity when the generation prompts are fixed. Other prompting strategies should be considered, such as using LAION alt-text or outputs from a captioning model.\n\n**A3** Thank you for the point! First, to clarify a potential miscommunication, the prompts used to generate synthetic images in our work are *not fixed* – rather, each image prompt is sampled from a probabilistic large language model (LLM) that is tasked to include additional relevant knowledge about the desired category (e.g., for the “dog” category, a sampled generation prompt might be “a photo of a dog playing with a ball in the park”). We will make this more clear in our paper.\n\nWe adopt this LLM-guided prompting strategy from SynCLR [57], which shows that it outperforms many alternative prompting strategies at the hundred-million scale, including prompting with LAION alt-text. We did not find any existing comparison between LLM-generated prompts and prompts from an image captioning model as proposed by the work you referenced [1], so we conducted this study ourselves.\n\nStarting from the unfiltered retrieved training set, we used BLIP-2 to caption each image and construct prompts of the form “a photo of {classname}, {BLIP-2 image caption}” following [1]. We then performed top-30% CLIP filtering on the resulting synthetic images. We compare the performance of training with filtered synthetic images generated with three prompting distinct strategies to our filtered retrieved data in Figure R3 of the rebuttal PDF. Specifically, we compare training with filtered synthetic images generated from (1) our original LLM prompts (orange lines), (2) BLIP-2 captions (red lines), and (3) LAION alt-text from retrieved data (green lines). Overall, among the three generation strategies, our original LLM prompts perform best on ImageNet, and perform comparably to BLIP-2 captions and LAION alt-text on Aircraft. All three synthetic strategies lag the performance of retrieved data. Thus, our conclusion—that existing synthetic image training methods do not surpass the retrieval baseline—holds under these variations of image generation strategy.\n\nWe were not aware of [1] at the time of submission — thank you for pointing it out to us! We will include a reference to that work and the above experiments motivated by [1]’s captioning method in our final paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tWaBr8zuoA", "reviewer": "Reviewer_6uYi", "summary": "This paper studies whether training on synthetic images from generative model can **truly** surpass the baseline of training on the retrieved real images that are used to train the generative model. It provides several key insights: 1) retrieved real images are significantly superior to synthetic images across a wide range of recognition tasks, 2) both data sources (retrieved and synthetic) are beneficial to original training images, and 3) adding synthetic images to retrieved images will ruin the gain achieved by the latter. It also analyzes two factors that may cause the inferiority of synthetic images.", "questions": "I love this paper, only with one minor question or suggestion. Indeed, it is somewhat expected that synthetic images may not be as good as retrieved synthetic images in standard benchmarks with widely existing class concepts. Could the authors provide more insights about the potential advantages of synthetic images in scenarios with rare or very complex concepts?", "limitations": "The authors have clearly discussed the limitations.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is very clearly written.\n2. The key insight (synthetic images are indeed inferior to the naive baseline of retrieved real images) is very interesting and timely. I believe it was previously totally overlooked by our community. There have been many works trying to utilize synthetic images for representation learning recently, but they mostly utilize synthetic images *blindly* and fail to dig into the critical question asked by this paper.\n3. The analysis for the poorer performance of synthetic images is convincing, especially the ablation study on the synthetically perturbed real images, which is quite inspiring.\n4. The scope and position of this paper are properly defined. It does not aim to uncover the uselessness of synthetic images, but to present a necessary baseline for future works to compare with. I believe this simple yet strong baseline will motivate future works to construct and leverage synthetic images more effectively. Besides, the authors also consider and discuss the scenarios where synthetic images are indispensable, e.g., scenarios with privacy concerns.", "weaknesses": "I do not think there are critical weaknesses in this paper. I only have one minor concern. See below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! We are honored to hear that you enjoyed our paper, and are grateful you found our research question “timely” and “totally overlooked” by our community.\n\nYour question—whether synthetic images can offer unique advantages for tasks with rare or complex concepts—is very exciting. Such concepts are precisely the concepts for which collecting labeled real data is difficult, and thus exactly where we have the greatest need for synthetic data. These directions are exciting topics for future work that we will discuss further in our final paper. For example, synthesizing very complex concepts might be made feasible by leveraging the compositional generalization abilities of generative models, which recent research promisingly suggests may be a unique boon of generative models compared to other models trained on the upstream data [1, 2] (references below).\n\nJust for curiosity’s sake, we also performed a preliminary study to try and relate the concept-wise performance of models trained on synthetic or retrieved data with the rarity of each concept. Specifically, given a concept denoted via a text string $c$ (e.g., “Airbus A320”), we approximate that concept’s frequency in LAION-2B by counting the number of image-text pairs with text containing $c$ as a (case-insensitive) substring. We scattered the concepts with their frequency on the $x$-axis and model accuracy on the $y$-axis. Unfortunately, we were not able to find any clean trends — with existing methods, the performance gap between synthetic and retrieved data does not appear to systematically decrease on rarer concepts. To speculate, this may be because generative models are trained on the same data pool that we retrieve from, and thus may also have difficulty learning rare concepts during general pretraining as they are seen less frequently. Recent work [3] corroborates this conjecture; however, we note there are asterisks to our preliminary study, as we did not extensively validate the robustness of our concept frequency metric. Nonetheless, if off-the-shelf pretrained generators are indeed less effective at generating rare concepts, a potential future direction would be to resolve this limitation by taking advantage of a small set of examples to adapt the generative model in a sample efficient manner (e.g., perhaps through textual inversion). Afterwards, we may be able to compose the concept with the generator’s existing knowledge to sample new data variations. \n\nThank you again for your time, and thank you for drawing our attention to these exciting future directions that may lead to gains beyond our retrieval baseline. We will discuss them thoroughly in the next version of the paper!\n\n\n* [1] Your diffusion model is secretly a zero-shot classifier. Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak.\n\n* [2] Text-to-image diffusion models are zero shot classifiers. Kevin Clark and Priyank Jaini.\n\n* [3] No \"Zero-Shot\" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance. Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip H.S. Torr, Adel Bibi, Samuel Albanie, Matthias Bethge.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their detailed feedback. We will incorporate all suggestions in the next version of our paper. Thank you all very much for your invaluable help in improving our work! 

Overall, we are thrilled to see that all reviewers found the research question we posed interesting and relevant to the community. The two main concerns were raised by **reviewers ZHX8 and rgD8**, who wondered if the retrieval baseline proposed in our paper is a necessary contribution as opposed to simpler and existing alternatives. 

For example, **reviewer ZHX8’s** primary concern questioned why we might want to compare synthetic data to data specifically retrieved from the generative model’s training set, as opposed to a simpler baseline of general high-quality real data that need not come from this specific source. We detail why this simpler baseline would not allow us to answer our research questions in response to the reviewer below. We will also clarify in our revised paper. To summarize, our goal is not just to understand whether training on synthetic data can improve model performance (in which case the simple baseline would suffice), but rather to better understand where any gains we observe derive from. Do gains from synthetic data come from the fact that we are implicitly subsampling relevant data from the generator’s huge real training set (which we can alternatively do via retrieval)? Or do gains come from the generative model truly adding some new information (e.g., through its inductive biases) that surpasses its training data? Comparison against the simple baseline cannot disentangle these two factors; any observed synthetic data gains versus the simple baseline may simply arise from improvements in the generative model’s training data quality. As interest in building SOTA vision models with synthetic data [3, 21, 51, 57, 58] is surging, we believe these are timely unanswered questions to tackle. Hence, we propose the retrieval baseline to enable studying them empirically.

**Reviewer rgD8** expressed concern that our retrieval baseline already implicitly exists in previous work like SynCLR [57], which has compared synthetic data to the *full* training set of the generative model. We detail why this baseline is also distinct from our baseline in response below. Mainly, the full dataset baseline does not control for the critical effect of data targeting, which has been implicitly conflated with synthetic data in past work [57] (L39-45). We show that by comparing synthetic data against our baseline, previous gains shown from synthetic data—while still exciting—largely go away. Thus, we believe our baseline is critical for understanding the true added value of synthetic training data.

**Reviewer rgD8 and reviewer Etje** both asked for additional empirical experiments to further validate the main findings of our paper. We have performed the requested experiments, and include all resulting figures and tables in the rebuttal PDF. We summarize the new experiments here:

* Train set decontamination (Figure R1, Table R1). **Reviewer rgD8** pointed out that retrieving data from LAION-2B may result in images from the benchmark training set (e.g., from ImageNet) being included in the retrieved data. Our retrieved sets are already decontaminated for the downstream test set; we thus performed additional train set decontamination and plotted the results of training on the train+test decontaminated data. Overall, train set decontamination has minimal impact on model performance and does not change our findings.

* CLIP score distributions (Figure R2). **Reviewer Etje** wondered whether there were significant discrepancies in the CLIP score of post-filtering synthetic and retrieved images. We find that if anything, CLIP judges synthetic data to be higher quality than retrieved data on average, despite its lagging training performance. We are optimistic that this new experiment will motivate future synthetic data filtering methods that do not rely on CLIP.

* Additional synthetic image prompt strategies (Figure R3). **Reviewer Etje** wondered whether generating synthetic images with other prompting strategies would yield higher training performance. We compare our original LLM-guided prompting strategy to prompts from the LAION alt-text or BLIP-2 captions of our retrieved data, and find that the original LLM-guided strategy performs better on ImageNet and similarly on FGVC-Aircraft compared to the two alternatives.

We believe the new experiments further corroborate our main findings, and we will include all of them in the next version of our paper. Once again, thanks to all the reviewers and chairs whose effort makes this process possible. We deeply appreciate your effort in helping improve our work! We are eager to further discuss with all reviewers during the discussion period.

Meta-review: This paper evaluates the effectiveness of training machine learning models using synthetic images generated by the Stable Diffusion generative model compared to real images directly retrieved from the LAION-2B dataset, which was used to train the generative model. The authors demonstrate that, while synthetic images can be beneficial for certain downstream tasks, real images generally outperform them, offering better accuracy and fewer artifacts. The study is backed by extensive empirical testing across various visual recognition tasks.

The paper addresses a timely and important question about the role of synthetic data in model training and proposes a strong baseline for future work in this area. Although the technical contributions are somewhat limited, the insights provided are valuable and likely to shape future research directions. Reviewers generally agree on accepting the paper, acknowledging its solid methodology and potential impact despite some noted weaknesses. Therefore, the Area Chair recommends acceptance.

The following suggestions are recommended for inclusion in the final version: 1) exploring scenarios where synthetic data may have advantages, such as in handling rare or complex concepts; 2) investigating the combination of synthetic and real data, and 3) improving the explanation of the contributions of retrieved real data as a baseline.
Decision: Accept (poster)