The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better

Authors: scott  geng
cheng-yu  hsieh
vivek  ramanujan
matthew  wallingford
chun-liang  li
♣  pang
wei  koh
ranjay  krishna
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 03:43:35

Abstract: Generative text-to-image models enable us to synthesize unlimited amounts of images in a controllable manner, spurring many recent efforts to train vision models with synthetic data. However, every synthetic image ultimately originates from the upstream data used to train the generator. Does the intermediate generator provide additional information over directly training on relevant parts of the upstream data? 
Grounding this question in the setting of image classification, we compare finetuning on task-relevant, targeted synthetic data generated by Stable Diffusion---a generative model trained on the LAION-2B dataset---against finetuning on targeted real images retrieved directly from LAION-2B. We show that while synthetic data can benefit some downstream tasks, it is universally matched or outperformed by real data from the simple retrieval baseline. Our analysis suggests that this underperformance is partially due to generator artifacts and inaccurate task-relevant visual details in the synthetic images. Overall, we argue that targeted retrieval is a critical baseline to consider when training with synthetic data---a baseline that current methods do not yet surpass. We release code, data, and models at [https://github.com/scottgeng00/unmet-promise/](https://github.com/scottgeng00/unmet-promise).

Main paper: 
 introduction: 
 The success of modern machine learning systems fundamentally relies on the quantity [33, 10, 28, 57, 51, 65] , quality [20, 72, 45, 44, 36] , and distribution [17, 22, 62, 70, 9] of the data they are trained on. However, acquiring large amounts of quality data remains challenging, due to the sheer cost of data collection and annotation. As demand for training data continues to rise, the field is actively exploring approaches to automatically curate data at scale [20, 2, 18] . One burgeoning approach is to source synthetic training data from conditional generative models. Generative models enable data to be tailored to specific requirements and generated at scale, presenting a promising alternative to the challenges of real data curation. Recent work highlights this potential: for example, in natural language processing (NLP), researchers prompt strong proprietary language models to cheaply synthesize large-scale datasets for instruction tuning [68, 29, 61] .Analogously, in computer vision-the focus of our research-many recent works train models on synthetic images from modern text-to-image generators, aiming to achieve state-of-the-art visual recognition performance [56, 63, 3, 23, 24] . For example, SynCLR [62] cleverly prompts Stable Diffusion for synthetic images tailored to pre-specified downstream image recognition domains; they find that a CLIP-like model trained from scratch on the resulting targeted synthetic images can outperform CLIP trained on LAION-2B, a significantly larger untargeted dataset of real images. This result is quite surprising. Stable Diffusion is also trained on LAION-2B, so by the data processing inequality, the synthetic images it generates cannot contain any additional information about LAION-Figure 1 : Given an upstream dataset of general real image-text pairs, we aim to curate a targeted dataset to train a learner on some target task. We can either (1) retrieve targeted real images directly from the upstream dataset, or we can (2) first train an intermediate generative model and then synthesize targeted synthetic images. By comparing these two approaches, our paper seeks to measure what value training on generated synthetic data adds. models and for privacy preservation. We release all code, models, and over 1TB of generated images to guide future work (https://github.com/scottgeng00/unmet-promise). 
related work: 
 Learning from synthetic data. Synthetic data has been widely explored in the context of many machine learning problems [68, 29, 25, 32, 61, 58, 5, 40] . For example, in NLP, synthetic data generated from strong large language models [47, 11] has been used to distill instruction-following behavior [68] and task-specific knowledge [30] into smaller models. In computer vision, prior works have sought to use synthetic data to improve the state-of-the-art across a breadth of visual tasks, such as object detection [48, 31] , semantic segmentation [54, 52, 8] , and optical flow [15] . Traditionally, this synthetic training data has been sourced from expert-crafted simulation and rendering pipelines [49, 15, 52, 8, 54, 31] . Recent advances in text-to-image synthesis via diffusion models [59, 27, 53] are changing this paradigm, inspiring a new wave of works that seek to train visual models on synthetic data algorithmically sampled from large-scale generative models [24, 63, 56, 23, 74, 37] . Recent works have also sought to algorithmically sample synthetic data from diffusion models trained on data-scarce domains [55] . This structural shift in the source of synthetic images from expert-supervised programmatic simulation to a learned generator that itself derives supervision from upstream data raises a critical question: does the intermediate step of training a generator and sampling synthetic data provide any added useful information over simply training on relevant parts of the upstream data directly? Our work formalizes and empirically grounds this question, contributing experiments and baselines to rigorously measure the benefits of training on modern data-derived synthetic data.Adapting pretrained vision models. Large-scale pretrained image models such as CLIP [50, 10] offer transferable visual features that benefit a wide range of downstream vision tasks. It is now common practice to use pretrained models as a starting point when deploying downstream taskspecific models instead of training them from scratch [69, 66] . From an algorithmic perspective, many methods have been proposed to adapt CLIP models to downstream tasks, each with varying trade-offs [71, 73, 42, 21, 4, 69] . We choose to study simple full-finetuning, centering our work on the data we adapt on as opposed to the algorithm. In particular, the quality and relevance of adaptation data has a crucial impact on downstream task performance; distribution shifts at inference time can significantly hurt performance [34] . Acquiring task-targeted data thus remains an active area of research [39, 67, 25] . Most related to our work is [39, 67] , who also employ retrieval as a technique for collecting task targeted-data. Our work builds upon these methods to construct baselines for systematically measuring the true added utility of model-generated synthetic training images.Retrieval as a baseline for synthetic data. Recent studies [74, 7] similarly use retrieval from generative model training data as a baseline for synthetic training images. For example, [74] adapts Stable Diffusion with real medical images and then generates synthetic medical images to train a ResNet from scratch; the resulting ResNet outperforms a ResNet trained on LAION-retrieved medical images. Consistent with our work, [74, 7] find that in the low-data regime of general image recognition tasks (e.g., subsampled ImageNet-1K, STL-10), augmenting real datasets with LAIONretrieved images outperforms augmenting with Stable Diffusion generated images when training ResNets from scratch. [7] further finds that this performance gap in the low-data regime persists across many synthetic image generation methods, including when they finetune Stable Diffusion with task-specific real data. In the medium-data regime (e.g. full ImageNet-1k), [7] finds augmenting with synthetic data matches but does not outperform augmenting with retrieved data. Our work generalizes the shared motivation of measuring the utility of synthetic training images to the modern data-rich regime and a wide range of visual classification tasks.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We describe limitations throughout the paper, such as the fact that we assume access to the generative model's upstream training data. We further dedicate a section to limitations in the Discussion section of our main text (Section 6).Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem setting and method: 
 Given a large dataset D of general real image-text pairs and a downstream visual classification task specified as a set of text class names C, we aim to algorithmically curate a targeted adaptation dataset D C of images x i and one-hot class labels y i to finetune and improve a pretrained vision model's performance on the downstream task. We compare two high-level approaches for sourcing this targeted data, shown in Figure 1: (1) we retrieve targeted real images directly from D, forming a targeted retrieved dataset D . We detail each approach below.We follow SynCLR [62] , a representative method for curating synthetic training data from off-theshelf text-to-image models. Given the set of visual class names C, we first synthesize a large corpus of corresponding image captions by prompting a large language model (details in Appendix C.1). For example, if the class name c ∈ C is "rose," then a generated caption might be "a close-up of a pink rose in bloom." We then use these captions as input for a text-to-image generator G trained on the upstream data D, yielding a large set of synthesized images x i . Each image is assigned a class label y i based on the class name c used to synthesize its caption. These synthetic images and labels {( x i , y i )} form our curated dataset D (synthetic) C .Rather than querying a generator trained on an upstream dataset D, we can directly train on parts of D itself by retrieving relevant data. D consists of image-text pairs (x i , t i ). To retrieve relevant pairs, we consider two strategies. We additionally deduplicate all retrieved images with respect to our evaluation datasets following [20] to minimize test set leakage. We apply NSFW filtering [57] .Strategy 1: hard substring matching. Inspired by [67] , we retrieve the set D (retrieved) C of all images x i whose corresponding caption t i contains at least one target class name c ∈ C as a substring:D (retrieved) C = {(x i , y i ) : (x i , t i ) ∈ D such that some class c ∈ C is a substring of t i } .Here, label y i is assigned based on the class c contained in t i . If an image-text pair (x i , t i ) ∈ D has text t i containing multiple class names c, c ′ ∈ C, then we simply retrieve x i multiple times and assign each instance a different label, once for each unique matched class name.Strategy 2: semantic k-NN retrieval. Hard substring matching is simple and effective when the target visual concepts c ∈ C are concrete entities that are likely to be described in text captions (e.g., c = "fire lily"), but may be less effective when the concepts are abstract (e.g., c = "lined texture"). Thus, we also consider semantic (soft) retrieval via CLIP image-text embedding space similarity 2 . We convert each target class name c ∈ C into a set of natural language search queries Q c based on the templates from the original CLIP paper [50] . For each query q c ∈ Q c , we use approximate k-NN search [16] to retrieve the set S qc of k-nearest image-text pairs (x i , t i ) ∈ D by CLIP similarity between the query q c and either the image x i or the text t i :S qc = arg top-k (xi,ti)∈D CLIP(x i , q c ) ∪ arg top-k (xi,ti)∈D CLIP(t i , q c ) .We assign each image-text pair (x i , t i ) ∈ S qc a class label y i based on the class name in query q c . We form the targeted dataset D (retrieved) C by unioning over all queries q c ∈ Q c and all classes c ∈ C:D (retrieved) C = c∈C qc∈Qc {(x i , y i ) : (x i , t i , y i ) ∈ S qc } .Data filtering has been shown to improve training performance for both real and synthetic data and is widely used [20, 24] . We filter both our synthetic and retrieved datasets following current best filtering practices for synthetic image data [24] . Given a curated dataset D C , we compute the CLIP similarity of each x i ∈ D C with text corresponding to its assigned label y i (e.g., "a photo of {class name}"), constructed using the CLIP zero-shot classification templates [50] . When there are multiple templates for a given class, we aggregate by taking the maximum similarity across templates. We ImageNet-1K are class-balanced by construction (i.e., we uniformly generate images for each class). We further postprocess the retrieved adaptation datasets D Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: The paper does not include theoretical results. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
main experiments: 
 We seek to measure the utility of learning from model-generated synthetic images. Grounding this question empirically, our experiments compare finetuning a pretrained CLIP model on (1) targeted synthetic images D Benchmarks. We focus on five downstream tasks where synthetic data has shown promise compared to similar scale untargeted real data [62] . We select (a) ImageNet-1K [14] and Describable Textures (DTD) [12] to evaluate recognition performance on broad categories and (b) FGVC-Aircraft [41] , StanfordCars [35] , and Oxford Flowers102 [46] to evaluate performance in fine-grained settings. We use standard pre-defined train, test, and validation splits when available, and otherwise randomly subset the training set to create missing train-validation splits (details in Appendix D.2).Finetuning data curation. For each downstream benchmark, we first curate an adaptation dataset D C (Section 3) by either (1) generating synthetic images with Stable Diffusion 1.5 [53] , trained on the LAION-2B dataset [57] , or (2) retrieving real images directly from LAION-2B. We treat the choice between our substring-based and semantic retrieval strategies as a hyperparameter, using downstream validation set accuracy to determine the best choice for each benchmark. Hyperparameters for retrieval are detailed in Appendix D.1.Model adaptation and evaluation. We adapt a LAION-2B pretrained CLIP ViT-B/16 [10] image encoder by finetuning on the curated adaptation dataset D C with a cross-entropy classification loss for a pre-set number of epochs. To elucidate the scaling trends of synthetic and retrieved data, we finetune across an order of magnitude of different adaptation dataset scales, subsampled from the full targeted adaptation dataset D C . We report zero-shot (ZS) and linear probing (LP) test set accuracy, using the benchmark train set to train the LP head. For both LP and ZS evaluation, we use the validation set to identify the best epoch and finetuning hyperparameters. For each data scale, we aggregate accuracy across the results of at least three random seeds, and report the standard deviation due to seed randomness. Please refer to Appendix D.3 for further evaluation details, and Appendix D.4 for further training and hyperparameter details. We present our main zero-shot and linear probing scaling results in Figure 2 .At equivalent data scales, finetuning with model-generated synthetic images can help, but is universally matched or outperformed by finetuning directly with images from the generator's training data. Consistent with prior research [62] , we find that training with targeted synthetic data can improve an unadapted model. For example, on FGVC-Aircraft-the setting where previous works have found strongest gains-finetuning with 139K Stable-Diffusion generated images improved downstream linear probing accuracy by an average of 3.8 percentage points over an off-the-shelf CLIP model (64.9% → 68.7%); on DTD, training with 110K synthetic images improves zero-shot accuracy by 3.3 points (56.3% → 59.6%).However, the gains from training on synthetic data are consistently matched or surpassed by training on retrieved real data. For instance, on FGVC-Aircraft, finetuning with an equivalent 139K LAION-2B retrieved images boosts performance by a massive 17.8 points (64.9% → 82.7%). Moreover, adapting with retrieved data can improve performance even when synthetic data does not (e.g., on ImageNet and Flowers102 zero-shot accuracy.) Finally, adapting with synthetic data can sometimes even hurt performance (ImageNet, StanfordCars, Flowers102 zero-shot), while targeted retrieved data improves or at least does not hurt performance across all settings considered. Given equal amounts of targeted retrieved and synthetic data, retrieved data is the clear winner.Synthetic data can sometimes decrease the gap with retrieved data given additional scale, but remains behind. The amount of data we can retrieve is fundamentally limited by the finite upstream data pool. For example, even after searching all 2 billion LAION-2B samples for images containing an FGVC-Aircraft class name in the caption, substring-based retrieval returned only 139K targeted images post-filtering. In contrast, it is straightforward to create ever-larger synthetic datasets by simply generating more data.Scaling the synthetic adaptation dataset size beyond the amount of retrieved data considered (illustrated in the gray-shaded regions of Figure 2 ), we find that increasing the amount of targeted synthetic data does not always improve performance. For example, on DTD, synthetic data exhibits U-shaped scaling, with performance positively scaling up to 110K synthetic training images, after which performance declines. On ImageNet, Flowers102, and StanfordCars, increasing the synthetic dataset size consistently hurts zero-shot accuracy and has minimal impact on linear probing performance.On Aircraft, scaling helps; there is a log-linear relationship between the size of the synthetic adaptation dataset and downstream linear probing accuracy (e.g., scaling from 139K → 250K synthetic images improves linear probing accuracy from 68.7% → 70.7%). However, synthetic data still lags behind retrieved data: matching the performance of a mere 15K retrieved aircraft images requires scaling the synthetic dataset to 500K images, reflecting a ∼33x difference in dataset size and required finetuning compute. Naively extrapolating this ratio, matching the performance of the full 139K retrieved adaptation dataset would require nearly 5M synthetic images after top 30% filtering. We note, however, that synthetic data is unlikely to truly scale infinitely, as synthetic data fundamentally derives from the (finite) training set of our generative model. Still, the performance of synthetic data is likely unsaturated at the 500K scale (i.e., accuracy is still trending up); due to compute limitations, studying whether further scaling can outperform retrieved data is left for future work.Synthetic data can improve a model's task representation without significantly improving the model's task performance. Broadly speaking, zero-shot task accuracy measures a model's ability to directly solve the downstream task, whereas linear probing accuracy measures the quality of the model's learned task-relevant representation. We find that even when training on synthetic data improves the model's representation (i.e., downstream linear probing accuracy), it may not significantly improve the model's zero-shot accuracy. In contrast, when training on retrieved data improves the model's representation, zero-shot accuracy also exhibits positive scaling. For example, CLIP adapted with either 15K retrieved images or 500K synthetic images both achieve a similar linear probing accuracy (∼ 72%), yet the model adapted with synthetic data achieves a much worse zeroshot accuracy (28.9% versus 39.5%). We discuss possible reasons for this qualitative discrepancy in model behavior in our analyses below (Section 5.1).Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We detail our data curation methods in Section 3 and our full experimental setup for measuring the added value of synthetic data at the beginning of Section 4. We additionally provide full hyperparameters and implementation details of data curation and model training in Appendix C and Appendix D. We have released all code necessary to replicate our study at https://github.com/scottgeng00/unmet-promise, and are currently uploading trained models and generated synthetic images from our experiments for release to mitigate computational cost of reproduction. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. Justification: We document all details necessary to safely reproduce our experiments, as referenced in the other questions. We take steps to introduce safeguards by filtering all of our web-scraped data with NSFW filtering. Additionally, we take care to not use the full version of LAION-5B, which potentially contains CSAM and has been deprecated.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
analysis: 
 In this section, we explore two questions to better understand our main results. First, what factors drive the underperformance of synthetic data? Second, do our findings hold under variations of our experimental setup? We focus our analysis experiments on ImageNet, to understand general image recognition performance, and FGVC-Aircraft, the sole benchmark where synthetic data exhibited strong positive log-linear scaling. Additional analysis experiments are presented in Appendix B.Qualitative visualizations. We visualize a random selection of images from our curated synthetic and retrieved adaptation datasets in Figure 3 . Compared to retrieved real images, we observe that the synthetic images ( 1) contain low-level generator artifacts, and (2) differ in visual content distribution, both in terms of semantic details and overall image composition. For example, although the synthetic FGVC-Aircraft adaptation images (top two rows of Figure 3 ) are recognizable as airplanes, the visual content often contains incorrect class-relevant semantic details: a correctly-depicted "Airbus A320" should have one engine per wing and two sets of wheels at its rear, yet our synthetic images often exhibit incorrect engine or wheel configurations. This qualitative discrepancy in visual detail precision may partially explain why training on synthetic data does not improve task zero-shot accuracy; synthetic images do not retain enough class-accurate details to directly teach the model the downstream task. In contrast, training on synthetic images can improve linear probing accuracy, because the synthetic images still broadly look like aircraft and thus may help align the model's representation to the downstream domain. 1) contain generator artifacts (e.g., the blur on the edges of the "Cessna 172", the eyes and mouth of the "Tabby Cat") and also (2) distort class-relevant visual content, such as the engine configuration of a true "Airbus A320" (i.e., exactly one engine per wing) and the entire visual appearance of a "Flute". We hypothesize that both factors contribute to synthetic training data's underperformance versus real training data.Synthetically perturbing retrieved real images. To disentangle the effect of low-level generator artifacts and visual content differences between synthetic and retrieved real images on downstream model performance, we trained on "hybrid" images that have similar semantic visual content as our retrieved real images but contain generator artifacts like our synthetic images. Following SDEdit [43] , we use Stable Diffusion to synthetically perturb our retrieved images to introduce model-specific artifacts present in the synthetic images Stable Diffusion generates. Given a noise strength parameter γ ∈ [0, 1] and a retrieved image x 0 , SDEdit adds Gaussian noise to x 0 according to timestep t = γ of Stable Diffusion's time-dependent forward process. We then denoise the noisy image using the same reverse diffusion process as in text-to-image generation, yielding a perturbed image x (γ) that looks semantically like x 0 while also containing Stable Diffusion-specific artifacts. Increasing γ increases the amount of Gaussian noise added to x 0 , thereby increasing the severity of visual artifacts introduced in the resulting x (γ) (see Appendix E.1 for further details). In pseudocode,x (γ) = StableDiffusion.denoise(StableDiffusion.add_noise(x 0 , γ), γ).Starting from the full targeted retrieved adaptation datasets D (retrieved) Cfor FGVC-Aircraft and Ima-geNet, we use SDEdit to introduce generator artifacts into the retrieved real images over a range of γ values. We visualize the resulting perturbed images in Figure 4 , and plot the results of training on these perturbed images across γ in Figure 5 . , where larger γ increases the severity of generator-specific artifacts added by the perturbation. When γ ≥ 0.6, the introduced artifacts can be strong enough to damage task-relevant visual details for finegrained tasks like FGVC-Aircraft (e.g., the airplane's engine and rear wheels).For broad tasks like ImageNet, artifacts have a lesser impact on class-relevant details; the "Tabby Cat" is recognizable as a cat even after perturbing with high γ. Our results suggest that generator artifacts do contribute to synthetic data's underperformance-any artifact level causes performance to drop below training on retrieved images (dashed blue line). Moreover, differences in visual content between synthetic and retrieved images also matter; even with relatively strong perturbations (γ = 0.5), training on artifact-afflicted perturbed images that retain the semantic content of retrieved images outperforms training on synthetic images (dashed orange line).Our results suggest three takeaways. First, generator artifacts indeed contribute to the underperformance of synthetic training images, especially for fine-grained classification tasks. On FGVC-Aircraft, any amount of added generator artifacts drops downstream accuracy. Second, the impact of artifacts is relatively lower for broad classification domains such as ImageNet, where downstream performance is not significantly impacted until we perturb with a relatively strong noise strength of γ = 0.5. Finally, visual content differences between synthetic and retrieved images also play a key role in the performance gap between synthetic and retrieved training data. When we perturb images with strength γ = 0.5, the resulting images are heavily afflicted with artifacts, but still retain the important class-relevant details of retrieved real images, such as correct airplane engine configurations. Training on γ = 0.5 images significantly outperforms training on synthetic images. Intriguingly, training on aircraft images perturbed beyond the point where class-relevant visual details are damaged (γ ≥ 0.6) still outperforms synthetic data; we speculate that this is because these heavily perturbed images still retain the overall image composition of retrieved images.For our main scaling experiments, we generate synthetic image datasets using Stable Diffusion 1.5 to maintain consistency with prior work [62, 23] . To what degree does our choice of generative model impact our findings? At the time of our study, Stable Diffusion 1.x models are the only modern textto-image models with open source training data available to retrieve from. Therefore, we focus our study here on Stable Diffusion (SD) versions 1.1, 1.3, and 1.5. Starting from SD v1.1, which is trained on the full LAION-2B dataset, SD v1.3 and v1.5 are derived by further finetuning on high-quality subsets of LAION-2B. This additional finetuning improves image generation fidelity [53] , but may lead to the model forgetting parts of the LAION-2B distribution [19] . Following our main experiment setup (Section 4), we use SD v1.1 and SD v1.3 to generate various-sized targeted synthetic adaptation datasets for ImageNet and FGVC-Aircraft. Results are plotted in Figure 6 . Overall, while training with synthetic data from different generative models yields varying performance, synthetic data from all generative models considered consistently fall short of retrieval. On FGVC-Aircraft, finetuning CLIP with either synthetic or retrieved data alone consistently improves downstream task accuracy. The gains from retrieved data are stronger than the gains from synthetic data across all data scales; however, synthetic data may improve CLIP in ways that is complementary to retrieved data, and thus present orthogonal value. and finetune a pretrained CLIP model with the resulting mix. We plot results in Figure 7 . We find that training on the mixed images outperforms training on synthetic images alone; however, training on a mix significantly drops performance compared to using retrieved data alone.  
discussion: 
 Our work sought to answer a key question: given that all model-generated synthetic images derive from the generator's upstream training data, does training on synthetic images provide value over training on the parts of the upstream real data directly? We contribute a set of rigorous experiments to ground this question empirically, and discover that training on upstream real images collected via our simple retrieval baseline significantly outperforms training on synthetic images. Our initial question is answered negatively. We therefore argue that retrieval is a critical baseline to surpass in order to show value from synthetic training data, and encourage comparison against it in future research.Importantly, we do not seek to make normative claims about whether training with synthetic images will ever surpass this baseline-future work may unlock gains that we have not yet found. As a first step, we contribute analyses of why synthetic training images underperform upstream real images, finding that both generator artifacts and semantic errors within synthetic images are key areas for future improvement. Furthermore, given that image retrieval is a strong alternative to image synthesis, a natural next step is to generate image compositions that are explicitly rare or absent from the generator's upstream training dataset; we are optimistic that synthesizing these "missing" images may offer unique value beyond what is present in the existing upstream real images. Such an approach leverages the compositional generalization abilities of the generator, which recent research promisingly suggests may be stronger than the compositionality of a discriminative model trained on the same upstream data [38, 13] .Finally, our findings assume access to the generative model's upstream training data, an assumption that may not always hold. The upstream pool may be proprietary or strictly regulated due to privacy concerns. In such settings, training directly on the upstream data is impossible; synthetic data from a generative model trained on this unavailable upstream data remains an exciting alternative to acquire otherwise inaccessible information.Limitations. As an empirical study, our compute budget limits the number of experimental variations we consider. We release all models and synthetic data from our work to benefit future research. All assets released are meant to be scientific research artifacts. Moreover, all models released are task-specific classification models, limiting potential for misuse. Nonetheless, we do not encourage the use or deployment of our models in practice.We further validate the main findings of our paper by (1) providing further analysis of the synthetic training data and (2) considering two additional variations upon our experimental setup: applying more aggressive decontamination, and generating synthetic images with alternative prompting strategies.Do the distributions of CLIP similarity scores in the final synthetic and retrieved training datasets significantly differ? Since we filter the synthetic and retrieved data independently of each other to maintain a clean experimetnal setup (i.e., we do not use any information from the retrieved images to inform the selection of synthetic data, and vice versa), there may be differences in CLIP similarity that potentially explain gaps in downstream training performance.We histogram the CLIP similarity score distributions of the resulting filtered synthetic and retrieved training data in Figure 8 . Overall, despite setting the filter score threshold for synthetic and retrieved data independently, we find that the distribution of post-filtering synthetic image CLIP scores is right-shifted compared to the distribution of post-filtering retrieved image CLIP scores. In other words, synthetic images have comparable or even higher CLIP scores than retrieved images on average; CLIP judges synthetic data to be higher quality on average. CLIP score differences alone do not explain the lagging training performance of synthetic data.Further corroborating this finding with a case study, we observed that CLIP assigns the synthetic "flute" images in Figure 3 relatively high similarity scores of 0.285, 0.265 and 0.263, despite the fact they they are obviously wrong to the human eye. For reference, a CLIP score of 0.249 reflects the top 30% of all retrieved 'flute' images. Overall, while the current best practice of using CLIP score to filter synthetic data does improve performance, CLIP score remains limited. Future work may explore synthetic data filtering methods that do not depend on CLIP. Targeted Retrieved Data Targeted Synthetic Data For our main experiments, we decontaminated all retrieved data for the downstream benchmark test sets to avoid test set leakage. However, retrieving from LAION-2B may also result in images from downstream benchmark training sets being included in the retrieved data. While this contamination will also affect synthetic data, as the generator is trained on the same data we retrieve from, the potential contamination is arguably more explicit when we retrieve directly. We thus further decontaminated all retrieved data for the benchmark train sets following [20] . We report the amount data removed by this additional decontamination step in Table 1 and plot the results of training on twice-decontaminated retrieved data in Figure 9 . Overall, while some retrieved images were indeed similar to train set data (an average 1.9% of retrieved data was removed), discarding them minimally impacted performance. Our main findings remain unchanged. Off-the-shelf CLIP Retrieved Data (Original, Test Decontaminated) Synthetic Data Retrieved Data (Train+Test Decontaminated)Figure 9 : We further decontaminate our original retrieved data (already test set decontaminated) for the benchmark train set and finetune off-the-shelf CLIP on the train+test decontaminated retrieved data. Overall, train set decontamination has little performance impact; synthetic data remains behind. Note, no train set contamination was found for DTD.In our main experiments, we generate synthetic data via LLM-generated image captions following the SOTA method of SynCLR [62] . SynCLR shows that prompting for synthetic images with LLM captions outperforms many alternative prompting strategies at the hundred-million scale, such as prompting with alt-text from LAION images. To ensure that we utilize the most performant synthetic data generation method available, we sought to validate this finding in our experimental setup. We further compare against the strategy suggested in [37] , which involves prompting with BLIP-2 generated captions of real images.Starting from the unfiltered retrieved training data, we used BLIP-2 to caption each image and construct prompts of the form "a photo of classname, BLIP-2 image caption" following [37] . We then performed top-30% CLIP filtering on the resulting synthetic images. We compare the performance of training with filtered synthetic images generated with three prompting distinct strategies to our filtered retrieved data in Figure 10 . Specifically, we compare training with filtered synthetic images generated from (1) our original LLM prompts, (2) BLIP-2 captions, and (3) LAION alt-text from retrieved data. Overall, among the three generation strategies, our original LLM prompts perform best on ImageNet, and perform comparably to BLIP-2 captions and LAION alt-text on Aircraft. For each concept name c ∈ C, we use three type of prompts to convert c into an image caption following [62] . For the sake of completeness, we detail the prompts here:1. c → caption. We prompt the language model (LM) to directly translate the class name into a caption using a prompt with 3 few-shot in-context examples.2. c, background → caption. We prompt the LM with an additional background attribute that is randomly sampled from a set that is predetermined based on the domain of C. For example, if C contains a list of flower names, then possible background attributes might include "garden," "meadow," or "forest." These background attributes are automatically generated by prompting a strong instruction-tuned language model such as GPT-4 [1] with the class names C. We provide the LM with 3 in-context examples of c, background → caption mappings.3. c, relation → caption. We prompt with an additional spatial relationship attribute that is sampled from a domain-invariant set of relationships, such as "next to," "below," "besides," etc. We provide 3 in-context examples of c, relation → caption mappings.Each of these captions are directly used as text input to Stable Diffusion 1.5 to produce our targeted synthetic dataset D (synthetic) C. When sampling from Stable Diffusion, we denoise for 50 DDIM [60] steps starting from Gaussian noise, using a classifier-free guidance [26] scale of 2.0. We choose Stable Diffusion as our generator because (1) it is pretrained on an open-source dataset, LAION-2B, and (2) to better maintain consistency with recent work [62, 63, 56, 23] .Generating the images can be computationally expensive; every 1M synthetic images generated (pre-filtering) requires around 12 hours of generation on 64 NVIDIA A100 GPUs. To lower the barrier for future research, we will release our generated synthetic images at https://github.com/scottgeng00/unmet-promise.We rank and filter synthetic data independently, using a per-class threshold to identify and keep the top 30% of images for each class and for each dataset. We adopt CLIP as our synthetic image filter as it is the current best filtering method we are aware of [24] .We perform k-NN retrieval with k = 2000 for every downstream benchmark except ImageNet-1K, where we use k = 500. We picked these values of k with a rough target of retrieving 10K images per class. In particular, the original CLIP paper [50] has a different set of class template strings for each benchmark; our query sets Q c for each benchmark are differently sized, and our values of k vary to reflect that.For class balancing, we set M = 10000. We do not tune k or M in our experiments.Choosing based on downstream validation set accuracy, we use our substring retrieval strategy for FGVC-Aircraft and Flowers102; we use our semantic retrieval strategy for ImageNet-1K, DTD, and StanfordCars.We use precomputed k-NN search indicies from LAION-2B [57] to query against OpenAI CLIP ViT-L/14 image embeddings. No search indicies are available for querying against text embeddings; we construct our own using FAISS [16] , using the configuration OPQ256 768,IVF131072 HNSW32,PQ256x8. Computing 2 billion OpenAI CLIP ViT-L/14 text embeddings for the captions in LAION-2B took approximately 2 hours on 100 GPUs of varying capacity. Computing the search index from the embeddings took approximately 12 hours on 128 CPU cores.We use the standard pre-defined train-test-validation splits for FGVC-Aircraft, DTD, and Flowers-102. Standard validation splits are not available for StanfordCars and ImageNet-1K. We construct a train-validation split for StanfordCars by randomly splitting 80% and 20% of the pre-defined training set (respectively) using torch.utils.data.random_split with random seed 42. We construct a validation set for ImageNet-1K by randomly subsampling 50K images from the pre-defined training set using torch.utils.data.random_split with random seed 42.When performing zero-shot evaluation, the finetuned model W • CLIP is directly used to inference the downstream test set without any additional tuning. When performing linear probing evaluation, we replace W with a randomly initialized linear head W ′ , freeze the CLIP encoder, and train W ′ with a standard cross entropy classification loss on the downstream benchmark train set; we perform inference with W ′ • CLIP.Finetuning details. To finetune CLIP for a specific downstream image classification task, we first initialize a linear readout head W using the weights from the text-based zero-shot CLIP model [10] . Concretely, we initialize W using the CLIP text embeddings of the class names for the desired downstream task. We then append the classification head W on top of CLIP's vision encoder, and train end-to-end using a standard cross entropy classification loss against one-hot labels.We could alternatively choose to finetune CLIP with a contrastive objective, where each positive pair is a synthetic or retrieved image alongside its corresponding caption. However, we find that cross entropy finetuning performs better across the board, so we use cross entropy finetuning for all experiments in our paper.A full adaptation dataset scale sweep for a single benchmark and a fixed set of hyperparameters takes approximately 24-36 hours on 2 NVIDIA A40 GPUs.Random seed. For our main experiments, generator ablations, and data mixing experiments we report results aggregated across at least three random seeds. The random seed is used to (1) seed the training algorithm, and (2) controls adaptation dataset subsampling.Hyperparameter details. We start with relatively standard hyperparameters from prior work [69] , and initially tune them in our setting by finetuning CLIP on a small-scale dataset of retrieved or synthetic images from each downstream benchmark and grid-sweeping by learning rate and batch size. From the hyperparameters we tried at this scale, we find the following work best for both synthetic and retrieved images across all downstream benchmarks:• Batch size: 512 • Warmup steps: 500 • LR schedule: Cosine decay • L2 weight decay: 0.1We find that models are sensitive to learning rate; there is no one optimal learning rate across all settings. Thus, for our full-scale experiments, we sweep learning rate across {5e-4, 1e-5, 1e-6}, and select the best learning rate for each downstream benchmark based on validation set accuracy.We train with an AdamW optimizer, using β 1 = 0.9, β 2 = 0.95.On all benchmarks except ImageNet, we finetune for a fixed 30 epochs. On ImageNet, we train for a fixed 10 epochs to save compute, as we found that validation set accuracy plateaued early on.We use SDEdit [43] to synthetically perturb the retrieved real images by adding Gaussian noise and then denoising with Stable Diffusion 1.5. We study the perturbation across 10 different choices of the noise scale (i.e. perturbation strength) parameter γ ∈ {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}. All other hyperparameters for the perturbation (e.g. guidance scale, noise scheduler) are fixed to the same hyperparameters used to generate synthetic images from scratch (detailed in Appendix C.1) When using SDEdit to perturb a retrieved real image, we additionaly require a text prompt to guide the denoising. We find that using an empty string for the prompt results in incoherent images. We thus use the string "a photo of {classname}", where {classname} is the label name assigned to the retrieved image being perturbed. Thus, even at noise scale γ = 1.0 (i.e., beginning the denoising at the endpoint of Stable Diffusion's forward diffusion process3 ), we would not expect model performance from training on perturbed retrieved images to be identical to model performance from training on our synthetically generated images; our synthetically generated images are synthesized using LLM-written prompts that contain richer information.ImageNet-1K is released under a custom license that specifies non-commercial research use only. Details can be found at https://www.image-net.org/download.php. Licensing information is unavailable for DTD, Flowers102, FGVC-Aircraft, and StanfordCars; all four datasets are products of academic research and are publicly available online for download.Stable Diffusion 1.1, 1.3, and 1.5 are all released under a CreativeML OpenRAIL M license. Open-CLIP models and OpenAI CLIP are released under MIT License. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Our paper is largely foundational. We describe broader impacts associated with our released assets in Appendix A.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: Our empirical study centers on LAION-2B, a web-scraped dataset that may contain NSFW and other sensitive content. We mitigate risk by filtering all raw data we download from LAION-2B with NSFW filtering, described in Section 3.2. Due to potential risk for misuse, we will not directly release our raw retrieved images. We instead ensure reproducibility by releasing code and k-NN search indices for the pipeline we used to perform the retrieval from the LAION-2B URL metadata, which is hosted publicly online.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: Full licensing information for all software, datasets, and existing models used in our work are provided in Appendix F.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: Our paper does not involve research with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.We perform semantic retrieval using precomputed LAION-2B embeddings from OpenAI CLIP ViT-L/14[50], the same model Stable Diffusion uses to embed text prompts during generation.Even at γ = 1.0, which maps input images to the endpoint of the forward diffusion process, Stable Diffusion's noise schedule does not convert the input image into pure isotropic Gaussian noise. This is due to the noise schedule parameters. 
 
 Appendix:  


Decision: This paper evaluates the effectiveness of training machine learning models using synthetic images generated by the Stable Diffusion generative model compared to real images directly retrieved from the LAION-2B dataset, which was used to train the generative model. The authors demonstrate that, while synthetic images can be beneficial for certain downstream tasks, real images generally outperform them, offering better accuracy and fewer artifacts. The study is backed by extensive empirical testing across various visual recognition tasks.

The paper addresses a timely and important question about the role of synthetic data in model training and proposes a strong baseline for future work in this area. Although the technical contributions are somewhat limited, the insights provided are valuable and likely to shape future research directions. Reviewers generally agree on accepting the paper, acknowledging its solid methodology and potential impact despite some noted weaknesses. Therefore, the Area Chair recommends acceptance.

The following suggestions are recommended for inclusion in the final version: 1) exploring scenarios where synthetic data may have advantages, such as in handling rare or complex concepts; 2) investigating the combination of synthetic and real data, and 3) improving the explanation of the contributions of retrieved real data as a baseline.

Reviews: review: paper_summary: paper_summary: This paper evaluates the performance of training machine learning models on synthetic images generated by the Stable Diffusion generative model compared to using real images directly retrieved from the LAION-2B dataset, which was used to train the generative model. The authors argue that while synthetic images can benefit some downstream tasks, they are generally outperformed by directly using targeted real images. The study involves extensive empirical testing across several visual recognition tasks, showing that real data often provides better accuracy and fewer artifacts than synthetic alternatives.
strength_weakness: strength_weakness: (1) The author aims to emphasize the importance of the original training data as an evaluation baseline, but has not yet compared the downstream fine-tuning results of the real-and-synthetic data of non-training data.  And not clear about the advantage of using upstream real data as baseline but not the othe dataset.  

(2) A lack of quantitative indicators to evaluate the quality of synthetic images, the authors only use CLIP for screening. For the generated image data, no more robust quantitative indicators are used to screen and eliminate images with low quality or obvious visual semantic noise. Because in most downstream methods based on synthetic data, generated images are usually screened by a lot of manual and computational methods to improve the fine-tuning quality. 

(3) Due to privacy restrictions, model open source and other issues, it is difficult to retrieve training data to compare with synthetic data, which may limit the future application of this work.
questions: questions: There is a main confuse that the key is "retrieved data - synthetic data" or "high quality data - low quality data" in this field, and the contribution to use retrieved real data as baseline should be more clear and specific. The inspiration this work provides for future work needs to be further clarified by the author, which is also the key to my rating.
limitations: limitations: The conclusions and exploration presented in this work is solid, but the fact that the original training data is of higher quality than the data generated by the generator is not a valuable conclusion.
paper_summary: paper_summary: The paper tries to answer the question of whether the progress of pretraining classification backbones with images obtained from generative models is due to the advances in generative image modeling or from the fact that these are implicitly sampled from huge image collections. To answer this question, the paper proposes a simple baseline consisting of querying the original databases on which the generative models are trained, finding nearest neighbors for the task at hand, and training on these neighbors instead of the generative samples. The paper shows that this simple baseline outperforms naively sampling from the image models.
Furthermore, the paper analyzes why the generated data underperforms real data, concluding that the downgraded performance is due (at least in part) to a lack of fine-grained detail (e.g. in the case of the FGVC dataset, the generated ) as well as artifacts in the generated images that introduce a domain gap with respect to real images.
strength_weakness: strength_weakness: - The main weakness of the paper is that it points out a deficiency of other papers, but its technical contribution is limited (i.e., nearest-neighbor retrieval). Although the question studied is really relevant to the community and the baseline pinpointed should always be considered in papers, the technical contributions of the paper seem limited.

- Sentences 33-35 are (at minimum) a bit ambiguous due to wording (the "over" in L34 is ambiguous), or they are wrong. By the data processing inequality, the generated samples cannot contain any additional information about the images in LAION-2B that is not contained in the images of LAION-2B themselves. However, the generated images can contain additional information that is not present in the original LAION-2B images, and that is useful for the downstream classification task. In fact, the images generated by the generative neural network also contain information about the neural network architecture and the training algorithm. If this was not the case, the mutual information between the generated images and the training algorithm would be 0, and thus synthetic and real images would be indistinguishable.
The *promise* of using generated images is precisely that training on synthetic images adds crucial extra information to the information present on LAION-2B about regularities of the world that are embedded in the implicit biases of the neural network and its training procedure. The implicit biases of neural networks add extra information about the composability of concepts (e.g. "a cow on the beach" is the same as "a horse on the beach" with the cow replaced by a horse, although there are no cows in the beach during training), stability with respect to small perturbations in both text and image space ("a cow in the beach" and "a cow in an island" are roughly the same) and other useful regularities of the world. I strongly encourage the authors to rephrase this sentence and explain that the gains in [57] are well motivated by this.

- Although interesting, the analysis in Section 5 is limited, and other SOTA image models should be tested. Only using StableDiffusion is well motivated in the experimental section (as it is required to have access to the training dataset), but in the analysis section, stronger public or proprietary models should be analyzed to see if they also suffer from the same deficiencies.

- Similar baselines have already been used in the past, and they are not correctly cited. StableRep [57] implicitly has the baseline proposed in the paper, as it compares against using the complete set of real data used for training the generative model (i.e. all the neighbors), showing that it outperforms it. Similarly, the preliminary work of [1] also compares against using the entire training dataset, showing that in this preliminary case, the generative models do not outperform the real datasets. 

- The comparison is just against a simple way to generate synthetic images, although several works have proposed more sophisticated ways of sampling that target the shortcomings studied in Section 5. For example, [2] trains on a particular modality with scarce data, instead of using a large-scale pretrained model (which may not have fine-grained details about all the classes). As seen in Figure 2, training with generative samples underperforms over directly using CLIP, while works like [57] outperform it.

- It would be good to study how close the retrieved samples are to the training splits of the datasets studied. Are the training samples of these datasets included in LAION-2B? If this is the case, and these are properly retrieved, it is expected that the real images are strictly upper bound (as they are perfect samples of the real distribution, and there won't be any distributional shift between training and testing).

Minor comments:
- L176 and Figure 2: The use of the term "Zero-shot" performance to refer to the text-query classification performance is a bit confusing, as the tested models have been finetuned with N-shots. If I understand correctly, this corresponds to applying the zero-shot classification technique of CLIP after finetuning on N samples for each class, but as the models have been finetuned with N-shots.
questions: questions: See weakness
limitations: limitations: Yes
paper_summary: paper_summary: There is a growing interest in training vision models using synthetic data. This paper explores the effectiveness of synthetic data compared to real images directly retrieved from image generator's training sets like LAION-2B. The experimental results indicate that, while synthetic data can be beneficial for some tasks, real data often matches or surpasses its performance. The paper suggests that these results are due to high-frequency generator artifacts and inaccuracies in task-relevant visual details present in synthetic data.
strength_weakness: strength_weakness: The proposed baseline using retrieved images is novel and inspiring. However, some experimental settings may limit the results of training on synthetic images, leading to unfair comparisons and potentially misleading conclusions.

- Settings regarding image quality:
  - Data filtering: The paper keeps images with top CLIP similarity as the training set (Section 3.3). Are synthetic and retrieved training data ranked and selected separately? What is the distribution of CLIP similarity in the final synthetic and retrieved training datasets?
  - Synthetic images with generation artifacts: As shown in Figure 3, synthetic 'flute' images have obvious generation artifacts. These images should be filtered for a fair comparison. What is the CLIP similarity of these images? What are the model results when training on a 'clean' dataset without these images?

- Settings regarding data distribution and variation:
  - Prompts: The current prompts for generating images may provide less information and variation than retrieved real data, influencing accuracy. How about sourcing the generation model using captions from retrieved data or other real images, as suggested in [1]?
  - Data scaling: Although synthetic images can be scaled efficiently, they tend to be similar when prompts are fixed. Do retrieved images have more variation with larger data scales than synthetic images? How do the results change if synthetic images in the scaling experiment are sourced using captions from retrieved images?

[1] Image Captions are Natural Prompts for Text-to-Image Models
questions: questions: Please address the weaknesses.
limitations: limitations: Limitations have been discussed in the paper.
paper_summary: paper_summary: This paper studies whether training on synthetic images from generative model can **truly** surpass the baseline of training on the retrieved real images that are used to train the generative model. It provides several key insights: 1) retrieved real images are significantly superior to synthetic images across a wide range of recognition tasks, 2) both data sources (retrieved and synthetic) are beneficial to original training images, and 3) adding synthetic images to retrieved images will ruin the gain achieved by the latter. It also analyzes two factors that may cause the inferiority of synthetic images.
strength_weakness: strength_weakness: I do not think there are critical weaknesses in this paper. I only have one minor concern. See below.
questions: questions: I love this paper, only with one minor question or suggestion. Indeed, it is somewhat expected that synthetic images may not be as good as retrieved synthetic images in standard benchmarks with widely existing class concepts. Could the authors provide more insights about the potential advantages of synthetic images in scenarios with rare or very complex concepts?
limitations: limitations: The authors have clearly discussed the limitations.

Comments: title: Thank you for your response! We are excited to discuss further.
Thank you for your reply! We apologize for the confusion and are happy to discuss further.
Thank you for your time!
Thanks for the clarification
Thank you for your response! We are happy to clarify and discuss further.
Thank you for your feedback!
Thanks for the Rebuttal
Thank you for response
Thanks for the rebuttal
Clarification on Weakness (1)
Clarification request on weakness point (1)
References clarification
Clarification request on references [1] and [2]
comment: Thanks for the further discussions. I will maintain my score.
Thank you for your time and continued interest in our work! We are glad that we were able to address parts of your concern and are very grateful to see that you’ve increased your score in favor of acceptance.

Overall, we are excited by the same possibility that you highlighted: that synthetic data can be effectively scaled, and that training on ever-larger synthetic datasets can improve our models. However, this goal remains highly non-trivial. Our work found that even when we scale up the amount of synthetic data generated from a current SOTA method [57] beyond the amount of retrieved data considered, synthetic data still lags retrieved data in terms of downstream training performance (**Figure 1, L199-220**). For example, training on a mere 30K retrieved images outperformed 500K synthetic images for FGVC-Aircraft. We further find that with current methods, scaling up synthetic data can often hurt performance, not help. Thus, we fully agree that scaling up synthetic data has exciting potential, but retrieved data is currently more performant than synthetic data when available. To realize the untapped potential of synthetic data, we need to find better methods. And to build better synthetic data methods, our field needs a principled target to aim for.

We believe that our retrieval baseline may be one such target. If any synthetic data method outperforms training on data retrieved from the generative model’s training set, then that strongly suggests that the gains come from improvements in our synthetic data methodology itself, as opposed to improvements in the generator’s real training data. In the second case, using the generator’s real training dataset directly would continue to be more practical when available; synthetic data would remain primarily useful under privacy constraints. 

While privacy is already an exciting application domain, we are optimistic about future improvements in our synthetic data methods to make them useful more broadly (i.e. even when the generator’s real train set is available). To this end, we also performed analysis experiments (**Section 5.1, Figures 3,4**) to better understand why synthetic data currently lags retrieved data. We show that both visual artifacts from the generator and high-level semantic content differences contribute to the underperformance of synthetic data, thus highlighting these two axes as future directions for improving synthetic data. Moreover, by conceptualizing retrieval as a target to beat, another natural future step could be to design methods that generate image compositions which are explicitly absent from the generator’s upstream training set; synthesizing these “missing" images may offer unique value beyond the existing upstream real images. Such an approach leverages the compositional generalization abilities of generative models, which recent research promisingly suggests may be a unique boon of generative models compared to other models trained on the upstream data [1, 2] (references below). We will thoroughly discuss these other ways in which our work may motivate future synthetic data improvements in our updated paper. Thank you for your thoughtful questions!

Thank you again for your time and valuable feedback, which has improved our work. We share your excitement about the potential of synthetic data, and are optimistic that our work will inspire subsequent works to find gains beyond our retrieval baseline. We would be eager to discuss further!

* [1] Your diffusion model is secretly a zero-shot classifier. Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak.
* [2] Text-to-image diffusion models are zero shot classifiers. Kevin Clark and Priyank Jaini.
Thank you for your time and continued interest in our work! We apologize for our confusing formatting — our initial response to your last question is folded into **Q3/A3** of the rebuttal text above. For your convenience, we summarize the relevant parts here:

* First, to clarify a potential miscommunication, our generation prompts are not fixed, but rather sampled from a probabilistic LLM that is tasked to generate image captions for each desired visual concept. We use these LLM captions as text-to-image prompts.
* We adopt the LLM-guided prompt method from SynCLR [57], a current SOTA work which shows that synthesizing images with LLM prompts can outperform alternative generation strategies (including synthesizing images from LAION alt-text) at hundred-million scale.
* We experimentally compared the training performance of filtered synthetic images generated using either (a) our original LLM prompts or (b) the LAION alt-text of our retrieved images. We report results of this new scaling experiment in Figure R3 of the rebuttal PDF. Synthetic images generated via LLM prompts outperform images generated with LAION alt-text prompts on ImageNet, and perform comparably on FGVC-Aircraft. Regardless of generation strategy, synthetic images lag the retrieval baseline.

We missed the part of your question about the diversity of retrieved versus synthetic images in our initial response. We sincerely apologize for the confusion! To clarify your question, we further compared the image variation in our final filtered Aircraft and ImageNet adaptation datasets, which consist of either (a) retrieved real images, (b) synthetic images generated with our original LLM prompts, or (c) synthetic images generated from the LAION alt-text of the retrieved images. We quantify image variation via the average pairwise cosine similarity of the CLIP image features for each dataset (i.e. lower average similarity → higher variation). To understand variation at large scale, we perform this analysis on the largest-sized version of each dataset . Results are as follows, with specific scale size in parentheses:

|   | ImageNet-1K | FGVC-Aircraft |
|---|----------|----------|
| **Retrieved Real Data** | 0.323 (2.5M images) | 0.506 (139K images) |
| **Synthetic Data (Original LLM Prompts)** | 0.369 (4M images) | 0.606 (500K images) |
| **Synthetic Data (LAION alt-text)** | 0.341 (2.5M images) | 0.527 (139K images) |

Overall, your intuition is correct: synthetic images generated with LLM prompts indeed exhibit higher average pairwise cosine similarity compared to retrieved images, suggesting deceased variation. This gap persists even though the scales of the LLM prompt synthetic datasets are significantly larger than the scales of the retrieved datasets (e.g. 4M vs 2.5M for ImageNet, 500K vs 139K for Aircraft). Moreover, generating synthetic images based on the LAION captions of the retrieved images does improve the measured variation of the resulting synthetic images. 

However, interestingly, improvements in synthetic image diversity alone do not directly translate into significant improvements in downstream model performance. As shown in Figure R1, models trained on the less-diverse LLM prompt generated images we originally considered perform better or comparably to models trained on the more-diverse LAION alt-text generated images. Nonetheless, we are excited that the experiments you suggested have uncovered another axis along which synthetic and retrieved images differ. We will include these experiments and detailed discussion in the updated version of our paper to help motivate future synthetic data work.
Thank you very much for your time and effort in this process! We enjoyed discussing our work with you, and are glad that we were able to address your concerns. We appreciate your score increase. We will clarify based on the discussed points in the next version of our paper. Thank you for helping us strengthen our work!
Thanks for the detailed clarification, you are right! If I understand correctly, and as specified in Section 4.1, SynCLR does create a LAION-like set of captions, and they train on all the captions once (using CLIP), and not retrain per each downstream task. Although this is less severe than training specifically for each downstream application (i.e. only training on Airplanes when the downstream is also Airplanes), it is true true that during the process of creating captions they use all the captions from all downstream tasks (with many variations and augmentations), so indeed, their process conflates the effects of the synthetic sampling and the targeting of (all) the downstream tasks they study.

After the discussions, authors have addressed my concerns, and I'm raising my score to acceptance. Still, I expect the final version of the manuscript to include clarifications on the mentioned points.
Thank you for your reply! We truly appreciate your time and continued interest. To illustrate what we mean by prior work “conflating targeted data sampling with synthetic data,” we detail why SynCLR's experiments [57] cannot answer our research question.

Our work asks: does synthetic training data contain useful added information beyond the training dataset of its generator? We believe this is timely as recent works (e.g. [57]) have shown that training on a synthetic dataset $D_S$ can outperform training on the full real dataset $D_R$ used to pretrain its generator. However, this prior finding—that $D_S$ outperforms full $D_R$—does not answer our question, as the experimental setup confounds two distinct independent variables. To see the confounded variables, let's recap SynCLR's method for generating a synthetic training dataset $D_S$:

1. Manually define a set of visual concepts that we want our model to perform well on. SynCLR's concept list is largely based on the classes in the main downstream evaluation tasks (Tables 6,11 in [57]). For example, since we want to perform well on FGVC-Aircraft and ImageNet, the concept set contains ‘Airbus A320’, ‘flute’, etc.
2. Use an LLM to generate image captions for each concept.
3. Generate an image for each caption via Stable Diffusion, a text-to-image model trained on LAION-2B.

Thus, $D_S$ is collected with so-called “targeted data sampling:” the synthetic dataset $D_S$ SynCLR trains on isn’t sampled unconditionally and is not intended to cover the full LAION-2B distribution. Rather, it is carefully tailored to specific tasks based on the manually-fixed concepts in step 1. As such, the synthetic dataset $D_S$ differs from LAION-2B, the real training set of Stable Diffusion (i.e. $D_R$), along two axes simultaneously. First and most apparent, (a) LAION-2B is real while (b) $D_S$ is sampled from a generator. More subtly but just as critical, (i) $D_S$ is targeted to the downstream eval tasks, while (ii) LAION-2B is a broad, task-agnostic distribution of images. To illustrate, $D_S$ is constructed such that 5% of the dataset is images of FGVC-Aircraft classes (30M total images). In contrast, we estimated the proportion of FGVC-Aircraft relevant images in LAION-2B to be just 0.056% (only ~1.1M total images). Thus, while SynCLR shows that training on $D_S$ outperforms training on full LAION-2B for FGVC-Aircraft and performs comparably on other benchmarks, any gains $D_S$ exhibits over full LAION-2B may not be because $D_S$ is generated while LAION-2B is real. Rather, it could simply be because full LAION-2B is task-agnostic, while $D_S$ is task-targeted. Importantly, whether data is targeted is not a unique property of synthetic data—real data can be targeted too (e.g. via retrieval as discussed below). With existing baselines, we cannot attribute the gains from training on $D_S$ to Stable Diffusion adding unique value on top of its LAION-2B training set; we cannot resolve our research question.

The conflation of these two independent variables (data can be synthetic or real; data can be targeted or untargeted) is not unique to SynCLR. Modern synthetic data methods often rely on prompting text-to-image models; synthetic data is thus often targeted via the generation prompts. For example, SynthCLIP [21] derives prompts from the MetaCLIP concept list, thus targeting synthetic data to MetaCLIP concepts. SynthCLIP also compares this targeted synthetic data to general real data (e.g. Conceptual Captions). 

The key innovation of our retrieval baseline is to enable disentangling these presently conflated variables, thus creating a more principled experimental setup. Through retrieval, we collect targeted real data from LAION-2B and compare targeted synthetic data $D_S$ head-to-head against this targeted real data. Our new setup now only varies one independent variable (i.e. synthetic versus real). Any gains $D_S$ exhibits over our retrieval baseline are thus properly attributable to $D_S$ having added information over the generator’s training set. However, we show that even SOTA synthetic data methods (we adopt SynCLR) lag the retrieval baseline. When we correctly control for synthetic data being targeted, previously observed gains vanish. In summary:

* Works like SynCLR compare targeted synthetic data to the generator's full untargeted real training set. This controls for the effect of seeing information from generator training data, but does not control for the equally critical effect of targeted data collection.
* We point out this gap and propose the retrieval baseline to further control for data targeting and resolve it.
* We show that gains from SOTA synthetic methods vanish against this principled baseline.

Our work contributes a way to measure if synthetic data has useful information beyond the generator's training set, thus yielding a clean target for future methods. We're eager to see the gains our field will find.

Thank you! We are happy to discuss further!
Thank you very much for your time and effort in this process! We appreciate your insightful comments, and will be sure to incorporate all other reviewer's valuable feedback in the next version of our paper. Thank you for helping make our work stronger!
Thank you for your rebuttal. Your reply has addressed my concerns regarding Q2 and Q3. 

Regarding your statement in A4, 'Many recent works position synthetic data as a potential drop-in replacement for real data, even when the retrieval baseline is possible,'. In my opinion,  the main advantage of synthetic data lies in its ability to scale effectively. By leveraging larger datasets, it can enhance performance in other tasks. From this perspective, whether considering data scaling capabilities or privacy risks, synthetic data holds more practical value than retrieving real data. 

Of course, the exploration of data quality is currently a major issue. The real focus of the field is how to make synthetic data superior to the real data retrieved in the training set not just using what kind of data as baseline. I appreciate your writing and experiments, could this work give us more deeper insights on how to make synthetic data better?
Thanks to the authors for the rebuttal. Did I miss the response to my last question, quoted below?

> Data scaling: Although synthetic images can be scaled efficiently, they tend to be similar when prompts are fixed. Do retrieved images have more variation with larger data scales than synthetic images? How do the results change if synthetic images in the scaling experiment are sourced using captions from retrieved images?
I go through other reviews and the authors' responses. I will keep my original score of 8. I believe this paper is worth presenting to our community since it highlights a competitive baseline in learning from synthetic images. This baseline will motivate subsequent works to more effectively explore the role of synthetic images in this new era. I think this big advantage has outweighed other minor disadvantages. The authors are also recommended to follow other reviews' advice to further polish the details of this paper.
Thanks for the rebuttal. Could authors give a bit more details about what they mean by targeted refinement and why  [57] and [21] "implicitly conflate the effects of training on synthetic versus real data with the effects of targeted data sampling". I still struggle to see the difference between [57] and "targeted refinement", is it simply that [57] targets the full training dataset instead of a subset like the case studied? Why is this substantially different?
I am glad to rephrase my concern in Weakness (1). I am curious about how text-image pairs from data outside the training set serve as the real-data baseline, compared with the results of the text prompt and generated images. If the result still shows that the quality of the generated data is inferior to real-data, it seems that other real-data from non-training dataset could also serve as a "baseline" for the claims in this article. In that case, what is the necessity of obtaining training data as declared in the article?

The question in the Question section revolves around whether the core answer to the performance on downstream CV tasks discussed in the article is "training and generated data" or "high-quality and low-quality data."? 

Thanks, hope my clarification will help the authors' further response.
Thank you very much for your time and effort in providing us feedback! We highly appreciate your review. Could you please clarify your point (1) under the weakness section? We would love to address your points as throughly as possible. Specifically, we were hoping to clarify the following:

*"... downstream fine-tuning results of the real-and-synthetic data of non-training data. And not clear about the advantage of using upstream real data as baseline but not the othe dataset."*

By "non-training data," do you mean that data outside the generative model's training dataset should also be used as the baseline? Does the main question under the Question section ("There is a main confuse...") refer to the same point? Thank you so much for your time!
[1] Generative models as a data source for multiview representation learning. Ali Jahanian, Xavier Puig, Yonglong Tian, Phillip Isola" (GenRep)

[2] Using diffusion models to generate synthetic labeled data for medical image segmentation. Daniel G Saragih, Atsuhiro Hibi , Pascal N Tyrrell

[3] StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners. Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, Dilip Krishnan

[57] corresponds to the submission citation (SynCLR). All three GenRep [1], StableRep [3] and SynCLR [57] "implicitly have the baseline proposed in the paper", by using the entire dataset (i.e. all the neighbors). In the three papers, the networks are pretrained with generative samples, and the full datasets are used to train the generative models. [1] does not outperform the baseline, while [3] and [57] do.
Thank you for your valuable review! We highly appreciate your feedback. To ensure that we can thoroughly address the review, could you please clarify what works [1] and [2] refer to? It seems that [1], [2] in the review text do not match references [1], [2] in our paper. Also, to double-check, [57] in our paper is a citation to SynCLR -- does "StableRep [57] implicitly..." in the review text refer to SynCLR, or does it refer to a different citation? Thank you very much for your time!