Image retrieval outperforms diffusion models on data augmentation
Authors: Max F Burg
Florian Wenzel
Dominik Zietlow
Max Horn
Osama Makansi
Francesco Locatello
Chris Russell
Venue: Accepted by TMLR ()

Abstract: Many approaches have been proposed to use diffusion models to augment training datasets for downstream tasks, such as classification. However, diffusion models are themselves trained on large datasets, often with noisy annotations, and it remains an open question to which extent these models contribute to downstream classification performance. In particular, it remains unclear if they generalize enough to improve over directly using the additional data of their pre-training process for augmentation. We systematically evaluate a range of existing methods to generate images from diffusion models and study new extensions to assess their benefit for data augmentation. Personalizing diffusion models towards the target data outperforms simpler prompting strategies. However, using the pre-training data of the diffusion model alone, via a simple nearest-neighbor retrieval procedure, leads to even stronger downstream performance. Our study explores the potential of diffusion models in generating new training data, and surprisingly finds that these sophisticated models are not yet able to beat a simple and strong image retrieval baseline on simple downstream vision tasks.

Published in Transactions on Machine Learning Research (11/2023)
Image retrieval outperforms diffusion models on data augmentation
Max F. Burgh-,* Florian Wenzel"", Dominik Zietlow", Max Horn Osama Makansi', Francesco Locatello* and Chris Russell"."
International Max Planck Research School for Intelligent Systems, Tübingen, Germany Institutee of Computer Science and Campus Institute Data Science, University of Göttingen, Germany Tübingen AI Center, University of Tübingen, Germany
'Mirelo AI
'ELLIS Institute Tübingen, Germany
"GlaxoSmithKline, AT&MI, Zug, Switzerland
Amaaoon Tübingen, Germany
A Institute of Science and Technology Austria
Oxffrd Internet Institute, University of Oxford, United Kingdom
WWrk mostly done Amazon Web Services, Tübingen, Germany
Contact: maxxbuugg@betthellbbbor
Reviewed on OpenRevicw:
Abstract
Many approaches have been proposed to use diffusion models to augment training datasets for downstream tasks, such as classification. However, diffusion models are themselves trained on large datasets, often with noisy annotations, and remains an open question to which extent these models contribute to downstream classification performance. In particu- lar, it remains unclear if they generalize enough improve over directly using the additional data of their pre-training process for augmentation. We systematically evaluate a range of existing methods to generate images from diffusion models and study extensions to assess their benefit for data augmentation. Personalizing diffusion models towards the tar- get data outperforms simpler prompting strategies. However, using the pre-training data of the diffusion model alone, via simple nearest-neighhor retrieval procedure, leads to even stronger downstream performance. Our study explores the potential diffusion models in generating new training data, and surprisingly finds that these sophisticated models are not yet able to beat a simple and strong image retricval bascline on simple downstream vision lasks.
1 Introduction
Data augmentation is a key component of training robust and high-performing computer vision models, and it has become increasingly sophisticated: From the early simple image transformations (random cropping, flipping, color jittering, and shearing) (Cubuk et al., 2020 over augmenting additional training data by combining pairs of images, such MixUp (hang et al. 2017 and CutMix (Yun et al, 2019 all the way to image augmentations using generative models. Augmentation via image transformations improves robustness towards distortions that resemble the transformation (Wenzel et al. 2022b and interpolating augmentations are particularly helpful in situations where diverse training data is scarce Ghiasi et al. 2021). With the success of generative adversarial networks (GANs), generative models finally scalod high-dimensional domains and allowed the generation of photorealistic images. The idea of using them for data augmentation purposes has been prevalent since their early successes and forms the basis of new set of data. augmentation

---

Published in Transactions on Machine Learning Research (11/2023)
A 10% ImageNet
B 100% ImageNet
63
top-1 accuracy (96)
top-1 accuracy (%)
ResNet-50
61
ResNet- 500
78
60
59
77
58
57
None (baseline)
Best DM (CLIP promen based
Best DM fine- tuned (pseudoword DM)
Retrieval from Laion 5b
None (baseline)
Azizi al, (2023)
Best prompt- based (CLP prompts)
Best DM fine-tuman to-e-d roseudoword DM)
Retrieval from Laion 5b
Augmentation type
Augmentation type
Figure 1: Are generalive methods beneficial for data augmentation? ? A. Each bar shows the accuracy and standard error of the downstream classifier (ResNet-50, He et al. (016 based on the best augmentation method within each family on the data-scarce 10% sub-sample of ImageNet. While diffusion model (DM) based techniques (orange: a prompting met based on Sariyildiz et al. (2022) and green: a fine-tuned DM) improve over the baseline of only using the original data (blue), a simple, more computationally efficient retrieval method using images from the diffusion model's pre-training dataset directly (red) performs best. B. Methods found be best on 10% ImageNet perform on full ImageNet as well. All methods suggested in this paper outperform Azizi al. (2023) in like-with-like comparison that matches data and classifier architecture (gray; ResNet-50 performance reported in their Table 3, column "Real+GGneratee
strategies Zietlow al.) 2022 Ghosh et al., 2022 Antoniou al.) 2017
Frid-Adar et al. 2018
Esteban
al. 2017 Motamed et al. 2021
Mariani al. 2018 Ramaswamy et al. 2021
Given the emergence of diffusion models (DMs) that outperform GANs in terms of visual quality and diversity RRames et al.] 2022 Saharia et al.) 20226 Rombach et al., 2022), using them for data angmentation is a natural next step. Diffusion models are trained on a large dataset of billions of filtered image text pairs retrieved from the internet, enabling them to gencrate images of unparalleled variety. Furthermore, these models can be easily conditioned using text queries, which allows for easy control of the data generation process.
We benchmark variety of existing augmentation techniques based on stable diffusion models (Rombach et al. 2022 on a randomly sampled 10% subset of the ImageNet Large Scale Visual Recognition Challenge 2012 LLSVRC2012) [Russakovsky et al.) 2015). simulating a low-data regime because adding data through augmentation helps most when training data is scarce (Hyams et al. 2019 Surprisingly, we find that these techniques are outperformed by simply retrieving nearest neighbors from the DM's training dataset (Laion 5b;
Schuhmann al. (202)) with simple prompts (using the same CLiP-like model (Radford et al. 2021 used in the DM). We propose new extensions and variants of slffuuioon-model-baseedd strategies each leading to improvements, however, none beat our simple retrieval baseline. We show that simple text prompts based class labels suffice for conditioning the DMs to improve the performance of standard classifiers (we chose ResNet-50; He et al. (2016)) compared to the unaugmented dataset. As images generated by simple prompts match the training distribution of the DM and not the training distribution of the classifiers, we test- test- ints-ired by related work on personalized DMs methods that fine-tune the DM conditioning and optionally the DMs denoising model component. These fine-tuned models outperformed the best prompting stratogics for their ability to create even better synthetic data for augmentation on ImageNet (Figure D Furthermore, for the best prompt-based and fine-tuned DM, we verified results obtained on 10% ImageNet (Figure TA) generalize
2

---

Published in Transactions on Machine Learning Research (11/2023)
to full ImageNet (Figure
3 outperforming Azizi et al. 2023), and found similar results Caltech256 (Figure T+ Griffin et al.
(007 ). Here, augmentation leads to smaller performance boosts compared to
the data-scarce setting, which is expected since abundant training data generally makes augmentation less important (Hyams et al.
2019 Although all investigated methods based on generating synthetic images
were sophisticated and compute-intensive none of them outperformed simple nearest. neighbor retrieval (Figures 1 andd -
As the Laion 5b dataset provides a compute-optimizze search index to retrieve images which are most similar to text prompt, our retrieval approach is computationally efficient (only training the classifier requires GPU). Importantly, retrieval does not require pre-downloading the entire datasct, rather, only storing the small search index and downloading nearest neighbor images referenced by the index is necessary. As such, even where the performance improvement is minor over diffusion model based augmentation methods (Figures 3 and G , we believe that the conceptual simplicity and efficiency of our approach makes it a compelling alternative.
2 Related work
Data augmentation is widely used when training deep networks. It overcomes some challenges associated with training on small datasets and improv- ing the generalization of the trained models (Zhang et al.
Data augmentation using generative adversarial networks.
2017 Manually designed augmentation methods
have limited flexibility, and the idea of using ML-generated data for training has attracted attention. Gener- ative adversarial networks (GAN) such as BigGAN [Brock et al. 2018) have been used to synthesize images for ImageNet classes (Besnier et al. 2020 Li et al., 2022). Despite early promising results, the use of GANs to generate synthotic training data has shown limited advantages over traditional data augmentation meth- ods (Ziellow et al., 2022). Diffusion models, on the other hand, might be better candidate since they are more flexible via general peneerl-puuposee purpose text-conditioning and exhibit larger diversity and better image quality
(Sohl-Dickstein et al.
2015 Ramesh et al. 2022 Saharia et al. 2022b Rombach ct al. 2022 Hence, in this
work, we focus on diffusion models.
Data augmentation using diffusion models. Recently, diffusion models showed astonishing results for synthesizing images (Sohl-Dickstein et al., 2015 Ramesh et al. 2022 Saharia et al,, 2022b Rombach et al, 2022). Numerous approaches have been published that adapt diffusion models better fit new images and that can be used for augmentation. We evaluate methods that employ diffusion models in guidance-free
manner prompt them without adapting the model: Luzi et al. 2022 generate variations of a given dataset by first adding noise to the images and then denoising them again, and Sariyildiz et al. (2022) generate a synthetic ImageNet clone only using the class names of the target dataset. We also evaluate methods for specializing (AKA personalizing) diffusion models in our study. Given a few images of the same object (or concept), Gal el al. (022)) learn a joint word embedding (pseudoword) that reflects the subject and can be used synthesize new variations of it (e.g., in different styles) and Gal et al. (2023) recently extended their method to significantly reduce the number of required training steps. Kawar et al. (2022) follow similar approach and propose a method for rext-conditioned image editing by fine-tuning the diffusion model and learning a new word embedding that aligns with the input image and the target text. We investigate the usefulness of "personalization" for data augmentation, which was not conclusively addressed in the original papers. We propose and evaluate extensions of these methods tailored to data augmentation and provide a thorough evaluation a unified setting. Finally, we benchmark these approaches against our suggested retricval bascline.
There have been multiple concurrently proposed methods to synthesize images for various downstream tasks that could not include in evaluation of their helpfulness for data augmentation to train an image classifier. Some methods focus on fine-tuning the diffusion model learning a unique identifier to personalize the DM to the given subject (Ruiz et al.] 2022), Shipard et al. (2023) create synthetic clones of CIFAR (Krizhevsky et al. 2009) and EuroSAT (Helber et al.) 2019) for zero-shot classification, other methods employ alternative losses for image generation to improve few-shot learning (Roy et al., 2022), and clones of medical data. Other methods optimize the features of the embedded images to augment specifically small datasets
Packhäuser et al. (2022); Ghalebikesabi et al. (2023) create synthetic,
3

---

Published in Transactions on Machine Learning Research (11/2023)
Zhang et al. 2022 Trabucco et al. (2023) additionally employ image synthesis, image editing (Meng all 2021h and in painting Lugmayr et al. 2022 Saharia et al. 2022a), Bansal & Grover (023)) generate new images conditioning the DM by prompts and example images leading to degraded performance when used to augment ImageNet, and other work focuses on medical data and sample curation (Akrout et al. 2023
He et al. (0023) use pretrained CLIP (Rdiord et al. 2021 as a base classifier and fine-tune it on synthetic images to better match the desired target distribution (e.g. ImageNet).. Their approach improved zero- and few-shot classification performance. To generate the images, they sampled from frozen pretrained DM with sophisticated class-name-bast based guidance and filtered them to keep only synthetic images with high CLIP classification confidence. In the few-shot casc, they used real images to initialize image generation and to remove generated images showing high classification probabilities for wrong classes. Our work does not address improving an already performing, pretrained CLIP model. Rather, we compared ResNet-50 classifier performance trained from scratch on datasets augmented by synthetic images, which we generated from pretrained and fine-tuned diffusion models, and by images retrieved from a web-scale datasct. Most similar to our study is the concurrent work of Aziii al. [2023 augmenting ImageNet with samples generated by fine-tuned Imagen diffusion model (Saharia et al. 2022b). For fine-tuning to ImageNet, they trained parameters of the generative DM while excluding the class- namn--am-aaaaaa based conditioning from optimization. In comparison, all of our augmentation methods performed better (Figure 3 Most importantly, none of the studies mentioned in this section compared against our suggested retrieval baseline.
3 Experimental protocol
We consider wide range of generative and retrieval-based augmentation techniques and evaluate their performance on downstream classification task (Figure AA).
Dataset. ImageNel. To extensively benchmark augmentation techniques, we simulate training data in low-data regime, as generally, augmentation is most helpful when training data is scarce (Hyams et al, 2019). Hence, unless explicitly stated, we use 10% sample (126,861 training images) of the ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC2012) (Russakovsky et al. 2015) training split retaining class imbalance throughout our experiments. As the retrieval method did not return sufficient samples for 10 of the 1,000 ImageNet classes, they were excluded from augmentation, model training, and evaluation. We additionally sample a disjoint set of images of the same size from the training split for hyperparameter optimization and model selection, which we refer to the validation split. When evaluating if results found on the 10% sub-sample generalize to full ImageNet, we include all but our validation split to the training set. All trained classification models are evaluated on the original ILSVRC2012 validation split. Caltech256. We additionally verify our findings on Caltech256 (Griffin et al. 2007 excluding five of the 256 classes for which retrieval did not return sufficient samples. We randomly sampled 80% of the data as train split and randomly partitioned the remaining 20% of the samples into equally sized disjoint validation and test splits.
Diffusion model backbone. To generate images, we used the pretrained Stable Diffussion v1.4 model, based on a latent diffusion architecture (Rombach et al. 2022). We discarded images marked as NSFW by the provided safety checker, replacing them with new samples. Stable Diffusion was trained on image sizes of 512 px, and we kept this resolution for all methods.
Data augmentation and classifier training. Diffusion models stochastically generate images, leading to a distribution of images that we wanted to capture when evaluating classifier performance. Hence, for each DM-based augmentation strategy, we generate 390 images per class ensuring that the mumber of images at least tripled per class, as in our sub-sampled ImageNet dataset each class contains between 74 and 130 images. For full ImageNet, we generate ten times more images. Then, we resample that data into five sets containing the same number of samples per class as our target dataset. We add each of the five additional datasets to the original training data for augmentation, obtaining five augmented datasets that we use for training a ResNet-50 classifier (He et al. 2016) on cach of them to derive variance estimates. On Caltech256 we generate three times as many images per class as in the training set and follow the same approach as for ImageNet. During training, the samples are further augmented by random resizing and cropping as is
4

---

Published in Transactions on Machine Learning Research (11/2023)
A Experimental protocol
B Taxonomy
Prompt learned condilioning
Diffusion model or dalasel retrieval
Prompt template provider
Unique pseudo-word representation
R
E.g. photo ClassName'
Pseudo-word
Uncondittooee Prompt-based Fine-tuned
Text encoder
10% ImageNet
Generated or retrieved
Classifier
Conditioning
10% ImageNet
Generated
Diffusion model
Reconstruction objective
Figure 2: Experimental protocol and laronomy of diffusion model based augmentation melhods. A. We generate images by guiding the Stable Diffusion model by text prompts by learned conditioning, retrieving images by nearest neighbor search in the CLIP embedding space of the DM's training data. We then train the downstream classifier on the original 10% ImageNet data augmented by the additional data and evaluate on the original ImageNet validation split. B. All considered methods adapt different components of the prompting, conditioning mechanismss and the fine-tuning of the diffusion model. We reference each method by circled number (see section 3.1 for details). Some methods edit the prompts while keeping the DM frozen: ① ② use single prompt for each class and 3 4 use multiple prompts from set of templates. Another family of methods optimize the conditioning vectors for the given images: W K only optimize the conditioning vector keeping the DM frozen, while ? also jointly fine-tunes the DM. Instead of optimizing the conditioning vector, S learns pseudo-word description of the class using multiple prompts keeping the DM frozen, while ④ additionally fine-tunes the DM. @ does not adapt any component of the DM and relies on encoding and docoding to create variations of the given images.
common for classifier training on ImageNet. On ImageNet we trained the classifier with batch size of 256 and a lcarning rate of 0.1. On Caltech256 we used a batch size of 1024 and after an initial lcarning rate optimization sweep we set the learning rate to 0.003. We divided the learning rate by 10 when validation accuracy failed to improve for 10 epochs. We stopped training when the validation accuracy did not improve for 20 epochs or after at most 270 epochs and used the highest validation accuracy checkpoint for final scoring.
3.1 Augmentation methods
The benchmarked augmentation methods can be grouped into four categories: (1) guidance-free diffusion model sampling, (2) simple conditioning techniques with prompts based on the objects' class label, and (3) personalization techniques that fine-tune the diffusion model conditioning and optionally the diffusion model itself to the classifier's 's data domain. We compare diffusion model approaches to simple baseline (4) using images retrieved from the dataset that the diffusion model was trained on.
Unconditional generation. Following the procedure of @ BOOMERANG (Luzi et al. 2022 we investi- gated guidanco-free method that does not require conditioning or updating the diffusion model. Instead, the approach adds noise to individual samples before denoising them.
5

---

Published in Transactions on Machine Learning Research (11/2023)
Prompt conditioning. We explore several prompt-based methods of guiding the DM to produce samples for specific class (Figure B). 1 SIMPLE PROMPT: We condition the model by simple prompis containing the object's class name n, prompting the DM with "A photo of n." (method proposed in this paper) and a version of it, ( SIMPLE PROMPT (NO ws), stripping whitespace, w(-), from class names, "A photo of w(n)." (proposed). 3 CLIP PROMPTS: We add sampling prompls from the set of CLIP (Radfrrd et al. 2021 text-encoder templates, e.g. "a photo of many w(n).", "a black and white photo of the w(n).", etc. (proposed) and ( SARIYILDIZ ET AL. PROMPTS: a set of templates proposed to create a synthetic ImageNet clone (Sariyildiz et al. 2022).
To spocify the class name, we leverage ImageNet's class name definitions provided by WordNet Miller 1995) synsets representing distinct entities in the WordNet graph. Each synset consists of one or multiple lemmas describing the class, where each lemma can consist of multiple words, e.g., "Tiger shark, Galeocerdo Cuvieri". We link each class via its synset to its class name. a synset consists of multiple lemmas, we separate them by comma, resulting in prompts like "A photo of tiger shark, Galcocerdo Cuvicri.', as we found that providing multiple lemmas led to better performance than using only the first. lemma of a synset. Whenever methods inserted the class name into prompt templates, sampled the templates randomly with replacement. Sariyildiz et al. (Sariyildiz et al. 2022 provided multiple categorics of prompt templates (e.g. class name only, class name with WordNet hypernyms, additionally combined with "multiple" and "multiple different" " specifications, class name with WordNet. definition, and class name with hypernyms and randomly sampled backgrounds from the places dataset (Zhou et al. 2017)) ). Here, we sampled for each category the same number of images and randomly across the background templates. For Caltech256 we used the class names as provided by the dataset.
Finc-tuning the diffusion model. We explore various methods for fine-tuning a DM for class- personalized sampling to improve reconstruction of the classification dataset (Figure B3. . FT CONDI- TIONING: freezing the DM and optimizing one conditioning per class (method proposed in this paper). FT CLUSTER CONDITIONING: optimizing multiple conditionings per class rather than just one (proposed). ? inspired by IMAGIC (Kawar et al., 2022), jointly fine-tuning the conditioning and the DM's denoising component. K TEXTUAL INVERSION (Gal et al. 2022): : instead of fine-tuning the conditioning, sampling prompt templates and combining them with optimizing a pseudo-word representing the class-concept. DDE combining the previous approach with optimizing the DM's denoising component.
We trained all models with the default Stable Diffusion optimization objective (Rombach et al. 2022 until model was trained for more than 40 epochs. When fine-
the validation loss stagnated or increased tuning the diffusion models on full ImageNet, the number of training samples increased 10-fold, thus, to keep the computational budget comparable across modalities, we stopped training after 3 epochs. We set hyperparameters in accordance with publishod works (Gal et al. 2022 Kawar et al.] 2022 Luzi et al. 2022) or existing code where available. Scaling to larger batch sizes was implemented by square-root scaling the learning rate to ensure constant gradient variance, and we performed fine-grained grid-search for the optimal learning rate. For the FT CLUSTER CONDITIONING models, we clustered the training images within class using k-means on the inception v3 embeddings. The conditioning vectors were initialized by encoding the SIMPLE PROMPT (NO WS) prompts with the DM's text encoder. For textual inversion [Gal el al.) 2022 we fine-tuned the text embedding vector corresponding to the introduced pseudoword and sample from the provided text templates (Gal et al. 2022 to optimize the reconstruction objective for our ImageNNtt subset, freezing all other model components. We initialized the pseudo- word embedding with the final word in the first synset lemma (i.c., for the class "tiger shark" we used "shark"). Where this initialization resulted in multiple initial tokens, we initialized with the mean of the embeddings. For each fine-tuning run we stored checkpoints with the best train and validation loss and those corresponding to 1, 2 and 3 epochs of training. In the case of IMAGIC (Kwarr et al. 2022) found that the validation loss was still decreasing after 40 epochs, however, as the quality of the reconstructed images significantly deteriorated after 8 epochs, we instead stored checkpoints for the first 8 epochs. This is similar to the Imagic training scheme, which optimized the embedding for 100 steps and the U-Net for 1,500 steps on a single image. Although we follow existing procedure as closely as possible, image quality might improve for longer model training runs. We selected the final model checkpoint based on the validation accuracy of a ResNet-50 classifier trained on the union of augmented samples and the target dataset.
6

---

Published in Transactions on Machine Learning Research (11/2023)
Table 1:
Overall performance of the data augmentation methods. We report the top-1 accuracy and standard
error of the downstream ResNet-50 classifier (He al. 2016 trained on the augmented 10% ImageNet
subset. The methods are described in section 3. and are grouped into families as described in the beginning
of section i The circles refer to the taxonomy in Figure 2.
Augmentation method
Accuracy (%)
10% ImageNett
57.2 + 0.2
20% ImageNett
70.2 + 0.3
Boomerang Luzi et al. 2022 0
56.3 + 0.3
Simple prompt (no ws; proposed)
60.0 0.3
Simple prompt (proposed) !
60.1 0.2
Sariyildiz et al. prompts Sariyildiz et al.
2022
60.8 0.2
CLIP prompts (proposed)
60.9 + 0.2
FT conditioning (proposed)
60.8 0.1
FT cluster conditioning (proposed) Imagic (conditioning & DM: Kawar al.
60.9 0.2
2022
61.0 0.3
Textual inversion (pscudoword; Gal et al.
(022
61.0 = 0.4
Pseudoword+DM (combining Gal et al.
(2022
Kawar et al. 2022 9
61.2 + 0.2
Retrieval (our suggested baseline)
10
62.6 0.1
Laion nearest neighbor retrieval. As baseline comparison, we suggest using (10 RETRIEVAL (method proposed in this paper) to select images from the Laion 5b (Schuhmann et al. 2022 dataset used to train the dilfusion model. This method finds nearest neighbor images to the SIMPLE PROMPT (NO ws) class name prompts in the CLIP embedding space.
Laion 5b is a publicly available datasct of 5 billion image-caption pairs extracted via web-crawler. The data was then filtered by only retaining images where the CLIP image embedding was consistent with the caption embedding. This filtering acts as a weak form of supervision, that retains a set of images where CLIP is more likely to work. The dataset provides CLIP embedding ncarest neighbors scarch index for cach instance, and official package [Beaumont] 2022) allows for fast search and retrieval. We used this to retrieve 130 images per class for ImageNet and the same number of samples per class for Caltech256. Unlike generating images with DMs, retrieving images is not stochastic, and we augment with those retrieved samples that are most similar to the class name search query. Images with a Laion aesthetics score of less than 5 were discarded to allow a fair comparison with Stable Diffusion 1.4 which was trained on this subset. For a fair comparison to our generative augmentation mothods, we used the same safcty checker model to discard images that were marked as NSFW. Due to changes in the availability of the images al the URLs in the dataset and the described filtering steps, it is often necessary to retrieve more than the desired number of images, which we do by increasing the number of nearest neighbors gradually from 1.4 130 to 10- 1.4 130 when not enough samples were found. To avoid using the same image multiple times, we applied the duplicate detector of the clip-retrieval package (Beaumont 2022 however, this does not detect all near duplicate images and some duplicates are still used.
Retrieval is computationally efficient. Just as simply prompting a pretrained diffusion model to gener- ate images avoids the need to download and train on a large datasct like Laion 5b, the use of cxisting retrieval search indices allows us to only download those images closest to the search prompt, without downloading and indexing the entire dataset ourselves. This makes any retrieval component of the pipeline computa- tionally efficient, and bottle necked by internet access rather than pure compute. As such, the performance numbers we report should be seen as representative, rather than being exactly replicable by someone using the same compute, because availability of images at the indexed internet addresses is not guaranteed.
7

---

Published in Transactions on Machine Learning Research (11/2023)
A ImageNet
B Boomerang
C Difference
Figure 3: Example images generated by BOOMERANG (Luzi et al. 2022p. The generated images lack diversity and effectively only add noise to the original ones. A. Example images from 10% ImageNet. B. The augmentations produced by Boomerang. C. There is only small difference between the original and augmented images (best viewed on screen).
We estimated compute requirements for all methods augmenting 10% ImageNet from log files and file time- stamps. For full ImageNet, requirements were approximately ten-fold higher. RETRIEVAL was the fastest method, only requiring a CPU machine with GB of CPU RAM and a 1.6 SSD disk store the scarch index, completing image retrieval in 28.5 Here, the bottleneck be avoided if a local copy of Laion 5b's image caption pairs is available, as nowadays is the case for many institutions. T'ine-roqirrmmnnt for DM fine-tuning depended on the specific method: FT CLUSTER CONDI- TIONING: 244 GPUh, h, FT CONDITIONING: 380 GPUh, h, TEXTUAL INVERSION: 368 GPUh, IMAGIC: 552 GPUh, h, PSRUDOWORD++DAE 720 GPUh (trained on 8 Nvidia V100 GPUs using distributed data parallel training, L.e 1 h corresponds to 1/8 = min training time on 8 parallel GPUs). Generating 390 images for each ImageNet class required 867 GPUh (Nvidia T4 GPU). Since ResNet-50 classifier training was early stopped, training times varied typically between 128 and 160 GPUh (trained 8 Nvidia T4 GPUs using distributed data parallel training). Overall, RETRIEVAL
downloading the images, which can
most efficient: 30x faster than the prompt-based DM methods, and 56x faster than the best performing and most compute intense PSEUDOWORD+ while not requiring an expensive GPU machine.
4 Results
We first analyze augmentation the data-scarce 10% ImageNet subset, providing results summary in Table I in which each block corresponds to the groups of augmentation methods introduced in Section 3.1
Section introduces the baselines, section benchmarks the previously suggested unconditional DM augmentation method and a method proposed create a synthetic ImageNet clone (Sariyildiz al. 2022
against our simple RETRIEVAL baseline method, section discusses improvements of conditioning the DM with (ext prompts, and section 1 discusses persona lization approaches to diffusson models. Finally, section compares RETRIEVAL synthetic augmentation data, explores scaling behavior, verifies that the methods in this paper generalize to full datasots, and certifies superior RETRIEVAL performance is not due to retrieving ImageNet duplicates for augmentation. We obtained all classification performance numbers by
training a 5e (He et al. 2016
4.1 Baselines
To establish the baseline classifier performance, we train on our 10% ImageNet subset (no added data), achieving a top-1 accuracy of 5 Since all augmentation methods double the size of the dataset, we consider an upper bound performance by using 20% of original ImageNet (no added data) with a classifier accuracy of 70.2 0.3.
4.2 Retrieval outperforms previously suggested diffusion-based augmentation
Unconditional generation. BOOMERANG Luzi et al. 2022 generates augmented images by sequentially adding Gaussian noise and uses a diffusion model to denoise elements of the target dataset, without altering the diffusion model or prompts. This results in a lower top-1 accuracy of 56.3 0.3 compared the original subset. of ImageNet. Figure 3 shows that this method does not. induce significant. diversity in the dataset
8

---

Published in Transactions on Machine Learning Research (11/2023)
ImageNet
Simple pr.
No ws pr.
CLIP pr.
E FT Cond.
Text. inv.
Imagic
Pseu.-DM
Retrieval
Papilon
Motor sopoter, scooter
Mailbag. posibag
Beer glass
Desktop computer
Figure 4: Exomple images obtained from the investigated augmentation methods. A. 10% ImageNet original
images. B.-D. Images generated by our prompt-based sampling techniques: (B) SIMPLE PROMPT, (C) SIMPLE PROMPT (NO ws) and (D) CLIP PROMPTS, respectively, where in (D) we show images the prompts "a bad photo of w(n)", "a black and white photo of the w(n)", "a cartoon w(n)", and "a photo of many w(n)". E.-H. Images generated by the fine-tuned diffusion models, specifically (E) FT CONDITIONING, (F) TEXTUAL INVERSION, (G) IMAGIC, and (H) PSEUDOWORD+ DM. I. Examples of RETRIEVAL from the diffusion model's training set. Best viewed when zoomed in.
and only slightly distorts the original images, leading to an overall decrease in performance. However, since this method is directly applied on the target dataset samples, it does not suffer from domain shift or class ambiguity.
Prompt-conditionig.. Sariyildiz et al. 2022 generated augmented training examples with a diffusion model adapted to the target dataset via conditioning mechanisms. Their method guides image generation by various prompts based on the class name (see section 3.1 for details) and achieves 60.8 0.2 classification accuracy, performing better than the 10% ImageNet baseline and worse than the 20% ImageNet upper bound (Table
Retrieving from the diffusion model pre-training data. We now established that diffusion models are helpful for creating augmentation data, however, it unclear how much value the DM's generative capabilities add compared to simply using their pro-training data directly for augmentation. To answer this question, we propose a simple RETRIEVAL method fetching images from the DM's pre-training data that are semantically closest to the SIMPLE PROMPT (NO ws) prompts (see section 3.1 for details). Augmenting with this data outperformed the sophisticated diffusion model based approaches at 62.6 0.11 top-1 accuracy, while being computationally less demanding.
Summary. Using diffusion models off the shelf, i.e., without adaptation or conditioning using the target dataset, not beneficial and can even deteriorate the classifier performance compared to the ImageNet bascline. Augmenting with generated samples adapted the target dataset by prompt-conditioning performs better than the unaugmented baseline. However, simply augmenting with images retrieved from the diffusion model's training data outperforms the DM based approaches, indicating that DM generated images have
9

---

Published in Transactions on Machine Learning Research (11/2023)
significant shortcomings. As retrieved images are real-world images, they show high photorealism, variety, and detail. However, retrieval underperforms the 20% ImageNet upper bound performance, which might be caused by a distribution shift in images between ImageNet and Laion 5b, and failures in retrieval, including both the retrioval of irrelevant images from the CLIP-based search index, and class ambiguity ("papillon" and "mailbag, postbag" in Figure 1
reflecting a mismatch of concepts from CLIP latent space to ImageNet classes as we measured semantic similarity as distance of CLIP embeddings.
4.3 Advanced prompt-conditioning improves but is not competitive with retrieval
Then, we embarked on a quest asking if we can improve diffusion model based methods over the strong RETRIEVAL baseline. To do so, we first explored several variations of text prompt conditioning strategies.
Simple prompt conditioning. We investigate SIMPLE PROMPT conditioning method that uses the prompt "A photo of n.", where is replaced by the class name, e.g., "A photo of tiger shark, Galeocerdo cuvieri." This improved over the 10% ImageNett baseline to top-1 classification accuracy of 60.1 + 0.2, but Figure PB shows clear problems with class ambiguity and a lack of diversity.
Tackling class ambiguity by white space removal. While generally mitigating class ambiguity is a hard problem, we focus on the ambiguity introduced by class names composed of multiple words. To this end, we investigated the variant SIMPLE PROMPT (NO ws) (no white space) that removed the white space from the class name used by SIMPLE PROMPT; e.g., "A photo of desklopcompuler.. While class ambiguily reduced for some classes (e.g., desktop computer; Figure AC) it was ineffective for others (e.g,, papillon) and overall resulted in slightly degraded performance 60.0 + 0.3 compared to koeping the white space. We explore more advanced methods in the following sections.
Improving sample diversity by diverse prompt templates. To increase the diversity of the samples, we made use of multiple prompt templates. We use CLIP PROMPTS which randomly selects one of the text templates provided by CLIP (Radford al., 2021 (see Figure D for examples and Radford et al.) (201)) for full list), increasing classification accuracy 60.9 0.2. This method performed best among all prompt-based techniques. Interestingly, the overall performance increased even though some prompts lead to synthetic images that did not match the style of ImageNet samples (e.g., "a cartoon photo of the papillon.") or images with texture-like contents instead of objects (e.g., papillon, image to the bottom right in Figure (D). Surprisingly, these slightly more elaborate templates just adding few more words compared to SIMPLE PROMPT (NO ws) (e.g., "a bad photo of a papillon.") improved class ambiguity in some cases (e.g., papillon; Figure !D).
Summary. Augmenting a dataset with images sampled by prompt-based conditioning techniques improves the downstream classifier performance beyond the unaugmented datasets, however, none of these methods improve over RETRIEVAL. Inspecting the generated samples, we find that various challenges remain: Ima- geNet is more than a decade old, and some images included in the dataset are older and of different style when compared to the images created by the recently trained Stable Diffusion model (e.g., desktop computer; Figure AA-D). In other cases, generated images do not match the domain of ImageNet samples, for instance because they are too artificial, sometimes even like a computer rendering (e.g., motor scooter; Figure A-D), their texture does not match (e.g., papillon; Figure A) or the prompt does not match the desired style (e.g., cartoon; Figure A). This tendency may have been amplified by Stable Diffusion v1.4 being trained on a subset of Laion, which was filtered to contain only aesthetic images
4.4 Personalizing the diffusion model improves further but does not outperform retrieval
In the previous section, we found that adapting the diffusion models by only editing the prompt offers limited improvement when used for augmentation. On our quest to improve diffusion models to narrow down the performance gap to the RETRIEVAL baseline, this section explores a more advanced set of methods that additionally fine-tune parts of the diffusion model to "personalize" it to the 10% ImageNet training images
http:://gitaab.cc///mpis/sssssss fifusioon
10

---

Published in Transactions on Machine Learning Research (11/2023)
by optimizing the DM reconstruction objective (c.f. Figure 3). Originally, these methods were proposed in the context of personalizing the model to a specific object or concept, i.e., a single or only a few images.
Fine-tuning the conditioning vectors. The general diffusion model architecture has multiple compo- nents that can be fine-tuned (see Figure 3). We start by fine-luning one conditioning vector per class to optimize the reconstruction objective for our ImageNet subsample, while keeping the other model weights fixed. This method, dubbed FT CONDITIONING (fine-tune conditioning). achieves an augmentation accuracy of 60.8 + 0.1 and is par with the best prompt editing method CLIP PROMPTS. Interestingly, while this method achieves good augmentation performance, the generated images do not look photorealistic, and suffer from noise and missing backgrounds (Figure
Fine-tuning clusters of conditioning vectors. Using a single conditioning vector for all images from one class might be insufficient to capture the full class variability. To see if this the case, we explore FT CLUSTER CONDITIONING. We generate k clusters of images per class, before fitting a conditioning vector to each individual cluster (see section 3.1 for details). We find that using & = 5 clusters slightly improves the accuracy by 0.1 percentage points over using k = 1. For larger k = 10 and k = 15, performance decrcased (60.8 = 0.2 and 60.6 = 0.3 accuracy). This might be due to the smaller number of images per cluster, making the fine-tuning more prone to over-fitting.
Textual inversion. To reduce the number of unrealistic images, we explored a variant of TEXTUAL INVER- SION (Gal et al. 2022p. This method learns a pseudoword n representing the class concept combined with a randomly drawn textual description of the generated image style (e-g., "a photo of n", "a rendering of a n², etc.; see Gal et al. (002)) for a full list). While this improves the photorealism of the generated samples (Figure 1 F), it only slightly improves the augmentation accuracy over the simple fine-tuning of conditioning vectors (FT CONDITIONING) by 0.1 percentage points (Table
Fine-tuning the denoising. We now explore fine-tuning the DMs denoising module jointly with the conditioning vector. This idea stems from IMAGIC (Kawar et al., 2022) and results in an augmentation accuracy of 61.0 + 0.3 (examples in Figure GG), which is on par with TEXTUAL INVERSION.
Finally, we combine the best-performing methods: jointly optimizing pseudo-word word per class (TEXTUAL INVERSION (Gal et al. 2022 and the DMs denoising module (IMAGIC (Kawar et al. 2022)). ). We denote this method by PSEUDOWORD
resulted in an accuracy of 61.2 + 0.2, outperforming all other DM basod techniques investigated. The generated images also show improved photorealism over TEXTUAL INVERSION and IMAGIC (Figure H
Summary. Personalizing diffusion models improves in matching the domain of the 10% ImageNet images better. For example, ImageNNet was collected about two decades ago, and while prompt-based techniques generate instances of the class "desktop computer" similar to modern computers (last row in Figure g 3--D), the fine-tuned models generate images depicting older computers (Figure ---H), better matching images from the time when ImageNet was created (Figure A more examples in Appendix Figure Furthermore, personalizing improves upon prompt-based techniques to reduce class-ambiguity (e.g. papillon and desktop computer showed ambiguity in Figure A but not in panels E-H),, show good sample variety and the best- performing PSEUDOWORD images. This model performs best across all generative augmentation techniques investigated, and suggests that we can leverage personalization techniques to combine the DM's knowledge of billions of annotated images with learning the domain distribution of the data we want to augment. At the same time, none of these sophisticated methods outperformed the simpler and computationally cheaper RETRIEVAL baseline.
method enhances photorealism and reduces the artistic style of generated
4.5 When does retrieval help?
Retrieval helps for most classes. To better understand the failure cases of the augmentation methods, we check for each class in ImageNet if the augmentation method is beneficial. To this end, for each class, we compute the performance improvement of the downstream classifier using the augmentation method com- pared to only using the original Tet samples. Figure 5A shows the distribution of improvements for
11

---

Published in Transactions on Machine Learning Research (11/2023)
A Classification accuracy improvement across classes
B Scaling behavior
Worst deterioration
IN DM
IN Ret.
Best improvement
IN DM
Ret.
Top-1 accuracy (%)
0.66
Fracton (%)
0.65
0.64
Psoudaword-DM
.e. Retrieval
20
0.61
Improvement (% top-1 acc.)
100
500 1000 2500
Data added for augmentation (%)
Figure Retrieval and diffusion model performance bution of classifier improvement for each class is shown for the best-performing DM-based method PSEU- DOWORD+DM (blue) and RETRIEVAL (orange). For each class, the improvement is computed by comparing the downstream classification performance using angmentations compared to only using the original 10% ImageNet samples. Images to the left and right show examples classes with the worst deterioration and greatest improvement. B. RETRIEVAL outperforms synthetic images across augmentation ranges. For 1000% augmentation, we could retrieve sufficient samples for 7 additional classes. Assuming classifier performance of 0% or 100% on these classes leads best- (V) and worst-case estimates.
classes and scaling behavior.
A. The distri-
top-1 accuracy (%)
66
ResNet- 50
64
62
60
None (basetine)
Best DM prompt- prompt-based based (CLIP prompts)
Best DM fire- tined (nseudoword undworrd DM)
Retrieval from Laton 5b
Figure Resulls on Callech256. Similarly to ImageNet, on Cal- tech256 diffusion model based approaches did not outperform our proposed simple RETRIEVAL baseline as well. Error bars: standard error of the mean across five runs.
Augmentation type
the RETRIEVAL augmentation method and the best DM-based method PSEUDOWORD+DM.. Both methods improve the performance for most of the classes, however, there are still many classes where the performance is decreased to 10%. The distribution of improvements for RETRIEVAL is similarly shaped as for PSEU- DOWORD+DM, but significantly pushed to the right. Systematically investigating these failure cases might a fruitful avenue to further improve generative augmentation.
Retrieval has better scaling behavior. As diffusion models can generate arbitrarily many samples, we explored our best-performing diffusion model's (pSEUDOWORD+DX)) scaling behavior for higher augmen- tation ratios. To compare its performance to RETRIEVAL, retrieved abundant samples and augmented with those images which CLIP embeddings the performance obtained by adding 100% retrieved images would require augmenting with approximately 400% to 500% synthetic images (estimated from Figure 5 3), highlighting retrieval's data efficiency. As more data added augmenting with samples from both methods, the performance gap between augmentation with retrieved and synthotic images increases substantially. Since the performance of classifiers trained on synthetic images saturates, is unlikely that there will be sufficient data for the current generative methods to close the gap in like-with-like comparison.
most similar to retrieval prompt (Figure 5B.. To reach
12

---

Published in Transactions on Machine Learning Research (11/2023)
Augmenting full datasets. So far, all presented results address the case of small dataset, simulated by drawing a subsample from ImageNet. Hence, we asked how well the best prompt-conditioned and per- sonalized DM augmentation techniques would perform compared to retrieval on large datasets. On full ImageNet (Figure [3), we found results consistent with the subsampled ImageNet: RETRIEVAL performed best (79.0 + 0.2) followed by PSEUDOWORD+ (78.8 0.2) and CLIP PROMPTS (78.7 + 0.1). All of our approaches outperformed the ResNet-50 bascline (76.1) and the concurrent study by Azizi et al. (2023) (78.17% ResNet-50 accuracy, see their Table 3, colurnn "Reaa-Genrrated"..
On full ImageNet, the performance boost of all methods narrowed compared to the data-scarce case: RE- TRIEVAL improved 2.9% accuracy on full ImageNet vs. 5.4% points on 10% ImageNet; similarly, PSEU- DMWORD+DM improved 2.7% on full ImageNet vs. 4% on 10% ImageNet (see Figure DA vs. 33). This does not mean that retrieval scales worse than DM based approaches. Rather, we generally expect augmentation to be less effective on large datasets, which would explain the smaller performance gains (Hyams et al., 2019
Additionally, we verified our results on Caltech256 Griffin et al. 2007 showing that diffusion models do not outperform our simple RETRIEVAL baseline across datasets (Figure
RETRIEVAL: 68.4 + 0.4, PSEU-
DOWORD+DM: 68.1 + 0.3, CLIP PROMPTS: 68.2 + 0.2, no augmentation: 59.3 + 0.7).
Retrieval adds negligible ImageNet duplicates. Cherti et al. (022) reported that one percent of ImageNet images are contained in Laion 400m, predecessor of the larger Taion 5b used in our study, raising two questions: (1) did RETRIEVAL perform better than diffusion models because it added real ImageNet images that were not available for training before, and more importantly (2) did test-set images lcak into the augmented training set? To address these questions, we leveraged perceptual image hashing (Zauner 2010) to compare images used for augmentation to ImageNet samples and considered any image-pair as a duplicate candidate if their hashes' Harmiing distance was less than ten (Cherti et al., 2022). Next, we refined this approach by manually inspecting all duplicate candidates. For 10% ImageNet, we found 0.009% of the test-set (11 of 126,861 test images) leaked to the RETRIEVAL augmented training data (Appendix Figure If we conservatively assume that these 11 test-images were classified correctly when included in the training-set, but would be misclassified if they were not present in the training-sel, we would see drop of 0.009% accuracy. 0.13% of the retrieved images were either in our validation split or the portion of ImageNet not used in the 10% training subsample. On the original test-set, which we did not use in our study, hashing found 2,377 potential duplicates. We inspected 500 of them and found four actual duplicates. Hence, we estimated 19 duplicates (0.01% % of the retrieved data) were used for augmentation. We hypothesize original test-set duplicates might be scarcer since original test-labels were never made public, thus, they might be inaccessible in the text to image nearest neighbor search index. For the best performing diffusion model (PSEUDOWORD+DMI we inspected 1,000 potential duplicates and did not find identical images (examples in Appendix Figure T. Overall, the minimal overlap of retrieved images and ImageNet cannot account for RETRIEVAL's ' large performance boost of 1.4% over DS.
For full ImageNet, we ran RETRIEVAL at a later point, and different images were returned because accessibility of the images at the internet addresses provided by Laion 5b fluctuates (sce Scction 3.1 for dotails). Here, we found that 0.002% test. images (3 of 126,861 lest. images) leaked into the augmented data. The only relevant influx of additional ImageNet images to the full ImageNet train-set could emerge from the validation set (we found 2 images, corresponding to 0.0002% of retrieved samples) or from the original test-set, for which we estimated that 187 real duplicates (0.02% of the retrieved data) were added by augmentation (estimated based on the true duplicate rate obtained in the previous paragraph). Again, these minimal numbers of duplicates cannot account for RETRIEVAL's superior performance, adding 0.2% accuracy over PSEUDOWORD+DDM. Overall, our results are in line with previous studies reporting only minor impact of duplicates RRdlord et al. 2021 Zhai et al. 2022
Li al. 2023
5 Discussion
Diffusion models have shown their effectiveness in many application areas, and using them for data augmen- fation is intriguing research direction. We have evaluated multiple methods to prompt and personalize
13

---

Published in Transactions on Machine Learning Research (11/2023)
diffusion models on their usefulness for data augmentation, showing that none of them beat the simple base- line of retrieving images from the diffuusio model's pre-training dataset. Why is the retrieval baseline so hard to beat? One reason might be that retrieval potentially accesses more information, as the pre-training dataset is usually much larger than the weights of the generative models trained on it. Although it has
been argued that diffusion models might partially compress the training data (Soeealli et al. 2022 Carlini et al. 2023) it is still unclear if the generative model captured all relevant information. However, diffusion models could possibly improve upon the so-far superior retrieval by generating a large amount of additional data and more diverse and compositionally novel images, for instance by generating out-of-domain samples (e.g.. "a photograph of an astronaut riding horse") (Ramesh et al., 2022 Saharia et al. 2022b Rombach et al. 2022). Furthermore, diffusion models allow, in principle, for more controlled adaptation than retrieval methods. We showed that personalization methods are a good step in this direction, however, they lypically focus only on creating variants of specific given images. To unlock the true potential of diffusion models for data augmentation, now methods that capture the target dataset manifold as a whole, are needed.
Limitations and future work. Diffusion models are an active research area, and now methods and applications are published on a daily basis. Hence, our experiments could only capture a subset of possible methods and newly proposed extensions of them (in total, eleven methods), and we evaluated all methods by training ResNet-50 classifier. However, performance might vary for other classifier choices. Similarly, replacing DM backbones by newer, better versions might improve performance for any augmentation method that is based on them. To simulate regime of scarce data, in which augmentation generally is most helpful, we used the same 10% random ImageNet subsample throughout our experiments. In principle, selecting another subsample from the same data distribution might slightly alter classifier performance. However, as this sub-split is shared across all augmentation methods, all models (including the baselines) would be systematically affected in the same way. Thus, we do not expect the particular choice of a specific sub-split would relevantly change performance differences between methods or affect our overall results. This notion is in line with our finding of results being consistent between 10% ImageNet and full ImageNet.
It would be interesting to extend our analysis to other specialized data for which collecting new samples might be difficult, for example medical images. Such data might not be available in web-crawled data like Laion 5b on which diffusion models were trained on. Regardless, previous work adapted DMs to medical data, generating privacy-preservee
datasot clones. Downstream classifiers trained on these synthetic clones exhibited a performance gap to real data (Packhäuser et al.) 2022 Ghalebikesabi et al. 2023) Augmenting by retrieved images might be even more severely impacted, as appropriate images might not be sufficiently available in the web-crawled data. Evaluating and developing generative and retrieval based augmentation methods for such specialized data might be an impactful avenue for future work. Similarly, exploring the investigated methods for -itto---ditributionn generalization might be promising future work as well (Qiu et al. 2022 Wenzel et al.) 2022al). We have shown that simple retrieval is a very strong competitor for data augmentation. Further improvements could be introduced by diversifying the retrieval set (Wenzel et al. 2020 Yue & Guestrin 2011), retrieving images using linear combinations of inputs (Zietlow et al.) 2022). generating more diverse retrieval search prompts leveraging off-the-shelf word-to-sentence models and data-filtering that were beneficial in CLIP finc-tuning setting (He et al. 2023
Conclusion. We showed that simple, computationally efficient retrieval baseline outperforms a wide range of iiffusiinnmodel-been angmentations. However, given the fast rate of progress in this field it is not possible to definitively say that retrieval can not be beaten, and we believe that diffusion models have the potential to improve over this bascline. Nonetheless, the strength of retricval's performance makes it clear that future works using diflusion models for augmentation should also compare against this baseline. We hope that our paper provides ground for researchers to benchmark generative augmentation methods and assess their benefit by comparing them with retrioval basclines. In the longer term, we hope that new methods can be developed that combine retrieval and generation, leading to greater improvements in the diversity and quality of augmented images.
14

---

Published in Transactions on Machine Learning Research (11/2023)
6 Broader impact
As our work uses Laion 5b (Schuhmann et al. 2022] and stable diffusion (Rombach et al. 2022 many of the ethical and societal problems of these works propagate lo our study. For example, explicit images of rape, pornography, stereotypes, racist and ethnic biases, and other toxic content have been found in Laion 400m Birhann ct al. 2021) and it is very likely that they are contained in its larger successor Laion 5b. As stable diffusion was trained on similar data, it might produce similarly problematic content (Carliii et al. 2023 We filtered images by the automatic safety-checkers provided by Laion 5b and stable diffusion and trained classifiers on them. As such, offensive images, even those missed by the safety checkers, will not be presented to users of the classifier. However, systematic biases present in the data are still likely to be present in the final classifier decisions. This is less of a concern for the sanitized labels used in the ISSVRC2012 ImageNet challenge which we evaluated on (the only person categories used are "scuba diver", "bridegroom", and "baseball player"; Yang et al. (20201) but for potential other tasks not studied in our work, such as labeling people as "hard-worker" vs. "lazy", racial and gender biases should be expected. These concerns are not unique the use of Laion 5b, and have been shown be present wide range of web-sourced datasots. In particular, the person subtree that used to be contained in the larger ImageNet dataset (this is a superset of the data used by the ImageNet Large Scale Visual Recognition Challenge (IISSRCC2022) that considered
(Yang
in our study) has been shown to contain offensive labels and biases Crawford & Paglen 2001 Yang et al. 2020
Acknowledgements
The authors would like to thank Varad Gunjal and Vishaal Udandarao. MFB thanks the International Max Planck Research School for Intelligent Systems (LMPRS-IS).
References
Mohamed Akrout, Bálint Gyepesi, Péter Holló, Adrienn Poór, Blága Kincsö, Stephen Solis, Katrina Cirone, Jeremy Kawahara, Dekker Slade, Latif Abid, Máté Kovács, and István Fazekas. Diffusion- based Data Augmentation for Skin Disease Classification: Impact Across Original Medical Datasets to Fully Synthetic Images, January 2023. URL http:///rxiv. http://axxvv..rr//aaaae 22t//1bs//301... 04802 arXiv:2301.04800 [cs, eess].
Antreas Antonion, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial net- works. arXin preprint arXiu: 3rXiv:1711.000 1.0344, 2017.
Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. Synthetic Data from Diffusion Models Improves ImageNet Classification, April 2023. URL http:://aaaiv..rr/////.. 0304.08466 arXiv:2304.08466 [cs].
Hritik Bansal and Aditya Grover. Leaving Reality Imagination: Robust Classification via Generated Datasets, February 2023. URL http:p//aaxxi..or///b//2/222 02500 02503 arXiv:2302.02500 [cs].
Romain Beaumont. Clip retrieval: Easily compute clip embeddings and build a clip retrieval system with them.
2022.
Victor Besnier, Himalaya Jain, Andrei Bursuc, Matthieu Cord, and Patrick Pérez. This dataset does not exist: training models from generated images. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2020.
Abeba. Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwee. Multimodal datasets: misogyny, pornography, and malignant stereotypes, October 2021. URL arXiv:2110.01963 [cs].
wwwpg///aa//...../////// 01963,
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. ar Xio preprint or 1rx::88000 1096, 2018.
15

---

Published in Transactions on Machine Learning Research (11/2023)
Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting Training Data from Diffusion Models, January 2023. URL
http://arxiv.ont www..org/abbo abs/a301. 13188 arXiv:2301.33188 [cs].
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhrnann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning, December 2022. URL https://arxiv... wttp://////.........///aaa
Kate Crawford and Trevor Paglen. Excavating ai: The politics of training sets for machine learning. The AI Now Institute, NYU, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognation workshops, pp. 702-703, 2020.
Cristóbal Esteban, Stephanie L Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional gans. arXiv preprint arXiv: (at:1706602633 2017.
Maayan Frid-Adar, Idit Diamant, Eyal Klang, Michal Amitai, Jacob Goldberger, and Hayit Greenspan. Gan- based synthetic medical image augmentation for increased enn performance in liver lesion classification. Newrocomputing, 321:321-331 2018.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen- Or. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion, August 2022. URL http:://axiiit// www.oogggg/o..ooo 2208 016118 DXXiv:2208001115 [cs].
Rinon Gal, Moab Arar, Yuval Atzmon, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. Designing an Encoder for Fast Personalization of Text-to-Imast Models, February 2023. URL http://arxiv. org/
bbs/2232..102 12228 arXiv:2302.1222 [cs].
Sahra Ghalebikesabi, Leonard Berrada, Sven Gowal, Ira Ktena, Robert Stanforth, Jamie Hayes, Soham De, Samuel L. Smith, Olivia Wiles, and Borja Balle. Differentially Private Diffusion Models Generate Useful Synthetic Images, February 2023. URL http::ttt::://t wwaeexiv..oggveee htngg/bs/2200 23022 13861, arXiv:2302.133. 13861 [cs, stat).
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF conference on compuler vision and pullern recognilion, pp. 2918 2928, 2021.
Partha Ghosh, Dominik Zietlow, Michael J Black, Larry S Davis, and Xiaochen Hu. Invgan: invertible gans. In Pattern Recognition: 44th DAGM German Conference, DAGM GCPR 2022, Konstanz, Germany, September 27 30, 2022, Proceedings, pp. 3 19. Springer, 2022.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognilion, pp. 770-778, 2016.
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition?, February 2023. URL http://arxiv. org/ ht5//bs//2111 22tp//272100707 arXiv:2210.07574 [cs).
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. (11772217-2226, 2019.
Gal Hyams, Dan Malowany, Ariel Biller, and Gregory Axler. Quantifying diminishing returns of amno- tated data: How much data do you really need?, September 23 2019. URL https://clear.nt ml/blog/
Accessed: 19 Oct 23.
16

---

Published in Transactions on Machine Learning Research (11/2023)
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Trani. Imagic: Text-Based Real Image Editing with Diffusion Models, October 2022. URL http::/arxixt org/ btg/abs/2210.09// 09276 arXiv:2210.0927: [es].
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alexander Cong Li, Ellis Langham Brown, Alexei A. Efros, and Deepak Pathak. Internet Explorer: Tar- geted Representation Learning on the Open Web. In Proceedings of the 40th International Conference Machine Learning, pp. 19385 19406. PMLR, July 2023. URL https: httpp:::/ppooeedingg...u wttp:/innng.......pss//////22 press/v22//
1i23c. html ISSN: 2640-3498.
Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Kreis, Sanja Fidler, and Antonio Torralba. Bigdataset- gan: Synthesizing imagenet with pixel-wise annotations. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognation, pp. 21330-21340, 2022.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timoste, and Lue Van Gool. RePaint: Inpainting using Denoising Diffusion Probabilistic Models. In 2022 IEEE/CVE Conference Computer Vision and Pottern Recognition (CVPR), pp. 11451-11461, June 2022. doi: 10.110/CVPR52288.202 2022..
01117. ISSN: 2575-7075
Lorenzo Luzi, Ali Siahkoohi, Paul M. Mayer, Josue Casco-Rodriguez, and Richard Baraniuk. Boomerang: Local sampling on image manifolds using diffusion models, October 2022. URL http//axvvv..or/aab/ abs/
2210. 12100 Siii:22100121000 [cs, stat].
Giovanni Mariani, Florian Scheidegger, Roxana Istrate, Costas Bekas, and Cristiano Malossi. Bagan: Data augmentation with balancing gan. arXiv preprint
2018.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun Zhu, and Stefano Frmon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference Learning Representations, 2021.
George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11):39-41 1995.
Saman Motamed, Patrik Rogalla, and Farzad Khalvati. Data augmentation using generative adversarial networks (gans) for gan-based detection of pneumonia and covid-19 in chest x-ray images. Informatics in Medicine Unlocked, 27:100779, 2021.
Kai Packhäuser, Lukas Folle, Florian Thamm, and Andreas Maier. Generation of Anonymous Chest Radio- graphs Using Latent Diffusion Models for Training Thoracic Abnormality Classification Systems, Novem- ber 2022. URL http://arxxv..rrg/ htsp:/bb//222111 01323 arXiv:2211.01322 (cs, eess).
Jielin Qiu, Yi Zhu, Xingjian Shi, Florian Wenzel, Zhiqiang Tang, Ding Zhao, Bo Li, and Mu Li. Are multimodal models robust image and text perturbations? arXiv preprint orXiv:2212.0080 08044, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Nya Sutskever. Learning Transferable Visual Modcls From Natural Language Supervision. arXiv:2103.000 00000 fes], February 2021. arXiv: 2103.00020.
URL httt:///axxiv wrt/absg////....t..ooooo..bb...s abs
http:/abs21003.000000 2tt///2110000000
Vikram Ramaswany, Sunnie SY Kim, and Olga Russakovsky. Fair attribute classification through latent space de- diabiisin In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 9301-9310, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents, April 2022. URL http://aaaxiv.oorraab///20.00..0 arXiv:2204.06125 [cs].
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
17

---

Published in Transactions on Machine Learning Research (11/2023)
Aniket Roy, Anshul Shah, Ketul Shah, Anirban Roy, and Rama Chellappa. DiffAlign Few-shot learning using diffusion based synthesis and alignment, December 2022. hetl.http://arxtte http://aaxxivvoorrr////2/13///1
wtggabs///aa. 224p//54242200 05404
arXiv:2212.05404 [cs].
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- Booth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation, August 2022. URL
www.:/..gggba.bb.ee. ht////bb/2200..... 2208 12242 arXiv:2208. 12242 [cs).
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Compuler Vision (IJCV), 115(3):211 2222,,12,,.11 2015. doi: 1100000/11160015-00////
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palctte: Image-to-Imal Diffusion Models. In ACM SIGGRAPH 2022 Conference Proceedings, SIGGRAPH '22, pp. 1-10, New York, NY, USA, July 2022a. Association for Computing Machinery. ISBN 378-1-4503-9337-- doi: 0111145/522228833507777 URL
8528233 3530757
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic Text-to-lmage Diffusion Models with Deep Language Understanding, May 2022b. URL http:://rrtit http://aavvvv..a/a///// hrggabb//2255 11487 arXiv:2205.1145 [cs].
Mert Bulent Sariyildiz, Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learning(s) from a synthetic ImageNet clone, December 2022. URL arXiv:2212.08420 [cs].
www.orgggg///.....////e// 08420,
Christoph Schuhmann, Romain Beaumont, Richard Veneu, Cade W Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Sri- vatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty- sizth Conference on Newral Information Processing Systems Datasets and Benchmarks Track, 2022. URL
https://openreviae wwt.://wwww....././///////////o.
Jordan Shipard, Arnold Wiliem, Kien Nguyen Thanh, Wei Xiang, and Clinton Fookes. Diversity Definitely Needed: Improving Model-Agnostie Zero-shot Classification via Stable Diffusion, April 2023. URL http: Slaxxiv.org/abs/230000 032998 arXiv:2302.03390 [es].
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. PMLR, 2015.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion Art Digital Forgery? Investigating Data Replication in Diffusion Models, December 2022. URL http:
//axxiv wttgg/abs/222..0. 03860 arX1v:2212.03360 [cs).
Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective Data Augmentation 07944 arXiv:2302.07944 [es].
With Diffusion Models, February 2023. URL
Florian Wenzel, Jasper Snock, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robust- ness and uncertainty quantification. In Advances in Neural Information Processing Systems, 2020.
Florian Wenzel, Andrea Dittadi, Peter Vincent Gehler, Carl-Johann Simon-Gabriel, Max Horn, Dominik Zictlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, Bernhard Schölkopf, and Francesco Locatello. Assaying Oft-OOf-Dstriiio Generalization in Transfer Learning, October 2022a. URL RRLttpp: //axxiv.. //aaxxiv.rg//a///2200 wwtp::/gg//bb/..ooo/o/// 09239 arXiv:2207.00239 [cs, stat).
18

---

Published in Transactions on Machine Learning Research (11/2023)
Florian Wenzel, Andrea Dittadi, Peter Vincent Gehler, Carl-Johann Simon-Gabriel, Max Horn, Dominik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, et al. Assaying out-of-distribution generalization in transfer learning. arxiv preprint vrXiu:2277.99233 2022b.
Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga. Russakovsky. Towards fairer datasets: Filtering and balancing the distribution of the people subtrec in the imagenct hicrarchy. In Proceedings of the 2020 conference fairness, accounlabilily, and Iransparency, pp. 547-558, 2020.
Yisong Yue and Carlos Guestrin. Lincar submodular bandits and their application diversified retricval. In Advances in Neural Informalion Processing Systems, 2011. URL httpps://procedingg. htt.://cceddinng....iiiis...... nwwrips..ccc.
hptp:///111h1///a000007777777722 pddf
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceedings of the IEEE/CVE international conference on compuler vision, pp. 6023 6032, 2019.
Christoph Zauner. Implementation and benchmarking of perceptual image hash functions. 2010.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Danicl Keysers, Alexander Kolesnikov, and Lucas Beyer. LiT: Zero-Shot Transfer with Locked-inage text Tuning, In 2022 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp.
New Orleans, LA, USA, June URL https://ieeexplore.
2022. LEEE. ISBN 978-1-66546-946-3, doi: 10.1109/CC BLL109/CVPR5668.22222282 leee wtt.:r//rcccmeee////8//588
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:17100 1eto::991009 2017.
Yifan Zhang, Daquan Zhou, Bryan Hooi, Kai Wang, and Jiashi Feng. Expanding Small-Seale Datasets with Guided Imagination, November 2022. URL http://arxiv org/ abaaaaaaa//// 2211 13976 arXiv:2211.1300 [cs].
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEFF transactions on pattern analysis and machine intelligence, 40(6):
1452-1464, 2017.
Dominik Zietlow, Michael Lohaus, Guha Balakrishnan, Matthäus Kleindessner, Francesco Locatello, Bern- hard Schölkopf, and Chris Russell. Leveling down in computer vision: Pareto inefficiencies in fair deep classifiers. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pallern Recognilion, pp- 10410-10421, 2022.
19


Reviews:
[{"review_id": "UrftihQGk1", "reviewer": "Reviewer_JdHi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- Tackles an actively studied problem in generative AI: Using synthetic data for training downstream models. Growing interest in this field.\n- A through comparison of 10 methods is presented for generating synthetic data for downstream classification tasks on 10% ImageNet and Caltech256.\n- An alternative method is proposed, which is based on retrieving training data for training the downstream classifiers.\n- The propsed method achieves superior performance compared to the baselines.\n\n### Weaknesses\n\n- The efficacy of generating data directly for downstream classification tasks has been nicely studied by [A, B].\nThese papers are cited in the related work section in a very shallow way without positioning the current submission with respect to those works, moreover, they are not considered in the model taxonomy and experiments.\nI would like to the authors to provide an in-depth discussion on the comparison against these works, and a comparison against these works in the experiments if possible.\n\n- As I mentioned below, [B] already claims that it is possible to fine-tune a diffusion model to provide synthetic data with improved utility.\nI wonder if all the \"personalization\" techniques evaluated in the experiments are trained under sufficient conditions.\nThis is especially important because in Figure-4, images generated by different fine-tuned diffusion models look similar.\nImageNet is collected almost two decades ago. Perhaps the domain gap between now and then can be visible for images of certain classes.\nIt would be nice to show how the output of pre-trained vs. fine-tuned Stable Diffusion change in that regard.\n\n- Following my previous point, I believe that the bold claims made in this paper should be revised clearly mentioning the type and quantity of data used in the experiments.\nCurrently, the experiments are performed on low-data regime (i.e., 10% of ImageNet and Caltech256) using the same number of additional generated or retrieved images.\nWhy low-data regime? Why 10%? Would the outcomes still be consistent with more data? How about the variance of the results (i.e., variance due to picking the particular 10% of ImageNet).\nHow much generated data would be needed to reach the performance of the proposed retrieval scenario?\n\n- The proposed retrieval method requires the whole training set of Stable Diffusion.\nI would expect this process to be computally challenging.\nSo it would be nice to compare compute requirements for all the groups of methods.\n\n[A] He et al. \"Is synthetic data from generative models ready for image recognition?\", ICLR 2023\nShows that with proper use of prompting, zero-shot and few-shot performance on downstream tasks can be improved using synthetic data from GLIDE.\nIt is reasonable to expect that the outcomes from this work would translate here, where Stable Diffusion (a better text-to-image model) is used.\n\n[B] Azizi et al. \"Synthetic Data from Diffusion Models Improves ImageNet Classification\", CVPR 2023\nDemonstrates that, if fine-tuned properly, diffusion models can be used to generate data for downstream classification tasks (ImageNet).\nCombining such synthetic data with real ones improve performance of the classifier trained solely using real data.\n\n### Some minor points\n\nOverall, I find the quality of text not so great.\nPlease see below\n- The conceptual comparison between GANs and diffusion models in introduction is misleading.\nGANs can condition on text too.\nThere the comparison is mainly between image generative vs text-to-image generative models, and the text should be revised accordingly.\n- The first sentence of Section-3 does not read well.\n- Figure-1 should be improved by mentioning what is measured in the figure.\n- Taxonomy of methods in Figure-2B is not clear, and rather confusing. I don't have a constructive feedback for this, sorry.\n- It would be much better to place Figure-3 and Figure-4 before the experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores the efficacy of diffusion models in generating training data for downstream classification tasks on 10% of ImageNet and Caltech256.\nA number of different techniques for generating data by diffusion models is compared.\nThese methods include simple prompting of frozen Stable Diffusion as well as fine-tuning it (i.e., personalizing it) for downstream tasks.\nMoreover, a retrieval method is introduced where instead of training a classifier for the downstream task using synthetic data, a set of images retrieved from the training set of Stable Diffusion (LAION-5B), to be used for training the classifier.\nExperiments reveal that while fine-tuning Stable Diffusion is better than keeping it frozen, simpler retrieval methods using the model's pre-training data are more effective.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Addressing the weaknesses I listed above would strengthen the work in my view.\nHowever, some of the changes (i.e., revising the claims, or extending the experiments in accordence with the issues I mentioned) might require work that is beyond the two-weeks discussion period or might require evaluating the submission from scratch again.", "broader_impact_concerns": "It is significantly important to openly discuss and acknowledge potential data- and model-related biases in scenarios involving generating synthetic data to tackle specific tasks. Although the proposed method does not directly synthetise images, it utilizes uncurated web-data, which raises similar risks. I suggest the authors to look at recent works in this field like [C], to develop a perspective, and add a Broader Impact Statement accordingly.\n\n[C] Sariyildiz et al. \"Fake it till you make it: Learning transferable representations from synthetic ImageNet clones\", CVPR 2023", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UNuZq1rsCJ", "reviewer": "Reviewer_ixUD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n- The paper is well written and easy to follow. \n- The experiments show relevant baselines (e.g. 20% of ImageNet)\n- The experiment design is simple but I liked the straightforward approach of testing different augmentation strategies.\n- The paper provides an interesting baseline for future evaluation of data augmentation methods.\n\n## Weaknesses:\n\n- The paper shows that the nearest neighbor retrieval baseline outperforms other diffusion methods.\nHowever, I think the usage of CLIP embeddings can be a major limitation hidden in the design of the experiments. CLIP was trained on pairs of images and their textual description crawled from the internet. Obviously, these embeddings are mostly trained on natural images. I would therefore question that the approach would work on more specialized datasets, such as medical images. However, \nspecialized dataset are exactly the ones where training would benefit the most from data augmentation.\nWhile medical images are shortly mentioned in the limitation sections, this issue should be discussed in more detail in the paper.\n\n\n- The description of the used data is unclear. \"We resample the additional data into 5 sets containing the same number of samples per class as our target dataset and train a ResNet-50 classifier (He et al., 2016) on each to derive variance estimates.\" Is the training on an generated clone of the original dataset or on the original dataset with additional samples? Please clarify this! The term \"augmentation\" would suggest that the original dataset is used. However, as I understand it, this is not the case?\n\n\n## Questions\n\n- In section 3, paragraph \"Data augmentation\" it is stated that for ImageNet 390 samples per class are sampled. However, in section 3.2 paragraph \"nearest neighbor retrieval\" it is stated that\n130 images per class for ImageNet were retrieved. Please clarify these mismatching numbers.\n\n- How long did the training take on a the NVIDIA T4 and V100 GPUs? (Useful to estimate the cost for reproducing the experiments)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper benchmarks diffusion models for data augmentation. In particular, it compares the performance achieved using different data augmentation strategies, including different prompting strategies and nearest neighbor retrieval. The paper finds that a suggested baseline using nearest neighbor retrieval on CLIP embeddings outperforms other methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Minor: I would recommend to merge the sections 3.1 and 3.2 and explain the details of each method right after it was introduced. For example, currently the \"Laion nearest neighbor retrieval\" is lacking detail and is hard to understand in section 3.1.\n\n- Please clarify my questions in the \"Weaknesses\" paragraph", "broader_impact_concerns": "I do not see any need to have a broader impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jJiBzBlde1", "reviewer": "Reviewer_Kp2i", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) Given the advances in synthetic data generation using diffusion models, this is a timely contribution to identifying effective strategies for data augmentation using large-scale pre-trained diffusion models like StableDiffusion.\n\n2) I found the overall experimental setup clear. The discussion in Section 4 is overall interesting (especially the discussion of different baselines for generating synthetic data).\n\nWeaknesses:\nSee Requested Changes below", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary: The paper presents a study of different diffusion model-based methods to identify effective strategies for data augmentation. The authors evaluate all the baselines in the context of classification accuracy on the ImageNet benchmark. Surprisingly, the authors find that a simple image-retrieval-based baseline outperforms more sophisticated diffusion model-based augmentation strategies on classification performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested Changes/Clarifications:\n\n1) Is it possible for the LAION-5B dataset to overlap with ImageNet? If yes, one potential concern is that the simple image-retrieval-based baseline might be using data from the ImageNet training set, in which case I would expect the performance of this baseline to be better than a dataset augmented with synthetic samples generated from a diffusion model. It would be nice if the authors could elaborate on this aspect in the paper when mentioning the retrieval-based baseline in Section 3.\n\n2) I think the difference between the retrieval baseline and the diffusion model baselines for both full ImageNet and Caltech datasets (as reported in Section 4.5) is insignificant (or marginal at best). In that sense, does this mean that the retrieval-based approach scales worse (in terms of dataset size) than the diffusion model-based augmentation methods and might be primarily valuable for augmentation in small datasets (which is also a valuable contribution)? If yes, I think it would be better to combine the sub-sections: `Retrieval has better-scaling behavior` and `Retrieval works on full data sets` into one sub-section describing the overall scaling behavior. Moreover, this should also be reflected in the claims that the authors presently make which currently indicate that the retrieval baseline outperforms diffusion model-based augmentation by a large margin when that might not be the case (as per the results in Section 4.5 on the full ImageNet dataset).\n\n3) Can the authors include some additional experiments on the 10% ImageNet subset but using a more recent classifier architecture like ViT (smaller variants) in the Appendix (just to rule out the possibility of results being specific to a classifier architecture)\n\n4) Are the diffusion model variants 5 (FT Conditioning) and 6 (FT Cluster Conditioning) proposed in this paper, as I do not see any citations for these baselines?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors made a substantial effort to address concerns raised by reviewers, and as a result reviewers feel that acceptance is warranted. I agree with them.
Decision: Accept (poster)