Effective Data Augmentation With Diffusion Models
Authors: Brandon Trabucco
Kyle Doherty
Max A Gurinas
Ruslan Salakhutdinov
Venue: ICLR 2024 poster (2024)
Keywords: Deep Learning
Primary area: general machine learning (i.e., none of the above)

Abstract: Data augmentation is one of the most prevalent tools in deep learning, underpinning many recent advances, including those from classification, generative models, and representation learning. The standard approach to data augmentation combines simple transformations like rotations and flips to generate new images from existing ones. However, these new images lack diversity along key semantic axes present in the data. Current augmentations cannot alter the high-level semantic attributes, such as animal species present in a scene, to enhance the diversity of data. We address the lack of diversity in data augmentation with image-to-image transformations parameterized by pre-trained text-to-image diffusion models. Our method edits images to change their semantics using an off-the-shelf diffusion model, and generalizes to novel visual concepts from a few labelled examples. We evaluate our approach on few-shot image classification tasks, and on a real-world weed recognition task, and observe an improvement in accuracy in tested domains.

Published as a conference paper at ICLR 2024
EFFECTIVE DATA AUGMENTATION WITH DIFFUSION MODELS
Brandon Trabucco 1 Kyle Doherty 2 Max Gurinas 9 Ruslan Salakhutdinov
Carnegie Mellon University, MPG Ranch, University Of Chicago, Laboratory Schools brandon@btrabuccoo com, rsa 1 ahuucce cmu. edu
ABSTRACT
Data augmentation one of the most prevalent tools deep learning, underpinning many recent advances, including those from classification, generative models, and representation learning. The standard approach to data augmentation combines simple transformations like rotations and flips generate new images from existing ones. However, these new images lack diversity along key semantic axes present the data. Current augmentations cannot alter the high-level semantic attributes, such as animal species present in scene, to enhance the diversity of data. We address the lack of diversity in data augmentation with image-to-image transformations parameterized by pre-trained rext-to-image diffusion models. Our method edits images to change their semantics using an aff-the-shelf diffusion model, and generalizes to novel visual concepts from a few labelled examples. We evaluate our approach on few-shot image classification tasks, and on real-world weed recognition task, and observe an improvement in accuracy in tested domains.
FUOTO
DA-Fusion
Semantically
Augment
Augmented Images For Training Models
Figure 1: Real images (left) are semantically modificd using publicly available off-thc-shelf Stable Diffusion checkpoint. Resulting synthetic images (right) used for training downstream classification models.
INTRODUCTION
An omnipresent lesson in deep learning is the importance of internet-scale data, such as ImageNet (Deng et al.) 2009). JFT (Sun et al., 2017). OpenImages (Kuznetoova et al. 2018), and LAION-5B (Schuhmann et al. 2022). which are driving advances in Foundation Models (Bommasani et al. 2021) for image generation. These models use large deep neural networks (Rombach et al., 2022)) to synthesize photo-realistic images for rich landscape of prompts. The advent of photo-realism in large generative models is driving interest in using synthetic images to augment visual recognition datasets (Azizi et al., 2023). These generative models promise to unlock diverse and large-scale image datasets from just handful of real images without the usual labelling cost.
Standard data augmentations aim diversify images by composing randomly parameterized image transformations Antoniou et al. [2017] Perez and Wang 2017 Shorten and Khoshgoftaar 2019 Zhao al., 2020). Transformations inc including flips and rotations are chosen that respect basic invariances present in the data, such as horizontal reflection symmetry for a coffee mug. Basic image transformations are thoroughly explored the existing data augmentation literature, and produce models that are robust to color and geometry transformations. However, models for recognizing coffee mugs should also sensitive to subtle details of visual appearance like the brand of mug; yet, basic transformations do not produce novel structural elements, textures, changes in perspective. On the other hand, large pretrained generative models have become exceptionally sensitive to subtle visual details, able to generate uniquely designed mugs from single example (Gal et al. 2022
Our key insight is that large pretrained generative models complement the weaknesses of standard data augmentations, while retaining the strengths: universality, controllability, and performance. We propose a flexible data augmentation strategy that generates variations of real images using text-to- image diffusion models FDA--usion). Our method adapts the diffusion model to new domains by

---

Published as a conference paper at ICLR 2024
Baseline Augmentation:
Real Train Image:
DA-Fusion:
Figure 2:
DA-Fusion produces task-relevant augmentations with no prior knowledge about the image content. Given image of train from PASCAL VOC (Everingham et al. 2209). we generate several augmentations using Real Guidance (He al. 2022
(top row), and compare these method (bottom row).
fine-tuning pseudo- prompts in the text encoder representing concepts augment. DA-Fusion modifies the appearance of objects in manner that respects their semantic invariances, such as the design of the grafliti on the truck Figure and and the design of the train in Figure 2 We test our method on few-shot image classification tasks with common and rare concepts, including real-world weed recognition task the diffusion model has not seen before. Using the same seper-parameters in all domains, our method outperforms prior work, improving data augmentation by up to + +10 percentage points. Our ablations illustrate that DA-Fusion produces larger gains for the more fine-grain concepts. Open- code is released at: httpp: /ttp://ggtttbb/ com www.///wwwnnn....oommoommmmon..e rabucco daa fusion
RELATED WORK
Generative models have been the subject of growing interest and rapid advancement. Earlier methods, including VAEs Kingma and Welling (2014) and GANs Goodfellow et al. (2014), showed initial
promise generating realistic images, and scaled up in terms of resolution and sample quality
Brock et al. (2019;; Razavi et al. (2019). Despite the power of these methods, many recent successes
photorealistic image generation
the result of diffusion models Ho et al. (2020); Nichol
and Dhariwal 2021); Saharia et al. (0222b) Nichol et al. (2022) Ramesh et al. (2022). Diffusion models have been shown to generate higher-quality samples compared their GAN counterparts
Dhariwal and Nichol (20TT), and developments like classifier free guidance Ho and Salimans (2022)
have made tex-ato-mmae generation possible. Recent emphasis has been on training these models
with internet-scale datasets like LAION-5B [Schuhmann al. (2022). Generative models trained
at internet-scale Rombach et al. (202)); Saharia et al. (2022b); Nichol al. (2022); Ramesh et al.)
(2022) have unlocked several application areas where photorealistic generation is crucial.
Image Editing Diffusion models have popularized image-editing. Inpainting with diffusion is one such approach that allows the user to specify what to edit as mask (Saharia et al. (2022a); Lugmayr jet al. (2022). Other works avoid masks and modify the attention weights of the diffusion process that generated the image instead Hertz et al. (2022); Mokady et al. (2022). Perhaps the most relevant technique to our work SDEdit Meng et al. 2022). where real images are inserted partway through the reverse diffusion process. SDEdit is applied by He et al. (002)) to generate synthetic data for training classifiers, but our analysis differs from theirs in that we study generalization new concepts the diffusion model wasn't 't trained optimize pseudo-prome promppromppppp (Li and Liang 2021 Gal et al. 2022) for each concept. Our strategy is more appealing than fine-tuning the whole model in Azizi al. (2023) since it works from just one example per concept (Azizi et al. (2023) require millions of images), and doesn't 't disturb the model's ability generate other concepts. Our fine-tuning strategy improves the quality of augmentations for common concepts Stable Diffusion has seen, and for fine-grain concepts that are less common.
To instruct the diffusion model what to augment,
Synthetic Data Training neural networks synthetic data from generative models popularized using GANs/Antonin et al.] (2017); Tran et al. (2017); Zheng et al. (2017). Various applications for synthetic data generated from GANs have been studied, including representation learning Jahanian
2

---

Published as a conference paper at ICLR 2024
al. 2022), inverse graphics Zhang et al. (2021a), semantic segmentation Zhang al. (2021b), lb. and
training classifiers Tanaka and Aranha (2019); ; Dat et al. (2019): Yamaguchi et al. (2020); Besnier al. (2020); Xiong al. (2020); Wickramaratne and Mahmud
(2021) p. More recently,
synthetic data from diffusion models has also been studied
few-shot setting He et al. (022)).
These works use generative models that have likely seen images of target classes and, to the best of our knowledge, we present the first analysis for synthetic data on previously unseen concepts.
3 BACKGROUND
Diffusion models Sohl-Dicksteil et al. (2015): Ho et al. 22000); Nichol and Dhariwal (021); [ ct al. (202I); Rombach al. (2022) are sequential latent variable models inspired by thermodynani diffusion Sohl-Dickstein et al. (2015). They generate samples via Markov chain with learned
Gaussian transitions starting from an initial noise distribution p(rr) ((rr;
T po (xo.r) = p((T)) II pa (
Transitions (=(x(11/2)) are designed to gradually reduce variance according to a schedule ........ BT so the final sample xo represents sample from the true distribution. Transitions often parameter- ized by fixed covariance Σ, = I and a learned mean (1 t) defined below.
1
3.
No (= t)
- (x,,, az
va
This parameterization choice results from deriving the optimal reverse process/Hoed et al. (2020), where <o(-) is neural network trained process noisy sample v; and predict added noise. Given real samples 2h and noise ~ ~O, I), one can derive I4 at an arbitrary timestep below.
xz (x00 e) Vārz VI ÖrE
Ho et al. (2020) define = -1 Br and 0₂ = IT ==1 These components allow training and sampling from the type of diffusion model backbone in this work. We use a pretrained Stable Diffusion model trained by |Rombach et al. (2022). Among other differences, this model includes text encoder that enables text-to-image generation (refer to Appendix for model details).
DATA AUGMENTATION WITH DIFFUSION MODELS
In this work we develop a flexible data augmentation strategy using text-to-image diffusion models. In doing we consider three desiderata: Our method is 1) universal: it produces high-fidelity augmentations for new and fine-grain concepts, not just the ones the diffusion model was trained on; 2) controllable: the content, extent, and randomness of the augmentation are simple to control and straightforward to tune; 3) performant: gains in accuracy justify the additional computational cost of generating images from Stable Diffusion. We discuss these in the following sections.
4.1 A UNIVERSAL. GENERATIVE DATA AUGMENTATION
Standard data augmentations apply to all images regardless of class and content Perez and Wang (0017). We aim to capture this flexibility with our cestusion-bbsed augmentation. This is challenging because real images may contain elements the diffusion model is not able to generate out-of-the- -ox- How do we generate plausible augmentations for such images? Shown in Figure 3 we adapt the diffusion model model, and fine-tuning only these embeddings to maximize the likelihood of generating new concepts.
concepts by inserting new embeddings in the text encoder of the generative
Adapting Generative Model the specified class name He et (2022). However, this is not possible for concepts that lie outside the vocabulary of the generative model because the model's 's text encoder has not learned words to describe these concepts. We discuss this problem in Section with our contributed reco-necoonntion task, which our pretrained diffusion model is unable generate when the class name is provided. A simple solution to this problem is to have the model's text encoder learn new words to describe new concepts. Textual Inversion (Gal al. (022) is well-suited for this, and we use it to learn a word embedding Wi from handful of labelled images for each class in the dataset.
When generating synthetic images. previous work uses prompt with
E [[Ic- col E, + v1 āge, t, photo of a ui, "]]]]
min Wo
3

---

Published as a conference paper at ICLR 2024
Image- To-Image Diffusion
Aighnot
Clas Labeh
Angreci
Dotiset
Dataset
XMM
XM
Class -
Sundanl Augmentatoon
DA-Fusion
Textual Inversion
Text Transformc
Real wis
Xc
Inown
Emboldiing cokkp
Pionplee
Syrahelo Eureplee
Fine-Tune Embeddings
Visual Concepts
"a photo of Class3
Train Downstream Model
Figure 3: How our data augmentation works. Given a dataset of images and their class labels, we generate M augmented versions of each real image using an image editing technique and pretrained Stable Diffusion checkpoint. Synthetic images mixed with real data when training downstream models.
image-editing
We initialize each new embedding Wi to class-agnostic value (see Appendix G). and optimize them to minimize the simplified loss function proposed by Ho et al. (2020). Figure shows how new embeddings Wi are inserted in the prompt given image of train. Our method is modular, and as other mechanisms are studied for adapting diffusion models, Textual Inversion can easily be swapped out with one of these, and the quality of the augmentations from DA-Fusion can be improved.
Generating Synthetic Images Many of the existing approaches generate synthetic images from scratch Antoniou et al. ( Taanka and Aranha (2019) : Besnier et al. (2020) Zhang et al. (202lba)) This is particularly challenging for concepts the diffusion model hasn't 't seen before. Rather than generate from scratch, we use real images as a guide. We splice real images into the generation process of the diffusion model following prior work in SDEdit Meng et al.) (2022). Given reverse diffusion process with S steps, we insert real image xf with noise ~ N (0. 7) timestep [Sto]. [0, 1] is hyperparameter controlling the insertion position of the image.
where to
T[Stol =V VoLsto 20
ref
[Sto]
We proceed with reverse diffusion starting from the spliced image at timestep [Sto]] and iterating Equation
2 until sample is generated timestep O. Generation is guided with a prompt that includes the new embedding Vे for the class of the source image (see Appendix G for prompt details).
4.2 CONTROLLING AUGMENTATION
Balancing Real & Synthetic Data Training models synthetic images often risks emphasizing spurious qualities and biases resulting from an imperfect generative model Antoniou al. (2017). The common solution assigns different sampling probabilities to real and synthetic images to manage imbalance He et al. (2022). We adopt a similar method for balancing real and synthetic data in Equation where denotes the probability that a synthetic image present the 1-th location in the minibatch of images B.
i~ U(11, NY). ~ u(fI,...,,
(6)
Bi41 - B,U,X, w.p. (1- else X
(7)
Here X E RxxxXXX3 denotes a dataset of N real images, and E specifies the index of particular image X,. For each image, we generate M augmentations, resulting in synthetic dataset X E R Xx with N M image augmentations, where X;, E HR enumerates the jth augmentation for the ith image the dataset. Indices and j are sampled uniformly from the available N real images and their M augmented versions respectively. Given indices ij, with probability (1 - ax) a real image image X; added to the batch B, otherwise its augmented image X,, is added. dlyper-paraede
details are presented in Appendix G and we find a = 0.5 to work effectively in all domains tested, which equally balances real and synthetic images.
4

---

Published as a conference paper at ICLR 2024
Improving Diversity By Randomizing Intensity Having appropriately balanced real and synthetic images, our goal is to maximize diversity. This goal is shared with standard data augmentation Perez and Wang (2017); : Shhorten and Khoshgoftaar (201)), where multiple simple transformations
used, yielding more diverse data. Despite the importance of diversity, generative models typically employ frozen sampling hyperparameters to produce synthetic datasets Antoniou al. (2017); Tanaka and Aranhal (2019); Yamaguchi et al. (2020); Zhang et al. (202 lb a): He et al. (2022). Inspired by the success of randomization in standard data augmentations (such as the angle of rotation), we propose to randomly sample the insertion position to where real images are spliced into Equation [ This can
be interpreted as randomizing the extent images are modified as tu - generations more closely
resemble the guide image.
k...... }), and observe consistent improvement in classification accuracy with k = 4 compared to fixing fo- Though the hyperparameter to is perhaps the most direct translation of randomized intensity to generative model-based data augmentations, there are several alternatives. For example, one may consider the guidance scale parameter used in classifier-free guidance (Ho and Salimans 2022). We leave this as future work.
In Section 6.2 we sample uniformly at random lo ~ U(( 2
$
DATA PREPARATION
Standard Datasets We benchmark our data augmentations on six standard computer vision datasets.
We employ C0 (Fei-Fei et al.) 2004), Flowers 102 (Nilsback and Zisserman 2008). FGVC Aircraft Maji et al. 2013), Stanford Cars (Krause et al. 2013). CocO et al. (2014), and PASCAL VOC Everiigggg et al. (2009). We use the official 2017 traming and validation sets of COCO, and the official 2012 2 training and validation sets of PASCAL VOC. We adapt these datasets into object classification tasks by filtering images that have at least one object segmentation mask. We assign these images labels corresponding to the class of object with largest area in the image, as measured by the pixels contained in the mask. Caltech101, COCO, and PASCAL VOC have common concepts like f "dog" and Flowers 12, FGVC Aircraft, and Stanford Cars have fine-grain concepts like "giant white arum lily" (the specific flower name). Additional details for preparing datasets are in Appendix G
Leafy Spurge We contribute a dataset of top- down drone images of semi-natural areas in the western United States. These data were gathered in an effort to better map the extent of a prob- lematic invasive plant, leafy spurge (Euphorbia esula), that a detriment to natural and agricul- tural ecosystems in temperate regions of North America. Prior work to classify aerial imagery accuracy of 0.75 Yang et al. (2020). To our knowledge, top-down aerial imagery of leafy spurge was not present in the Stable Diffusion training data. Results of CLIP-retrieval [eaumont (2022) returned close-up, side-on images of members of the same genus (Figure 4) in the top 20 results. We observed the first instance of our target species, Euphorbia esula, as a 35th result. The spurge images we contribute are semantically distinct from those in the CLIP corpus, because they capture the plant and landscape context around it from 50m distance above the ground, rather than close-up botanical features. Therefore, this dataset represents a unique opportunity to explore few-shot learning with Stable Diffusion, and developing a robust classifier would directly benefit efforts to restore natural ecosystems. Additional details are in Appendix H
sample from the Spurge dataset (the first on the left), compared with top results of CeiP-retrieval queried on the prompt: "a drone image of Icafy spurge". Closcup images from members of the same genus (sec- ond, and third) are in the top 20 results and a closcup of the same species for the 35th result (fourth). No top- down aerial images of the target plant were revealed.
Figure 4:
of leafy spurge achieved
DA-FUSION IMPROVES FEW-SHOT CLASSIFICATION
Experimental Details We test shw-shot classification on seven datasets with three data augmenta- tion strategies. RandAugment (Cubuk et al. (020) employs no synthetic images, and uses the default hyperparameters in torchvis sin. Real Guidance (He et al. 2022) uses SDEdit on real images with to = 0.5, has a descriptive prompt about the class, and shares hyperparameters with our method to ensure fair evaluation. DA-Fusion is prompted with "a photo of a <w,>" where the embedding for <w;> is initialized to the embedding of the class name and learned according to Section 4.1
Each real image is augmented M times, and a ResNet50 classifier pre-trained on ImageNet is fine- tuned mixture of real and synthetic images sampled as discussed in Section 4.2 We vary the
5

---

Published as a conference paper at ICLR 2024
C1
Plowes112
Airceai
Stanfard
0.00
Accuracy (1A))
Accurasy (1a1)
Accurasy (N5)
ACCORTRS (VAD
0.00
COCO
PASCAL
Lesty Spurge
Overall
15
Accamne (Na))
Accuracy (VaI)
Normaltted Score
Accuracy
2
Examples Per Class
Examples Per Class
Exumatee Per Class
Examples Per Class
RandAAgmennt (Cubuk 3028)
Guilbore (Heet 2022)
DA-Fuisu (Ours)
Figure 5: Few-shot classification performance with full information. DA-Fusion consistently outperforms RandAugment (Cubuk et al. 2020). and Real Guidance (He et al. 2022 with a descriptive prompt. In fine-grain domains such Flowers I02, which represents classification of flowers into subelasses like "giant white lily." Real Guidance performs no better than traditional data augmentation. In contrast, DA-Fusion performs consistently well across a variety of domains with common concepts (COCO, PASCAL VOC, Caltech101) rare concepts (Flowers102. FGVC Aircraft, Stanford Cars) and novel concepts to Stable Diffusion (Leafy Spurge).
number of examples per class used for training the classifier on the x-axis in the following plots, and fine-tune the final linear layer of the classifier for 10, 000 steps with a batch size of 32 and the Adam optimizer with learning rate 0.0001. We record validation metrics every 200 steps and report the epoch with highest accuracy. Solid lines in plots represent means, and error bars denote 68% confidence intervals over 4 independent trials. An overall score is calculated for all datasets after (d) normalizing performance using y,"" (d) + (v) (d) / / (Umax Umin ) where d represents the dataset, Vmnx is the maximum performance for any trial of any method, and ymin is defined similarly.
Results in Figure [ show DA-Fusion improves accuracy in every domain, often by significant margin when there are few real images per class. We observe gains between +5 and +15 accuracy points in all seven domains compared standard data augmentation, Our results show how generative data augmentation can significantly outperform color and ceometry-based transformations like those in RandAugment (Cubuk et al. 2020). Despite using a powerful generative model with a descriptive prompt, Real Guidance He et al. (2022) performs inconsistently, and in several domains fails to beat RandAugment. To understand this behavior, we binned the results by whether dataset contains common concepts (COCO, PASCAL VOC, Caltech101), fine-grain concepts (Flowers 102 FGVC Aircraft, Stanford Cars), or completely new concepts (Leafy Spurge), and visualized the normalized scores for the three data augmentation methods in Figure 6
Interpreting Results
Concepts
Vine-Girain Concopts
Concepts
6.43
030
8.55 Narmaliced Score
Normalized Score
Normalized Score
RandAugment (Cubuk et al, 2028)
Reil Guidunce (Be et al. 2022)
DA-Fuseon (Ours)
Figure 6: Performance stratified by concept novelty. Real Guidance (Hc et al. 2022 uses descriptive prompt to instruct Stable Diffusion what to augment, and works for common concepts Stable Diffusion was trained on. However, for harder-to-describe concepts, this strategy fails. DA-Fusion works well at all novelty levels, improving by 12.8% for common concepts, 24.2% for fine-grain concepts, and 20.8% for unseen concepts.
Class Novelty Hinders Real Guidance rreveass a systematic failure mode in Real Guidance (He et al., 2022) for novel and fine-grain concepts. These concepts are harder to describe in prompt than common ones consider the prompts "a top-down drone image of leafy spurge taken from 100ft in the air above grassy field" versus "a photo of cat." DA-Fusion mitigates this by optimizing escudo-prompts. formatted as "a photo of a <w,>", " that instruct the diffusion model on what to generate, and has the added benefit of requiring no prompt engineering. Our method works well at all

---

Published as a conference paper at ICLR 2024
Intermet Data
"car"
Diffusion
Prompt: "A photo of car"
Model
Erase: "airplanc"
Synthetic
Prompt: "A photo of an airplane"
Leakage
Krase Concepts Prevent
Figure 7: Leakage of internet data to downstream models. Large generative models trained at internet scale may produce synthetic similar their training data tested on common concepts (right). show result of erasing common concepts by fine-tuning the attention layer weights Stable Diffusion's UNet (right).
levels of concept novelty, and produces larger gains the more fine-grain concepts are, improving by 12.8% for common concepts, 24.2% for fine-grain concepts, and 20.8% for novel concepts.
PREVENTING LEAKAGE INTERNET DATA
Previous work utilizing large pretrained generative models produce synthetic data (He et al. 2022 has left an important question unanswered: are we sure they are working for the right reason?
Models trained on internet data have likely seen many examples of classes in common benchmarking datasets like ImageNet Deng et al. (2009). Moreover, Carlini et al. (2023) have recently shown that pretrained diffusion models can leak their training data. Leakage of internet data, as in Figure 7 risks compromising evaluation. Suppose our goal is to test how images from diffusion models improve few-shot classification with only few real images, but leakage of internet data gives our classifier access to thousands of real images. Performance gains observed may not reflect the quality of the data augmentation methodology itself, and may lead to drawing the wrong conclusions.
We explore two methods for preventing leakage of Stable Diffusion's 's training data. We first consider model-centric approach that prevents leakage by editing the model weights remove class knowledge. We also consider data-centric approach that hides class information from the model inputs.
Model-Centric Leakage Prevention Our goal with this approach to remove knowledge about concepts in our benchmarking datasets from the weights of Stable Diffusion. We accomplish this by fine-tuning Stable Diffusion in order to remove the ability to generate concepts from our benchmarking datasets. Given list of class names in these datasets,
utilize recent method developed by Gandikota et al. (2023) that fine-tunes the UNet backbone of Stable Diffusion so that concepts specified by a given prompt can no longer be generated (we use class names as such prompts). In particular, the UNet is fine-tuned to minimize the following loss function.
min E [[e,( "class name") ) Eg- (re, 17(Eg. (xz, t, "class name") Eg. (Ie, t))
(8)
Where "class name" is replaced with the actual class name the concept being erased, represents the parameters of the UNet being fine-tuned, and 0* represents the initial parameters of the UNet. This procedure, named ESD by Gandikota et al. (2023). can be interpreted as guiding generation in the opposite direction of classifier free- rre-gguidacee, and erase variety of types of concepts.
Data-Centric Leakage Prevention While editing the model directly to remove knowledge about classes strong defense against possible leakage, it also costly. our experiments, erasing single class from Stable Diffusion takes two hours on single 32GB V100 GPU. As an alternative for situations where the cost model-centric defense by removing all mentions of the class name from the inputs of the model. In practice, switching from a prompt that has the class name to new prompt omitting the class name is sufficient. Section goes into detail how to implement this defense for different types of models.
high, we can achieve weaker defense
Results With Model-Centric Leakage Prevention Figure g shows results when erasing class knowledge from Stable Diffusion weights. We observe a consistent improvement in validation accuracy by as much as +5 percentage points on the Pascal and COCO domains when compared to the standard data augmentation baseline. DA-Fusion exceeds performance of Real Guidance He
7

---

Published as a conference paper at ICLR 2024
COCO
PASCAL
Lealy Sparge
Overaal
Acceracy (V3))
Acuurayy (VA4)
Normatteed Store
Acceraty 82
Examples Per Class
Examaples Per Class
Exumatees Per Class
Examples Per Class
Basdinc
Guidance
DA-Fusica
Figure 8: Few-shot classification performance with model-centric leakage prevention. We evaluate DA-Fusion on three classification datasets and outperform classic data augmentation and a competitive method from recent literature. We report validation accuracy as function of the number of images in the observed training set in the top row, and the average accuracy over the interval (the number of examples per class) in the bottom row.
COCO
PASCAL VOC
Leafy Sporge
Overall
Accorary (Na)
Acceraa (NAD)
Nacrtaal Score
Examples Per Class
Examples Per Class
Eexmples Per Class
Examples Per Class
Basdinu
Real Guidaace (He et 2022)
DA-Fusiou (Ours)
Figure 9: Few-shot classification performance with data-centric Icakage prevention. DA-Fusion unilaterally outperforms the standard data augmentation baseline, and competitive method from recent literature. Gains with DA-Fusion are larger with this weaker defense against training data leakage compared to the stronger model-centric defense. We discuss how to interpret this larger improvement in Section 6
et al. 2022) overall while utilizing the same hyperparameters, without any prior information about the classes in these datasets. In this setting, Real Guidance performs comparably to the baseline, which suggests that gains in Real Guidance may stem from information provided by the class name. This experiment shows DA-Fusion improves few-shot learning and suggests our method generalizes to concepts Stable Diffusion wasn't trained on. To understand how these gains translate to weaker defenses against training data leakage, we next evaluate our method using a data-centric strategy.
Results With Data-Centric Leakage Prevention Figure 9 shows results when class information is hidden from Stable Diffusson inputs. As before, we observe consistent improvement validation accuracy, by as much as +10 percentage points on the Pascal and COCO domains when compared to the standard data augmentation baseline. DA-Fusion exceeds performance of Real Guidance He et al. (2022) in all domains while utilizing the same hyperparameters, without specifying the class name as an input to the model. With weaker defense against training data leakage, we observe larger gains with DA-Fusion. This suggests gains are due in part to accessing Stable Diffusion's 's prior knowledge about classes, and highlights the need for strong leakage prevention mechanism when evaluating synthetic data from large generative models. We next ablate our method to understand where these gains come from, and how important each part of the method is.
6.2
How IMPORTANT ARE RANDOMIZED INTENSITIES?
Our goal this section understand what fraction of gains are due to randomizing the intensity of
our augmentation based on Section 4.2 We employ the same experimental settings as in Section 6
using data-centric leakage prevention, and run our method using fixed insertion position to
(labelled k = in Figure 10). following the settings used with Real Guidance. In Figure 10 we
report the improvement in average classification accuracy on the validation set versus standard data augmentation. These results show that both versions of our method outperform the baseline, and
randomization improves our method in all domains, leading to an overall improvement of 51%.
6.3 DA-FUSION IS ROBUST To DATA BALANCE
We next conduct an ablation to understand the sensitivity of our method to the balance of real and synthetic data, controlled by two hyperparameters: the number of synthetic images per real image M E N, and the probability of sampling synthetic images during training a E [0, 1). We use a = 0.5 and M 10 throughout the paper. Insensitivity to the particular value of a and M is desireable
8

---

Published as a conference paper at ICLR 2024
Pascal
COCO
Sparge
Overall
Acorrary Calleed (1a))
0003
Acourccy Galaed (Val)
4065 1000 8000
Galsed (Nai)
Nemmllied
0015
PAsPosio (k
PA-Pusten (k
Figure 10: Ablation for randomizing augmentation intensity. We vary the number of intensities from (A 4) the main experiments, to deterministic insertion position tu 0.5. We report the improvement average few-shot classification accuracy over standard data augmentation, and observe consistent improvement.
Pascal
Pascel
Accuracy (1ai)
Accuracy (Y54)
Accurscy (Vab
Ascuracc (Yal)
0.4
0.70
Examples Clas
Examples
DA-Fusion PA-Pusten to 0.5)
Guidance Real Guldence to 0.3)
DA-Fuiio
Real Guidance
DA-Fmion (24 5)
Real Guidance (1d-5 5) Real Guidance (M 21)
DA-Fuudon 0.7)
Real Guidance 8.7)
DA-Fuion (M-
Figure 11: Ablation for data balance sensitivity. We our method with a (0.3, 0.7) and M (5, 10, 20) and report the improvement in few-shot classification accuracy over Real Guidance using the same settings. DA-Fusion is robust to the balance of real and synthetic data and outperforms prior work in each setting.
trait because it simplifies hyper-parameter tuning and facilitates our data augmentation working out-of-the-box with no domain-specific tuning. We test sensitivity to and M by comparing runs of DA-Fusion with different assignments Real Guidance with the same a and M. Figure shows stability as a and M varies, and that a 0.7 performs marginally better than a 0.5, which suggests our method improves synthetic image quality because sampling them more often improves accuracy. While M = 20 performs marginally better than M = 10, the added cost of doubling the number generative model calls for a marginal improvement suggests M 10 is sufficient.
7 DISCUSSION
We proposed a flexible method for data augmentation based on diffusion models, DA-Fusion. Our method adapts pretrained diffusion model to semantically modify images and produces high quality augmentations regardless of image content. Our method improves few-shot classification accuracy tested domains, and by up to +10 percentage points on various datasets. Similarly, our method produces gains diffusion model. To understand these gains, we studied how performance is impacted by potential leakage of Stable Diffusion training data. To prevent leakage during evaluation, defenses that target the model and data respectively, each on different sides of a trade-off between defense strength and computational cost. When subject both defenses, DA-Fusion consistently improves few-shot classification accuracy, which highlights its utility for data augmentation.
contributed wreedrecogniitee
dataset that lies outside the vocabulary of the
presented two
There several directions improve the flexibility and performance of method as future work. First, our method does not explicitly control how an image is augmented by the diffusion model. Extending the method with a mechanism to better control how objects in an image are modified, e.g. changing the breed of a cat, could improve the results. Recent work in prompt- based image editing Hertz al. (2022) suggests diffusion models can make localized edits without pixel-level supervision, and would minimally increase the human effort required to use DA-Fusion. This extension would let image attributes be handled independently by DA-Fusion, and certain attributes could be modified more extremely than others. Second, data augmentation is becoming increasingly important in the decision-makee setting Yarats et al. (022).. Maintaining temporal consistency is an important challenge faced when using method in this setting. Solving this challenge could improve the few-shot generalization of policies in complex visual environments. Finally, improvements to our diffusion model backbone that enhance image photo-realism are likely to improve DA-Fusion.

---

Published as a conference paper at ICLR 2024
REFERENCES
Giuseppe Amatulli, Sami Domisch, Mao-Ning Tuanmu, Benoit Parmentier, Ajay Ranipeta, Jeremy Malczyk, and Walter Jetz. suite of global. cross-scale topographic variables for environmental and biodiversity modeling. Scientific data, 5:1 80040, March 2018. ISSN 2052-4463. doi: 10.1038/sdatt.2018...
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks, 2017. /RL/nttpss//rrx...or/////..
04340
Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David Fleet. Syn- thetic data from diffusion models improves imagenet classification. CoRR, abs/2304.08466, 2023. doi: 10.485550aXxx.2234400666 URL http://///.....g//1.111555//0///
2304.0844 66
Romain Beaumont. Clip retrieval. 2022.
Victor Besnier, Himalaya Jain, Andrei Bursuc, Matthieu Cord, and Patrick Pérez. This dataset does not exist: Training models from generated images. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020, pages 1-5. IEEE, 2020. doi: 10. /Ttp://AAASS7777.2222.0.000
1109/CCASSP407 76. 2020. 9053146
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji. Annie Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel F. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and et al. On the opportumities and risks of foundation models. CoRR, abs/2108.07255. 2021. URL https: www.vv.vvvv..///aia//aa//aaaaavv org/ abs 2008 07258
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL httptst://opeeeeeeiiiiiee 00F
Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. CoRR, abs/2301.13 2023, doi: 10.48550/arXiv.2301.. 1311 URL 48550/arXiv 2301. 13188
Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with a reduced search space. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Floring Balcan, and Hsuan- Lin, editors, Advances in Neural Information Pro- cessing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020. URL / htte://85566600 haw.ae//11400000000/////88800000
neurips. cc/ html
paper/2020/hashh
Pham Thanh Dat, Anuvabh Dutt, Denis Pellerin, and Georges Quénot. Classifier training from generative model. In 2019 International Conference on Content- Multimedia Indexing (CBMI), pages 1 1-6, 2019. doi: 10.1109/CBM1.219..0 88774777
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009). 20-25 June 2009, Miami, Florida, USA, pages 248-255. IEEB Computer Society, 2009. doi: 10. (009///VP..2009550666..... URL htt.::tpsss/ii...ggg/////
11 /O/////R.... 2009. 2009.
10

---

Published as a conference paper at ICLR 2024
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 4780-87944 2021. URL /ttpttp::/prroceeeeeeeee ngs. heurgi....erii.. neurips CC/ httpp/ppppr/22222 /
Abst html
Kyle D. Doherty, Marirose P. Kuhlman, Rebecca A. Durham, Philip W. Ramsey, and Daniel L., Mummey. Fine-grained topographic diversity data improve site prioritization outcomes for bees. Ecological Indicators, 132:108315, 2021. ISSN 1470-160X. doi: https://boi.rg//000011///
acolind.2021. 10835, 108315, URL https: www.seceeeeeeeeeeettcooooiieeeee
Mark Everingham, Luc Van Gool, Christopher K. I. Williams. John Winn, and Andrew Zisserman.
The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88: 303-308, September 2009. URL https: wwwpw//wwwsow..cooooo.oo com
mublicattoo/tteepassee
wwwewsssllieeo -ob wow.:auaoeooeetttaaasaeeeeeeeeee clce--l-llss------ccaaa..eeg
Printed version publication date: June 2010.
Li Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 Conference Computer Vision and Pattern Recognition Workshop, pages 178-178, 2004, doi: 10. 10.11.9/.
2004.383.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing texl-to-image generation using textual inversion, 2022. URL http:htttt//tttt /ttp:p/////aiioooor/r/////////// 2208. 01618
Rohit Gandikota, Joanna Materzynska,
Jaden Jaden from diffuusion models. CoRR, abs/2303.003345. 2023, doi: N0448500aaXx..223307334. URL
and David Bau. Erasing concepts
Ian Goodfellow, Jean Pougct-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. /
Ayaan Haque. EC-GAN: low-sample classification using sumi-supervised algorithms and gans (student abstract). In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty- Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 15797-15798. AAAI Press, 2021. URL
aaai .ww.aaiioor//ndeee/eeeeeee index
php/ AAAI
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition?, 2022. URL https
//arxiv. hww.:/org/aaa 2210.0222000 07574
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt- to-prompt image editing with cross attention control, 2022. URL stttps://axxiv.rgg/ 2208. 01626
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. URL httht://p//xxi
org/aba/ 2207. 12598
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. / http://ooccediin......eerrr..oaa http:/pppp..ppppppp/////////////
html

---

Published as a conference paper at ICLR 2024
Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models data source for multiview representation learning In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL
ew net/ wow js7dc js7dCL.
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http:
1/a //arxi farxiiv.. www. 1312. 6114
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 2013 IEEE International Conference on Computer Vision Workshops, pages doi: 10.1 19911/CV..2213..77
554-561, 2013.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Tust-, Shahab Kamali, Stefan Popev, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. The open images dataset V4: unified image classification, object detection, and visual relationship detection at scale. /tttppp//axxivoorr//b///////
CoRR, abs/181 1.00982, 2018. URL /attp: wwtp://w.//bbbbb.ee.eoeeee 00982
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNPP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pages 4582-4597, Association for Computational Linguistics, 2021. doi: 1e111165511222111 lol-- URL hwwph:tttt//cttc wtt.:ooo......../1//...o acl-long: 353
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pages 740-755. Springer, 2014. doi: 1007/ 319 10602-1
978-3-319-100 11119--0602-111 16020-1128.8.-------111 L8. URL htttps://doi..e http:////oo..orr1111000////////
48
Andreas Lugmayr, Martin Danelljan, Andrés Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 11451-11461 IEEE, 2022. doi: 10.1109/CVPR52688 2022222688202220111 0111 URL https //doi oro.iooog/100. http:/1119////2... 2022. 01117
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, net htt.:/orrmmmiaacaeceeeeeeeeeeeee 2022. OpenReview.net, 2022. URL httlhttts////peeeeee/ ott.:///wee.....eeoooooooocccaaa
E
Ron Mokady, Amir Hertz, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models, 2022. URL jatps:://azxvv.o/g/ abs//
2211.00 09794
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 24 July 2021, Virtual Event. volume 139 of Proceedings of Machine Learning Research, pages 8162 8171. PMLR, 2021. URL http: http::::/pocceednngsss
mlr. mtr.press/vl39/nm cho 21a hhtll
12

---

Published as a conference paper at ICLR 2024
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image genera- tion and editing with text-guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference Machine Learning, ICML. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 16784-16804. PMLR, 2022. URL ht attps::/proceecinngg mir ptw.eiigg.........s.....eeeeeeee press/ seress/vieee https////////nnooeeeeeeaaa html
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning, 2017. URL /RLL:tttss//rrrav.or////aa//
04621
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022. UtLtttts:://axxtt ftttt:::/aariv.orpgp www.iii.g..o...oaae.oooooooooett abs /2004 06125
Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity im- ages with VQ-VAE-2. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence d' d'Alché-Bus Buc, Emily B. Fox, and Roman Garnett, editors, Advances Neural In- formation Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages nwwrips......... cccpaper/201 9/hash/ htmml
14837-14847, 2019. URL (ww22fa1718ddbbbccc wttpo//cccccccccccaaaaaooeeeeee.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IFEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA. USA, June 18-24, 2022, pages 10674-10685, IEEE, 2022. doi: 10. ALL.L//CVPR2222.222222000 10.1109/CVPR522 688 2022. 01042
URL https: /doi.
Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In Munkhtsetseg Nandigjav, Niloy J. Mitra, and Aaron Hertzmann, editors, SIGGRAPH '22: Special Interest Group on Computer Graphics and Interactive Techniques Conference, Vancouver. BC, Canada, August 7 - 11, 2022, pages 15:1-15:10. ACM, 2022a. doi: 10. 10.1145/32223333307577 URL https: /tt.//oi..oo /ttp://....rg/11111153335333362. 35307757775
/10
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. 2022b. URL
0ess/2205.1144
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022. URL
https::/arxiv. wttpg//bbb////o// 08402
Connor Shorten and Taghi M. Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data 6( x60, Jul 2019, ISSN 2116-1115. doi: 10. 00.11666440337-019-099--00 URL https:///o:: org wttprrg/..11111111 www.1118614400 00144005377-99--0000 -19-0197-0
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 2256 2265. JMLR.org, 2015. htl/atttpp//proceedd.gg mlr. wttppp//////ssoo--iiittcccee./ee html
13

---

Published as a conference paper at ICLR 2024
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th Interna- tional Conference on Learning Representations, JCLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL /RLIItpps///opeeeeeeee. wttp:eaaeweeeeeeeeeeeoaaaaaaaaaa
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness data in deep learning era. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 843 643-852. IEEE Computer Society, 2017. doi: 10.1109//CC22207..97. URL https: oor.oooooo/ooooooooooooooooooooo 100001100000 wttpp/////....//../////// 2017.97
Fabio Henrique Kiyoiti dos Santos Tanaka and Claus Aranha. Data augmentation using gans, 2019.
URL httthhtt:
org/ absa 1904. 09135
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sublayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-2 28-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 10347-10357. PMLR, 2021. URL v139/Eouvron21e html
mlr press/
Toan Tran, Trung Pham, Gustavo Carneiro, Lyle Palmer, and Ian D. Reid. A bayesian data augmentation approach for learning deep models. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 2797-28066 2017. URL https: httpps:://pcceeee..ngteo//// newripeuipp... ips hccpaaper/2117/haall 07 176033ddc118ccffaccffffl.
704 79a at html
Sajila Wickramaratne and Md Shaad Mahmud. Conditional-gan based data augmentation for deep learning task classifier improvement using fnirs data. Frontiers in Big Data, 4:659146, 07 2021. doi: 10.3389/fdata.22 2021 2221.659111666
Wei Xiong, Yutong He, Yixuan Zhang, Wenhan Luo, Lin Ma, and Jiebo Luo. line-grained image-to- image transformation towards visual recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Shin's ya Yamaguchi, Sekitoshi Kanai, and Takeharu Eda. Effective data augmentation with multi- domain learning gans. In The Thirty-Fourth AAAI Conference Artificial Intelligence, AAAI 2020, The Thirty-Seconl Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI
2020, New York, NY, USA, February 7-12, 2020, pages 6566-6574. AAAI Press, 2020. URL
https: //os: aaai. 1. org/ index. php/ AAAI
6131
Xiaohui Yang. Anne M. Smith, Robert S. Bourchier, Kim Hodge, and Dustin Ostrander. Flowering leafy spurge (cuphorbia esula) detection using unmanned acrial vehicle imagery in biological control sites: Impacts of flight height, flight time and detection method. Weed Technology, 34(4):
575-588, 2020. doi: 10.wet 2020.8.
Denis Yarats, Rob Fergus, Alessandro Lazaric and Lerrel Pinto. Mastering visual continuous control: Improved augmented reinforcement learning. In The Tenth International Conference Learning Representatiinns, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
URL
net. ftrett/oooffidddeo SJ- _Yyes
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe.
Cutmix: Regularization strategy to train strong classifiers with localizable features. In 2019 Computer Vision, ICCV 2019, Seoul. Korea (South), October 27 November 2, 2019, pages 6022 6031, IEEE, 2019. doi: 00.110///CCV222999006..
IEEE/CVF International Conference
URL https://////oo .rr. 10. 110...1000 10100///CCCC 2tc.cc2220110 2019. 00612
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, André Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. The visual task adaptation benchmark. CoRR, abs/1910.04867, 2019. URL http://arxii www.oaog/////oo.ooooo/.o../..../ / 1910 04867
14

---

Published as a conference paper at ICLR 2024
Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, and Sanja Fidler. Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 202 OpenReview.net, 2021a. URL forum?id=ywe JJuHX1
www.iieweeee.. net/
Yuxuan Zhang, Huan Ling. Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antomio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 10145-10155. Computer Vision Foundation / IEEE, 2021b. doi:
URL ht htttoo/...tttttt///////ee http:::://pppcccssssette thecvf. com/ content/ CVPR2021/ssmm/2haat Satps_aaaasattttee Efficiens Labelett__abeeeeeeeeeae Fateeaaaoraaaa_aaattooor With Minimal Human__aaan__aaa wwweerfffreeeeeeee CVPR 2021_papers www.upaaeer.hcc .tmml
WWW0//CCVRR4437.222.0000000
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient GAN training. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL hash/ 1ash/55547 1B57/EFFS55000EE221117000---.1.2 cttpppps..cppppp///0000///
Zhedong Zheng, Liang Zheng, and Yi Yang. Unlabeled samples generated by GAN improve the person re-identification baseline in vitro. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 3774-3782. IEEE Computer Society, 2017. 2017.405
doi: 10.1 1011199C.2221..4440 405. URL wwtp://0....../.....0..... 1101111111000 ICW./CCC.... ICt../C1VV......0000
15

---

Published as a conference paper at ICLR 2024
A LIMITATIONS & SAFEGUARDS
As generative models have improved in terms of fidelity and scale, they have been shown to occasion- ally produce harmful content, including images that reinforce stereotypes, and images that include nudity or violence. Synthetic data from generative models, when it suffers from these problems. has the potential to increase bias in downstream classifiers trained on such images if not handled. We employ two mitigation techniques to lower the risk of leakage of harmful content into our data augmentation strategy. First, we use a safety checker that determines whether augmented images contain nudity or violence. they do, the generation is discarded and re-sampled until a clean image is returned. Second, rather than generate images from scratch, our method edits real images, and keeps the original bigh-level structure of the real images. In this way, we can guide the model away from harmful content by ensuring the real images contain no harmful content to begin with. The combination of these techniques lowers the risk of leakage of harmful content, but is not a perfect solution. In particular, detecting biased content that encourages racial gender stereotypes that exist online much harder than detecting nudity or violence, and one limitation of this work is that we can't yet defend against this. We emphasize the importance of curating unbiased and safe datasets for training large generative models, and the creation of post-training bias mitigation techniques.
B ETHICAL CONSIDERATIONS
There are potential ethical concerns arising from large-scale generative models. For example, these models have been trained on large amounts of user data from the internet without the explicit consent of these users. Since our data augmentation strategy employs Stable Diffusion (Rombach et al. 2022). method has the potential to generate augmentations that resemble or even copy data from such users online. This issue is not specific to our work; rather, it is inherent to image generation models trained scales large Stable Diffusion, and other works using Stable Diffusson also face this ethical problem. Our mitigation to this ethical problem is to allow deletion of concepts from the weights of Stable Diffusion before augmentation. Deletion removes harmful, or copyrighted material from Stable Diffusion weights to ensure it cannot be copied by the model during augmentation.
C BROADER IMPACTS
RandAAgment (Pascal)
CutMix (Pascal)
Accuracyy (Val)
1:0
Accuracy (VaD)
8:22
Data augmentation strategies like DA- Fusion have the potential to enable training vision models of variety of types from limited data. While we studied classification this work. DA-
2:70
c.0.
16
Exumplex Per
Examples Per
Fusion may also be applied to video classification, object detection, and visual reinforcement learning. One risk associated with improved few- shot learning on vision-based tasks is that synthetic data can be generated targeting particular users. For exam- ple, suppose one intends to build a person-identification system used to record the behavior patterns of a spe- cific person in public. Such a system trained with generative model-based data augmentations may only need one real photo to be trained. This poses a risk to privacy, despite other benefits that few-shot learning provides. As another example, suppose one intends build system capable of generating pornography of specific celebrity. Few-shot learning makes this possible with just a handful of real images that exist online. This poses risk to personal safety and bodily autonomy of the targeted person.
Baseline
Baseline
Real Guidance (He et al., 2022)
Real Guidance (He al., 2022)
DA-Fusion (Ours)
DA-Fusion (Ours)
Results with stronger data-angmentation basclines. DA-Fusion improves over both RandAugment (using default Py- Torch settings), and CutMix (with default setting from Yun et al. (019)). Note that Rand results use model-centric leakage prevention, and CutMix results use data-centric leakage prevention, showing improve over stronger baselines in both regimes.
Figure 12:
16

---

Published as a conference paper at ICLR 2024
Caltechl0l (Mati-Cratri)
Flowers102 (oatii
Cattech181 (Data-Contrie)
Flewers102
Accursey (Vai)
Accurney (Na
Accuracy (Vai)
Accurscy
Examples Class
Essmples
Essmples
Esamples
Caltechlii (Made-Crattic)
Flowers (Mudtl-Coatrio
Caltech101 (Data-Cuutric)
Flewers102 (Data-Cautric)
0788
4167
Artrayy (14n)
1780
4750
0.79
617
Baseline
Guidlacee 2023)
DA-Pusiaa (Ours)
Figure 13: Few-shot classification performance with both kinds of leakage prevention on additional datasets. DA FA-Fusion outperforms the standard data augmentation baseline, and a competitive method from recent literature. These results reinforce the message in the main paper: RA-Fusion is effective data augmentation strategy.
D ADDITIONAL RESULTS
We conduct additional experiments on the Caltech101 (Fei-Fei et al. 2004). and Flowers 102 (Nilsback) and Zisserman 2008) datasets, two standard image classification tasks. These tasks are commonly used when benchmarking few-shot classification performance, such in the Visual Task Adaptation Benchmark (Zhai et al. 2019). Results on these datasets are shown in Figure 13] and show that DA-Fusion improves classificatio performance both when using model-centric defense against training data leakage, and a data-centric defense, described in Section 6.1 of the paper.
E STRONGER AUGMENTATION BASELINES
DeiTT Backbone (Pascal)
In the main paper, we considered data augmentation base- lines consisting only of randomized rotations and flips. In this section, we compare against two stronger data aug- mentation methods: RandAugment (Cubuk al., 2020),
Accuracy (Val)
0.75
0.70
0.65
and CutMix Yun et al., 2019). Results are presented in Figure [2] and show that DA-Fusion improves over both RandAugment and CutMix on the Pascal-based task.
0.60
0.55
F DIFFERENT CLASSIFIER ARCHITECTURES
Examples Per Class
Results in the main paper use ResNet50 architecture for the image classifier. In this section, we consider the Data-Efficient Image Transformer (DeiT) (Touvron et al., 2021), and evaluate DA-Fusion with data-centric leakage prevention on the Pascal task. Results in Figure 14 show that DA-Fusion improves the performance of DeiT, and suggests that gains generalize different model architec- tures, including both convolution- based models (such as ResNet50), and attention based ones (such VIT).
Bascline
Real Guidance (He et al., 2022)
DA-Fusion (Ours)
Figure 14: Few-shot results with a stronger classification model. DA-Fusion improves DeiT when compared to standard data aug- mentation baseline, and Real Guidance.
G HYPERPARAMETERS
Our method inherits the hyperparameters of text-to-image -ot-iii-iiggeeeeee diffusion models and SDEdit Meng et al.) (2022). In addition, we introduce several other hyperparameters in this work that control the diversity of the synthetic images. Specific values for these hyperparameters given in Table
17

---

Published as a conference paper at ICLR 2024
Hyperparameeerr Name
Value
Synthetic Probability a
0.5
Real Guidance Strength to Num Intensities &
0.5
Intensities Distribution to Synthetic Images Per Real M Synthetic Images Per Real M (spurge) Textual Inversion Token Initialization Textual Inversion Batch Size
u(10.25, 0.5, 0.75, 1.0))
10
"the"
Textual Inversion Learning Rate
0.0005
Textual Inversion Training Steps
1000
Class Agnostic Prompt
a photo"
Standard Prompt
"a photo of a <class name>"
Textual Inversion Prompt
"a photo of ClassX"
Stable Diffusion Checkpoint
CompVis/stable-diffune v1 4
Stable Diffusion Guidance Scale
7.5
Stable Diffusion Resolution
512
Stable Diffusion Denoising Steps
1000
Classifier Architecture
ResNet50
Classifier Learning Rate
0.0001
Classifier Batch Size
32
Classifier Training Steps
10000
Classifier Early Stopping Interval
200
Table 1: Hyperparameters and their values.
We uniformly at random select 20 classes per dataset for evaluation, turning them into 20-way classification tasks. This reduces the computational cost reproducing the results in our paper, and the exact classes used in each dataset can be found in the open-source code.
H LEAFY SPURGE DATASET ACQUISITION AND PRE-PROCESSING
In June 2022 botanists visited areas in western Montana, United States known to harbor Ieafy spurge and verified the presence absence of the target plant at 39 sites. We selected sites that represented range of elevation and solar input values as influenced by terrain. These environmental axes strongly drive variation in the structure and composition of vegetation Amatulli al. (018)) Doherty al. (2021). Thus, stratifying by these aspects of the environment allowed us to test the performance of classifiers when presented with diversity of plants which could be confused with our target.
During surveys, each site was divided into 3 x grid of plots that were 10m on side (Fig. 15). and then botanists confirmed the presence or absence of leafy spurge within each grid cell. After surveying we flew a DJI Phantom 4 Pro at 50m above the center of each site and gathered still RGB images. All images were gathered on the same day in the afternoon with sunny lighting conditions.
We then cropped the the raw images to match the bounds of plots using visual markers installed during surveys as guides (Fig. 16). Resulting crops varied in size because of the complexity of terrain. F.G., ridges were closer to the drone sensor than valleys. Thus, image side lengths ranged from 533 1059 pixels. The mean side length was 717 and the mean spatial resolution, or ground sampling distance, of pixels was 1.4 cm.
In our initial hyperparameter search we found that the classification accuracy plot-scale images was less than that of classifier trained smaller crops of the plots. Therefore, we generated four 250x250 pixel crops sharing a corner at plot centers for further experimentation (Fig, 17). Because spurge plants were patchily distributed within plot, botanist reviewed each crop in the present class and removed cases which cropping resulted in samples where target plants were not visually apparent.
18

---

Published as a conference paper at ICLR 2024
5
8
9
Figure 15: A drone image of surveyed areas containing leafy spurge. At each site botanists verified spurge presence or absence in a grid of nine spatially distinct plots. Note that cell five is rich in leafy spurge.
19

---

Published as a conference paper at ICLR 2024
Figure 16: Markers installed the corners of plots were used to crop plots from source images.
20

---

Published as a conference paper at ICLR 2024
At each plot image center cropped four 250x250 pixel sub-plots. We did this to amplify our data and improve classifier performance. The crops of plots with spurge present labels were inspected by botanist to filter out examples where cropping excluded the target plant the plants were not apparent.
Figure 17:
21

---

Published as a conference paper at ICLR 2024
Figure 18: Here we show examples of synthetic images generated from the leafy spurge dataset with DA-Fusion methods. The top row shows output where images are pooled to fine-tune single token per class. The bottom row shows examples where tokens are generated specifically for each image. Source images, inference hyperparameters, and seed are otherwise identical in each column.
I BENCHMARKING THE LEAFY SPURGE DATASET
We benchmark classifier performance here the full leafy spurge dataset, comparing a baseline approach incorporating legacy augmentations with our novel DA-fusion method. For 15 trials we generated random validation sets with 20 percent of the data, and fine-tuned pretrained ResNet50 on the remaining 80 percent using the training hyperparameters reported in section ?? for 500 epochs. From these trials compute cross validated mean accuracy and 68 percent confidence intervals.
In the case of baseline experiments, we augment data by flipping vertically and horizontally, as well as randomly rotating by much 45 degrees with probability of 0.5. For DA-Fusion augmentations we take two approaches (Fig. 18) The first we refer to as DA-Fusion Pooled, and we apply the methods Textual Inversion Gal et al. (2022). but include all instances of class in single session of fine-tuning, generating one token per class. In the second approach we refer to as DA-Fusion Specific, we fine-tune and generate unique tokens for each image in the training set. the specific case, we generated 90, 180, and 270 rotations well as horizontal and vertical flips and contribute these along with original image for Stable Diffusion fine-tuning to achieve the target number of images suggested to maximize performanceGal et al. (2022). In both DA-Fusion approaches we generated ten synthetic images per real image for model training. We maintain a 0.5, evenly mixing real and synthetic data during training. We also maximize synthetic diversity by randomly selecting 0.25, 0.5, 0.75, and 1.0 to values. Note that we do not apply concept erasure here as in few-shot experiments from the body text.
Both approaches to DA-Fusion offer slight performance enhancements over baseline augmentation methods for the full leafy spurge dataset. We observe 1.0% gain when applying DA-Fusion Pooled and a 1.2% gain when applying DA-Fusion Specific(Fig, [9). It is important to note that, as implemented currently, compute time for DA-Fusion Specific linearly related to data amount, but DA-Fusion Pooled compute is the same regardless of data size.
While pooling was not the most beneficial in this experiment, we support investigating further. This is because fine-tuning leafy spurge token in a pooled approach might help to orient our target in the embedding space where plants with similar diagnostic properties, such as flower shape and color from the same genus, may be well represented. However, the leafy-spurge negative cases do not correspond
single semantic concept. but a plurality, such as green fields, brown fields, and wooded areas. It is unclear if fine-tuning single token for negative cases by pooled method would remove diversity from synthetic samples of spurge-free background landscapes, relative to an image-specific approach. For this reason, we suspect hybrid approach of pooled token for the positive case and specific tokens for the negative cases could offer further gains, and support the application of detecting weed invasions into
22

---

Published as a conference paper at ICLR 2024
0.965
0.95
Accuracy (Val)
Accuracy (Val)
0.960
0.90
0.955
0.85
0.950
0.80
0.945
100
200
300
400
500
Epochs
Baseline
DA-Fusion Pooled
Du-Fuuioon Specifie
Figure 19: Cross-validated accuracy of leafy spurge classifiers when trained with baseline augmen- tations versus DA-Fusion methods on the full dataset. In addition to the benefits of DA-Fusion in few-shot contexts, we also find our method improves performance on larger datasets. Generating image-specific tokens (green line and bar) offers the most gains over baseline, though at the cost of greater compute.
23


Reviews:
[{"review_id": "JbsLQb9RfC", "reviewer": "Reviewer_rXxt", "summary": "The paper introduces a novel approach to data augmentation in deep learning, addressing the issue of limited diversity in traditional augmentation methods. The authors propose a method called DA-Fusion, which leverages large pretrained generative models to generate variations of real images while respecting their semantic attributes. They fine-tune pseudo-prompts to instruct the diffusion model on what to augment, and their approach is tested on few-shot image classification tasks across various domains, including a real-world weed recognition task.\n\nThe authors provide a comprehensive review of related work, highlighting the advantages of diffusion models in image generation and editing. They also discuss the challenges of preventing leakage of internet data when using large pretrained generative models for synthetic data generation.\n\nThe experimental results show that DA-Fusion consistently improves accuracy in various domains, outperforming traditional data augmentation methods. The paper includes a detailed analysis of the results, including a breakdown by the presence of common, fine-grain, or completely new concepts in the datasets.\n\nThe paper concludes by suggesting future directions for improving the flexibility and performance of their method, such as better control over image augmentation, maintaining temporal consistency in decision-making settings, and enhancing the photo-realism of the diffusion model backbone.\n\nOverall, the paper presents a promising approach to data augmentation that addresses the limitations of traditional methods and demonstrates its effectiveness in various domains. The discussion of potential future improvements adds value to the work.", "questions": "I believe the idea is nice and clear. Paper is well written. Some might be of the opinion that the idea is not novel. However, I do like the effective use of pre-trained diffusion models and textual inversion for data augmentation. I like the detailed analysis that the authors have done for their approach.\n\nOne thing I would like to see is how this method can be used to generate positive and negative samples for contrastive learning based approaches. I understand this can be a herculean task for the authors to do now so not required as of now. But curious to know about the effect of this data aug strategy in SSL.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The work on DA-Fusion offers several strengths and introduces significant novelty in the context of data augmentation:\n* Novel Data Augmentation Technique: DA-Fusion introduces a unique approach to data augmentation by leveraging large pretrained generative models. It goes beyond traditional data augmentation methods that mainly involve geometric transformations. This novelty lies in utilizing generative models to create diverse and semantically meaningful variations of real images.\n* Semantic Preservation: Unlike traditional data augmentation techniques that focus on basic geometric transformations, DA-Fusion aims to respect the semantic attributes of images. It modifies images in a manner that retains their underlying semantics, such as the design of objects or specific visual details, making it more relevant for real-world recognition tasks.\n* Few-shot Learning Improvement: The paper demonstrates the effectiveness of DA-Fusion in enhancing few-shot image classification tasks. It is particularly valuable in scenarios where only a limited number of real images per class are available. This is a significant advantage as few-shot learning is a challenging problem with practical applications.\n* Fine-Grained Concepts: The authors show that DA-Fusion is especially beneficial for fine-grained concepts, where traditional data augmentation methods may not provide sufficient diversity. This highlights the method's potential in improving the recognition of subtle visual differences in image classification tasks.\n* Leakage Mitigation: The paper addresses the important issue of preventing leakage of internet data, which is often a concern when utilizing large pretrained generative models. The proposed defenses to mitigate leakage during evaluation contribute to the robustness and reliability of the method.\n* Generalization to New Concepts: DA-Fusion's ability to generalize to new concepts not seen during the diffusion model's training is a noteworthy aspect. This is a crucial capability, as it allows the method to adapt to a wide range of recognition tasks without the need for extensive additional data.\n* Clear Experimental Validation: The work provides a thorough experimental validation, including results on a variety of datasets spanning common, fine-grained, and completely new concepts. The consistent improvement in accuracy across different domains demonstrates the practical applicability and versatility of the approach.", "weaknesses": "Here are some of the weaknesses of this work:\n* Complexity and Computational Cost: The proposed method involves fine-tuning pseudo-prompts for each concept, which can be computationally expensive and time-consuming. This approach might not be as practical as traditional data augmentation techniques that are computationally efficient and easy to implement.\n* Lack of Control Over Augmentations: While DA-Fusion introduces the concept of modifying images while respecting their semantic attributes, it does not explicitly provide fine-grained control over how the augmentations are performed. This lack of control may limit its applicability to specific use cases where precise image modifications are required.\n\nSome nitpicks: Citations starting from second page are not in brackets like that in the first page. Would be nice to have consistency and keep them in brackets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BpsmIVdNVc", "reviewer": "Reviewer_KxYv", "summary": "The paper introduces DA-Fusion that generates additional training data using Stable Diffusion. It involves learning a word embedding that represents each class in the dataset and using SDEdit for generation. Experimental results are shown for few-shot classification on different types of datasets e.g., common concepts, rare concepts, etc, with supportive results.", "questions": "In addition to the questions in weakness, some clarifications about the experiments:\n- How does the method perform zero shot? If the learned tokens did capture the class characteristics, and there is no distribution shift between training and test, it seems like DA-Fusion should do well.\n- Is the leafy spurge task a binary classification problem i.e., detecting if there is leafy spurge in the image or not? If that is the case, how often do other plants occur in the data? I was wondering how much fine-grained details can textual inversion capture.\n- How much variance is there in the class word embedding? If textual inversion is performed again with a different seed, how different would the generations be? If they are, it can be another way to introduce diversity.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper was well written and easy to follow.\n- Performing textual inversion for novel concepts seems like a promising idea.", "weaknesses": "1. Generalizability of the method.\n    - It seems like the learned tokens are helpful for learning dataset specific biases, as it can add additional information on the general “style” of the dataset and of its classes. However, it seems to make certain assumptions about the datasets, e.g., that the images are object centric - only the target class is present, they have standard poses/viewpoints. It is not clear if there is substantial variations within the images of that class, e.g. in iwildcam, where the images from each class can be from different camera traps, thus, different backgrounds, camera parameters, the animals can have very different poses, or can be highly occluded. In this case, what would the single word embedding learn?\n\n2. Desideratas (sec 4)\n    - Controllable (content)\n        - Is textual inversion (as it is used in the paper) controllable? I.e. Is it easy to control for whether the token learns the style or object? Or if certain objects tends to co-occur with another, e.g. train and rails, it may not be necessarily learning the token for trains but also the objects that co-occur with it.\n    - “performant: gains in accuracy justify the additional computational cost of generating images from Stable Diffusion”\n        - There does not seem to be any comparisons with compute cost relative to the baseline Rand Aug. DA-Fusion requires performing textual inversion, which may be expensive.\n        - It would also be interesting to compare the performance with standard augs like Rand Aug for different M. E.g., [1] showed that training on synthetic data outperforms when the generated dataset size much larger than the original dataset size. It is possible that at smaller M, standard augmentations outperform synthetic data, with the additional benefit that it is also fast to compute.\n\n3. Experiments\n    - Randaug, which includes several color based augmentations does not seem to be a good baseline for fine grained datasets like Flowers102, that may require the original color for classification.\n\n\n[1] Sariyildiz et al. Fake it till you make it: Learning transferable representations from synthetic ImageNet clones. CVPR’23", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jebeeNk7qD", "reviewer": "Reviewer_hrLX", "summary": "This paper propose a diffusion model-based data augmentation technique for image classification. The method is based on a pretrained diffusion model. Textual inversion is applied to learn word embeddings for the classes in the target dataset for which the pretrained diffusion model may not learn before. During the generative process, real image is inserted at some time step to guide the generation. When training the target model, real image and synthetic image are mixed by probabilistic sampling. The paper further discuss some design choices for the proposed method, including the time step at which the real image is inserted during generation, strategies to prevent leakage of internet data, mixing ratio of real and synthetic images and the number of augmentations generated for each image. The proposed method is benchmarked on seven classification datasets and shown to outperform RandAugment and Real Guidance.", "questions": "1. I am not clear how the data-centric leakage prevention is performed. The paper mention that \"switching from a prompt ... is sufficient\" and \"Section 4.1 goes into detail\". If data-centric leakage refers to using prompt like \"a photo of a Class3\", what is the setting difference between Fig.5 and Fig.9?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method that combines several existing techniques for data augmentation is interesting.\n2. The paper provide insightful analysis on several design choices, including the time step at which the real image is inserted during generation, strategies to prevent leakage of internet data, mixing ratio of real and synthetic images and the number of augmentations generated for each image.\n3. The paper will release code and an aerial imagery dataset of leafy spurge, which will facilitate future study in this direction.", "weaknesses": "1. The proposed technique is only applicable for classification tasks. It is not clear how it can be applied for object detection and segmentation tasks. My thinking is that one of the major drawbacks of such generative model-based augmentation method vs. traditional method may be that it can not simultaneously generate the segmentation mask and bounding box annotation for the augmented images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qCWlPODEya", "reviewer": "Reviewer_5pTF", "summary": "The authors proposed DA-Fusion, which uses an image-to-image diffusion model to generate new synthetic images to assist classification tasks. DA-Fusion utilizes Textual Inversion to learn the word embeddings of unseen concepts and employs the data balancing and random intensity trick to improve the augmentation results further. The empirical shows the effectiveness of DA-Fusion in the low-shot setting on common concepts, fine-grained concepts, and rare concepts.", "questions": "- The authors used a different number of augmented images per real image, $M$ ($M$=50 for spurge and $M$=10 for other data). Are there any guidelines for selecting $M$?\n\n- As pointed out in the weakness, the technical contribution of this work is slightly limited. It is recommended to compare DA-Fusion with more and stronger data augmentation baselines (instead of just RandAugment and Real Guidance) to claim a more significant empirical contribution.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper shows positive empirical results on several classification tasks covering common, fine-grained, and rare concepts.\n\n- The paper is well-written and easy to follow.", "weaknesses": "(1) The technical contribution is limited. DA-Fusion combines several existing methods, like the image-to-image diffusion model, the Textual Inversion, and the data balancing technique.\n\n(2) DA-Fusion outperforms the much simpler RandAugment method mostly on extremely low-shot settings (less than 16 images per class). The improvements of DA-Fusion seem to be marginal when there are more than 16 images per class. Also, it is unfair to compare with RandAugment, which only uses the default hyperparameters, while DA-Fusion is “fine-tuned” on the target data (i.e., textual inversion, selecting $M$, and other parameters). The authors should search for the optimal number of operations and augmentation magnitude for RandAugment for a fairer comparison.\n\n(3) The authors tested their method on COCO and PASCAl VOC datasets for pure classification but not object detection. Being a general image-to-image diffusion model, the generation step in DA-Fusion may alter the position of the objects in input images. It seems that DA-Fusion can only be applied to classification tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)