Diversify Your Vision Datasets with Automatic Diffusion-based Augmentation
Authors: Lisa Dunlap
Alyssa Umino
Han Zhang
Jiezhi Yang
Joseph E. Gonzalez
Trevor Darrell
Venue: NeurIPS 2023 poster (2023)
Keywords: data augmentation
diffusion
vision and language
TL;DR: V&L models can summarize spurious features in your data with language, which can be used to augment your training data

Abstract: Many fine-grained classification tasks, like rare animal identification, have limited training data and consequently classifiers trained on these datasets often fail to  generalize to variations in the domain like changes in weather or location.  As such, we explore how natural language descriptions of the domains seen in training data can be used with large vision models trained on diverse pretraining datasets to generate useful variations of the training data. We introduce ALIA (Automated Language-guided Image Augmentation), a method which utilizes large vision and language models to automatically generate natural language descriptions of a dataset's domains and augment the training data via language-guided image editing. To maintain data integrity, a model trained on the original dataset filters out minimal image edits and those which corrupt class-relevant information. The resulting dataset is visually consistent with the original training data and offers significantly enhanced diversity. We show that ALIA is able to surpasses traditional data augmentation and text-to-image generated data on fine-grained classification tasks, including cases of domain generalization and contextual bias. Code is available at https://github.com/lisadunlap/ALIA.

Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentation
Lisa Dunlap Alyssa Umino
Han Zhang
Jiezhi Yang
Joseph E. Gonzalez
Trevor Darrell
UC Berkeley
Abstract
Many fine- ine-grained classification tasks, like rare animal identification, have limited training data and consequently classifiers trained these datasets often fail to generalize to variations the domain like changes in weather or location. As such, explore how natural language descriptions of the domains seen in training data can be used with large vision models trained on diverse pretraining datasets to generate useful variations of the training data. We introduce ALIA (Automated Language-guided Image Augmentation), a method which utilizes large vision and language models to automatically generate natural language descriptions of a dataset's domains and augment the training data via language-guides image editing. To maintain data integrity, a model trained on the original dataset filters out minimal image edits and those which corrupt class-relevant information. The resulting dataset visually consistent with the original training data and offers significantly enhanced diversity. We show that ALIA is able to surpasses traditional data augmentation and -o-tte--mmae generated data on fine one-grained elassification tasks, including cases of domain generalization and contextual bias. Code is available https: //gtthub......... com htmmm/isadunlaa///am AALIAA.
Introduction
While modern pretraining data are incredibly diverse, datasets for specialized tasks such fine- grained animal identification are often much less so, resulting in trained classifiers that fail when encountering new domains such as change in weather or location. An effective method to address this is to add more training data from the test domain (37), but obtaining and labeling this additional data is often costly and it can be challenging to determine which domains to gather data from.
To address this, recent works have utilized image generation models trained on large pretraining datasets to supplement incomplete training sets by generating diverse examples from generative models fine-tuned on the training set (8; 14; 1). Furthermore, previous work in language-guice guided data augmentation with vision and language models relies on user-supplied domain descriptions (7) or descriptions generated from word-to-sentence models (9).
Although these methods can be effective in increasing image diversity, they either require finetuning the image generation model, which can be prohibitively expensive, generating images which have no visual grounding in the original training data. While the latter may be suitable for common benchmarks, such as ImageNet (6). proves be much less effective when we move a specialized setting where generative models like Stable Diffusion (30) cannot recreate images which resemble the training data from text alone, as shown Figure In our work, we foeus on how to utilize pretrained vision and language models for image captioning and generation as a translation layer between task-specific image data and task-agnostic natural language descriptions of domains. Since these high-level domain descriptions are well-represented by image generation models like Stable Diffusion, we can use them to perform language-guin guided image editing of the specialized training data.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

A photo of a bird...
"A photo of [] bird the wild."
pprrched on fence"
"interacting with flowers"
perched on birdfeeder"
Original
CutMix
RandAug
Text-to-image generation
Traditional data augmentation
(Ours) Automated Language-guided Image Editing
Figure 1: Overview. Example augmentations using text-to-images generation, traditional data augmentation methods, and our method, Automated Language-guin guided Image Editing (ALIA) on CUB (41). Images generated by ALIA retain task-relevan information while providing more domain diversity as specified by the prompts. Within the prompts, indicates the specific class name.
This produces images which are visually consistent with the training data, vary the task-agnostic domains, and preserve the task-relevant information present in the original image.
Specifically, our method ALIA (Automated Language-guided Image Augmentation) first generates captions for each image, summarizes the captions into short list of domain descriptions with large language model (LLM), and then uses these descriptions to generate edits of the training data with Stable Diffusion. To ensure data quality, we use classifier trained on our original training set to remove images which (I) do not change the domain of the original image desired or (2) corrupt task-relevant information. After filtration, we are left with an edited dataset visually consistent with the original data and representing all domains seen in testing (Figure 1). ALIA does not require finetuning the image captioning or image generation model, nor does it require user-supplied prompts.
We evaluate on fine- ine-grained bird classification (CUB (41)), domain generalization (iWildCam (17)), and contextual bias (Waterbirds (32)) datasets. We show that the addition of ALII generated data outperforms traditional data augmentation techniques and text-to-image generated data by up to 7%, even heating the performance of adding real data on iWildCam. Furthermore, we investigate how domain descriptions produce more useful edits than user-provided prompts, and examine the effect of filtering and the choice of image editing techniques (Section 5.4).
Related Works
Supplementing Training Data with Generative Models. Using generative models for data augmen- tation is well-explored area of research, specifically medicine (8; 33), domain adaptation (12) and bias mitigation
(34). These methods train or use pretrained GAN to generate images from the desired distribution. Furthermore, GANs have been used to supervise dense visual alignment (28) and generate pixel-level annotations from small amount of labels (45; 18; 39). Recently, several works (9; 35; 38) have shown diffusion models' ability to generate training data zero or few shot settings as well as generate hard training examples (13). While these works do show the promise of diffusion-ged -enerrate data, models trained on diffusion-generatte accuracy than models trained on real datasets unless finetuned for that specific task 38). In contrast, we use diffusion models do image editing with text rather than generating images from text alone, resulting in augmentations that closely resemble the training data without finetuning.
data obtain significantly worse

---

Traditional Data Augmentation. Traditionally, data augmentation techniques involve random flipping, cropping, color shifting, etc. manipulate the original images and create new versions of these images (36). More recently proposed mixup-based data augmentations aim to make the augmented data more diverse by cutting and pasting patches of two input images (42), applying
convex combinations (43), and adaptively learning sample mixing policy (10; 20; 15; 5). These methods often result in images that look unnatural as shown Figure I, while our method aims to create augmentations which are visually consistent with the original training data.
Language-guidedt Data Augmentation for Classification. Recent works (7; 9: 29) have explored how natural language descriptions of the domains and classes seen during testing can be used to improve performance and generalization typically employ text-to-imee -ottgggg generation with prompts sourced from language models (9), or augment training images using user-provided descriptions
few-shot or domain adaptation setting, These methods
both training and unseen test domains in shared vision and language space (7). Similar to our work, these techniques use the high-level domain knowledge of large models to diversify image data with language, but crucially they either generate descriptions and augmentations that are not
rely on user-provided domain descriptions grounded in any real image data. In contrast, the foundation of our work is to ground both the domain descriptions and the generated training data in the given training set.
3 ALIA: Automated Language-guided Image Editing
Given labeled training dataset, aim to augment the dataset with images that are edited to improve representation of various domains. We define domain to be any aspect of an image that is not intended to be used for classification (e.g. location, weather, time of day). The key insight of our method to utilize image captioning and image generation models trained on large amounts of pretraining data to summarize the domains in the training set and use those descriptions to augment training data using text-conditioned image editing.
Our method consists of 3 stages: generating domain descriptions, generating useful image edits, and filtering out poor edits. An overview of our method is shown Figure 2.
Caption Training Data and Summarize Unique Domains
Generate Useful Variations Domain Descriptions
Filter w/ Model Confidence
Training Data
Text-to- Pout-to-laage Image Model
Captioning Model
camera trap photo an animal...
L.M Summary
Classifier
"in field dirt and shrube"
"walking on a trail at night"
X
lake field"
Figure 2: ALIA. Given a specialized dataset, we caption all images in our dataset using a pretrained captioning model, and feed these captions into large language model to summarize them into small (<10) set of natural language descriptions. Utilizing these descriptions, we perform text- text-guided image augmentation via pretrained text-to-image model, thereby generating training images that align with the described settings. Finally, we apply two filtering techniques: CLIPP-aasd semantic filter to eliminate obvious edit failures, and a confidence-basee filter that removes more subtle failures by filtering edits confidently predicted by classifier trained on the original dataset.
3.1 Generating Domain Descriptions
key idea of our method is to use captioning and language models trained on large amounts of pretraining data to summarize the potential domains. We assume knowledge of the superclass that
3

---

encapsulates all classes, such as "bird" for CUB or "animal" for iWildCam and prefix to guide the format of the desired caption form. for example "a photo of a bird...". ". Once these prompts are generated, we found that we often achieve higher quality edits by adding the class name into the prompts after the fact (e.g. "a photo of Scott Oriole bird...").
To generate a concise list of domain descriptions, we first caption each image in the dataset using a pretrained captioning model. This produces a comprehensive set of captions, which may highlight potential domains seen at test time. Note that these captions do not need to accurately describe the task speciffc information, such as the species of the bird, as their purpose is to provide broad overview of the context, such as the environment the bird is in or the actions of the bird. Additional image data of possible domains seen in testing but that don't contain any televaele information can also be used in this step. For example, when performing animal classification, one can add in background images of different locations that may seen deployment.
We assume that the amount of training data is not small (< 100 samples). Therefore, we use an LLM summarize these captions into list of domains which are agnostic of the class. Due constraints on context length, we randomly sample 200 unique captions and use the following prompt:
"I have a set of image captions that want to summarize into objective descriptions that describe the scenes, actions, camera pose, zoom, and other image qualities present. My captions are [CAPTIONS]. want the output to a handful of captions that describe unique setting, of the form [PREFIX]"
We then ask a refinement question to ensure each caption is of only one setting and agnostic of the class with the following prompt:
"Can you modify your response so each caption is agnostic of the type of [SUPER- CLASSJ. Please output less than 10 captions which cover the largest breadth of concepts."
The end result is a list of less than 10 domain descriptions that often include domains seen in the test data (e.g. "A camera trap photo of an animal near lake."). These descriptions serve as the foundation for the subsequent text-conditioned image editing stage of our method. We use BLIP (19)
captioning model and GPT-4 (25) for summarizing the captions. A complete list of prompts for each dataset is given in Section 5.
3.2 Editing Images with Language Guidance
Once the domain descriptions are generated, they are used to condition the edits of original images in the training set. In experiments, we employ two editing techniques based on Stable Diffusion; however, ALIA can be used with any text-conditioned image editing method. The two techniques deployed for our experiments are follows:
Image to Image with Text Guidance (Img2Img) (30; 2; 22): This technique first uses an image encoder to translate provided image into a latent representation. Then, leveraging diffusion model, this latent representation is progressively modified through a series of transformations, conditioned on the user-provided text. Finally, the modified latent representation is decoded to generate augmented image that incorporates the desired modifications specified in the prompt.
Instruct Pix2Pix (3); Given an edit instruction (e.g. "put the animals in the forest") and an image, this method generates a modified image adhering the specified instruction. This accomplished by training a conditional diffusion model on dataset composed of paired images and edit instructions.
Among the many existing image editing techniques (23; 11; 4; 44; 26), we selected the two above for their ease of use and quality of outputs. Notably, the methods highlighted here share a common goal: generate coherent, visually-grounde augmentations conditioned on the natural language descriptions extracted from the dataset.
3.3 Filtering Failed Edits Using Semantic and Visual Features
As depicted in Figure 3, there are three distinct failure cases for the text-conditioned augmentation: (I) total failure, where the edited image is vastly different from the original, (2) identity failure, where

---

Total Failure
Identity Failure
Class Corruption Failure
Prediction
incornet
Confidence
high
Identity
Corrupt
low
Acceptable Edits
Figure 3: Filtering. Semantic filtering eliminates instances total failure (left), but often misses instances of minimal edits (center) or edits which corrupt class information (right). Meanwhile, our confidence- based filtering mechanism is able to remove these failures by leveraging the prediction confidence of classifier trained on the original dataset applied the image edits.
the edited image is nearly identical to the original, and (3) class corruption failure, where the edit sigmificantly changed task-specific features. While previous work (9) utilized CLiP-based filtering to remove low-quality images, it only removes instances of total failure in settings where CLIP does not have a good representation of the classes. As such, we also employ confidence-batt filtering technique which uses a classifier trained on the original training set to determine instances of identity and class corruption failures.
Semantic Filtering, We use CLIP to predict whether the generated image is related to the task or not. For example, the CUB dataset, we provide the text prompt "a photo of a bird" well as the filtering prompts "a photo of an object", "a photo of scene", "a photo of geometric shapes", "a photo", and "an image". All images that are not classified as photo of a bird" are removed.
Confidence-basel Filtering. We take inspiration from previous work in identifying mislabeled examples using model confidence (24). After training model f the original dataset, caleulate a confidence threshold by for each class y by averaging the softmax score of the correct label for each image in the training out if confidence (x'), 9) 2 tg. In the case that the image was correctly predicted (y = Ø). since the model is already confident its prediction, the edited image likely contain roughly the same information as the original, and thus should be filtered out. In cases where y + 9. high confidence suggests that the edit has corrupted the class a point that more highly resembles another class.
Specifically, given edited image x' with label and prediction ÿ. it is filtered
In our experiments, we randomly sample portion the filtered augmentations to incorporate back into the training set, leading a dataset size expansion between 20-100%: Given that the prompts and the image edits are grounded in the original training set, ALIA is able to preserve visual consistency and encompass broader array of domains.
4 Experimental setup
4.1 Implementation.
We fine-tune a ResNet50 for the CUB (41), (411,,WW iWildCam (17), and Waterbirds (32) datasets. We use the Py PyTorch pretrained models (27) on ImageNet with an Adam optimizer (16) and cosine learning rate scheduler. For each method, we do hyperparameter sweep across learning rate and weight decay and choose the parameters with the highest validation performance. We train on 10 GeForce RTX 2080 Ti GPUs.
For all the diffusion- -aased editing methods, we use Stable Diffusion version 1.5 (31) from Hugging- Face (40) with the default hyperparameters aside from the edit strength (how much to deviate from the original image) and the text guidance (how closely the generated image should align with the text prompt). For these parameters, we search over different values each for edit strength and text guidance, visualizing the resulting generations for a random sample (10 images) across 4 random seeds (40 generated images in total). We pick the parameters which generate the most diverse images that both retain the task-relevant information and remain faithful to the edit instruction. More details on this selection process including visualizations as well as results on training on data generated with different edit strengths and text guidance are in the Appendix. Results are averaged over 3 random seeds and further details on the hyperparameter search space and final choice of hyperparameters are listed in the Appendix.
5

---

4.2 Baselines.
The Baseline model trained on the original training dataset, without any data augmentation. We additionally compare adding in the generated data with our method to the original training set (+ALIA), adding real data from the test distribution (+Real), adding diffusion generated data from text alone (+ (+Tx12mmg), and two traditional augmentation baselines (+ CuúMix, +RandAug). In order to perform a fair comparison, we keep the number of images added per class the training set. consistent across methods.
For the +ALIA and +Tx12Ing results, we generate twice as much data as the original training set for each language prompt to ensure enough data is available after filtering. For +ALIA, this entails generating edits for every image. We then randomly sample from these datasets match the class distribution of the held-out real data used for the +Real baseline. For the + +Txr21mg baseline, we generate images with hand-crafted prompt for each dataset, and apply our semantic filtering to the generated images to remove low-quality samples. We provide the amount of data added and the prompts used for the + Imx22mmg baseline in the subsequent sections.
Lastly, we compare to two data augmentation baselines taken from recent literature: (1) CutMix (42), which generates mixed samples by randomly cutting and pasting patches between training images to encourage the model to learn more localized and discriminative features and (2) RandAngment (5), an automated data augmentation technique that reduces the search space of the type and magnitude of augmentations to find the best image transformations for given task. For these baselines, we use the implementations the Py Torch Torchvision library (21).
Experiments
We evaluate on 3 specialized tasks: domain generalization via camera trap animal classification (iWildCam), fine-grained bird classification (CUB). and bird classification with contextual bias (Waterbirds). Details of each dataset are listed in subsequent sections, with more statistics in the Appendix.
camera trap photo
"near large body water the middle field"
"In grassy field with bushes"
Original
84.87
Accuracy
77.56
70
72/97
75.59
74.10
67.51
60
50
+CutMix +RandAug
+Tx22mg
+AUA
RRaal
Figure 4: iWildCam Subset. Left plot visualizes the original training images and the corresponding data generated from ALJA based on the generated prompts. Right plot shows accuracy for adding in generated data from ALIA and baselines as in Section 4.2. ALIA significantly outperforms all the baselines including adding in real data from the test distribution, while +Tx12Img and +RandAug see smaller improvements.
5.1 Domain Generalization [iWildCam (17))
The iWildCam dataset is large-scale collection of images captured from camera traps placed in various locations around the world. We subsample the dataset to create a 7-way classification task (background, cattle, elephant, impala, zebra, giraffe, dik-dik), with 2 test locations that are not in the training or validation set. The training set has 6,000 images with some classes having as few as 50 images per example, and our + +eal data baseline contains roughly 2,000 images from locations not seen during training. We generate our domain descriptions from the background images from the test domain, but do not use these images in training (details in Section 8.3). Since many images in the

---

original training set contain watermarks or timestamps while the text-to- text-to-image generated data usually does not, we crop the top and bottom of each image in prepossessing.
The prompts generated by ALIA are "a camera trap photo of a I...": (I) "in grassy field with trees and bushes.", (2) "in a forest in the dark.", (3) "near a large body of water in the middle of a field.", (4) "walking on dirt trail with twigs and branches.".
We use "a camera trap photo of in the wild." as the prompt for the + +Tx2Img baseline.
As shown in Figure ALIA not only outperforms all baselines, with 17% performance improvement over training on the original data, but even exceeds the performance of adding in the same amount of real data. Note the improved performance over real can be somewhat attributed to that fact that our prompts are taken from empty images from the test domain rather than the training set.
5.2 Fine-grained Classification [CUB (41))
CUB is a one-grained bird classification dataset comprised of photos taken from Flickr. We remove of the 30 images from each of the 200 classes from the training for the +Real comparison.
The prompts generated by ALIA are "a photo of ( bird...": (1) "interacting with flowers.", (2) "standing by the waters edge.", (3) "perched on fence., (4)
standing on a rock.", , (5) "perched on a branch." " (6) "flying near tree, sky as the backdrop.", (7) perched birdfeeder."
We use "an iNaturalist photo of a { bird in nature." " as the prompt for the +7xx21mg baseline.
As shown in Figure 5, ALIA outperforms all the baselines aside from RandAug and adding in real data, while adding text-to til-t-g-iagee generated data results in similar performance with the baseline. Significantly, these results show that ALIA provides performance improvements even in conditions devoid of domain shifts, thereby broadening its utility.
"A photo of bird.."
"standing the waters edge" a rock"
"standing
"perched fence"
Original
74
72
72.89
73.75
Accuracy
70.56
69.16
69.21
Baseline
+CutMix
+RandAug +Ta12mm
+AUA
+Real
Figure 5: Bird Classification (CUB). Left plot visualizes the original training images and the corresponding data generated from ALIA based on the generated prompts. Right plot shows accuracy for ALIA and baselines. ALIA outperforms the baselines aside from RandAug and adding real data, while adding text-to-image generated data results in similar performance with the baseline.
5.3 Contextual Bias [Waterbirds (32)]
Waterbirds is a synthetically constructed dataset which creates contextual bias by taking species of landbirds and waterbirds from the CUB-200 (41) dataset and pasting them onto forest and water backgrounds from the Places (46) dataset. The classification task landbirds versus waterbirds, and the spurious correlation is the background. The training and validation sets have all landbirds appearing on forest backgrounds and all waterbirds appearing on water backgrounds, while the test set has representation of backgrounds and bird types. Our Real data baseline contains only images from the out-oi-domain groups, namely landbird on water and waterbird on land.
Interestingly, we found that the edits generated by the Img2Img editing method were very low quality (visualizations in the Appendix). Thus, we use InstructPix2Pixx prompting the L.LM with the prefixes "a photo of a bird" and editing the final prompts to fit the instructional format of the editing method.
7

---

"A photo of Landbird....
"A photo of Waterbird..."
"sxcmminng in mountains background"
"standingg
"standing beach looking
green background"
stumpp
Original
Original
100
67.34
Cutt0
#Randoug
30
Accurast
71.40
62.24
63.42
63
97
63.64
Baaciro
AAJA
MMal
0C0 Accurasy
Figure 6: Waterbirds Results. Right plot shows the breakdown of in-domain (waterbirds on water, landbirds on land) and out- of-domain performance (waterbirds on land, landbirds on water). ALIA able to roughly match in-domain accuracy of the other methods while drastically improving out-of -f-ma-n accuracy of all other augmentation methods. This results in 7% improvement for class- bass-balanced accuracy when compared to other augmentation baselines.
Left plot shows class-balanced accuracy for ALIA and baselines.
The prompts generated by ALIA are "a photo of a J...":
(I) "in a bumboo forest with a green background."" (2) "flying over the water with a city skyline in the background.", (3) "perched on "standing on a tree stump in the woods.". (6) "swimming in a lake with mountains the background.", (7) "standing the beach looking up."
window." (4) "standing in the snow in forest.", (5)
We use "an iNaturalist photo of a { in nature." " as the prompt for the +7xt21mg baseline.
As shown Figure 6, ALIA is able to approximately match the in-domain accuracy of the other augmentation methods while outperforming all other methods besides the real data baseline for overall accuracy. While ALIA's performance on this dataset is much lower than the state of the art methods, we believe that these results point promising future direction in how modify ALIA to identify and amplify specific class-domain pairings which are underrepresented in the training set.
5.4 Ablations
Effect of Prompt Quality and Filtering. We ablate the quality of ALIA generated prompts by comparing the edit prompts generated by ALIA against those provided by users. For the user-supplied prompts, we draw on the engineered prompts that were used for the + +Xx22Img baseline. As shown in Table descriptions generated by ALIA outperform eser-provided prompts, especially in the contextual bias setting, indicating that our generated prompts are able to accurately describe key domain-specific features and variations, resulting in more effective data augmentation for mitigating biases and improving model generalization. Moreover, Table also demonstrates how our semantic and confidence-basee filtering further improves accuracy.
Dataset
User Prompt
ALIA Prompts
ALIA Prompts + Filtering
iWildCam
79.92+4.22%%
82.57= 19%
84.87++ 1.92%
CUB
71.02+0.479%
71.25=0.58m 0.86%
72.70++0. 10% 71.40-:1.859
Waterbirds
63.64 + 1.43%
70.22=0.53%
Table 1: Effect of ALIA Generated Prompts and Filtering.

---

Choice of Image Editing Method. As mentioned in Section 5.3, the suitability of image editing methods can vary based on the specific dataset. As such, we explore the impact of the editing method on performance by modifying the iWildCam subset from Section 5.1 with InstructPix2Pix. We employ the same prompts as the Img2lmg case, reformatted to the instructional syntax required by InstructPix2Pin (e.g. "a camera trap photo of {) in grassy field with trees and bushes" becomes "put the (} in a grassy field with trees and bushes"). Figure 7 presents examples of edits generated using InstructPix2Pixx Although InstructPix2Pix can produce edits with more significant color variations than Img2Img, frequently applies an unrealistic airbrush-like effect or erases most of the information present in the original image. This outcome can be attributed to the fact that InstructPix2Pix is fine-tuned on a dataset of paired images and edits, which primarily consist of stylistic changes to artistic images, making the iWildCam images fall outside their domain. When comparing the performance of InstructPix2Pix, we find that it achieves an accuracy of 76.23 23 + 4.78%, compared to 84.87 + 1.92% obtained using Img2Img edits. These results indicate that the choice of image editing method has significant impact on the final performance of ALIA, and thus it is crucial to profile the quality of image edits from various techniques one's specific dataset.
"put the near large body of water in the middle of a field."
Figure 7: InstructPix2Pix edits on iWildCam. Despite offering more color variation, the method often imparts an artificial airbrush-like effect or removes substantial original image details.
Amount of Augmented Data Added, Figure 8 depicts the accuracy on CUB as function of the number of generated images added where the grey line is the baseline accuracy. ALIA is able to achieve accuracy gains up to 1000 images (20% of the original training set), at which point accuracy starts to decline. contrast, images generated from text alone see a decrease in accuracy almost immediately, reaching below the accuracy of the original training set at 2000 images. We suspect this because much of the text image data is from a different distribution than the regular training data, and thus adding small amounts can increase robustness while large amounts cause the model to overfit to this distribution. Since image to image edits use language to edit the training images directly, these images are less likely to be out of distribution as compared to the text image data.
74
Txx22mmg
Baseline
Accuracy
72
70
68
66
500
1000
1500
2000
2500
3000
3500
4000
Number of Generated Samples Added
Figure 8: Size of Augmented Dataset (CUB). Results are shown for adding 200, 400, 1000, 2000, and 4000 generated images to the training set. The grey line depicts the baseline accuracy on the original 4994 images. ALIA is able to achieve accuracy gains up to 1000 images, at which point accuracy starts to decline. In contrast, images generated from text alone see a decrease in accuracy almost immediately, reaching below the accuracy of the original training set.
6 Limitations
While ALIA able to generate impressive augmentations, there are limitations. As ALIA depends on large pretrained vision and language models to translate between task-specific image data and task agnostic natural language descriptions of domains, performance of the method is bottlenecked by the quality of the captioning model, LLM, and image editing method (see Section 5.4). Furthermore, as we assume that the task-specific information the training data cannot be easily generated via text alone, attempts to change aspects like the pose of a bird in CUB is likely to result in a failed edit. Finally, determining the optimal quantity of augmented data to reincorporate into the training set remains an unresolved question, and is we look forward to addressing in future research.

---

7 Conclusion
We present ALIA, a novel approach to data augmentation that leverages the high-level domain knowledge of large language models and text-conditioned image editing methods. By grounding both the domain descriptions and the augmented training data in the provided training set, our method has demonstrated impressive capabilities in several challenging settings, including domain adaptation, bias mitigation, and even scenarios without domain shift. Our findings offer exciting potential for future research, suggesting that the utilization of structured prompts generated from actual training data can be effective strategy for improving dataset diversity. As the capabílities of captioning, LLMs, and image editing methods grow, we expect the efficacy and scope of our approach to increase.
Acknowledgements. We thank Suzie Petryk for her invaluable feedback on the manuscript. This work was supported in part by the NSF CISE Expeditions Award (CCF-1730628), DARPA's SemaFor, PTG and/or LwLL programs, and BAIR's 's industrial alliance programs.
References
[1] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. Synthetic data from diffusion models improves imagenet classification, 2023.
[2] Andreas Blattmann, Robin Rombach, Kaan Oktay, and Björn Ommer. Retrieval-augmented diffusson
models, 2022.
[3] Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. November 2022.
[4] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion- biffusion-based
semantic image editing with mask guidance. arXiv preprint arXiv:2200.1427, 2022.
[5] Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In Larochelle, M. Ranzato, Hadsell, M.F. Balean, and Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18613-18624. Curran
Associates, Inc., 2020.
[6] Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.
Jeee, 2009.
[7] Lisa Dunlap, Clara Mohri, Han Zhang, Devin Guillory, Trevor Darrell, Joseph E. Gonzalez, Anna Rohrbach, and Aditi Raghunathan. Using language extend unseen domains. International Conference on Learning Representations (ICLR), 2023.
Maayan Frid-Adar, Idit Diamant, Eyal Klang, Michal Amitai, Jacob Goldherger, and Hayit Greenspan. Gan- based synthetic medical image augmentation for increased enn performance liver lesion classification. Neurocomputing. 2018.
[9] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr. Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? arXiv preprint arXiv:2210.07574
2022.
[10] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. AugMix: A simple data processing method to improve robustness and uncertainty. Proceedings of the International Conference on Learning Representations (ICLR), 2020.
[11] Amir Hertz, Ron Mokady, Jay Tenenhaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint orXiv:2208.1 00660, 2022.
[12] Sheng- Wei Huang, Che-C Tsung Lin. Shu-Ping -in- Chen, Yen-Yi Yi Wu an Po-Hao Hsu, and Shang-Hong Lai.
Auggan: Cross domain adaptation with gan-based data augmentation. European Conference on Computer Vision, 2018.
[13] Saachi Jain, Hannah Lawrence, Ankur Moitra, and Aleksander Madry. Distilling model failures as directions in latent space. In ArXiv preprint arXiv:2206. /4754, 2022.
[14] Hygg Kim. Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning (ICML), 2020.
[15] Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning, pages 5275-5285. PMLR, 2020.
[16] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412,6980. 2014.
[17] Pang Wei Koh, Shiori Sagawa, Henrik Marklund. Sang Michael Xie, Marvin Zhang. Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, al. Wilds: benchmark in- the-wild distribution shifts. In International Conference on Machine Learning. pages 5637-5664. PMLR, 2021.
[18] Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Kreis, Adela Barriuso, Sanja Fidler, and Antonio
Torralba. Bigdatasetgan: Synthesizing imagenet with pixel-wise annotations, 2022.
[19] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman -imgge pre- mre-trannig for unified rision-languan understanding and generation, 2022.
10

---

[20] Zicheng Liu, Siyuan Li, Di Wu, Zihan Liu, Zhiyuan Chen, Lirong Wu, and Stan Z Li. Automix: Unveiling
the power of mixup for stronger classifiers. In Computer Vision-ECCV 2022: 17th European Conference, Aviv, Israel, October 23-27. 2022, Proceedings. Part XXIV, pages 441-458. Springer, 2022.
[21]
TorchVision maintainers and contributors. Torchvision: Pytorch's computer vision library. https:
2016.
[22] Chenlin Meng. Yutong He. Yang Song. Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.
SDEdit:
Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
[23] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2022.
[24] Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. In Proceedings of the 35th Conference on Neural Information Processing Systems Track on Datasets and Benchmarks, December 2021.
[25] OpenAI. Gpt-4 technical report. 2023.
[26]
Dong Huk Park, Grace Luo, Clayton Tostc, Samanch Azadi, Xihui Liu, Maka Karalashvili, Anna Rohrbach, and Trevor Darrell. Shape-guided diffusion with inside-outside attention. arXiv, 2022.
[27] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep
learning library. Advances in neural information processing systems, 32, 2019.
[28] William Peebles, Jun- Yan Zhu Richard Zhang, Antonio Torralba, Alexci Efros, and Eli Shechtman.
Gan-supervised dense visual alignment. In CVPR, 2022.
[29]
Suzanne Petryk, Lisa Dunlap, Keyan Nasseri, Joseph Gonzalez, Trevor Darrell, and Anna Rohrbach. On guiding visual attention with language specification. In Conference on Computer Vision and Paterr Recognition (CVPR), 2022.
[30] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution
image synthesis with latent diffiusion models. 2021.
[31]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and BjÄîm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[32] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Perey Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. International Conference on Learning Representations (ICLR), 2020.
[33] Swami Sankaranaray anaan Yogesh Balaji, Carlos Domingo Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. Conference on Computer Vision and Pattern Recognition (CVPR). 2018
[34] Viktoriia Sharmanska, Lisa Anne Hendricks, Trevor Darrell, and Novi Quadrianto. Contrastive examples
for addressing the tyranny of the majority. CORR, abs/2004.06524 2020.
[35] Jordan Shipard, Arnold Wiliem, Kien Nguyen Thanh, Wei Xiang, and Clinton Fookes. Boosting zero-shot classification with synthetic data diversity via stable diffusion. arXiv preprint arXiv:2302. 032980 2023. [36] Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning.
Journal of big data, 6(1):1-48 2019.
[37] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Mea- suring robustness to natural distribution shifts in image classification. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[38] Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmentation
with diffusion models, 2023.
[39] Nontawat Tritrong, Pitchaporn Rewatbowornwong, and Supasorn Suwajanakorn. Repurposing gans for shot semantic part segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[40] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https://gi tht.///./tttuu.../ com/
2022.
[41] Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongic. The caltech-uesd
birds-200-2011 1 dataset, 2011.
[42] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In International Conference on Computer Vision (ICCV), 2019.
[43] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference Learning Representations, 2018
[44] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023. [45] Yuxuan Zhang. Huan Ling, Jun Gao, Kangxue Yin, -ran-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In CVPR,
2021.
[46] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEFE Transactions on Pattern Analysis and Machine Intelligence,
40, 2018.
11


Reviews:
[{"review_id": "pgsdQsBOa6", "reviewer": "Reviewer_u22M", "summary": "This paper presents a data augmentation methods. It first leverages the advanced image captioning and large language models to generate  prompts. Next, it uses diffusion-based models to generate more images. In addition, it trained a model to filter failure or similar cases during image generation. Extensive experiments over 3 specialised tasks demonstrate the effectiveness of the proposed method. ", "questions": "1. It have been studied in [8, 35] that images generated from diffusion models could help to improve the image classification performances. The author claims that \"we use diffusion models to do image editing with text rather than generating images from text alone, resulting in augmentations that closely resemble the training data without finetuning.\" However, I'm thinking stable diffusion models (text2image models) could generate more diverse data as compared to text editing diffusion models (e.g. InstructPix2Pix). Then why text editing methods are better? Experimental results to compare the augmented data through text2image and text editing methods are expected. \n2. This paper aims to improve the performances on specialised tasks that have very few training data. However, some of these tasks may contain out-of-domain data that may be difficult to be generated through diffusion models. Discussion regarding to these cases are expected.\n3. It is mentioned \"We fine-tune a ResNet50 for the CUB [40] and iWildCam [15] datasets and a ResNet18 for the Planes [21] dataset\". Why use different networks for different datasets?\n4. In Appendix, the generated images through img2img of Airbus and Boeing look very strange. Discussion regarding to these results are expected.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1. This paper is general clear and easy to follow\n2. The proposed method is simple and technically sounds\n3. The proposed method effective improves the classification performances over the reported three tasks", "weaknesses": "This paper studied the generated images through diffusion models on \"specialised\" classification tasks. However, generated images through diffusion models have been proved to be effective on improving \"general\" classification tasks [8, 35]. Therefore, the novelty and contribution of the proposed method seems to be incremental.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your review and you bring up important questions on how our work differs from the several works that explore using diffusion generated data to improve \"general\" classification tasks. We aimed to address this in the introduction of our paper and we hope to provide more clarity below. \n\n**Motivation for Image Editing/OOD data.** We agree that text2image models produce more diverse images than image editing methods like InstructPix2Pix, and thus in a setting where the training/test images are well represented in the training set of Stable Diffusion, using images generated from text alone is likely to result in bigger gains than edited data. \n\nWhile this is an exciting avenue to explore, we are interested in how well these text conditioned generative models can do when the task is out-of-domain; that is, when text to image models cannot produce images similar to those in the training set. In these settings, it is unclear if/how one can utilize these powerful models to improve accuracy. The key insight of our work is that grounding the augmentation in the training data itself is an effective method of utilizing the high domain-level knowledge of Stable Diffusion (e.g. background, weather, lighting, etc) to vary existing images, thus maintaining the class-relevant features. As shown qualitatively, images generated from text alone look dissimilar to real training images while image edits result in images that are less diverse but more similar to the types of images seen in this specialized task. On datasets of various sizes, we show quantitatively that this results in an augmentation technique that beats both traditional augmentation strategies as well as using Txt2Img for augmentation. \n\n**Why use ResNet18 for Airbus VS Boeing.**\nWe chose ResNet18 because the dataset is rather small and it made the experiments much less compute intensive. We will update the manuscript with the ResNet50 results for the camera ready. \n\n**The generated images for Airbus and Boeing look very strange.**\nOur hypothesis as to why img2img does not work well for the Aircraft dataset is because of the lack of patterns in the image. Many of the planes are displayed against a uniform blue background and are mostly white themselves. We saw a similar effect when trying to edit simple cartoons, or plane pictures taken randomly from Google. That being said, we predict that this problem will go away with better diffusion models or models trained on more images similar to the plane images. We will add this explanation to the appendix.\n"}, {"review_id": "FiXI5AVjJV", "reviewer": "Reviewer_za7y", "summary": "This paper explores improving robustness to variations in domains with the use of modern large vision and language models. They propose ALIA (Automated Language-guided Image Augmentation) which generates automated descriptors of data domains and augments training data with language-guided image editing. They use a model trained on the original dataset as a quality filter to ensure class-relevant information is maintained. They show this approach leads to nice gains on fine-grained datasets or data with significant clutter, for both classification and detection.\n\nAfter reading the authors rebuttal and discussing with them, I will maintain my score of 7.", "questions": "Could performance be further improved by including domain descriptors for test data (assuming access to test data at inference time, but not test labels) for cases with domain shift between train and test?\n\nIn some cases, context is highly important and valuable species identification information for human experts, and certain species would not ever been seen, eg, perched on a birdfeeder, out at night, in the water. Have you considered learning class-appropriate subsets of language templates per-species, to reduce potentially confusing and unrealistic generated context?\n\nI noticed that you considered only a subset of iWildCam classes and camera locations, what was the reasoning behind the choice of subsampling? Did you artificially balance the dataset at all (ie removing rare categories or capping common ones) or did you keep a realistic shift in subpopulation distribution across different domains? How were the real test-domain images sampled for comparison, uniformly across the test camera sites and categories of interest or?, Did you make sure that the test data was identical when comparing (ie remove that added test data from evaluation for all models)? It would be good to be very explicit about all of these choices whenever you use a non-standard split of a dataset \n\nDoes performance ranking across baselines change if we consider different metrics? For example, if we consider top-1 accuracy or break down performance per class on iWildCam instead of looking at the macro f1? Does this method improve more for rare classes or common ones?", "limitations": "The limitations section is well done, and points out several clear potential failure modes of this method. More detailed analysis of failure cases would be appreciated. For which classes does this method help, and for which does it hurt? Are there any consistent patterns in data that the model finds difficult to predict accurately, even after augmentation?  It would help the reader build additional intuition for where gaps remain and in which cases this method is ready for deployment.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "It’s clever to combine recent large models in this way, using image captioning to generate appropriate background or context caption templates across based on what is seen in the dataset, and allowing language-guided editing to provide diversity in a more realistic manner than randaug or cutmix based on those templates. In particular, the use of semantic features and visual features (via classifier confidence) to filter failures both where the image is only minimally changed or where the image is changed in a way that corrupts class information is quite nice. The experiments, across several datasets and dimensions of challenges including domain shift and contextual bias, were nicely done and visualizations were clear and informative.", "weaknesses": "Since the domain descriptions are build with the existing dataset, it seems that this method may not work as well for cases where the dataset in question is not representative of all potential domains seen in practice, or where the dataset has significant bias or gaps. Additionally, since the diverse augmented data is conditioned on training examples and the category in question often is minimally changed, there is still limited diversity for rare species with few examples during training, particularly rare pose diversity. It would be nice to explore this more deeply on a large-scale fine grained dataset with significant imbalance, such as iNaturalist.\n\nThe biggest weakness of this paper is lack of clarity and transparency about some of the data choices made. In particular, it wasn’t clear what protocols were used when selecting subsets of iWildCam for train and test.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your appreciation of our work! We agree that ALIA is not suitable for cases were there are unseen domains which are significantly different from the domains seen in training or cases where the class relevant features need to be significantly augmented (e.g. pose change). As we will detail below, our subset of iWildCam does include class imbalance, but since the animals are mostly well represented by Stable Diffusion, we are unsure of how well ALIA can handle class imbalance for rare species. Due to time constraints we were not able to test ALIA on iNaturalist, but we will strive to have results for the camera ready. \n\n**Including Domain Descriptors for Test Data.** Yes! For our iWildCam experiment, we did exactly this: took the unlabeled background images for the new domains and used those to generate the domain descriptions. This is an especially useful tool in cases of a large domain shift like sim to real transfer, and the user can also explore using their own constructed prompts if they anticipate a particular test domain but have no data for it. \n\n**Class-Specific Domain Descriptions.** This is another great idea that we have been exploring. For instance, one could generate domain descriptions per class and use these descriptions to generate diversity within a class, or inversely could use these to uncover potential biases in the dataset and direct the augmentation to a subset of classes. There are a lot of interesting avenues around selecting which images to augment with which prompts, and we see this as exciting future work. \n\n**iWildCam Subset Construction.**\nWe apologize for not being clearer on how and why this dataset was constructed. We chose to construct a subset of iWildCam because (1) the experimentation and generation time for the entre wilds dataset was prohibitively expensive and (2) some classes didn’t have enough examples for us to split into a train, val, test, and extra set. While we did not artificially balance the classes, we did want classes which would have at least 40 examples in each split. \n\nIn an effort to keep this constrained setting as close to the original iWildCam dataset as possible, we constructed the train/val/test/extra splits such that each split contains non-overlapping locations and we did not subsample within locations to balance the class support. \n\nSince iWildCam has a train, id_test, and test split already, we sampled the locations with at least two classes to put into our subset, ensuring that the train subset was sampled from the train set, the val subset was sampled from the id_test set. We split the sampled locations from the iWildCam test set into two disjoint groups, which formed the test set and extra set for our subset. These new splits were set before experimentation, so all methods were trained, validated, and evaluated on the same data. We have included a breakdown of the class counts of each split below. We will also include this description in our final manuscript. Please let us know if you have any further questions surrounding the construction of this dataset. \n\n| Split | Background | Cattle | Elephant | Impala | Zebra | Giraffe | Dik-Dik |\n| :---  |:----: |:---: | :----: | :---: | :---: | :----: | :---: |\n| Train | 2210 | 801 | 366 | 981 | 720 | 460 | 514 |\n| Val   | 2006 | 206 | 53 | 140 | 119 | 58  | 244 |\n| Test | 127 | 1416 | 2003 | 4553 | 144 | 47 | 213 |\n| Extra | 402 | 506 | 91 | 625 | 85 | 47 | 468 |\n\n**iWildCam Metrics.** Below we include the accuracy for our iWildCam experiments, which again shows that our method outperforms all baselines as well as real data. Since we do not add in the same number of images per class in our experiment, it would be hard to draw useful conclusions on which classes our method does well for, but our intuition is that the edited data for classes which stable diffusion cannot recreate well will likely be of lower quality even after filtering, and thus may result in worse performance. We will strive to provide further quantitive analysis on the failure modes of ALIA by the camera ready.\n\n| Dataset  | Accuracy          |\n| :---     |   :---:           |\n| Baseline | 67.51(6.15)       |\n| +CutMix  | 77.56(4.77)       |\n| +RandAug | 72.97(3.87)       |\n| +Txt2Img | 75.59(3.36)       |\n| +ALIA    | **84.87(1.92)**    |\n| +Real    | 74.10(3.37)       |\n"}, {"review_id": "ss4leMJQpZ", "reviewer": "Reviewer_p3WR", "summary": "The paper proposes a method to augment an existing vision dataset with samples that are likely reflective of task-relevant variations that are potentially missing in the same. To do this, the authors propose a method, ALIA (Automated Language-guided Image Augmentation), which utilizes off-the-shelf image-generation and language model pipelines to first generate diverse domain descriptions using a captioning + LLM pipeline which can then be used to edit an existing image to generate diverse variations of the same. Specifically, ALIA consists of three steps – (1) prompting an LLM to generate domain descriptions (likely capturing visual variations) based on a collection of generated captions associated with the original dataset, (2) using an image editing pipeline (Img2Img / Instruct Pix2Pix) to generate edits and (3) automated filtration steps to ensure the generated images and visually consistent and can reliably augment the existing dataset. From experiments conducted across three benchmarks – domain generalization (iWildCam), fine-grained classification (CUB) and classification in the presence of contextual bias (custom split of FGVC-Aircraft) – the authors show that ALIA guided expansion of the data distribution is most-effective in improving performance over a vanilla baseline and other prior augmentation strategies considered. Additional ablations outline the extent to which prompting, filtration and the editing mechanism impact performance.", "questions": "The points outlined under strengths and weaknesses influence my rating. Regarding weaknesses, my suggestions are intended more towards improving completeness and significance of the results presented in the current submission. Among these, I think (1), (3) and (4) are crucial weaknesses which are perhaps central to the claims of the paper. Addressing these would definitely help me in improving my rating of the paper.", "limitations": "The only potential negative societal impact that I foresee is that models trained on ALIA augmented datasets can “unintentionally” inherit biases present in the base models (LLM, Diffusion) in the ALIA pipeline. While this may not matter as much for the datasets being considered for experiments in the submission, it is perhaps crucial for other work attempting to build on top of ALIA for sensitive situations. Adding a discussion (with obvious disclaimers) would improve the current submission. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The following points outline the strengths associated with the submission.\n\n- The paper is generally well-written and easy to follow. The authors do a good job of motivating the base observations – (1) circumventing additional data curation by using generated images and (2) adopting conditional text-guided edits as a more structured way to generate samples to augment an existing dataset. The introductory section does a good job of outlining the necessity of individual components in ALIA – the necessity of invoking task-agnostic language descriptions to guide edits, avoiding fine-tuning of individual large-scale models and including an automated filtration step. \n- In my opinion, compared to prior work (as noted by the authors as well), the novelty of the proposed approach lies in automating the pipeline to generate augmented samples with minimal interventions. While it remains to be seen the extent to which “avoiding fine-tuning” will translate to more complex settings (images with multiple objects)  and tasks (structured prediction), ALIA seems like a novel step in this direction of expanding data distributions by ensuring broad coverage of variations likely to be seen in test-time settings (modulo the obvious caveats acknowledged by the authors in Section 6).\n- The proposed method seems to work and leads to improvements over a vanilla baseline, other diffusion-guided (conditional / unconditional) editing schemes and prior augmentation strategies, and is often competitive with an oracle setting where one has access to data from the test distribution. With the exception of the points raised under weaknesses, the significance of ALIA lies in the fact that it is a straightforward and timely combination of existing techniques in language modeling and image generation / editing / personalization, and works fairly well off-the-shelf across multiple settings. Additionally, I particularly like the filtration process – designed by first identifying potential failure modes and subsequent methods to counter those.\n- The ablations (coupled with underlying hypotheses wherever applicable) are useful and provide actionable insights about the extent to which components in ALIA are sensitive to the data and the model at hand – for instance, while InstructPix2Pix is more useful compared to Img2Img, in the contextual bias settings, it leads to artifacts for iWildCam edits. These observations are likely going to be useful for future work attempting to build on top of ALIA.\n", "weaknesses": "The following points outline the weaknesses associated with the submission. Most of these points are either associated with the significance and completeness towards the intended goal.\n\n- Given that the description of a “domain” is left somewhat open-ended in the current version (L83-85), the paper would benefit from including the discussion (supported perhaps by quantitative results) surrounding – (1) what kind of semantic / stylistic variations are missing in the base-dataset and (2) whether ALIA explicitly counters those scarcities by introducing relevant edits. While the examples provided for the contextual bias settings in Figure 7 help highlight how over-represented settings are pruned (and under-represented ones are highlighted), a discussion surrounding all different settings would significantly strengthen the submission.\n- Following up on the previous point, in addition to above, including an analysis informing the extent to which one needs to augment an existing dataset would be useful as well – is it necessary to consider an expansion of 20-100% in most settings? It may not matter as much for small-scale settings, but for someone intending to build on top of ALIA, it may be useful to know if one runs into a “diminishing return” scenario for different datasets at any point – i.e., if gains obtained by adding more diverse samples become increasingly marginal beyond a point.\n- L215-216 states that domain descriptions for the background images from the test-set were used to generate augmented samples in iWildCam. Coincidentally, iWildCam is the only setting where +ALIA > +Real. Since assuming access to the specific “kind” of target domain variations during training is not entirely fair (one does not know the test-time variation apriori), the paper would benefit from relaxing claims surrounding “beating real data” for this specific setting.\n- For the contextual bias experiments, it might be beneficial to consider settings where spurious correlations have been studied heavily (for instance, the NICO [A] benchmark). Not only would that involve a more diverse “base” dataset, but also help compare with more sophisticated algorithms (a subset of which are designed to explicitly counter spurious correlations). This is motivated from the fact that supplementing with “targeted” data may not always be the best solution in contextual bias settings.\n\n\n[A] – Towards Non-I.I.D. Image Classification: A Dataset and Baselines\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback and suggestions, we hope to have addressed each of your concerns with the following:\n\n**More clarity around domain shift.** We agree that more clarity around what each domain shift is, in addition to quantitative proof that ALIA improves accuracy under this domain shift, is crucial to showing the efficacy of our method. Below we describe the shifts in each dataset (if available) as well as proof that ALIA improves performance in each of these cases. Note that Cub2011 has no explicitly defined domain shift. \n* *iWildCam:* This dataset is constructed such that the locations of the camera trap differ from train to test; specifically, the locations containing (1) a lake in the distance or (2) a dirt trail with trees are two locations not present in the training set. A sample of images from these locations are included in the global rebuttal PDF. We see that our prompt generation technique does produce domain descriptions that describe the test domain, such as “a camera trap photo of a {} near a large body of water in the middle of a field.”, “a camera trap photo of an {} in a forest in the dark.”, and “a camera trap photo of a {} walking on a dirt trail with twigs and branches.” We further see in Figure 4 of the paper that when evaluating on these two locations, we get high performance compared to other approaches. \n* *Airbus VS Boeing:* The domain shift in this dataset is the existence of Airbus planes on grass and Boeing planes on road in the test set. More explicitly, examples considered in-domain are Airbus on road, Boeing on grass, Airbus in the sky, and Boeing in the sky, which appear in both the training and test set. An exact breakdown of the number of samples in each group are in the Appendix. As shown in the global response and PDF, our method is able to improve on the in-domain performance of all augmentation methods, while also beating the baseline and traditional data augmentation methods on the out-of-domain performance. \n* *Waterbirds:* as described in the global rebuttal, the domain shift here is the presence of Landbirds on Water and Waterbirds on Land in the test set. As shown in the global response and PDF, our method is able to roughly match the in-domain accuracy of the other methods while drastically improving the accuracy of all other augmentation methods. \n\n**Amount of Generated Data VS Accuracy.** Although we did not have time to investigate this for all datasets, we were able to experiment on how accuracy changes with the amount of generated data added on Cub2011. As shown in the results (contained in the global rebuttal PDF), ALIA is able to achieve accuracy gains up to 1000 images (20% of the original training set), at which point accuracy starts to decline. In contrast, images generated from text alone see a decrease in accuracy almost immediately, reaching below the accuracy of the original training set at 2000 images. We will work to get similar plots for the other datasets by the camera ready. \n\n**Claims of beating real data.** We agree that we should provide more context to the claim of ALIA beating real data, especially considering that we do have knowledge of the test domain. We will update our manuscript accordingly.\n\n**Spurious correlation datasets.** While we agree that NICO is a more diverse benchmark that simulates real world domain shifts, we chose not to use it in our evaluation because the images seemed as though they could be easily mimicked with Stable Diffusion. In these cases, the best approach would likely use Txt2Img data, as it is more diverse than image editing. "}, {"review_id": "GQzXqGLhM8", "reviewer": "Reviewer_JLbC", "summary": "The paper introduces ALIA to generate domain descriptions of the dataset from captions of each image. These captions are then used to generate more data using Stable Diffusion. There is also a filtering process to remove corrupted images or those with minimal edits. ALIA improves performance over the baselines on several datasets.", "questions": "Other than the questions above,\n- There are existing datasets with contextual bias e.g. Waterbirds, CelebA. I was wondering why the authors chose to create one with the Aircraft benchmark.\n- Is it interesting that img2img does not work well for the Aircraft dataset, do the authors have any intuition why?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper was well written and easy to follow.\n- Fine-grained classification is an interesting and difficult problem. It is also nice to see a different setting compared to existing papers trying to improve robustness on IN-Sketch/R etc. Furthermore, the method performs well compared to the baselines.\n- I like the discussion on filtering failed edits. It is important but usually not given alot of attention.", "weaknesses": "- Data and computational efficiency\n    - How many new images were generated for the baselines and ALIA? It would be useful to have a plot of how the performance of the methods changes with generated data and compute time. Augmentation methods like CutMix, RandAug have the advantage of being relatively efficient and more scalable than generation methods. This may matter if the difference in accuracy is not as large e.g. between RandAug and ALIA in Fig 5. \n- Background domain from test images\n    - ALIA seems to be informed of the test domain, at least for iWildCam, while the other baselines are not. Domain adaptation methods that makes use of unlabelled target images, or test time adaptation methods may be more relevant baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. We hope that we have adequately addressed your concerns below: \n\n**Data and Computational Efficiency.** We generate 2 edits per image per prompt in the training set for ALIA. Depending on the output of the LLM, the number of prompts vary from 4-10. Using the HuggingFace implementation of Img2Img editing on a GeForce RTX 2080 Ti, we can produce 2 edits of a given image in approximately 2 seconds, resulting in a generation time of around 15 minutes for the smallest dataset (Airbus VS Boeing) and around 7 hours for the largest dataset (iWildCam Subset). We will include the time it took to generate our datasets in the appendix. While our method does require a lengthy generation time if the dataset is large, our primary focus in this paper was to test the quality of our augmentations rather than the efficiency. \n\n**Background Domain from Test Images.** For iWildCam we do use unlabeled test images to generate the domain prompts for editing, but crucially we only use images from the background class, so we do not obtain any data of the animals in the test domain. While there do exist domain adaptation baselines which can make use of this test data, our paper is focused on data augmentation methods specifically.\n\n**Choice of Contextual Bias Benchmark.** We chose the Aircraft dataset because it is a challenging fine-grained dataset, with contextual bias that wasn't artificially created like Waterbirds. However, we do have results for the Waterbirds dataset, which are included in the global response and pdf. As you can see, ALIA outperforms all baselines aside from adding real data in the case of 100% contextual bias. \n\n**Img2Img Failure on Aircraft dataset.** Our hypothesis as to why img2img does not work well for the Aircraft dataset is because of the lack of patterns in the image. Many of the planes are displayed against a uniform blue background and are mostly white themselves. We saw a similar effect when trying to edit simple cartoons, or plane pictures taken randomly from Google. That being said, we predict that this problem will go away with better diffusion models or models trained on more images similar to the plane images. We will add this explanation to the appendix."}, {"review_id": "0CbWl7era4", "reviewer": "Reviewer_Aa4b", "summary": "The paper proposes a generative data augmentation method, ALIA, which utilizes large image captioning and language models to summarize the domain description and employs language-guided image editing methods to create augmented training data. Experiment results show that ALIA outperforms recent data augmentation methods on fine-grained classification tasks.", "questions": "- Q1. While being a generic data augmentation method, it seems that each tested dataset is comprised of similar objects, e.g., CUB contains birds, FGVC-Aircrafts contains aircraft and iWildCam contains animals. Does ALIA assume the dataset classes to be similar? On a more diverse dataset like ImageNet, can the large language model summarize good domain descriptions and create valid edits?\n\n- Q2. In line 118. The authors constrain the number of domain descriptions to be less than 10. Can the authors explain more about the choice of the number of domain descriptions? How does it affect the quality of the augmented samples?\n\n- Q3. In lines 191-192. The authors mention generating twice as much data as the original data to ensure “enough” data is available after filtering. How do we know whether the amount of filtered data is “enough” or not? Should we generate more data if the original dataset size is smaller?\n", "limitations": "I find no negative societal impact in this work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- S1. The paper is well-written and easy to follow.\n- S2. The idea of using prompts to extract domain information and class-agnostic descriptions is novel.\n- S3. The empirical result of the iWildCam experiment is promising. It is impressive that the task model trained on additional ALIA-generated images outperforms that uses real samples.\n", "weaknesses": "- W1. Limited technical contribution. Although the idea of using prompts to extract domain information and class-agnostic descriptions is novel, most building parts of ALIA are off-the-shelf methods, for example, BLIP, GPT-4, Img2Img and Instruct Pix2Pix. \n\n- W2. The proposed method relies on pre-trained models heavily. As mentioned in the limitation section, ALIA is likely not as effective for unseen test domains. If ALIA performs poorly on a target dataset, no feedback can be sent to fine-tune ALIA on the dataset.\n\n- W3. ALIA is only tested on three datasets, where the dataset classes are mostly similar (see Q1).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and agree that our method is dependent on the ability of the pretrained models we use. We address your other questions and concerns below. \n\n**Datasets.** Our datasets consist of similar objects because we focused on the difficult setting of fine-grained classification. This choice was to see if diffusion could maintain the fine grained features of the class while changing the spurious features. That being said, ALIA does not need the classes to be similar, and could be applied to improve performance on datasets like ImageNet. Furthermore, we agree that more datasets are better and thus we added in an additional contextual bias dataset Waterbirds, which shows that ALIA outperforms everything but the real data baseline. \n\n**Choice of number of domain descriptions.** Our choice of less than 10 domain descriptions was due to compute constraints, since we generate two edits for each image in the dataset for each prompt. While the number of domain descriptions doesn't affect the quality of the augmented samples, having a really small number of prompts could potentially affect the diversity of the resulting augmented set.\n\n**Amount of data to generate.** The choice to generate twice as much data as the dataset (2 edits per image) is largely due to the real data baseline. Because we set aside real data to compare to (+Real), we needed all generated data techniques to add in the same amount of data per class as this real data baseline. In practice one could edit a small percentage of their dataset and see what percentage of images were filtered out, then use that to determine roughly how many images to edit in order for the filtered dataset to be of the desired size. For example, if one wanted to add in 10 samples per class and profiling on a small generated dataset shows that around 50% of the images get filtered out, at least 20 images should be edited per class. \n\n"}]

Author rebuttal:
We thank all reviewers for their detailed reviews. We are delighted that they found the paper easy to follow and saw value in our discussions surrounding failed edits and different data augmentation techniques. 

As requested, we have run a few more experiments to ensure a comprehensive evaluation. We have plotted the resulting data tables and included all plots in the attached PDF. 


### Waterbirds [Contextual Bias]

Waterbirds[1] is a synthetically created dataset which creates contextual bias by taking species of landbirds and waterbirds from the CUB-200[2] dataset and pasting them onto forest and water backgrounds from the Places[3] dataset. For the training and validation sets, all landbirds appear on forest backgrounds and all waterbirds appear on water backgrounds, while the test set has an even representation of backgrounds and bird types. Further experimental details are contained in the PDF.

As shown below and in Figure 1 of the PDF, ALIA is able to roughly match the in-domain accuracy of the other methods while drastically improving the accuracy of all other augmentation methods. Note that we don't bold the +Real numbers in the table because it is considered an oracle baseline. 

| Dataset | ID Accuracy | OOD Accuracy | Class Balanced Accuracy |
| :---        |   :----:   |   :---: | :----:   |   
| Baseline      | 97.34(0.23) | 27.14(3.06) | 62.24(1.58) |
| +CutMix   | **98.01(0.33)** | 28.84(5.29) | 63.42(2.49)  |
| +RandAug | 97.61(0.51)  | 30.32(6.31) | 30.32(6.31) |
| +Txt2Img | 97.61(0.11) | 29.66(3.81) | 63.64(1.92) |
| +ALIA | 96.17(0.47) | **46.63(3.96)** | **71.40(1.84)** |
| +Real | 91.45(0.45) | 89.40(0.49) | 90.43(0.47) |

[1] Sagawa, et al. "Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization" 

[2] Wah, et al. "The Caltech-UCSD Birds-200-2011 dataset"

[3] Zhou, et al. "Places: A 10 million image database for scene recognition" 



### Number of generated images VS accuracy on Cub2011

While our main experiments restrict the number of generated images added to the dataset, one of the benefits of diffusion generated augmentation is that one can create infinite amounts of data. Figure 2 in the PDF depicts the accuracy on CUB as a function of the number of generated images added (200/400/1000/2000/4000), where the grey line is the baseline accuracy on the original 4994 images. 

ALIA is able to achieve accuracy gains up to 1000 images (20% of the original training set), at which point accuracy starts to decline. In contrast, images generated from text alone see a decrease in accuracy almost immediately, reaching below the accuracy of the original training set at 2000 images. We suspect this is because much of the text to image data is from a different distribution than the regular training data, and thus adding small amounts can increase robustness while large amounts cause the model to overfit to this distribution. Since image to image edits use language to edit the training images directly, these images are less likely to be out of distribution as compared to the text to image data. 

### Airbus VS Boeing ID/OOD accuracy breakdown

In order to show that ALIA does improve accuracy on the unseen domain, we break down the results from Figure 6 of the main paper into in-domain accuracy and out of domain accuracy. Note that we don't bold the +Real numbers in the table because it is considered an oracle baseline. 

As shown below as well as in Figure 3 of the PDF, ALIA is able to improve on the in-domain performance of all augmentation methods while beating the baseline and traditional data augmentation methods on the out-of-domain performance.

| Method   | ID Accuracy        | OOD Accuracy       | Class Balanced Accuracy |
| :---     |   :---:            |   :---:            |   :---:                  |
| Baseline | 85.79(1.99)        | 31.68(0.60)        | 64.64(0.75)              |
| +CutMix  | 85.96(1.30)        | 32.11(1.50)        | 64.96(1.35)              |
| +RandAug | 87.23(1.07)        | 30.17(0.09)        | 64.87(0.65)              |
| +Txt2Img | 81.52(1.45)        | **41.62(0.21)**   | 65.85(1.12)              |
| +ALIA    | **88.11(0.84)**       | 36.20(0.36)        | **68.84(0.76)**              |
| +Real    | 87.07(1.16)        | 44.34(2.25)        | 71.81(1.29)              |


Meta-review: The reviewers did not reach a full consensus. However, the main criticism of the rejecting reviewer is that it is already known diffusion models can generate images which help training. Indeed, this is shown as a baseline in the paper and the paper proposes editing training images instead. The AC finds this different and interesting enough to warrant publication.
Decision: Accept (poster)