SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 6: Waterbirds Results. Left plot shows class-balanced accuracy for ALIA and baselines. Right plot shows the breakdown of in-domain (waterbirds on water, landbirds on land) and out-of-domain performance (waterbirds on land, landbirds on water). ALIA is able to roughly match in-domain accuracy of the other methods while drastically improving out-of-domain accuracy of all other augmentation methods. This results in  $7\%$  improvement for class-balanced accuracy when compared to other augmentation baselines.
The prompts generated by ALIA are "a photo of a  $\{\}...$
(1) "in a bamboo forest with a green background.", (2) "flying over the water with a city skyline in the background.", (3) "perched on a car window.", (4) "standing in the snow in a forest.", (5) "standing on a tree stump in the woods.", (6) "swimming in a lake with mountains in the background.", (7) "standing on the beach looking up."
We use "an iNaturalist photo of a  $\{\}$  in nature." as the prompt for the  $+Txt2Img$  baseline.
As shown in Figure 6, ALIA is able to approximately match the in-domain accuracy of the other augmentation methods while outperforming all other methods besides the real data baseline for overall accuracy. While ALIA's performance on this dataset is much lower than the state of the art methods, we believe that these results point to a promising future direction in how to modify ALIA to identify and amplify specific class-domain pairings which are underrepresented in the training set.
5.4 Ablations
Effect of Prompt Quality and Filtering. We ablate the quality of ALIA generated prompts by comparing the edit prompts generated by ALIA against those provided by users. For the user-supplied prompts, we draw on the engineered prompts that were used for the  $+Txt2Img$  baseline. As shown in Table 1, descriptions generated by ALIA outperform user-provided prompts, especially in the contextual bias setting, indicating that our generated prompts are able to accurately describe key domain-specific features and variations, resulting in more effective data augmentation for mitigating biases and improving model generalization. Moreover, Table 1 also demonstrates how our semantic and confidence-based filtering further improves accuracy.
Dataset	User Prompt	ALIA Prompts	ALIA Prompts + Filtering
iWildCam	79.92±4.22%	82.57±2.19%	84.87±1.92%
CUB	71.02±0.47%	71.25±0.86%	72.70±0.10%
Waterbirds	63.64 ±1.43%	70.22±0.53%	71.40±1.85%
Table 1: Effect of ALIA Generated Prompts and Filtering.
Choice of Image Editing Method. As mentioned in Section 5.3, the suitability of image editing methods can vary based on the specific dataset. As such, we explore the impact of the editing method on performance by modifying the iWildCam subset from Section 5.1 with InstructPix2Pix. We employ the same prompts as in the Img2Img case, reformatted to the instructional syntax required by InstructPix2Pix (e.g. "a camera trap photo of a {} in a grassy field with trees and bushes" becomes "put the {} in a grassy field with trees and bushes"). Figure 7 presents examples of edits generated using InstructPix2Pix. Although InstructPix2Pix can produce edits with more significant color variations than Img2Img, it frequently applies an unrealistic airbrush-like effect or erases most of the information present in the original image. This outcome can be attributed to the fact that InstructPix2Pix is fine-tuned on a dataset of paired images and edits, which primarily consist of stylistic changes to artistic images, making the iWildCam images fall outside their domain. When comparing the performance of InstructPix2Pix, we find that it achieves an accuracy of  $76.23 \pm 4.78\%$  compared to  $84.87 \pm 1.92\%$  obtained using Img2Img edits. These results indicate that the choice of image editing method has significant impact on the final performance of ALIA, and thus it is crucial to profile the quality of image edits from various techniques on one's specific dataset.
"put the {} near a large body of water in the middle of a field."
Figure 7: InstructPix2Pix edits on iWildCam. Despite offering more color variation, the method often imparts an artificial airbrush-like effect or removes substantial original image details.
Amount of Augmented Data Added. Figure 8 depicts the accuracy on CUB as a function of the number of generated images added where the grey line is the baseline accuracy. ALIA is able to achieve accuracy gains up to 1000 images (20% of the original training set), at which point accuracy starts to decline. In contrast, images generated from text alone see a decrease in accuracy almost immediately, reaching below the accuracy of the original training set at 2000 images. We suspect this is because much of the text to image data is from a different distribution than the regular training data, and thus adding small amounts can increase robustness while large amounts cause the model to overfit to this distribution. Since image to image edits use language to edit the training images directly, these images are less likely to be out of distribution as compared to the text to image data.
Figure 8: Size of Augmented Dataset (CUB). Results are shown for adding 200, 400, 1000, 2000, and 4000 generated images to the training set. The grey line depicts the baseline accuracy on the original 4994 images. ALIA is able to achieve accuracy gains up to 1000 images, at which point accuracy starts to decline. In contrast, images generated from text alone see a decrease in accuracy almost immediately, reaching below the accuracy of the original training set.
While ALIA is able to generate impressive augmentations, there are limitations. As ALIA depends on large pretrained vision and language models to translate between task-specific image data and task-agnostic natural language descriptions of domains, performance of the method is bottlenecked by the quality of the captioning model, LLM, and image editing method (see Section 5.4). Furthermore, as we assume that the task-specific information in the training data cannot be easily generated via text alone, attempts to change aspects like the pose of a bird in CUB is likely to result in a failed edit. Finally, determining the optimal quantity of augmented data to reincorporate into the training set remains an unresolved question, and is an area we look forward to addressing in future research.
We present ALIA, a novel approach to data augmentation that leverages the high-level domain knowledge of large language models and text-conditioned image editing methods. By grounding both the domain descriptions and the augmented training data in the provided training set, our method has demonstrated impressive capabilities in several challenging settings, including domain adaptation, bias mitigation, and even scenarios without domain shift. Our findings offer exciting potential for future research, suggesting that the utilization of structured prompts generated from actual training data can be an effective strategy for improving dataset diversity. As the capabilities of captioning, LLMs, and image editing methods grow, we expect the efficacy and scope of our approach to increase.
Acknowledgements. We thank Suzie Petryk for her invaluable feedback on the manuscript. This work was supported in part by the NSF CISE Expeditions Award (CCF-1730628), DARPA's SemaFor, PTG and/or LwLL programs, and BAIR's industrial alliance programs.