ASPIRE: Language-Guided Data Augmentation for Improving Robustness Against Spurious Correlations (2024)

Abstract: Neural image classifiers can often learn to make predictions by overly relying on nonpredictive features that are spuriously correlated with the class labels in the training data. This leads to poor performance in real-world atypical scenarios where such features are absent. This paper presents ASPIRE (Languageguided Data Augmentation for SPurIous correlation REmoval), a simple yet effective solution for supplementing the training dataset with images without spurious features, for robust learning against spurious correlations via better generalization. ASPIRE, guided by language at various steps, can generate non-spurious images without requiring any group labeling or existing non-spurious images in the training set. Precisely, we employ LLMs to first extract foreground and background features from textual descriptions of an image, followed by advanced language-guided image editing to discover the features that are spuriously correlated with the class label. Finally, we personalize a text-toimage generation model using the edited images to generate diverse in-domain images without spurious features. ASPIRE is complementary to all prior robust training methods in literature, and we demonstrate its effectiveness across 4 datasets and 9 baselines and show that ASPIRE improves the worst-group classification accuracy of prior methods by 1% 38%. We also contribute a novel test set for the challenging Hard ImageNet dataset .

heading: 1 Introduction
text: Spurious correlations are unintended associations or biases learned by models between the input image and the target label, often resulting from factors like data selection biases (Torralba and Efros, 2011; Jabri et al., 2016). The repeated co-occurrence of
1Code and data: https://github.com/Sreyan88/ASPIRE ∗Equal Technical Contribution.
certain features (like foreground objects or backgrounds), with a more than average chance, within instances of a particular class leads the model to learn shortcuts and focus on these spurious nonpredictive features for prediction than core ones. For example, most of the images in ImageNet dataset (Deng et al., 2009) labeled as Dog Sled also show a dog, and image classifiers trained on ImageNet fail to correctly identify an image of a dog sled without a dog in it.
Instances of a class in the training set where the co-occurring spurious features are present are commonly known as majority groups, while atypical instances where such spurious features are absent are known as minority groups. Deep neural networks trained on these datasets poorly generalize on minority groups (naturally due to their scarcity) and thus can exhibit significant performance degradation on minority groups in the test (Sagawa et al., 2019), or in real-world scenarios when encountering domain shift (Arjovsky et al., 2019). When training over-parameterized deep neural networks, there are multiple solutions with the same loss values at any given training stage, and the optimizer usually gravitates towards a solution with lesser complexity (or tends to learn a shortcut) (Wilson
386
et al., 2017; Valle-Perez et al., 2018; Arpit et al., 2017; Kalimeris et al., 2019). When faced with co-occurring spurious features, the optimizer may preferentially utilize them, as they often require less complexity than the anticipated semantic signals of interest (Bruna and Mallat, 2013; Bruna et al., 2015; Brendel and Bethge, 2019; Khani and Liang, 2021). Even powerful classifiers like CLIP and ViT undergo a significant drop in performance when exposed to minority group images in the test (Yang et al., 2023; Kirichenko et al., 2023). Teaching meaningful data representations to deep neural networks that yield good performance on a target downstream task while avoiding over-reliance on spurious features remains a central challenge in CV.
Motivation. Learning classifiers robust to spurious correlations is an active area of research (Sagawa* et al., 2020; Liu et al., 2021a; Kirichenko et al., 2023), and has the potential to improve various Computer Vision (CV) applications such as visual question-answering (Liu et al., 2023d), retrieval (Kong et al., 2023; Kim et al., 2023), classification (Liu et al., 2021a), etc. have shown to consistently In prior work, researchers generally employed different learning techniques with the assumption that annotated data for the minority groups existed in the training dataset. Most of these works are built on the same base principle: improved generalization on minority groups can lead to a more robust classifier. Despite extensive research in deep learning indicating that more data may lead to better generalization, little effort has been made to leverage this principle specifically for building robust classifiers. Additionally, we argue that it is impractical to manually collect and label minority group images for real-world, large-scale datasets. For example, in more complex datasets like the Hard ImageNet, beyond the commonly evaluated CelebA (Liu et al., 2015) and Waterbirds (Welinder et al., 2010), a single class of images may have multiple spuriously correlated features. Thus, identifying all such features through human perception to collect and label minority group images is a difficult task.
Main Contributions. In this paper, we present ASPIRE, a novel technique to augment existing image classification datasets with diverse non-spurious images for building robust image classifiers. Intuitively, our solution exploits the fact that more data can lead to better generalization on minority groups (Sagawa et al., 2020; Liu et al., 2021b).
Guided by language, ASPIRE does not depend on any additional image annotations or humanlabeled non-spurious data and only requires a training dataset and a standard model trained using Empirical Risk Minimization (ERM) to identify most of the spurious correlated features for each class in the training dataset. ASPIRE first selects a small portion of the total instances in the training set, misclassified by a classifier after ERM training. These selected images are then captioned, and an LLM extracts the tokens from the caption that describe the foreground objects and background. This is followed by editing the image using advanced language-guided image editing pipelines to remove or replace one object at a time and predicting the class of the edited image using the standard ERM classifier. We attribute the objects or background features that lead to the highest miss-prediction (due to its absence) as plausible spurious correlations learned by the model. Finally, we personalize a diffusion model on the edited images to generate diverse in-domain synthetic images for each class with our desired features, i.e., without the plausible spurious correlations detected by ASPIRE. To summarize, our main contributions are as follows:
• We propose ASPIRE, a method to expand existing datasets with non-spurious images to build more robust image classifiers. ASPIRE is dataset-agnostic (works with any dataset with one or multiple spuriously correlated features per class), training-method agnostic (complements all other methodologies proposed in prior work), and does not need any additional labeled supervision of spurious features or non-spurious images.
• We extensively evaluate ASPIRE on 4 datasets and 9 baselines and show that augmentations generated by ASPIRE improve the worstgroup accuracy of all baselines. Additionally, we perform extensive qualitative analysis to prove the effectiveness of ASPIRE.
• We contribute a novel test set for the Hard ImageNet dataset (Moayeri et al., 2022) equally balanced with spurious and non-spurious images to promote research in this space.

heading: 2 Methodology
text: Preliminaries. In this section, we provide an overview of our proposed approach. Fig. 2 pictorially describes the various steps in ASPIRE.
1 We first train a base classifier E using ERM on the entire training set and extract images with features that are spuriously correlated to construct Dhold. 2 We caption each image in Dhold. 3 We feed the caption to a LLM and extract the foreground objects and background for each image. 4a We remove one foreground object at a time and predict the class of the edited image E . If E predicts incorrectly, we consider the object as a plausible spurious correlation learned by E for that class. 4b We edit the image to change its original background with an alternative background suggested by the LLM and follow the process to similar to 4a . 5 We personalize a text-to-image diffusion model using edited images from the previous step for the top-k unique items leading to the highest number of wrong predictions. 6 We re-train E using the generated augmentations to obtain Ê .
Let’s assume we have a training dataset Dtrain = {xi,yi}, where every group of images belonging to a particular class predominantly has images with co-occurring spurious features, also known as the majority group. Dtrain might have a much smaller number of non-spurious images, or might not, which is also known as the minority group. We do not assume our training dataset to have any group labeling or additional supervision, like labeling for spurious objects. We also have a test dataset Dtest = {xi,yi} where both groups are represented equally. Additionally, we have a model E trained on Dtrain, using naive Empirical Risk Minimization (ERM). Thus, E would already identify the majority group images in Dtest with remarkable accuracy; our primary objective is to improve the performance of the classifier on the minority group without hurting the model’s overall performance. The next subsections describe each step in detail. (1) Extracting Dhold using E . We use E to extract a small hold-out set from Dtrain, which we denote as Dhold. Dhold should consist of images with spurious correlations in the train (or the majority group), which we will use in our later stages to detect the specific features that are the spurious correlations. Precisely, we first identify training examples that are correctly classified by a standard ERM model and then randomly select p% of them for constructing Dhold. We are inspired by prior work in this space and build on the heuristic that
a well-trained classifier tends to have low majority group loss (and subsequently high majority group accuracy) (Liu et al., 2021a; Nam et al., 2020).
(2) Image Captioning on Dhold. As mentioned earlier, ASPIRE depends on language guidance to achieve its primary objective of generating synthetic, non-spurious images. Thus, in this step, we generate a textual description of each image in Dhold, which can capture foreground and background information in the image. To accomplish this, we use a state-of-the-art image captioning model, GIT (Wang et al., 2022). We expect our image description to include information about most of the visible foreground objects and the predominant background, and we found captions generated by GIT to meet these requirements and not suffer from spurious correlations themselves. As captioning tools get better, we acknowledge that replacing GIT with its improved counterparts will improve the performance of ASPIRE even further.
(3) Extracting objects and backgrounds from captions. After captioning, we use LLMs to extract the phrases in the caption that correspond to foreground and background objects and the single predominant background. We assume our search space for identifying spurious correlations to be bounded within them, which is a reasonable assumption in most real-world cases and also in line with most prior work in this space (Joshi et al.,
2023). Recent LLMs have been shown to possess superior reasoning abilities, and we employ GPT-4 for our task (OpenAI, 2023). LLaMa-2 70B (Touvron et al., 2023) also proved to be competitive in this task. However, GPT-4 made fewer mistakes. An example of the input and output of this step of ASPIRE is as follows: Original Caption: “A man with two dogs and a sled in the snow.", Original Label: “Dog Sled". Output: foreground: [“man",“dogs"], background:[“snow"], alternate background:[“desert"]. For simplicity, let us denote the list of identified foreground and background objects for image xi as Fi and the predominant background as Bi (more about the alternate background in Step 4.b.). The task of extracting objects and backgrounds from text captions is effectively an information extraction task that involves understanding the structure of the sentence and the relationship between the words, and we found LLMs to deal better with anomalies and outof-distribution text scenarios than traditional NLP methodologies (algorithmic details about the traditional NLP method can also be found in Appendix A.2). Additionally, we want the identified objects or background to ignore the actual class label. This is crucial for the ASPIRE pipeline, as we do not want to edit the core feature in the image (discussed in detail in the next subsections). This can also be challenging as the class label may or may not exactly appear in the caption. However, we found LLMs to complete this task with remarkable accuracy and ASPIRE to be able to handle minor errors (due to top-k selection described later in this section). We use a single generic prompt with annotated exemplars for all datasets, which can be found in Appendix A.6.
(4.a.) Identifying spurious foreground objects. The primary objective of this step is to identify one or several unique features per class that are plausible spurious correlations. To achieve this, we build on recent advancements in language-guided image in-painting to remove one object at a time identified in Fi followed by allowing E to predict the class of the edited image. If E predicts the image correctly, we do not do anything with that image. If E predicts an image incorrectly, we identify the removed object as a plausible spurious correlation for the class c in the dataset and add the image to a set Dsynth (which we later use to personalize textto-image generation). Additionally, we add the text phrase of the spurious object to another set Tsynth.
Precisely, for every fore-ground object f in Fi, we first localize the object using Grounding DINO (Liu et al., 2023c), which takes as input the text phrase of f identified from the caption and outputs a bounding box bb for f . This is followed by extracting the segmentation map M for f using Segment Anything (Kirillov et al., 2023), which accepts bb as the segmentation prompt. M is then used to remove f from xi using LaMa image inpainting (Suvorov et al., 2022). For detailed information on the workings of Grounding DINO, Segment Anything, and LaMa, we request our readers refer to the original paper.
(4.b.) Identifying spurious backgrounds. The primary objective of this step is to identify if the predominant background of the image xi serves as a spurious correlation for the particular class c of images in the dataset to which xi belongs. Following a hypothesis similar to (4.a.), we assume that if removing the background b in Bi from xi can lead E to a wrong prediction, b can be a plausible spurious correlation. However, removing the background altogether (and keeping just the foreground items) not only disrupts the image semantics but is also not representative of real-world cases. Prior work also shows that removing the background forces the model to pay attention to foreground objects (Kirichenko et al., 2023) which are not suitable for our use case. Thus, we employ recent advances in instruction-based image editing to achieve this task. We first leverage the superior reasoning abilities of an LLM to suggest an alternate contrasting background b̃ for the image from its caption. Next, we instruct InstructPix2Pix (Brooks et al., 2023) to convert the background of xi from b to b̃. Similar to the previous step, if E predicts the image correctly, we do not do anything with that image. However, if E predicts an image incorrectly, we identify the original background as a plausible spurious correlation and add the edited image to Dsynth while we add the original text phrase of the background from the caption to Tsynth. (5) Non-spurious augmentation generation. The primary objective of this step is to generate indomain non-spurious images Daug for every class in the dataset Dtrain. These generated augmentations can then be used to supplement the training dataset Dtrain followed by re-training E to reduce its reliance on the spurious correlations. Generating in-domain augmentations without non-spurious features is crucial to the success of our approach
as out-of-distribution samples may adversely affect model performance (Trabucco et al., 2023). The most trivial approach would be to generate Daug by prompting any open-source text-to-image model. However, there exist two primary roadblocks to this approach: (1) Open-source diffusion models trained on internet-scale data generate diverse images for diverse prompts. Thus, prompting these models does not confirm the consistency of generations with the underlying distribution. (2) These models also posses spurious correlations or biases themselves (Trabucco et al., 2023). For example, prompting Stable Diffusion with the prompt: “picture of a dog sled” generates dog sleds with dogs most of the time. Attaching negative words with the prompts (like “picture of a dog sled without a dog”) often leads to the same spurious images as Stable Diffusion (and most image generation models) are known to not adhere to such negation in prompts (Tong et al., 2024).
To overcome the aforementioned problems and generate in-domain images with the desired nonspurious features, we resort to personalizing a textto-image generation model. Specifically, we train Stable Diffusion using textual-inversion (Gal et al., 2023) with samples from top-k phrases in Tsynth, and their corresponding images in Dsynth. Textualinversion effectively learns concepts and style from a small set of images for each class in Dsynth by just fine-tuning a single token in the embedding layer (which in our case is just the original class label) without over-fitting the generation model. Dsynth is the perfect candidate for extracting this small set as it contains non-spurious images, i.e., images without spurious features and concepts. Finally, we prompt the fine-tuned model to generate n× diverse samples for Daug. Top-k selection. Recall that Dsynth and their corresponding text phrases in Tsynth represent all wrongly predicted edited instances, i.e., they have a diverse set of foreground objects and backgrounds for each class. Thus, we attribute only the top-k unique items in Tsynth with the highest frequencies as the spurious correlation associated with that class and use images from only the top-k items for diffusion personalization. However, due to diversity in generated captions, text phrases corresponding to the same type of objects and backgrounds may be represented in Tsynth in diverse forms, for e.g., [“dogs”,“dog”,“two dogs”, ⋯]. Thus, before selecting the top-k items, we first collapse all the
similar phrases to one by first finding the root for all phrases in Tsynth by stemming and then calculating the cosine similarity between the glove embedding of the roots (to account for dissimilar roots, for e.g., “snow” and “snowy mountain”). Items with a cosine similarity of ≥ 0.90 are collapsed into one. (6) Re-training the base classifier E . Once we have generated Daug, we add the generated images to the existing Dtrain to re-train our standard classifier E . As mentioned earlier, the ASPIRE augmentation methodology is training-method-agnostic, and the augmentations generated can be coupled with any existing training approach from literature. The next Section describes how we add ASPIRE augmentations to our baseline training pipelines.

heading: 3 Experimental Setup
text: Datasets. To evaluate the effectiveness of ASPIRE, we experiment on 4 benchmark datasets, including Waterbirds (Sagawa et al., 2019), CelebA (Liu et al., 2015), SPUCO Dogs (Joshi et al., 2023) and Hard ImageNet (Moayeri et al., 2022). The Waterbirds dataset, generated synthetically by combining images of birds from the CUB dataset (Wah et al., 2011) and backgrounds from the Places dataset (Zhou et al., 2017), has 4 groups of images in training and testing datasets including waterbirds on water background, waterbirds on land background, landbirds on water background and landbirds on land background. The minority groups for the dataset (groups with the least number of samples in the training set) are waterbirds on land and landbirds on water. The main challenge is correctly identifying the minority groups in the test. For CelebA, we perform the hair color prediction task, which has 4 groups of images, including blond and non-blond males and blond and non-blond females. The minority group is blond males. SPUCO Dogs has 4 groups of images, including big dogs in indoor and outdoor settings and small dogs in indoor and outdoor settings. The minority groups are big dogs indoors and small dogs outdoors. The Hard ImageNet dataset has images from 15 ImageNet synsets and is more complex than the other 3 datasets, does not have group labeling, and has multiple spurious correlations for each class. For more details, we request our readers to refer to Moayeri et al. (2022). Since the dataset does not have a test set, we contribute a novel expert-annotated test dataset with 25 spurious and 25 non-spurious images per class. The spurious and non-spurious
features for each class were inspired by the original paper. More details about dataset statistics and annotation can be found in Appendix A.1.
Baselines. To prove the efficacy of ASPIRE augmentations, we add ASPIRE augmentations to the original training pipeline for various robust training methods proposed in literature. Precisely, we employ Group DRO (Sagawa et al., 2019), SUBG (Idrissi et al., 2022), Just Train Twice (JTT), Learning from Failure (LfF) (Nam et al., 2020), Correct-n-Contrast (CnC) (Zhang et al., 2022), Deep Feature Reweighting (DFR). (Kirichenko et al., 2023) and MaskTune (Asgari et al., 2022), To this list, we add the standard Empirical Risk Minimization (ERM) baseline, trained using SGD without any additional modifications. Additionally, we compare ASPIRE augmentations with augmentations generated using the methods proposed by Gowal et al. (2021) and Azizi et al. (2023). More details on baselines and how ASPIRE augmentations were added for training can be found in Appendix A.4. We do not experiment with LVLMs like LLaVa (Liu et al., 2023a) as there is no simple method to fine-tune them for robustness against spurious correlations proposed in literature. AS-
PIRE is only meant to complement methods proposed on the standard framework of fine-tuning vision encoders for image classification Hyper-parameters. For training the base ERM model, we train the model for 100 epochs with a learning rate of 1e−3 using the SGD optimizer with a weight decay of 1e−4. For training all other baselines, we use the original hyper-parameter settings proposed by the authors in their original paper. This includes the seed settings and the number of runs for every model. We use just 1 × augmentations of non-spurious images. Though this is possible for us as all our current datasets are also annotated with group labels, the number of ASPIRE augmentations to be added can be decided using hyper-parameter search, and we noticed no signs of over-fitting till 3× augmentations (see Appendix 1). For top-k, we resort to k=3 post a hyper-parameter
search among k={1,2,3,4,5}. k=3 seemed to capture the most major spurious correlations while ignoring the minor ones. Examples of extracted top-k can be found in Figure 4 and Appendix 1. For prompting InstructPix2Pix, we use Text CFG=7.5 and Image CFG=1.5. Prompt in Appendix A.6.

heading: 4 Results and Analysis

heading: 4.1 Quantitative Analysis
text: Table 1 compares the results of 9 baselines trained with and without ASPIRE augmentations. Worstgroup accuracy corresponds to the accuracy of minority groups (or non-spurious images) in the test. As we clearly see, with just 1× augmentations, ASPIRE improves the average accuracy of our baselines by 0.1%-22.2% and the worst-group accuracy of our baselines by 1.2%-37.5%. ASPIRE consistently achieves higher gains in worst-group accuracy and only undergoes a slight drop in average accuracy in some settings, which is in line with prior art and our primary motivation of improving robustness against spurious correlations. We notice the highest gains in Hard ImageNet, a fundamentally more difficult dataset with no minority group images in the training dataset and multiple spurious correlations per class. Our standard ERM model also witnesses the highest gains among all other baselines. On average, our 2-stage training baselines improve by a higher margin on average than 1-stage baselines due to improved explicit generalization over ASPIRE augmentations. The method proposed by Gowal et al. (2021) consistently underperforms ASPIRE, thereby highlighting that explicitly removing spurious features in the generated dataset improves robustness. On the other hand, the method proposed by Azizi et al. (2023) significantly underperforms ASPIRE in worst group accuracy but outperforms in average accuracy in some settings. This is due to the fact that standard data augmentation amplifies spurious correlations already present in the training set as it generates images with similar features to those on which it is conditioned.
Ablations. Table 2 removes certain key components in the ASPIRE pipeline to prove their efficacy. As we can see, the ASPIRE performance decreases significantly when the image generation step is removed (only edited images are used to train the robust classifier). Additionally, ASPIRE undergoes a sharper drop in performance when foreground identification is removed than the back-
(a) Without augmentations. (b) With augmentations.
(c) Without augmentations. (d) With augmentations.
Figure 3: GradCAM visualizations of the features used by the standard ERM model trained with and w/o ASPIRE augmentations on the Hard ImageNet dataset (Balance beam top and Volleyball bottom). As clearly visible, when trained with ASPIRE augmentations, the model tends to focus better on core features than spurious ones (more in Appendix B).
ground, which we attribute to the design of the test set minority groups of existing datasets.

heading: 4.2 Qualitative Analysis
text: Fig. 3 illustrates the GradCAM visualizations of the features used by the standard ERM model trained with and w/o ASPIRE augmentations for two classes from the Hard ImageNet dataset, Volleyball and Horizontal Bar. When trained with ASPIRE augmentations, the model tends to focus better on core features corresponding to the actual class than spurious ones. Fig. 4 illustrates examples of original images, edited images (edited by the ASPIRE pipeline), and ASPIRE-generated augmentations. ASPIRE successfully captures the major spurious cues learned by a model (shown in top-k) and generates diverse images without them. We show more examples in Appendix B and C and illustrate some failure cases in Appendix D.

heading: 5 Literature Review
text: Geirhos et al. (2020) provides a detailed survey on how image classification models perform poorly when trained on datasets with spurious correlations. Following this, a lot of works explore SGD training dynamics and inductive biases of such models in the presence of spurious correlations (Nagarajan et al., 2021; Pezeshki et al., 2021; Rahaman
et al., 2019). Shah et al. (2020) shows how deep neural networks, trained using ERM, can take shortcuts and learn to rely on spurious features rather than core features for a class. They call this phenomenon the extreme simplicity bias. Hermann and Lampinen (2020) and Jacobsen et al. (2019) further present examples with both natural and synthetic images, highlighting instances where these networks overlook core features. Shinoda et al. (2023) explore the types of shortcuts that are more likely to be learned. A plethora of methods in literature propose novel training strategies for improving robustness against spurious correlations (Ben-Tal et al., 2011; Hu et al., 2018; Sagawa* et al., 2020; Oren et al., 2019; Zhang et al., 2021). A detailed explanation of all
these methods can be found in Section 3 and Appendix A.4.
The use of synthetic data to improve the performance of downstream CV tasks has been explored extensively in the past. For data-driven generative models, GANs have remained the predominant approach to date (Brock et al., 2018; Li et al., 2022). Very recently, He et al. (He et al., 2022) employ large-scale text-to-image models like GLIDE (Nichol et al., 2021) to augment training data with synthetic images and show improvement in image classification performance.
Prior work explores language guidance for image generation for varied objectives. For example, Prabhu et al. (2023) proposes to generate counterfactual images for stress-testing image classification models. On similar lines, Wiles et al. (2022) and Vendrow et al. (2023) propose to identify failure cases and spurious correlations using augmented data generated using language guidance. Finally, Dunlap et al. (2023) proposes to adapt a model to new domains using augmented data. To the best of our knowledge, generative data augmentation with or without language guidance for improving robustness against spurious correlations has not yet been explored.

heading: 6 Conclusion
text: In this paper, we present ASPIRE, a novel data augmentation methodology to augment existing datasets with non-spurious minority group images to build robust and de-biased image classifiers. We evaluate ASPIRE on 4 benchmark datasets with 9 baselines and show that ASPIRE augmentations improve the worst-group accuracy of all baselines while maintaining average accuracies.

heading: Limitations and Future Work
text: As part of future work, we would like to address the current limitations of ASPIRE, which include:
1. ASPIRE is limited to how well image captioning models can describe the image. Though captioning models improve over time, we would like to explore novel ways to resolve this bottleneck. For example, Large MultiModal Language Models like LLaVa (Liu et al., 2023b) have been shown to perform exceptionally well at generating detailed captions of input images.
2. The edited images used to personalize text-toimage generation may sometimes be of low quality, leading to poor augmentations in more complex datasets, and we would like to explore ways to resolve this bottleneck. We also acknowledge that the advancement of text-toimage diffusion models to better follow text prompts will eventually lead to performance improvement of ASPIRE.
3. The different components of ASPIRE add computational overhead to the ASPIRE pipeline (over just the ERM classifier). However, it should be noted that a wealth of literature in offline data augmentation for NLP and CV tasks (through synthetic data generation) almost always employs computationally expensive foundation models for additional data generation. Textual-inversion fine-tuning of diffusion models used in our experiments is also computationally cheaper than full finetuning. Lastly, as part of future work, we would like to explore computationally cheaper alternatives to an LLM for information extraction from captions. Additionally, the augmentation process is completely offline and needs to be done just once for each dataset.
4. We also illustrate some failure cases of ASPIRE in Appendix D.

heading: Ethics Statement
text: Image generation models are prone to generating harmful, obscene and offensive context for certain classes pf objects, we prevent this from happening in ASPIRE by using a safety checker for the Stable Diffusion model which estimates whether a generated images could be considered offensive or harmful. For the CelebA dataset, ASPIRE performs modification where genders of people are swapped to debias the model towards certain attributes related a class. This approach is used only to improve the fairness and debiasing of the model.

heading: A Appendix

heading: A.1 Dataset Details
text: Table A.1 shows dataset details for all 4 datasets used in our experiments. As clearly visible, there is a notable disparity between the number of images representing minority groups (non-spurious images) and those representing majority groups (images with spuriously correlated features). In contrast, the test set for each dataset maintains a balanced representation between the minority and majority groups. This can lead classifiers to quickly adopt spurious correlations, resulting in sub-optimal performance on the test set.

heading: A.2 Algorithm
text: Algorithm 1 describes algorithmically the ASPIRE pipeline. Readers can refer to the algorithm for a detailed step-by-step understanding of the workings of ASPIRE.

heading: A.3 Traditional NLP algorithm details
text: Introduction Extracting foreground objects and the background from a caption using traditional Natural Language Processing (NLP) techniques and libraries like SpaCy involves several steps. Here’s a general approach: Text Preprocessing First, preprocess the text to ensure it’s in a suitable format for analysis. This might include:
• Lowercasing all words.
• Removing punctuation and special characters.
• Tokenization: Breaking the text into individual words (tokens).
Part-of-Speech Tagging Use SpaCy to perform part-of-speech (POS) tagging, which identifies the grammatical parts of speech for each word (e.g., noun, verb, adjective). This is crucial for identifying potential objects and elements of the scene. Named Entity Recognition (NER) Employ Named Entity Recognition to identify named entities in the text, which can include names of people, places, organizations, or other proper nouns. These entities can be part of the foreground or background. Dependency Parsing Dependency parsing helps understand the grammatical structure of the sentence, showing how words relate to each other. This is useful to distinguish between main subjects (likely foreground objects) and contextual elements (possibly background). Chunking or Phrase Detection Use chunking or phrase detection to group together contiguous sequences of tokens that form meaningful phrases.
Noun phrases, in particular, are often key in identifying objects and scene elements. Identifying Foreground and Background Foreground Objects Typically, these are nouns or noun phrases that are the main subjects or objects of the sentence. They often appear with adjectives and are part of active clauses. Background Information This can include descriptions of settings, locations, or contexts. Adverbial phrases and clauses, as well as descriptive language, can signal background details. SpaCy Implementation Here’s a simple implementation using SpaCy:
import spacy
# Load the SpaCy model nlp = spacy.load("en_core_web_sm")
def extract_foreground_background(text): doc = nlp(text)
foreground = [] background = []
for token in doc: # Check for nouns and proper nouns
for foreground if token.pos_ in ["NOUN", "PROPN"]:
foreground.append(token.text)
# Background might be set by adverbial phrases or adjectives
if token.pos_ in ["ADJ", "ADV"]: background.append(token.text)
# Check for named entities if token.ent_type_: if token.ent_type_ in ["PERSON",
"ORG", "GPE"]: foreground.append(token.text) else: background.append(token.text)
return foreground, background
# Example Usage text = "The cat sat on the mat in the sunny room." foreground, background = extract_foreground_background(text) print("Foreground:", foreground)
print("Background:", background)

heading: A.4 Details on Baselines
text: To maintain training efficiency, for training each baseline with ASPIRE augmentations, we add only 1× more augmentations to the original dataset for CelebA, Waterbirds, and SPUCO Dogs, or effectively or effectively double the number of nonspurious minority group images in each dataset. These 3 datasets have labeled minority groups, and thus, the number of augmentations to be added amounted to the total minority group images in each class of the original dataset. For Hard ImageNet, we add as many more augmentations as the total number of original training samples in each class of the original dataset. We elaborate on the rationale behind the choice of our baseline setup in Appendix A.5, where we also describe why we choose not to compare ASPIRE with large multi-modal models. We next describe how we add ASPIRE augmentations to the original training pipeline for different baselines. Emperical Risk Minimization (ERM) For this baseline, we compare a ResNet-50 model trained using ERM (with SGD) on the original dataset with a ResNet-50 model trained on the original dataset augmented with ASPIRE augmentations. For ERM, we just add ASPIRE augmentations to the initial training set. 1-stage training baselines. Group DRO (Sagawa et al., 2019) is a state-of-the-art method that uses group information on train and adaptively upweights worst-group examples during training. SUBG (Idrissi et al., 2022) is ERM applied to a random subset of the data where the groups are equally represented, which was recently shown to be a strong baseline. We also add ASPIRE augmentations to the initial training set for both baselines. 2-stage training baselines. Just Train Twice (JTT). JTT follows a 2-stage training process wherein they first identify training examples that are misclassified by a standard ERM model, and then train the final model by upweighting the examples identified in the first stage. Learning from Failure (LfF). (Nam et al., 2020) Similar to JTT, LfF follows a 2-stage training process wherein they first identify training examples that are misclassified by a biased ERM model, and then train the final model by re-weight training samples using the relative difficulty score based on the loss of the biased model. Correct-n-Contrast (CnC) (Zhang et al., 2022) detects the minority group examples similarly to JTT and uses a contrastive objective to learn representations robust to spurious correlations. Deep Feature Reweighting (DFR). (Kirichenko et al., 2023) DFR follows a 2-stage training process wherein they first fine-tune a pretrained ResNet model (pre-trained on the entire ImageNet dataset) using ERM on the entire train split followed by re-training the last layer using a small set from the train with an equal number of instances for both majority and minority groups. MaskTune. (Asgari et al., 2022) follows a 2-stage training process, wherein they first fine-tune a ResNet model on a dataset using ERM on the entire train split followed by re-training the model with new masked data for one full epoch. For all these baselines, we add ASPIRE augmentations to the set used in the second stage of training.

heading: A.5 Choice of Baselines
text: To the best of our knowledge, there exists no prior method in literature that generates minority group images to expand the training set. Most work has focused on devising novel training methods for robust classification, all of which are complementary to ASPIRE and compared to our method in this paper. As also mentioned in Section 5 of our paper, generative data augmentation for improving overall accuracy has been explored but is unrelated to our method. Additionally, the primary aim of ASPIRE is to improve the downstream performance of image classification models. We acknowledge that other types of models, like instruction-tuned Vision-Language Models (Liu et al., 2023b), might identify and classify the image correctly into a predefined class given specific prompts (again, this is an underexplored area in CV), but comparing this is beyond the scope of this paper and experimental setting. Our setting is consistent with most prior art in (methods listed in Table 1).

heading: A.6 Prompts
text: GPT-4.The general-purpose prompt we use for GPT-4 is listed as follows: I will provide you with a list of tuples. Each tuple in the list has 2 items: the first is a caption of an image and the second is the label of the image. For each, you will have to return a JSON with 3 lists. One list should be the list of all phrases from the caption that are objects that appear in the foreground of the image but ignore objects that correspond to the actual label (the label for the phrase might not be present exactly in
Algorithm 1 ASPIRE Data Augmentation Algorithm Data: Image Classification Dataset Dtrain → {xi (Image), yi (Label)}; E = Classifier(xi, yi) // Image classification model C = Captioning(xi) // Image captioning model L = LLM(Prompt, Captions, y) // LLM to extract foreground and background objects. BG = InstructPix2Pix(b, b̃, xi) // InstructPix2Pix to convert background of an image. G = GroundingDino(f, xi) // Creates bounding box around objects. S = SegmentAnything(bb) // Extracts image segmentation maps from bounding boxes. I = InpaintAnything(M, xi) // Removal of objects corresponding to segmentation maps. Dcorrect ← ∅ for xi in Dtrain do // Consider only the images which are predicted correctly.
if E(xi) == yi then Dcorrect ← Dcorrect ∪ {(xi, yi)};
end end Sample p% of Dcorrect to create Dhold. Dcaptions ← C(Dhold) // Caption the images in the holdout set. Fi,Bi ← L(Prompt,Dcaptions,Dyhold) // Extract foreground and background objects by prompting the LLM. Dsynth ← ∅, Tsynth ← ∅; for {f} in Fi do
bb ← G(f, xi); // Create the bounding boxes. M ← S(bb); // Extract the segmentation maps. x mod i ← I(M); // Modify image by removing the foreground object.
// Consider only the images which are predicted wrong after modification. if ycorrecti ≠ E(xmodi ) then
Dsynth ← Dsynth ∪ {xmodi }; Tsynth ← Tsynth ∪ {f};
end end for {b, b̃} in Bi do
x mod i ← BG(b, b̃, xi);// Change image background as suggested by the LLM. // Consider only the images which are predicted wrong after modification. if ycorrecti ≠ E(xmodi ) then
Dsynth ← Dsynth ∪ {xmodi }; Tsynth ← Tsynth ∪ {b};
end end // Collapse synthetic dataset based on text phrases that are similar to each other. Select top-k items that have the highest count per image class in the dataset. T ksynth,Dksynth ← TopK(Col(Tsynth,Dsynth)) Train the Stable Diffusion model SD using Dksynth. Generate Daug from SD. // Creating a new training dataset by combining the augmentations with the original training data. Dnewtrain ← Dtrain ∪Daug; // Retrain the original image captioning model on the new training data. Retrain E on Dnewtrain.
the caption) (named ’foreground’). The second list should have the single predominant background of the image to the foreground objects (named ’background’). If you do not find a phrase that corresponds to the background, return an empty list for the background. The third is an alternative background for the image, an alternative to the background you suggested earlier (named ’alt’). Here are some examples which also show the format in which you need to return the output. Please just return the JSON in the following format: Exemplars ⋯ and here is the caption:. We will provide the exemplars on our GitHub.
InstructPix2Pix.The prompt we use for InstructPix2Pix is: turn the background from original background to alternative background.
A.7 Examples of top-k identified by ASPIRE
Table 4 shows the top-k spuriously correlated features (or groups of features) for each class and for each dataset. As mentioned earlier, due to diversity in captions, the same kind of foreground object or background may be expressed with different phrases. ASPIRE thus returns groups of top-k items rather than a single top-k item for each k.

heading: A.8 Collection of Test-Set for Hard ImageNet
text: Our institution’s Institutional Review Board(IRB) has granted approval for the data collection. We followed the following steps for collecting a test set of the Hard ImageNet dataset:
1. We first identified spurious features in the Hard ImageNet and verbalized them. These features were identified from annotations in the original proposed dataset by Moayeri et al. (2022).
2. 3 annotators with extensive vision and language experience collected 1/3rd of the total 750 images. The annotators were not hired from any crowdsourcing platform and, in fact, were volunteers from our organization. The only instruction that was provided was that the image should have the primary target label of the image, and while majority group images should have the identified spurious features, minority group images should not.
3. Post this step, each annotator validated the images collected by the other annotators.

heading: B GradCam Visualizations
text: While Fig. 6 shows examples of the majority group images from the test set with spurious features, Figure 7 shows examples of minority group images without any spurious features. For Fig. 7 we show examples where the ERM classifier predicted the class of the image incorrectly (due to the absence of spurious features) while the one trained with ASPIRE predicted the class correctly. As we clearly see, in both cases, when trained with ASPIRE augmentations, the model learns to focus on core features rather than spurious ones while making predictions.

heading: C Generation Examples
text: Table 8 shows examples of original images from the train set, the edited images from the ASPIRE pipeline and augmentations generated using ASPIRE. To the left of the generated augmentations, we also mention the top-k spurious correlations discovered by ASPIRE for the particular class. ASPIRE generates diverse augmentations with the de-
sired non-spurious features that can be used to train robust models

heading: D ASPIRE Failure Cases
text: This section lists some failure cases of our proposed ASPIRE framework. As APSIRE leverages
external models in its pipeline, the success of ASPIRE at times depends on the capabilities of these models. For generating augmentations, we notice the following failure cases:
1. Superimposition of other foreground ob-
jects on the foreground object of interest. Recall that ASPIRE detects foreground objects to remove (for spurious correlation detection) by parsing captions. These objects are then removed to detect if the object is spurious or not. In cases where another foreground object in the image is superimposing the foreground object, though the language-grounded pipeline is able to detect it properly, the inpainting model is at times unable to precisely remove just that object without not removing the superimposing foreground object and removes both the original object and the object superimposing it. An example is a human wearing spectacles, where we only want to remove the human, but the inpainting model removes both the human and the spectacles it is wearing. We provide an example of this case in Figure 5 (bottom row).
2. Foreground objects change on changing background. InstructPix2Pix, at times, tends to change the foreground object when prompted to change the background significantly, for example, changing outdoor background → indoor background. We provide an example of this case in Figure 5 (top row).
3. Bias in Stable Diffusion. Although our Stable Diffusion fine-tuning step, with textual inversion, helps overcome its current biases, limited samples present for fine-tuning sometimes hurt this adaptation. For example, even after fine-tuning, Stable Diffusion might generate images of dog sled with dogs in it.

heading: E Additional Details

heading: E.1 Model Parameters
text: Git-Large-Coco has ≈ 300M parameters with a CLIP/ViT-L/14 image encoder and a 6 layer transformer decoder with 12 attention heads and 768 hidden-state. Stable Diffusion is a ≈ 860M parameter UNet and ≈ 123M parameter text encoder model. ResNet-50 has ≈ 25M parameters with 50 layers.

heading: E.2 Compute Infrastructure
text: All our experiments are conducted on NVIDIA A100 GPUs. We batch prompted LLaMa-2 13B, with a BS of 16, where LLaMa-2 performed distributed inference on 4 A100 GPUs. That translates to 52.51 TFLOPs per batch. Fine-tuning SD with
textual inversion with a BS of 8 takes and an avg. of ≈1 hour. For generating 1× augmentation, we use 1 A100 GPU for an average ≈1.2 hours in total.
E.3 Implementation Software and Packages We implement all our models in PyTorch2 and use the HuggingFace3 implementations of ERM, Git (Wang et al., 2022), LLaMa-2 13B (Touvron et al., 2023) and InstructPix2Pix (Brooks et al., 2023). We also use the following code/components in our pipeline Grounding DINO4 (Liu et al., 2023c), Segment Anything5 (Kirillov et al., 2023) and Stable Diffusion using textual-inversion6 (Gal et al., 2023).
We also use the following repositories for running the baselines: Group DRO7 (Sagawa et al., 2019), SUBG8 (Idrissi et al., 2022), JTT9 (Liu et al., 2021b), Learning from Failure 10 (Nam et al., 2020), Correct-n-Contrast11 (Zhang et al., 2022), Deep Feature Reweighting12 (Kirichenko et al., 2023), MaskTune13 (Asgari et al., 2022) and DivDis14 (Lee et al., 2022).
All the above GitHub code has been released under an MIT license, free for research use.

heading: E.4 Dataset Links
text: We use the Waterbirds 15, SPUCO Dogs 16, Hard ImageNet 17 and the CelebA 18 dataset. All the datasets are free for research use.

heading: E.5 Potential Risks
text: Generative models are prone to hallucinate and potentially generate nonsensical, unfaithful or harmful content to the provided source input that it is conditioned on.
2https://pytorch.org/ 3https://huggingface.co/ 4https://github.com/IDEA-Research/GroundingDINO 5https://github.com/facebookresearch/segment-anything 6https://github.com/rinongal/textual_inversion 7https://github.com/kohpangwei/group_DRO 8https://github.com/facebookresearch/BalancingGroups 9https://github.com/anniesch/jtt
10https://github.com/alinlab/LfF 11https://github.com/HazyResearch/correct-n-contrast 12https://github.com/PolinaKirichenko/deep_feature_reweighting 13https://github.com/aliasgharkhani/masktune 14https://github.com/yoonholee/DivDis 15https://www.vision.caltech.edu/visipedia/CUB-200.html 16https://github.com/BigML-CS-UCLA/SpuCo 17https://openreview.net/forum?id=76w7bsdViZf 18https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html
(a) Dog sled w/o ASPIRE (b) Dog sled w/ ASPIRE (c) Puck w/o ASPIRE (d) Puck w/ ASPIRE
(e) Blonde female w/o ASPIRE
(f) Blonde female w/ ASPIRE
(g) Blonde female w/o ASPIRE
(h) Blonde female w/ ASPIRE
(a) Seatbelt w/o ASPIRE (b) Seatbelt w/ ASPIRE (c) Dogsled w/o ASPIRE (d) Dogsled w/ ASPIRE
(e) Male blonde w/o ASPIRE
(f) Male blonde w/ ASPIRE
(g) Male blonde w/o ASPIRE
(h) Male blonde w/ ASPIRE

heading: Future Work
text: As part of future work, we would like to address the current limitations of ASPIRE, which include:
1. ASPIRE is limited to how well image captioning models can describe the image. Though captioning models improve over time, we would like to explore novel ways to resolve this bottleneck. For example, Large MultiModal Language Models like LLaVa (Liu et al. , 2023b) have been shown to perform exceptionally well at generating detailed captions of input images. 2. The edited images used to personalize text-toimage generation may sometimes be of low quality, leading to poor augmentations in more complex datasets, and we would like to explore ways to resolve this bottleneck. We also acknowledge that the advancement of text-toimage diffusion models to better follow text prompts will eventually lead to performance improvement of ASPIRE. 3. The different components of ASPIRE add computational overhead to the ASPIRE pipeline (over just the ERM classifier). However, it should be noted that a wealth of literature in offline data augmentation for NLP and CV tasks (through synthetic data generation) almost always employs computationally expensive foundation models for additional data generation. Textual-inversion fine-tuning of diffusion models used in our experiments is also computationally cheaper than full finetuning. Lastly, as part of future work, we would like to explore computationally cheaper alternatives to an LLM for information extraction from captions. Additionally, the augmentation process is completely offline and needs to be done just once for each dataset. 4. We also illustrate some failure cases of ASPIRE in Appendix D