RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models

Authors: maya  varma
jean-benoit  delbrouck
zhihong  chen
akshay  chaudhari
curtis  langlotz
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 17:44:04

Abstract: Fine-tuned vision-language models (VLMs) often capture spurious correlations between image features and textual attributes, resulting in degraded zero-shot performance at test time. Existing approaches for addressing spurious correlations (i) primarily operate at the global image-level rather than intervening directly on fine-grained image features and (ii) are predominantly designed for unimodal settings. In this work, we present RaVL, which takes a fine-grained perspective on VLM robustness by discovering and mitigating spurious correlations using local image features rather than operating at the global image level. Given a fine-tuned VLM, RaVL first discovers spurious correlations by leveraging a region-level clustering approach to identify precise image features contributing to zero-shot classification errors. Then, RaVL mitigates the identified spurious correlation with a novel region-aware loss function that enables the VLM to focus on relevant regions and ignore spurious relationships during fine-tuning. We evaluate RaVL on 654 VLMs with various model architectures, data domains, and learned spurious correlations. Our results show that RaVL accurately discovers (191% improvement over the closest baseline) and mitigates (8.2% improvement on worst-group image classification accuracy) spurious correlations. Qualitative evaluations on general-domain and medical-domain VLMs confirm our findings.

Main paper: 
 introduction: 
 Contrastive vision-language models (VLMs) (e.g., CLIP [36] and ALIGN [24] ) are a powerful class of models that jointly learn relationships between images and text. VLMs are generally pretrained on web-scale datasets with millions of image-text pairs and have been shown to exhibit impressive capabilities on a wide range of downstream tasks. In particular, VLMs have the ability to perform tasks in a zero-shot manner without utilizing explicit task-specific training data; this is accomplished by modeling downstream tasks (e.g., image classification, text-to-image retrieval) as image-text matching tasks [36] .However, pretrained VLMs can exhibit poor zero-shot performance when compared to state-of-the-art task-specific models, particularly on challenging or out-of-domain downstream tasks [36, 7, 17, 19] . As a result, pretrained VLMs are often fine-tuned on domain-specific vision-language datasets in order † Equal senior authorship. 1 to improve zero-shot performance on tasks of interest. For instance, recent works have fine-tuned the CLIP VLM [36] on vision-language datasets consisting of (i) chest X-rays and paired physician reports [45] , (ii) pathology data and paired text [17, 19] , and (iii) product images and paired captions from online fashion retailers [7] .Domain-specific vision-language datasets used to fine-tune VLMs may be small in size, preventing VLMs from gaining the robustness benefits that come with training on diverse, web-scale data [6, 14] . As a result, fine-tuned VLMs may capture spurious correlations between image features and textual attributes [56] . For instance, consider a VLM fine-tuned on an animal image-text dataset where the presence of butterflies is closely correlated with the presence of flowers (Figure 1 ). Consequently, the VLM may learn to incorrectly associate the image features corresponding to flower with the textual attribute butterfly. At test time, the VLM is likely to exhibit degraded zero-shot classification performance on (i) images of butterflies without flowers and (ii) images of other animals with flowers.Improving robustness of fine-tuned VLMs to spurious correlations is challenging for the following two reasons. First, existing automated approaches primarily discover and mitigate spurious correlations at the global image level rather than intervening directly on fine-grained image features. Such approaches discover spurious correlations by identifying coherent groups of misclassified images in an automated fashion [13, 43, 22, 42] ; then, the identified spurious correlation can be mitigated during training using data augmentation or robust optimization [43, 39, 22, 56] . However, recent works have suggested that such global image-level strategies (i) discover spurious correlations that align poorly with human-interpretable attributes [25] and (ii) may not effectively enable models to ignore spurious correlations during training [15, 18] . Second, existing approaches for discovering and mitigating spurious correlations are predominantly designed to improve robustness of unimodal image classification models [39, 43] or pretrained VLMs [60, 49] . These settings differ substantially from the fine-tuned VLM setting, which presents several unique challenges such as the absence of class and subgroup labels in the training set and the inclusion of free-form text.In this work, we address these challenges by introducing Region-aware Vision-Language learning (RAVL), an approach for improving the robustness of fine-tuned VLMs to spurious correlations. RAVL takes a fine-grained perspective on VLM robustness by discovering and mitigating spurious correlations using local image features, rather than operating at the global image level. Our contributions are:• First, given a fine-tuned VLM, RAVL discovers learned spurious correlations between image features and textual attributes. Using a labeled classification dataset, we decompose images into candidate regions, utilize the VLM embedding space to group visually-similar regions into feature clusters, and quantitatively evaluate the effects of each feature on zero-shot classification errors. • Second, given a ranked list of image features that the VLM has learned to spuriously correlate with one or more textual attributes, RAVL mitigates the identified spurious correlations. Our key insight is that region-level information can be leveraged during VLM fine-tuning in order to improve model robustness. To this end, we introduce a novel region-aware loss function 
discovering spurious correlations in fine-tuned vision-language models: 
 In this section, we present the first stage of RAVL, which aims to discover learned spurious correlations in VLMs. In Section 3.1, we discuss our region-aware approach for discovering fine-grained spurious correlations. Then, in order to quantitatively evaluate the efficacy of spurious feature discovery methods, we introduce a large-scale evaluation framework in Section 3.2. Finally, in Section 3.3, we use our evaluation framework to demonstrate that RAVL outperforms prior approaches in discovering fine-grained spurious correlations between image features and textual attributes.The first stage of RAVL aims to identify spurious correlations between image features and textual attributes learned by a fine-tuned VLM M. In contrast to prior works that have incorporated humans in the loop in order to identify spurious correlations [56, 30] , RAVL is a fully automated approach. Additionally, whereas previous automated methods for discovering spurious correlations focus predominantly on identifying groups of images with high error rates [22, 13] , our approach identifies specific image features that model M has learned to spuriously correlate with a textual attribute. Our goal is to discover precise spurious correlations that can be easily interpreted by humans.As discussed in Section 2, a model M that has learned a spurious correlation between an image feature e a and a textual attribute y will demonstrate low zero-shot performance on (i) images in I i in D V into a set of candidate regions R i .There are a variety of ways in which an image can be decomposed into regions, such as dividing images into equal-sized segments (e.g., quadrants) or using region proposal networks (RPNs) [38] . Ideally, regions should capture key features in the image; however, we emphasize that RAVL does not require groundtruth region-level annotations. We then apply RoIAlign [16, 63] to the image encoder of M to extract embeddings for each region. Zero-shot classification is performed using the computed region embeddings, resulting in a softmax-normalized region score distribution matrix S Ri ∈ R |Ri|×|Y| .Given region-level embeddings for all candidate regions in D V , we next aim to identify coherent groups of image features that occur consistently throughout the dataset (e.g., features corresponding to "flower" or "butterfly" in Figure 1 ). To this end, we cluster the computed region-level embeddings using the K-Medoids algorithm with cosine distance. The optimal number of clusters is selected in an automated fashion using Silhouette distance. The resulting clusters (denoted as C) capture key image features in D V . For feature cluster c ∈ C, let e c denotes the set of features in cluster c.Identifying candidate image features that directly contribute to classification errors. We now seek to identify features that, when present in an image, are directly responsible for prediction errors.Let R c represent the set of regions assigned to cluster c and let I c represent the set of images associated with the regions in cluster c. We identify labels for images in I c ; we designate this label set as Y c . For each class label y ∈ Y c , we identify all images in I c with label y, and we designate zero-shot classification accuracy on this subset of n y in images as p y in . Then, we identify all images in D v with label y that do not have a region included in cluster c, and we designate zero-shot classification accuracy on this subset of n y out images as p y out . We now introduce the cluster influence score, which evaluates the extent to which features e c contribute to mispredicted image classification labels. We restrict our evaluation to only include mispredicted images in I c with ground-truth labels y such that p y in < p y out ; we will refer to this subset as I err c ⊂ I c . For each image I i ∈ I err c , we extract (i) the image score distribution vector s Ii and (ii) the region score distribution matrix S Ri . We use s Ii to identify the predicted image class ŷ, and we then identify the region r max i in R i with the highest score for class ŷ.Definition 1 (Cluster Influence Score). For cluster c and label y, the cluster influence score is the proportion of images I i ∈ I err c with label y where the identified highest-scoring region r max i is part of cluster c (i.e., r max i ∈ R c ):EQUATIONThe final cluster influence score for cluster c is computed as the maximum over all labels y as H c = max y∈Yc H y c . High values of H c show that features e c are similar to the incorrect label in the vision-language embedding space; this suggests that for a given image with an incorrect prediction, feature e c is more likely to contribute to the misprediction than other features in the image. On the other hand, low values of H c are likely to indicate that feature e c represents a core feature associated with the class label or a neutral feature that does not affect predictions.Given H c for each feature cluster, we prune all clusters with influence scores below a threshold of τ l , which we set to 0.25 in all experiments.Ranking image features by degree of learned spurious correlation. For each remaining feature cluster, we next aim to determine the extent to which the presence or absence of features e c affects classification performance; we introduce the cluster performance gap metric to this end.Definition 2 (Cluster Performance Gap). For cluster c and label y, the cluster performance gap is the weighted difference between zero-shot classification accuracy on images with features e c and images without features e c :EQUATIONwhere w y is a simple weighting factor computed as w y = 2 × [min(n y in , n y out )/(n y in + n y out )]. Since spurious correlations result in consistent errors as opposed to isolated misclassifications, the weighting factor is designed to prioritize stronger spurious correlations that result in a larger number of errors. G y c ranges between 0 and 1. The final performance gap metric for cluster c is computed across all labels as G c = y∈Yc |G y c |. A high value of G c suggests that the presence or absence of features e c contribute to large class-level variations in image classification performance.Given G c for each feature cluster, we rank clusters in order from highest to lowest values. The output of this stage is a ranked list of image features that model M has learned to spuriously correlate with one or more class labels in Y.We now discuss our approach for evaluating RAVL. Evaluating the accuracy of predicted spurious correlations is challenging because the ground-truth spurious correlations learned by a model M are typically unknown. Previous works on VLM robustness evaluate discovered spurious correlations with qualitative experiments, human-in-the-loop evaluations, or small-scale datasets [56] . Our aim in this section is to introduce a large-scale experimental setup where the ground-truth spurious correlations learned by VLMs are known and annotated in advance; this can then enable us to determine whether the features discovered by RAVL in Section 3.1 accurately align with the groundtruth. Our evaluation framework is motivated by prior work [26, 13] ; however, in contrast to existing approaches, we introduce evaluation settings that are designed (i) for evaluating robustness approaches at the fine-grained region level rather than the global image-level, and (ii) for evaluating VLMs rather than unimodal models.Designing Controlled Evaluations: Our evaluation framework artificially induces spurious correlations in the VLM fine-tuning data; then, given the known pre-defined spurious correlation and a VLM that learned the desired spurious correlation, we can quantitatively evaluate the extent to which RaVL discovers the correlation.We create a set of evaluation settings using data from two domains: (1) synthetic data (MNIST [11] and FashionMNIST [51] ) and (2) real-world data (COCO [27] ). Each evaluation setting consists of the following components:1. Predefined spurious correlation: We define a spurious image feature and textual attribute pair (e eval , a eval ). For MNIST and FashionMNIST, e eval represents a red rectangle; a eval is generated from the set of class labels {zero, one, two, three, four five, six, seven, eight, nine} for MNIST and {t-shirt, trouser, pullover, dress, coat, sandal, shirt, sneaker, bag, ankle boot} for FashionMNIST.For COCO, we sample e eval and a eval from the list of annotated attributes. 2. Fine-tuning dataset: We construct a vision-language fine-tuning datasetD eval F = {(I i , T i )} m i=1with images I i and text T i . Dataset D eval F is sampled from the training sets of MNIST, Fashion-MNIST, or COCO such that the presence of image feature e eval is closely correlated with the presence of text attribute a eval as measured by Cramer's V [57] . = {(I i , y i , R i , L i )} ni=1 with images I i , class labels y i , region bounding boxes R i , and region-level labels L i . In particular, a eval must be included in the class label set, and e eval must be annotated in the region-level label set. Since D eval V is designed to reflect a real-world setting, we assume that a correlation between a eval and e eval does not exist. Dataset D eval V is constructed from the test sets of MNIST, FashionMNIST, or COCO.Given the four components listed above, we classify an evaluation setting as valid if model M learned the intended spurious correlation. In order to measure this, we first identify images with label a eval in D eval V and compute the performance difference between images with feature e eval and images without feature e eval ; we designate this value as ϵ 1 . Then, for labels y ̸ = a eval , we compute the maximum performance difference between images without feature e eval and images with feature e eval ; we designate this value as ϵ 2 . Large values of ϵ 1 and ϵ 2 suggest that model M has learned the desired spurious correlation between image feature e eval and textual attribute a eval , as defined in Section 2. We remove settings where ϵ 1 or ϵ 2 are below some predefined performance threshold τ eval . The performance threshold τ eval serves as a quantitative indicator of learned correlation strength.Implementation Details: In total, we generate 620 fine-tuning datasets D eval F (100 synthetic; 520 real-world). We then fine-tune model M on each dataset with three random seeds, resulting in 1860 candidate evaluation settings. Finally, we filter out settings where model M does not consistently learn the spurious correlation; to this end, we only retain settings where both ϵ 1 and ϵ 2 exceed τ eval = 10 across all three random seeds. We repeat this procedure across various pretrained VLMs M, resulting in 654 valid experimental settings. Additional implementation details are provided in Appendix B.Comparisons to Prior Approaches: Given an evaluation setting with a predefined spurious correlation (e eval , a eval ), a fine-tuned VLM M, and an evaluation dataset D eval V , our goal is to determine the extent to which RAVL can discover the correlation between e eval and a eval .To this end, we use the labeled zero-shot classification dataset D eval V , which includes ground-truth region bounding boxes and associated region labels. We provide the ground-truth bounding boxes as input to RAVL, which returns a single top-ranked cluster of regions likely to include spurious features. We rank regions within the cluster based on similarity to the cluster medoid, and we utilize the provided region-level labels in D eval V to evaluate the proportion of top-K regions that contain the desired spurious feature e eval . In line with prior work [13] , we report performance with Precision@K metrics. We note that given an identified spurious feature e eval , the correlated textual attribute a eval can be detected by identifying the class label in D eval V where the absence of feature e eval leads to degraded performance.There are few existing approaches for performing automated detection of fine-grained spurious features learned by VLMs. Here, we compare RAVL with four previously-developed methods: Distilling Failures [22] , George [43] , Domino [13] , and Spurious-Aware Detection [56] . Distilling Failures, George, and Domino are state-of-the-art approaches for automatic identification of model failures resulting from spurious correlations; although these methods operate at the global image level and are designed for unimodal settings, we adapt these approaches for our setting by utilizing regions and zero-shot classification scores as input. Spurious-Aware Detection operates at the fine-grained region level by computing class-based performance gaps resulting from the presence or absence of particular features. To enable a fair comparison with RAVL, we provide the same set of regions and associated embeddings as input to all baselines. We also compare RAVL with a random baseline, where the ranked list of regions is shuffled randomly. Table 1 summarizes mean Precision@10 metrics across all 654 evaluation settings. Results demonstrate that RAVL consistently outperforms prior approaches in discovering spurious correlations between image features and textual attributes, contributing to a 191% improvement over the closest baseline. In Table 1 , we evaluate the effects of learned spurious correlation strength by varying the error threshold τ eval from 10 to 40 and reporting performance for the subset of valid evaluation settings. Results show that RAVL is particularly effective when VLM M learns a strong spurious correlation; as learned correlation strength increases, performance of RAVL increases by 47% whereas most baselines degrade in performance. We also observe that Domino, George, and Distilling Failures often achieve performance near or below the random baseline across our evaluation settings; this suggests that methods designed for detecting errors resulting from spurious correlations at the global image-level cannot be easily adapted for fine-grained region-level discovery. Figure 2 demonstrates that our findings hold for both synthetic and real-world data.Ablations: Our ablation study evaluates the role of the cluster influence score H c and the cluster performance gap metric G c (Section 3.1) in enabling accurate discovery of spurious correlations between image features and textual attributes. We compare the following three metrics for ranking clusters: (1) an unweighted cluster performance gap metric where w y is set to 1, (2) the cluster performance gap with w y computed as in Section 3.1, and (3) a combination of the cluster performance gap and cluster influence metric as used in RAVL. As shown in Table 2 , the metrics utilized by RAVL consistently demonstrate the best performance across various learned correlation strengths (τ eval ). Our results suggest the utility of both the performance gap metric and the influence score in identifying fine-grained spurious correlations.Evaluations in the Wild: In addition to our controlled evaluations, we evaluate the ability of RAVL to surface spurious correlations learned by 12 off-the-shelf VLMs [12, 36, 20] ; this presents a realistic and uncontrolled evaluation setting. We consider two zero-shot classification tasks D V : (1) a 397class scene classification task on SUN397 [52] and (2) binary classification of cardiomegaly in chest X-rays from ObjectCXR [23] . We use the cluster performance gap metric G c , introduced in Section 3.1, to quantify the degree of the learned spurious correlation.Our results demonstrate that all evaluated models, which span a range of architecture, training data, and parameter counts, show evidence of having learned spurious correlations; this is demonstrated by nonzero values of the cluster performance gap metric G c . On average across the evaluated models, the top-ranked spurious feature cluster discovered by RAVL on SUN397 achieves a cluster performance gaps (G c ) of 9.9 ±3.2 (minimum = 5.1, maximum = 14.0). On ObjectCXR, the mean value of G c is 0.08 ±0.04 (minimum = 0.04, maximum = 0.12). 2 Our results support findings from previous work suggesting that all models may learn spurious correlations [30] .In Figure 3 We fine-tune each model M on dataset D eval F using a single NVIDIA A100 GPU with an initial learning rate of 5e-5. We use a batch size of 128 and train for 100 epochs with early stopping. We set the loss temperature as τ = 0.07. In line with prior works that explore the benefits of locked image-text training [2, 46] , we freeze the text encoder and only learn weights for the image encoder. 
mitigating spurious correlations in fine-tuned vision-language models: 
 In this section, we present the second stage of RAVL, which aims to mitigate learned spurious correlations in VLMs. In Section 4.1, we discuss our methodology for mitigating fine-grained spurious correlations with a novel region-aware loss function. In Section 4.2, we use the evaluation framework previously introduced in Section 3.2 to demonstrate that RAVL substantially outperforms prior approaches in mitigating spurious correlations between image features and textual attributes.As described in Section 3, Stage 1 of RAVL discovers image features that VLM M has learned to spuriously correlate with textual attributes. We next aim to mitigate the spurious correlation. Motivated by prior work on fine-grained VLMs [58, 46] , our key insight is that utilizing region-level information during VLM training can enable models to focus on relevant image-text relationships and ignore spurious correlations.Since dataset D F exclusively consists of images and text, ground-truth subgroup and class labels are not available. As a result, we first assign plausible (i) region-level subgroup labels and (ii) image-level class labels to the vision-language fine-tuning dataset D F . To assign subgroup labels, we decompose each image I i in dataset D F into a set of candidate regions R i . We then fit the trained K-Medoids clustering model from Section 3.1 on R i and identify all spurious regions associated with the top ranked cluster. We represent the identified spurious regions as R s i and remaining non-spurious regions as R r i such that R s i ∪ R r i = R i . In order to assign plausible class labels, we parse the paired text T i associated with each image to identify samples that reference the class labels included in the zero-shot classification label set Y; we refer to the assigned class label for image I i as ŷi .We now introduce a novel region-aware contrastive loss function for training VLM M new . For batch B, we define R s B as the set of all spurious regions in the batch: R s B = Ii∈B R s i . For image I i ∈ B, the first loss component L i R encourages high embedding similarity between non-spurious regions R r i and assigned class label ŷi when compared to other class labels.EQUATIONHere, for region embedding function f and text embedding function g, σ m (A, b) = exp(max a∈A (⟨f (a), g(b)⟩ /τ )) with temperature τ . The term P (R s B ) is a penalty that enforces embedding-level dissimilarity between spurious regions and correlated class labels.A encourages high embedding similarity between non-spurious regions R r i and assigned class label ŷi when compared to other regions. We define σ(a, b) = exp(⟨f (a), g(b)⟩ /τ ) with temperature τ .EQUATION)The final loss is expressed asL = λL CL + (1 -λ) |B| i=1 (L i R + L i A ).Here, λ is a hyperparameter and L CL takes the form of the original loss function used for training M; in our experiments, L CL is the CLIP objective [36] . Extended formulations of our loss function are provided in Appendix C.Comparisons to Prior Approaches: We use the evaluation framework previously introduced in Section 3.2 to compare RAVL with prior approaches. There are few existing approaches for mitigating spurious correlations in the setting of fine-tuned VLMs. Here, we compare RAVL with standard VLM fine-tuning, upsampled VLM fine-tuning, ERM, GDRO [39] , and Spurious-Aware Mitigation [56] . Since ERM and GDRO are traditionally used in unimodal classification settings, we adapt these approaches for our setting by adding a contrastive vision-language objective and using zero-shot classification scores during fine-tuning; we refer to these approaches as VL-ERM and VL-GDRO respectively.Table 3 summarizes mean zero-shot classification results across our real-world evaluation settings. Since performance of mitigation methods is dependent on the accuracy of the discovered spurious correlations in Stage 1, Table 3 displays results for two evaluation categories: (i) the 192 settings where RAVL Stage 1 Precision@10 is greater than 0.6, and (ii) the 106 settings where RAVL Stage 1 Precision@10 is greater than 0.8. In line with prior works on robustness [39, 56] , we report image overall performance and image worst-group performance. Additionally, in order to evaluate the extent to which the VLM understands fine-grained features, we introduce two new metrics: region overall performance and region worst-group performance. Region-level accuracies are computed by performing zero-shot classification with region embeddings and comparing predicted labels to the ground-truth region-level labels provided in the zero-shot classification dataset.Results show that RAVL consistently outperforms prior approaches in mitigating spurious correlations. Across the two evaluation categories in Table 3 , RAVL contributes to an improvement of up to 8.2% on image worst-group performance and 10.8% on region worst-group performance over the nearest baseline. Improvements in region worst-group performance are particularly notable, suggesting that RAVL can better interpret fine-grained features when compared to prior approaches. Additionally, as the accuracy of the discovered spurious correlations in Stage 1 increases, the performance of the RAVL mitigation approach increases proportionally. Our results demonstrate the efficacy of our fine-tuning procedure in mitigating spurious correlations when compared to prior approaches.We construct zero-shot classification datasets D eval V from the test sets of MNIST, FashionMNIST, and COCO. For MNIST and FashionMNIST, we generate region bounding boxes using equally-sized quadrants. For COCO, we use the ground-truth bounding boxes and associated labels. Evaluation datasets are sampled to ensure that a correlation between a eval and e eval does not exist. For MNIST, we perform prompt ensembling for zero-shot classification using the following prompts: A PHOTO OF THE 
conclusion: 
 In this work, we introduced RAVL, a fine-grained region-aware approach for addressing spurious correlations in VLMs. We demonstrate through large-scale, controlled experiments as well as in-thewild evaluations that RAVL can discover (191% improvement in identified correlations) and mitigate (8.2% improvement on worst-group performance) spurious correlations in VLMs. We hope that our work can help (i) diagnose and correct critical failure modes in VLMs prior to deployment and (ii) drive progress towards the development of fine-grained approaches for model robustness.  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All existing data (MNIST, FashionMNIST, COCO, SUN397, ObjectCXR), models (CLIP VLMs, OpenCLIP VLMs, PubMedCLIP VLMs), and code used in this paper have been cited and are available under CC BY, CC BY-SA, CC BY-NC, or MIT licences.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Machine learning models often learn spurious correlations (also known as shortcuts) between image features and class labels. For instance, models have been shown to rely on the presence of chest tubes rather than disease features when identifying collapsed lungs in chest X-rays [34] ; surgical skin markings when detecting melanoma from skin lesions [48] ; and environmental features when performing object recognition tasks [3] . Models that learn spurious correlations will generalize poorly to real-world settings. Our work builds on several recent research directions for (i) discovering and (ii) mitigating spurious correlations.Discovering Spurious Correlations. In the unimodal setting, prior works have developed automated methods for identifying systematic errors resulting from learned spurious correlations in vision models. Using a labeled validation set, these approaches utilize clustering algorithms [13, 43] or lightweight models [42, 22, 31] to identify subgroups of images with high error rates; for instance, in the example in Figure 1 , images containing butterflies without flowers may be identified as one such subgroup. Given a set of images in the identified subgroups, a user can then identify the common features and rectify the data or model. However, recent work has suggested that it is often challenging for humans to interpret identified subgroups and accurately determine the shared features resulting in model failure [25] . Additionally, such methods often focus solely on identifying images with high error rates (e.g. butterflies without flowers) rather than identifying the specific class of features contributing to the error (e.g. flowers). A related line of work has aimed to identify spurious features using human supervision [41] or external concept banks [50] .In the vision-language setting, Yang et al. use an external off-the-shelf object detector to annotate features [56] . Then, for each feature, the difference in zero-shot classification accuracy between images containing the feature and those without the feature is measured; high performance gaps are used to signal spurious features. However, the efficacy of this approach is reliant on the quality of the object detector and a human-in-the-loop is used to verify results; also, as we show in this work, performance gaps alone are not always sufficient for discovering spurious features.Mitigating Spurious Correlations. There is a line of work aiming to mitigate spurious correlations in the context of deep learning [61, 39, 32, 9, 28, 33, 21] . These works explore strategies like data augmentation [55, 59, 57, 22, 50] and instance upsampling [39, 43] . While these approaches have been explored widely in unimodal tasks [29, 53] , mitigating spurious correlations in vision-language settings has not been extensively studied. Some previous works have studied this problem within the context of pretrained VLMs [60, 49, 1] ; however, their setting differs markedly from the fine-tuned VLM setting, where datasets are composed of image-text pairs with no class or subgroup labels. In the fine-tuned VLM setting, existing works predominantly operate at the global image-level [56] , which is unlikely to be sufficient for mitigating fine-grained spurious correlations.We create 654 evaluation settings using data from two domains: (1) synthetic data (MNIST [11] and FashionMNIST [51] ) and ( 2) real-world data (COCO [27] ). Below, we provide implementation details for the four components included in each evaluation setting:1. Predefined spurious correlation: We define a spurious image feature and textual attribute pair (e eval , a eval ). For MNIST and FashionMNIST, e eval represents a red rectangle; a eval is generated from the set {zero, one, two, three, four five, six, seven, eight, nine} for MNIST and {t-shirt, trouser, pullover, dress, coat, sandal, shirt, sneaker, bag, ankle boot} for FashionMNIST. For COCO, we sample e eval and a eval from the list of annotated attributes.2. Fine-tuning dataset: Vision-language fine-tuning datasets D eval F are sampled from the training sets of MNIST, FashionMNIST, and COCO such that the presence of feature e eval is correlated with the presence of text attribute a eval as measured by Cramer's V. For MNIST and FashionMNIST, we synthetically generate text captions by randomly sampling from the following pre-defined prompt templates:. In order to reflect real-world settings where spurious features (e.g. skin markings in dermoscopic images [48] ) may not be annotated in text, text captions in our synthetic settings solely refer to class labels and do not describe the spurious feature. For COCO, we use the provided text captions.Our 654 evaluation settings are summarized in Table 4 . Datasets are licensed under CC BY, CC BY-SA, CC BY-NC, or MIT licenses. In Figure 4 , we provide examples of both synthetic and real-world evaluation settings. and a real-world evaluation setting (bottom row). The example synthetic evaluation setting consists of a predefined spurious correlation between a red rectangle (spurious image feature e eval ) and nine (textual attribute a eval ). This spurious correlation is visible in the vision-language fine-tuning dataset, where the presence of red rectangles and nines are strongly correlated, but not in the evaluation dataset.Similarly, the example real-world evaluation setting consists of a predefined spurious correlation between a person (spurious image feature e eval ) and couch (textual attribute a eval ). Again, this spurious correlation is visible in the vision-language fine-tuning dataset, where the presence of people and couches are strongly correlated, but not in the evaluation dataset.In this section, we extend Section 4.1 by providing additional descriptions of our region-aware loss function.For batch B, we define R s B as the set of all spurious regions in the batch: R s B = Ii∈B R s i . For image I i in batch B, the first component of our region-aware loss function L i R is designed to maximize embedding similarity between non-spurious regions R r i and assigned class label ŷi ; simultaneously, L i R will minimize embedding similarity between non-spurious regions R r i and other class labels in the batch. We formulate L i R as follows:EQUATIONwhere P (R s B ) is a penalty term that encourages dissimilarity between spurious features and correlated class labels as expressed below. Including this term in the denominator of L i R is meant to pull embeddings of spurious regions away from correlated class labels.EQUATIONThe formula for L i R includes two similarity functions: σ and σ m . We define σ and σ m as follows. Let f represent a region embedding function (associated with the image encoder of VLM M) and let g represent a text embedding function (associated with the text encoder of VLM M). Then, for an arbitrary region a, the function f (a) will generate region embedding f (a) ∈ R d with embedding dimension d. For an arbitrary class label b, g(b) will generate text embedding g(b) ∈ R d . Given this notation, we establish the following definitions for σ and σ m :EQUATIONIn the loss term L i R , the function σ m (R r i , ŷi ) will compute the maximum similarity between regions in R r i and class label ŷi . We specifically use the maximum operation in this computation since there are likely to be regions included in R r i that do not reflect the class label; for instance, in the example provided in Figure 1 , there may be non-spurious regions such as trees or leaves included in R r i , which do not align with the animal class labels. The maximum operation ensures that the similarity between at least one region in R r i and the class label should be high. The second component of our region-aware loss function L i A is designed to maximize embedding similarity between non-spurious regions R r i and assigned class label ŷi ; simultaneously, L i A will minimize embedding similarity between other regions in the batch and class label ŷi . We formulate L i A as follows:EQUATION)In this section, we extend the results provided in Section 3.3 with additional evaluations of Stage 1 of RAVL. Our goal is to evaluate the ability of RAVL to discover fine-grained spurious correlations between image features and textual attributes.We implement RAVL according to the details provided in Section 3.1. RAVL includes a clustering step that identifies groups of visually-similar regions. For all evaluation settings, we identify the optimal number of clusters by sweeping all cluster sizes ranging between |Y| * 2 and |Y| * 5; we then select the optimal number of clusters using Silhouette scores. We select these bounds to be larger than the class label set size by several multiples in order to ensure that clusters adequately separate distinct features. Prior works [13, 43] have also utilized overclustering approaches for this objective. We note that users can adjust the bounds based on the composition of their dataset; for instance, complex datasets with diverse features may require a larger range. For MNIST and FashionMNIST, the size of the label set |Y| is 10; For COCO, the size of the label set ranges between 2 and 5.For all baselines, we utilize the official implementations provided by the authors. We adapt Domino, George, and Distilling Failures for our setting by providing region embeddings as input rather than image embeddings.In Figure 5 , we provided an extended version of Figure 2 . We demonstrate that RAVL consistently outperforms baselines across two domains (synthetic images and real images), two model initializations (CLIP-RN50 and CLIP-RN101), and four learned correlation strengths (measured by varying τ eval ∈ {10, 20, 30, 40}).Evaluations on scene classification: Here, we provided extended details on our in-the-wild evaluations performed on scene images (Section 3.3).We leverage ten off-the-shelf VLMs as our model M: CLIP-RN50, OpenCLIP-RN50, CLIP-RN101, OpenCLIP-RN101, CLIP-ViTB/32, OpenCLIP-ViTB/32, CLIP-ViTB/16, OpenCLIP-ViTB/16, CLIP-ViTL/14, and OpenCLIP-ViTL/14 [36, 20] . The four RN models utilize ResNet vision encoders and the six ViT models utilize Vision Transformer backbones. The CLIP models were trained on a proprietary dataset with 400M image-text pairs. OpenCLIP ResNet models were trained on YFCC15M [44] , and OpenCLIP ViT models were trained on LAION2B [40] .We select SUN397 as our zero-shot classification dataset D V [52] . SUN397 consists of scene images from 397 classes. We use the test data from official partition number 1, which consists 19,850 images.We then use an off-the-shelf region proposal network [63] to identify candidate regions.For each VLM M, we perform 397-class zero-shot scene classification on SUN397. We use a prompt ensemble consisting of two prompt templates as provided by CLIP [36] . Due to the large size of the zero-shot classification dataset D V , we perform clustering using the CLARA (Clustering for Large Applications) algorithm, which is an efficient implementation of K-Medoids, and fix the number of clusters as |Y| * 2, which is 794 in this case.Evaluations on chest X-ray classification: Here, we provided extended details on our in-the-wild evaluations performed on medical images (Section 3.3). In recent years, a range of vision [47, 54, 37] and vision-language [45, 5, 4, 62] models have been proposed for learning diagnostic patterns in medical images, and there is a critical need for methods capable of identifying spurious correlations in this domain. Our goal is to determine if RAVL can effectively surface spurious correlations learned by real-world fine-tuned VLMs developed for medical image interpretation.We leverage two off-the-shelf variants of the PubMedCLIP model [12] as our VLM M: PubMedCLIP-RN50 and PubMedCLIP-ViTB/32. The PubMedCLIP-RN50 model utilizes a ResNet-50 vision encoder and was fine-tuned from the CLIP-RN50 model. The PubMedCLIP-ViTB/32 model utilizes a Vision Transformer backbone for the vision encoder and was fine-tuned from the CLIP-ViTB/32 model. Both variants of PubMedCLIP are fine-tuned using ROCO, a large radiology dataset consisting of images and captions collected from PubMed [35] .We select Object-CXR as our zero-shot classification dataset D V . Object-CXR is a dataset of 10,000 frontal chest X-rays compiled from around 300 township hospitals in China [23] . Twelve radiologists with 1-3 years of experience annotated the images, identifying foreign objects within the lung field using bounding boxes, ellipses, or masks, excluding support devices. We retain only bounding boxes and exclude chest X-rays without annotations, resulting in 8,726 object annotations across 4,372 images. For our evaluations, we use the Object-CXR dev split, which includes 974 object annotations across 489 images. We assign image-level labels to the dataset using torchxrayvision [8] , a library that includes a variety of pretrained chest X-ray models. Specifically, we use the XRV-DENSENET121-DENSENET121-RES224-ALL pretrained model to produce multi-class labels for a variety of diseases, including Enlarged Cardiomediastinum, Cardiomegaly, Lung Opacity, Lung Lesion, Edema, Consolidation, Pneumonia, Atelectasis, Pneumothorax, Pleural Effusion, and Fracture.A disease is identified as present if it meets a confidence threshold of 0.60.For each PubMedCLIP VLM M, we perform binary zero-shot classification of cardiomegaly in Object-CXR. Cardiomegaly is a medical condition characterized by the presence of an enlarged heart.After performing a manual search over the text prompt space, we identify CARDIOMEGALY and NORMAL as the prompts that lead to the highest zero-shot classification accuracy for both model variants. The PubMedCLIP-RN50 model achieves an overall zero-shot classification accuracy of 74.2, with an accuracy of 14.0 on the group of images with cardiomegaly and an accuracy of 91.9 on the group of images without cardiomegaly. The PubMedCLIP-ViTB/32 achieves an overall zero-shot classification accuracy of 39.4, with an accuracy of 80.4 on the group of images with cardiomegaly and an accuracy of 28.0 on the group of images without cardiomegaly. Interestingly, given the selected prompts, we note that the PubMedCLIP-RN50 and the PubMedCLIP-ViTB/32 models exhibit inverse trends, with PubMedCLIP-RN50 achieving higher performance on the class of images without cardiomegaly and PubMedCLIP-ViTB/32 achieving higher performance on the class of images with cardiomegaly.Given VLM M and zero-shot classification dataset D eval V , we apply RAVL in order to surface learned spurious correlations. Similar to our controlled evaluations on synthetic datasets, we perform K-Medoids clustering with the number of clusters ranging from 20 to 50. The optimal number of clusters is selected using Silhouette distance; we use 24 clusters for PubMedCLIP-RN50 and 20 clusters for PubMedCLIP-ViTB/32. The final cluster performance gap metric G c associated with the top-ranked spurious feature cluster is 0.041 and 0.119 for the PubMedCLIP-RN50 and PubMedCLIP-ViTB/32 models respectively. Extended Results: In Figure 6 , we extend the qualitative results provided in Figure 3 with additional examples of spurious correlations surfaced by RAVL in off-the-shelf VLMs. We make the following observations:• For the OpenCLIP ViT-L/14 model, RaVL surfaces a feature cluster consisting of green plants and fences. We observe a performance gap of 18.3 points between images with class label outdoor chicken coop that contain the RaVL-identified feature and those that do not contain the feature. This suggests that the OpenCLIP ViT-L/14 model can better classify an outdoor chicken coop scene when green plants and fences are present. • For the CLIP ViT-B/32 model, RaVL surfaces a feature cluster consisting of people. We observe a performance gap of 24.3 points between images with class label pub (indoor) that contain the RaVL-identified feature and those that do not contain the feature. This suggests that the CLIP ViT-B/32 model can better classify a pub (indoor) scene when people are present. • For the OpenCLIP ResNet-101 model, RaVL surfaces a feature cluster consisting of chairs.We observe a performance gap of 23.3 points between images with class label restaurant patio that contain the RaVL-identified feature and those that do not contain the feature. This suggests that the OpenCLIP ResNet-101 model can better classify restaurant patio scenes when chairs are present.We train model M new using a single NVIDIA A100 GPU with an initial learning rate of 5e-5. We use a batch size of 128 and train for 100 epochs with early stopping. We set the loss temperature as τ = 0.07 and use λ = 0.8 in loss function L. In line with prior works that utilize locked image-text fine-tuning [2, 46] , we freeze the text encoder and solely learn weights for the image encoder. We generate candidate regions for the fine-tuning dataset D eval F using a region proposal network with identical settings to prior work [63] .Below, we provide additional implementation details for the five mitigation baselines we explore in this study. Since there are a limited number of existing approaches designed for mitigating spurious correlations in fine-tuned VLMs, we adapt several existing methods for our setting in order to train model M new :• Standard VLM Fine-Tuning: We perform standard VLM fine-tuning with the original loss function L CL used to train model M. In our experiments, L CL is the CLIP objective [36] . evaluation settings, we observe embedding generation to take a mean of 24.5 seconds on a single A100 GPU. • Finally, given candidate regions and corresponding embeddings, the remainder of the RaVL discovery procedure (clustering and computation of metrics) can be run completely on CPU. Across a set of 10 evaluation settings on COCO and FashionMNIST, we observe that clustering and computation of metrics require a mean of 3.4 seconds to run.The mitigation stage of RAVL requires finetuning a VLM M new . Across a set of 10 evaluation settings on COCO and FashionMNIST, we observe that the inclusion of our fine-grained region-aware loss function at this stage adds an average of 0.15 seconds per training step (on a single A100 GPU) in comparison to the original fine-tuning procedure for M.Societal Impact: The goal of our work is to improve robustness of fine-tuned VLMs to spurious correlations. As VLMs become more commonplace in society, we hope that our approach can enable users to better detect and mitigate model failures prior to deployment. We also note that our work includes a series of evaluations on medical images; rigorous clinical testing is necessary before robustness approaches are deployed in healthcare settings.In line with prior works in vision-only and vision-language settings, our method is specifically designed to surface and mitigate local, fine-grained spurious features. There may be some sources of spurious signal that do not manifest in this way; for instance, features like image brightness or gender can be considered global features, where the spurious signal is not localized to a particular image region. Our approach is not designed for these global spurious features. Rather, our problem setting is inspired by the many real-world, practical examples of region-level spurious features that have been demonstrated in literature to affect model performance, such as image-level markings in dermoscopic images [48] , medical devices in radiographs [34] , and text markers in medical images [10] . • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).We note that since the formula for Gc involves a summation over class labels, raw values of Gc for our 2-class chest X-ray classification task are lower than those for our397-class scene classification task. 


Decision: RaVL is a method to detect and mitigate spurious correlations in VLMs. The method tries to figure out what features contribute to zero-shot classification errors and then try to remedy those. The reviewers agree this is an important problem, the paper is well written and presents a contribution to the literature. There was some discussion with one of the reviewers about the impact of batch size on performance, but I do not think this changes the main findings of the paper. Overall the method seems solid, experimental evidence seems satisfactory. For these reasons I vote to accept this paper.

Reviews: review: paper_summary: paper_summary: This paper tackles spurious correlations between image features and textual attributes in fine-tuned VLMs. It proposes an approach to discover and mitigate spurious correlations using local image features (image regions rather than a whole image). Experiments are done in both controlled settings and realistic settings.
strength_weakness: strength_weakness: 1) Looking into the regional features may make the approach computationally expensive. Computational complexity analysis is missing in the paper. 
2) A lot of experiments were done under controlled settings. More evaluations in the wild would make the results stronger.
3) A small error in writing, line 259 says “are designed for unimodal settings” - ref [50] is designed for multi-modal settings.
questions: questions: Have you tried the approach on other settings, other than fine-tuned VLM?  For example, spurious correlations can happen in other vision classification models as well.
limitations: limitations: There are discussions on limitations in the supplementary material.
paper_summary: paper_summary: Vision-language models (VLMs) tend to exhibit poor zero-shot performance when compared to task-specific models. However, fine-tuned VLMs may capture spurious correlations in domain-specific datasets which may be small in size. The paper proposes an automated spurious correlation detection and mitigation method for fine-tuned VLMs. The method first discovers spurious correlations by leveraging a region-level clustering approach to identify precise image features contributing to zero-shot classification errors. Then, it mitigates the identified spurious correlation with a region-aware loss function that enables the VLM to focus on relevant regions and ignore spurious relationships during fine-tuning. Experiments demonstrate the effectiveness of this method on general domain and medical-domain VLMs.
strength_weakness: strength_weakness: There is a lack of computational complexity analysis. To detect fine-grained spurious correlations, RAVL needs to segment images into multiple regions, cluster all the regions per class, and calculate the contributions of each cluster to mispredictions. Therefore, the computational complexity seems high. It would be better to analyze the time cost of the proposed method.
questions: questions: 1. How to obtain region-level labels when constructing the evaluation dataset (Line 220)?

2. How to determine the hyperparameters used in the spurious correlation discovery and mitigation stages?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper introduces RAVL, a method designed to identify and address spurious correlations in Vision-Language Models (VLMs). The authors emphasize two key areas: (a) prioritizing local image features over global image-level features, and (b) concentrating on the fine-tuning phase rather than the pre-training phase. They implement the first idea through the following steps: 1. Utilizing the VLM embedding space to extract candidate image features from the validation set. 2. Filtering these candidate features to pinpoint those that directly contribute to classification errors. 3. Ranking the remaining image features based on the extent of their learned spurious correlations. To mitigate spurious correlations, the authors regularize the dissimilarity between the spurious regions and correlated labels while encouraging high embedding similarity.
strength_weakness: strength_weakness: 1. This framework is limited to mitigating geometric spurious correlations by focusing on regional candidates. While some studies such as [1] highlight texture and background information as significant contributors to spurious correlations, it is more desirable to address all potential sources of spurious correlations.ss all the potential spurious correlations.

2. Although they used RoIAlign, there are no qualitative visualization results to show which regions affect which classes. I believe this would be important for understanding the model's behavior from a human vision perspective.

[1] Geirhos, Robert, et al. "ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness." arXiv preprint arXiv:1811.12231 (2018).
questions: questions: 1. What if this framework, instead of using RoIAlign, considers feature maps as candidates and filters them channel-wise? If this approach is effective, it could provide a more general framework from the perspective of Weakness 1, as feature maps are activated differently, representing various features [1].

2. Why did the authors use K-Medoids clustering? Additionally, I am curious about the authors' choice to use softmax-normalized embeddings. Feature maps might offer more diverse information for comparison during clustering.

3. Is this framework robust on batch size?


[1] Jeon, Myeongho, Myungjoo Kang, and Joonseok Lee. "A Unified Framework for Robustness on Diverse Sampling Errors." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
limitations: limitations: The authors discussed the limitations and potential negative societal impacts of their work in the Appendix. I fully agree with their assessment.

Comments: title: Response to Reviewer LC5i
Response to Reviewer ONXK (2/2)
Response to Reviewer ONXK (1/2)
Response to Author Feedback
Response to Author Feedback
comment: Thanks for the response. My concerns have been well addressed.
Below, we respond to the points raised by Reviewer LC5i during the discussion period. 

**Definition of spurious correlations. “For some “true” image features, when the following are true, they may also appear to have the property (e.g., see lines 124 - 127 in the paper) mentioned above. (1) The “true” image features only appear in a subset of the class y. This can happen because not all images in the same class have the same features. (2) The training data and test data are sampled differently: the “true” features only appear (or appear more often) in the training data with class y, and appear less often in the test data with class y. How do you distinguish between spurious image features and the “true” image features with different appearing frequencies in training and test data?”**

Our definition of spurious correlations is consistent with prior work (e.g. [1,2]). In reference to the specific cases listed by the reviewer above, it is standard practice to consider a "true" image feature as one that appears consistently within the class. This means that “true” features will (1) appear in all images rather than just a subset and (2) will be consistently associated with the class label $y$ at both training and test time. For example, when considering images from the class “bird”, it is reasonable to assume that all images will include the true features of feathers, wings, and a beak, which are defining characteristics of birds. This will be true for all birds, independent of whether they are in a training or testing dataset.

We refer the reviewer to Singla et al. [2], which formally defines true features as those that are "always a part of the" class definition; meanwhile, spurious features are defined as those that "are likely to co-occur" with the true features but are not a part of the class definition. We follow these definitions in our work. We demonstrate with 654 evaluations in both synthetic and real-world settings that our approach can differentiate between spurious features and true features, as shown in Figure 2 and Table 1; we corroborate this further with in-the-wild experiments.

We hope that this clarifies the definitions of true and spurious features.

[1] Eyuboglu et al. “Domino: Discovering Systematic Errors with Cross-Modal Embeddings” ICLR 2022. 

[2] Singla, S. et al. Salient ImageNet: How to discover spurious features in Deep Learning? ICLR, 2022.
**Feature Maps. “(i) Is it not possible to mitigate shortcut correlations (SCs) in an automated manner using feature maps? (ii) This makes sense. (iii) Why do you believe that feature maps represent global features? Given the receptive field, feature maps typically capture local features within the feature space (e.g., the first channel corresponds to the top-left, while the last channel corresponds to the bottom-right).”**

Our approach uses RoIAlign to extract relevant regions of the feature maps, which we then compress into a low-dimensional embedding. These embeddings (1) can then be accurately and efficiently processed in an automated fashion and (2) are explicitly designed to characterize local, region-level features. We emphasize that using RoIAlign to embed regions of interest is a standard technique grounded in prior literature (e.g. [3]).

(i) Performing automated identification of region-level spurious correlations using raw feature maps would be computationally challenging due to the presence of large numbers of feature maps; for instance, a ResNet-50 model produces 2048 distinct 2D feature maps per image at the last layer. For a small validation dataset with only 1K images, this would result in over 2 million 2D feature maps. Strategies for reducing the number of feature maps (e.g. via pruning) can result in the loss of important information. Using ROIAlign and region-level embeddings enables us to reduce dimensionality of feature maps while preserving semantic meaning, enabling efficient automated analysis.

(iii) We first clarify that a feature map produced by a model has three dimensions: height $H$, width $W$, and channels/depth $C$. For a ResNet-50 with an input image size of 224x224, the feature map dimensions at the last layer are $H=7$, $W=7$, and $C=2048$. The channels themselves do not explicitly represent image geometry and locations (i.e. the top left or bottom right); rather, each channel corresponds to a 2D feature map of size $7 \times 7$ that characterizes a feature within the input image. These 2D feature maps exhibit significant variations in feature granularity;  the features characterized in each map can range in granularity from global (e.g. color, texture) to local (e.g. individual leaves on a tree). This variability limits the use of raw feature maps for our task. On the other hand, ROIAlign is an effective strategy for embedding specific regions of interest, focusing exclusively on local, fine-grained features.

We hope that the above responses address your concerns. Thank you for your time.

[1] Tiu, E., et al. Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning. Nat. Biomed. Eng (2022).

[2] Huang, Z et al. A visual–language foundation model for pathology image analysis using medical Twitter. Nat. Medicine (2023).

[3] Zhong et al. RegionCLIP: Region-based Language-Image Pretraining. CVPR 2022.
Below, we respond to the points raised by Reviewer ONXK during the discussion period.

**Batch Size. “With only a brief experiment provided in Q5, I am not convinced that RaVL outperforms other baselines across various batch sizes. The impact of batch size is not a trivial factor.”**

In our paper, we used a batch size of 128 across all mitigation experiments (Section 4). We selected a batch size of 128 for the following reasons. First, a batch size of 128 is the largest size that will fit on a single A100 GPU; we specifically used a single A100 GPU for all experiments in order to demonstrate the ease of use and cost-effectiveness of RaVL. Second, a batch size of 128 is either larger or equivalent to the batch sizes used by prior works that fine-tune VLMs for real-world applications (e.g. medical settings [1,2]) demonstrating that our choice of the batch size hyperparameter is consistent with real-world practice. Third, the Spurious-Aware Mitigation baseline, which is the only other previously-developed approach for the fine-tuned VLM setting (Appendix A), is designed to operate with a batch size of 128; consequently, using a batch size of 128 across evaluations enables a fair comparison with this baseline. As shown in Table 3 in the paper, RaVL Stage 2 outperforms all mitigation baselines.

Beyond the batch size justification shared above, to be further responsive to the provided reviewer comment, we extend the batch size ablations provided with our original rebuttal by evaluating RaVL across three batch sizes (32, 128, and 256) and six baselines. We evaluate multiple COCO evaluation settings for each ablation. Across a total of 72 trained VLMs $M_{new}$ evaluated in the tables below, **we show that RaVL outperforms all baselines across all evaluated batch sizes**. We thank the reviewer for this suggestion, and we will be sure to update the manuscript with this ablation. 

| Batch Size | Method | Img. Overall 	| Img. Worst Group | Reg. Overall | Reg. Worst Group|
|----------------------|------------------|------------------|------------------|------------------|------------------|
| 32 | Standard FT   	| 59.1 	| 25.2	  | 69.3	 |44.1 	 | 
| 32 | Upsampled FT	| 60.3 	| 28.0	 | 65.2 	| 34.3	| 
| 32 | VL-ERM 		|58.4	 |21.6	  | 67.7	 | 37.2	| 
| 32 | VL-GDRO 	 	| 59.6 	|24.3	 | 69.3	 | 35.2	| 
| 32 | Spurious-Aware 	|54.2 	|12.2	 | 66.9	 | 36.2	| 
| 32 | RaVL (Ours)		| **64.3**| **39.7**| **75.5** |**44.7**| 


| Batch Size | Method | Img. Overall	 | Img. Worst Group | Reg. Overall | Reg. Worst Group|
|----------------------|------------------|------------------|------------------|------------------|------------------|
| 128 | Standard FT    	| 67.6 	| 33.6	 | 77.3 	| 46.0 	|
| 128 | Upsampled FT 	| 69.5	 | 37.1	 | 79.6 	| 52.9	 |
| 128 | VL-ERM		 | 68.2 	| 29.0	| 78.1	| 46.3	|
| 128 | VL-GDRO 		 | 69.7	 | 31.5	 | 77.5 	| 44.6	|
| 128 | Spurious-Aware 	| 69.4	 | 31.8	 | 78.6 	| 48.1	|
| 128 | RaVL (Ours) 	| **71.0** | **42.9** | **82.1** | **57.4**|


| Batch Size | Method | Img. Overall 	| Img. Worst Group | Reg. Overall | Reg. Worst Group|
|----------------------|------------------|------------------|------------------|------------------|------------------|
| 256 | Standard FT    	| 64.3 	| 41.4	 | 73.1 	| 53.9 	|
| 256 | Upsampled FT 	| 66.8	 | 48.8	 | 74.0 	| 58.5	 |
| 256 | VL-ERM		 | 62.5 	| 24.9	| 73.2	| 44.9	|
| 256 | VL-GDRO 		 | 63.5	 | 29.8	 | 75.2 	| 45.1	|
| 256 | Spurious-Aware 	| 66.7	 | 34.3	 | 74.6 	| 44.5	|
| 256 | RaVL (Ours) 	| **68.1** | **53.9** | **79.0** | **66.7**|

**Visualizations. “Thank you for including the additional qualitative experiment. However, I find the statement "The new VLM more precisely relies on core features" unclear. Could you please specify which core features you are referring to?”**

The images in the figure are obtained from the test set of a COCO evaluation setting in which the fine-tuned VLM $\mathcal{M}$ learned a strong spurious correlation between the class label "chairs" and the spurious feature "person". Visualizations show that before mitigation, the VLM is unable to precisely localize the core features associated with the class labels, often relying on the presence of people or various background details. After the RaVL mitigation procedure, the VLM $\mathcal{M}_{new}$ more precisely relies on core features (namely, the chairs).
Thanks again for the rebuttal.

This paper is based on the assumption that, “a model M that has learned a spurious correlation between an image feature e_a and a textual attribute y will demonstrate low zero-shot performance on (i) images in D_V with label y without the feature e_a and (ii) images in D_V with other labels Y \ {y} with the feature e_a. ”  (e.g., see lines 124 - 127 in the paper)

I have one more question on the paper.  For some “true” image features, when the following are true, they may also appear to have the property (e.g., see lines 124 - 127 in the paper) mentioned above.  
1)  The “true” image features only appear in a subset of the class y. This can happen because not all images in the same class have the same features.
2) The training data and test data are sampled differently: the “true” features only appear (or appear more often) in the training data with class y, and appear less often in the test data with class y. 

How do you distinguish between spurious image features and the “true” image features with different appearing frequencies in training and test data?
Thanks for the detailed rebuttal.
Thanks for your detailed responses. My concerns are well addressed.
Please let me know if I misunderstood anything.
Thank you for your feedback. However, after further consideration, I have become less favorable toward this submission and have decided to downgrade my initial score for the following reasons:

- In response to Q1, the authors suggest that the empirical results are a key justification. However, with only a brief experiment provided in Q5, I am not convinced that RaVL outperforms other baselines across various batch sizes. The impact of batch size is not a trivial factor.

- Thank you for including the additional qualitative experiment. However, I find the statement "The new VLM more precisely relies on core features" unclear. Could you please specify which core features you are referring to?

- (i) Is it not possible to mitigate shortcut correlations (SCs) in an automated manner using feature maps? (ii) This makes sense. (iii) Why do you believe that feature maps represent global features? Given the receptive field, feature maps typically capture local features within the feature space (e.g., the first channel corresponds to the top-left, while the last channel corresponds to the bottom-right).

- The justification provided for the use of K-medoids clustering seems appropriate to me.