RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models
Authors: Maya Varma
Jean-Benoit Delbrouck
Zhihong Chen
Akshay S Chaudhari
Curtis Langlotz
Venue: NeurIPS 2024 poster (2024)
Keywords: vision-language models
robustness
spurious correlations
fine-grained
Primary area: evaluation
TL;DR: We present RaVL, an approach for discovering and mitigating spurious correlations in fine-tuned vision-language models

Abstract: Fine-tuned vision-language models (VLMs) often capture spurious correlations between image features and textual attributes, resulting in degraded zero-shot performance at test time. Existing approaches for addressing spurious correlations (i) primarily operate at the global image-level rather than intervening directly on fine-grained image features and (ii) are predominantly designed for unimodal settings. In this work, we present RaVL, which takes a fine-grained perspective on VLM robustness by discovering and mitigating spurious correlations using local image features rather than operating at the global image level. Given a fine-tuned VLM, RaVL first discovers spurious correlations by leveraging a region-level clustering approach to identify precise image features contributing to zero-shot classification errors. Then, RaVL mitigates the identified spurious correlation with a novel region-aware loss function that enables the VLM to focus on relevant regions and ignore spurious relationships during fine-tuning. We evaluate RaVL on 654 VLMs with various model architectures, data domains, and learned spurious correlations. Our results show that RaVL accurately discovers (191% improvement over the closest baseline) and mitigates (8.2% improvement on worst-group image classification accuracy) spurious correlations. Qualitative evaluations on general-domain and medical-domain VLMs confirm our findings.

RAVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models
Maya Varma Stanford University
Jean-Benoit Delbrouck Stanford University; Hugging Face jodel@stanford. edu
mayavarma@cs. stanf owwfoor.ce.. edu
Zhihong Chen Stanford University zhihongo@stanford.oo edu
Akshay Chaudhari! Stanford University
Curtis Langlotz i Stanford University
akshayac@@tannord.cco edu
langlotz@stanf wareetannooooooodoooooo edu
Abstract
Fine-tuned vision-language models (VLMs) often capture spurious correlations between image features and textual attributes, resulting in degraded zero-shot performance at test time. Existing approaches for addressing spurious correlations primarily operate the global image-level rather than intervening directly on fine-grained image features and (ii) are predominantly designed for unimodal settings. In this work, we present RAVL, which takes a fine-graines perspective on VLM robustness by discovering and mitigating spurious correlations using local image features rather than operating the global image level. Given a fine-tuned VLM, RAVL first discovers spurious correlations by leveraging region-level clustering approach to identify precise image features contributing to zero-shot classification errors. Then, RAVI. mitigates the identified spurious correlation with a novel region-aware loss function that enables the VI.M to focus on relevant regions and ignore spurious relationships during fine-tuning. We evaluate RAVL on 654 VLMs with various model architectures, data domains, and learned spurious correlations. Our results show that RAVL accurately discovers (191 % improvement over the closest baseline) and mitigates (8.2% improvement on worst-group image classification accuracy) spurious correlations. Qualitative evaluations general- domain and medical-domain VLMs confirm our findings
Introduction
Contrastive vision-language models (VLMs) (c.g., CLIP (36] and ALIGN (44 are powerful class of models that jointly learn relationships between images and text. VLMs are generally pretrained on web-scale datasets with millions of image-text pairs and have been shown exhibit impressive capabilities on a wide range of downstream tasks. In particular, VLMs have the ability to perform tasks in zero-shot manner without utilizing explicit task-specific training data; this is accomplished by modeling downstream tasks (e.g., image classification, text-to-image retrieval) as image-text matching tasks (36]
However, pretrained VLMMs can exhibit poor zero-shot performance when compared to state-of-the--e task-specific models, particularly on challenging or out-of- downstream tasks [36] 7 17 19]. As a result, pretrained VLMs are often fine-tuned on domain -ppecfi vision- languuge datasets in order
Equal senior authorship.
Code: https://gittub.com http://iiituucccoeeeeeeeeeeeeeee wwt.:/ooot/ttaaaeeeeeee AIM./od-----MIA//I-Ia
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Stage 1: Discovery
Stage Mitigation Gitelliaag- Oiéective
Our Approach: RaVL
Huttberfty
bird
Setting is -unee dataset Dp where the prosence of butterflies is comclated with the presence flowers
buttberlyy
hind
Visio Encader
butterfly
Gratned Region-
lind
X
renaribe caniaarity
butterfly
sind
Goal: Identiify learned spurious correlations betwween image features (flowar) and textual attributes (buttarfly)
Goak: Mitigate the spurious correlation
Figure I: Region-aware Vision-Languer learning (RAVL). RAVL takes a fine-grained perspective on VLM robustness by discovering and mitigating spurious correlations using local image features.
to improve zero-shot performance on tasks of interest. For instance, recent works have fine-tuned the CLIP VLM (36 on vision-language datasets consisting of (i) chest X-rays and paired physician (ii) pathology data and paired text 17 19]. and (iii) product images and paired captions
reports 45
from online fashion retailers
Domain-specific vision-language datasets used to fine-tune VLMs may be small in size, preventing VLMs from gaining the robustness benefits that come with training on diverse, web-scale data 66 As result, fine-tuned VLMs may capture spurious correlations between image features and textual attributes [56]. For instance, consider a VLM fine-tuned on an animal image-text dataset where the presence of butterflies is closely correlated with the presence of flowers (Figure D. Consequently, the VLM may leamn to incorrectly associate the image features corresponding to flower with the textual attribute butterfly. At test time, the VLM is likely to exhibit degraded zero-shot classification performance on (i) images of butterflies without flowers and (ii) images of other animals with flowers.
Improving robustness of fine-tuned VLMSs to spurious correlations is challenging for the following two reasons. First, existing automated approaches primarily discover and mitigate spurious correlations the global image level rather than intervening directly on fine-grained image features. Such approaches discover spurious correlations by identifying coherent groups of misclassified images in an automated fashion 1K
then, the identified spurious correlation can be mitigated
during training using data augmentation or robust optimization 45 39 22 56]. However, recent works have suggested that such global image-level strategies (i) discover spurious correlations that align poorly with human-interpree interpretable attributes [5] and (ii) may not effectively enable models to
ignore spurious correlations during training [5 Second, existing approaches for discovering
and mitigating spurious correlations are predominantly designed to improve robustness of unimodal
image classification models [39 4 or pretrained VL.Ms (60 49). These settings differ substantially from the fine-tuned VLM setting, which presents several unique challenges such as the absence of class and subgroup labels in the training set and the inclusion free-form text.
In this work, we address these challenges by introducing Region-aware Vision-Languag learning (RAVL), an approach for improving the robustness of fine-tuned VLMs to spurious correlations. RAVL takes a fine tree-grained perspective on VLM robustness by discovering and mitigating spuri- ous correlations using local image features, rather than operating at the global image level. Our contributions are:
First, given a fine-tuned VLM, RAVL discovers learned spurious correlations between image features and textual attributes. Using labeled classification dataset, we decompose images into candidate regions, utilize the VLM embedding space to group visually-similar regions into feature clusters, and quantitatively evaluate the effects of each feature on zero-shot classification errors. Second, given a ranked list of image features that the VLM has learned to spuriously correlate with one or more textual attributes, RAVL mitigates the identified spurious correlations. Our key insight is that region-level information can be leveraged during VL.M fine-tuning in order to improve model robustness. To this end, we introduce novel region-aware loss function

---

that encourages the VLM to focus on relevant regions and ignore spurious relationships during fine-tuning.
In order to evaluate RAVL, we introduce large-scale evaluation framework for controlled, fine- grained evaluations of VLM robustmess on synthetic and real-world data. Our framework consists of 654 fine-tuned VLMs paired with annotations for the ground-truth spurious correlations learned by each VLM. Across these evaluation settings, (i) RAVL accurately discovers spurious correlations, achieving a 191% % improvement over the closest baseline, and (ii) RAVL effectively mitigates spurious correlations, achieving up to an 8.2% improvement on worst-group image classification accuracy. Qualitative evaluations on general-domain and medical-domain VLMs confirm the utility of RAVL.
This paper is organized follows. In Section 2 we introduce our problem setting. Then, in Section 3 we present Stage 1 of RAVL, including our proposed methodology for discovering spurious correlations, our large-scale evaluation framework, and experimental results. In Section we introduce Stage 2 of RAVL, including our proposed methodology for mitigating spurious correlations as well as experimental results. Finally, we conclude in Section
Related Work: Our work builds on several recent research directions for discovering and mitigating spurious correlations. We provide analysis of related works in Appendix Section
2 Preliminaries
In this section, formally describe our problem setting. Datasets used for fine-tuning VLMs can be expressed as DF = { H= where 4, represents image inputs and Tū represents paired free-form text. We do not assume access any class subgroup labels.
The performance of fine-tuned VLMs can be characterized with zero-shot classification tasks. line with prior work [13 22 (56,, assume that the zero-shot classification dataset includes validation split Dv = {(I,, y )) with images I, and known ground -truthhtuthhhuo class labels y/s E V. where y denotes the set of all possible class labels. At evaluation time, classification performance is computed by encoding class labels in y as text and matching images to the closest class label using embedding similarity. We do not assume access to any subgroup labels.
Fine-tuned VLMs may learn spurious correlations between image features and textual attributes. Let e. represent the image features corresponding to visual concept (e.g.. flowers in Figure and y E y represent a class label (e.g., "butterfly" in Figure [T] such that e, and y share no causal relationship. Then, fine-tuned VLM that has learned spurious correlation will be unable to disentangle e, and at evaluation time. This will manifest in low zero-shot classification performance on the following two subgroups of data: (i) images from class label y without the feature e, and (ii) images from other class labels y {y} with the feature e..
However, since neither the fine-tuning dataset Dr nor the evaluation dataset Dy include subgroup labels corresponding to visual concepts a, discovering and mitigating such spurious correlations poses a challenge. For instance, in Figure [ there are no annotations for flowers in datasets Dp and Dy, making it challenging to identify and address the learned spurious correlation between image features corresponding to flowers and the textual attribute corresponding to "butterfly".
In the following sections, we will discuss our automated approach RAVL, which aims address this challenge by employing Aine-grained region-level information to discover (Section 3) and mitigate (Section 4 spurious correlations in fine-tuned vision- langugee models.
3 Discovering Spurious Correlations in Fine-Tuned Vision-Languaae Models
In this section, we present the first stage of RAVL, which aims to discover learned spurious correla- tions in VLMs. In Section 3.1 we discuss our region-aware approach for discovering fine- ane-ggrined spurious correlations. Then, in order to quantitatively evaluate the efficacy of spurious feature dis- covery methods, we introduce large-scale evaluation framework in Section 3.2 Finally, in Section we use our evaluation framework to demonstrate that RAVI outperforms prior approaches in discovering fine-grained spurious correlations between image features and textual attributes.
3

---

3.1 Our Approach: Discovering Spurious Correlations
The first stage of RAVL aims to identify spurious correlations between image features and textual attributes learned by a fine-tuned VLM M. In contrast to prior works that have incorporated humans in the loop in order identify spurious correlations [56] 30 RAVL is fully automated approach. Additionally, whereas previous automated methods for discovering spurious correlations focus predominantly on identifying groups of images with high error rates [3] our approach identifies specific image features that model M has learned to spuriously correlate with textual attribute. Our goal is to discover precise spurious correlations that can be easily interpreted by humans.
As discussed in Section 2 a model M that has learned a spurious correlation between an image feature e and textual attribute y will demonstrate low zero-shot performance on (i) images in Dv with label y without the feature e,, and (ii) images in Dv with other labels y \ {y} with the feature ea. The key challenge lies in identifying such relationships when no annotations are provided for visual concepts a. RAVL addresses this challenge by (1) obtaining candidate image features in Dv. (2) identifying the candidate image features that, when present in an image, directly contribute to classification errors, and (3) ranking the identified image features by degree of learned spurious correlations.
Obtaining candidate image features. RAVI. first utilizes the zero-shot classification dataset Dv the fine-tuned VI.M .M to extract an
to identify candidate image features. To this end, image embedding for each image 7, in Dy and a text embedding for each class y E y Zero-shot classification is performed using the computed embeddings; this results in voftmax-normallin image score distribution vector $1, RP',, where [)]] represents the number of classes. Then, we decompose each image I, in Dv into a set of candidate regions R2. There are a variety of ways in which image can be decomposed into regions, such as dividing images into equal-sized segments (e.g.. quadrants) or using region proposal networks (RPNs) [38]. Ideally, regions should capture key features in the image; however, we emphasize that RAVL does not require ground- truth region-level annotations. We then apply RolAlign [66 63 to the image encoder of M to extract embeddings for each region. Zero-shot classification is performed using the computed region embeddings, resulting in ceftmax-normalized region score distribution matrix SR, [
Given region-level embeddings for all candidate regions in Dy, we next aim to identify coherent groups of image features that occur consistently throughout the dataset (e-g., features corresponding to "flower" or "butterfly" in Figure To this end, we cluster the computed region-level embeddings using the K-Medoids algorithm with cosine distance. The optimal number of clusters is selected in an automated fashion using Silhouette distance. The resulting clusters (denoted as C) capture key image features in Dy. For feature cluster e E C, let e, denotes the set of features in cluster c.
Identifying candidate image features that directly contribute to classification errors. We now seek to identify features that, when present in an image, are directly responsible for prediction errors.
Let R2 represent the set of regions assigned to cluster and let I. represent the set of images associated with the regions in cluster c. We identify labels for images in Iz; we designate this label set as Ve. For each class label y E Ve. we identify all images in I, with label y, and designate zero-shot classification accuracy on this subset of n", images as ph Then, we identify all images in D. with label y that do not have a region included cluster c, and we designate zero-shot classification accuracy on this subset of nuut images as pout-
We now introduce the cluster influence score, which evaluates the extent to which features C, contribute to mispredicted image classification labels. We restrict our evaluation to only include mispredicted images in I. with ground-truth labels y such that ph, < Pime: we will refer to this subset as Zerr C I,- For each image I, E I we extract (i) the image score distribution vector s,, and (ii) the region score distribution matrix SR,- We use Sr, identify the predicted image class ÿ. and then identify the region rmaaz in R4 with the highest score for class ý.
Definition I (Cluster Influence Score). For cluster e and label y. the cluster influence score is the proportion of images 4, E Ierr with label where the identified highest-scoring region r.maz is part of cluster e (i.e., yaa E R.y:
H
Σ LEIS
1 mar E Rel
(I)
14 E Ierr 1ya
wH
4

---

The final cluster influence score for cluster e is computed as the maximum over all labels as H. = mar Hy. High values of H. show that features e, are similar the incorrect label in the vision-language embedding space; this suggests that for a given image with an incorrect prediction, feature e, is more likely to contribute to the misprediction than other features in the image. On the other hand, low values of He are likely to indicate that feature represents a core feature associated with the class label or a neutral feature that does not affect predictions.
Given He for each feature cluster, we prune all clusters with influence scores below a threshold of n which we set to 0.25 in all experiments.
Ranking image features by degree of learned spurious correlation. For each remaining feature cluster, next aim to determine the extent to which the presence or absence of features e, affects classification performance; we introduce the cluster performance gap metric to this end.
Definition 2 CClsterr Performance Gap). For cluster c and label y, the cluster performance gap is the weighted difference between zero-shot classification accuracy on images with features e, and images without features e::
G
Wy x (p% pout):
where Why is a simple weighting factor computed as Wiy = 2 x [min(n",. n. (nn + m.tt)l
Since spurious correlations result in consistent errors opposed to isolated misclassifications the weighting factor is designed to prioritize stronger spurious correlations that result in a larger number of errors. G ranges between O and 1. The final performance gap metric for cluster e is computed across all labels as Ge = Eyex, Gy A high value of Ge suggests that the presence or absence of features e, contribute to large class-level variations image classification performance.
Given Ge for each feature cluster, we rank clusters in order from highest to lowest values. The output of this stage is ranked list of image features that model M has learned spuriously correlate with one or more class labels in V
3.2 Experimental Setup: Designing Large-Scale Evaluation Framework
We now discuss our approach for evaluating RAVL. Evaluating the accuracy of predicted spurious correlations is challenging because the ground-truth truth spurious correlations learned by model M are typically unknown. Previous works on VLM robustness evaluate discovered spurious correlations with qualitative experiments, human-in-the-loog evaluations, or small-scale datasets (56). Our aim in this section is to introduce a large-scale experimental setup where the ground-truth spurious correlations learned by VLMs are known and annotated in advance; this can then enable us to determine whether the features discovered by RAVL in Section 3.1 accurately align with the ground- truth. Our evaluation framework motivated by prior work [26 13:: 3: however, in contrast to existing approaches, we introduce evaluation settings that are designed (i) for evaluating robustness approaches at the fine mee-grained region level rather than the global image-level, and (ii) for evaluating VLMs rather than unimodal models.
Designing Controlled Evaluations: Our evaluation framework artificially induces spurious corre- lations in the VLM fine-tuning data; then, given the known pre-defined spurious correlation and VLM that learned the desired spurious correlation, we can quantitatively evaluate the extent to which RI discovers the correlation.
We create set of evaluation settings using data from two domains: (1) synthetic data (MNIST and FashionMNIST (51)) and (2) real-world data (COCO (27)). Each evaluation setting consists of the following components:
1. Predefined spurious correlation: We define a spurious image feature and textual attribute pair (ceval m.a)). ). For MNIST and FashionMNIST, eval represents red rectangle; a ana is generated from the set of class labels {zero, one, two, three, four five, six, seven, eight, nine) for MNIST and [t-shirt, trouser, pullover, dress, coat, sandal, shirt, sneaker, bag, ankle boot) for FaSHionMNIST. For COCO, we sample eval and a from the list of annotated attributes.
2. Fine-tumee tine-tunig dataser: We construct a vision-language fine-tuning dataset DF = {(4,, (A))))) with images I; and text T. Dataset Deral is sampled from the training sets of MNIST, Fashion- MNIST, or COCO such that the presence of image feature evaal is closely correlated with the presence of text attribute a eval as measured by Cramer's V [57].
5

---

Domain: Synthetic Data
Domain: Real-Worid Data
RaVi (Oursi
Spurious Aware
Domino
04
George
Distiling Failures
0.2
Random
00
Strength Learned Spurious Correlation (tea))
Strength of Learned Spurious Correlation (tead)
Figure 2: RAVL accurately identifies spurious correlations. Using our evaluation settings, we show that RAVL consistently outperforms prior methods in discovering learned spurious correlations between image features and textual attributes. Here, we provide Precision @ 10 metrics for a CLIP- RN50 model fine-tuned synthetic data (129 settings) and real-world data (171 settings).
Table 1: Mean Precision 10 metrics demon- strate the efficacy of RAVI. in discovering spu- rious correlations. On average across 654 evalu- ation settings, RAVL consistently outperforms baselines.
Table 2: Ablations show the utility of the clus- ter performance gap and influence metrics. We report Precision 10 metrics for a CLIP-RNSO model fine-tuned on real-world data (171 set- tings).
Method
Correlation Strength (Troat)
10
20
30
40
Ablation
Correlation Strength (Frcat)
Num. Eval Settings
654
369
234
168
10
20
30
40
Random
21.2
18.2
15.5
12.5
Distilling Failures
20.1
16.2
8.5
1.5
Unweighted G. Only
21.2
30.0
36.2
55.0
George
19.3
15.9
10.9
7.7
G. Only
40.9
51.7
63.8
66.7
Domino
17.1
15.0
11.7
9.0
G. & H. (RAVL)
46.0
54.8
72.4
83.3
Spurious-Awar Aware Detection
20.0
25.3
32.1
42.0
RAVL (Ours)
61.8
76.2
84.2
91.1
3. Fine-tuned VLM: A VLM M is fine-tuned on Derai
4. Evaluation dataset: Model . is evaluated using a zero-shot classification dataset Dynal {[T; Vi, R., == with images J,, class labels yi, region bounding boxes Ri, and region-level labels C. In particular, a eval must be included in the class label set, and eval must be annotated in the region-level label set. Since Deval is designed to reflect real-world setting, we assume that a correlation between eval and evall does not exist. Dataset Deval is constructed from the test sets of MNIST. FashionMNIST, or COCO.
Given the four components listed above, we classify an evaluation setting as valid if model M learned the intended spurious correlation. In order to measure this, we first identify images with label eval in Degal and compute the performance difference between images with feature eoal and images without feature g we designate this value as Then, for labels y j a aval we compute the maximum performance difference between images without feature eval and images with feature eeval, we designate this value 62. Large values of 61 and é2 suggest that model M has learned the desired spurious correlation between image feature eval and textual attribute a coal as defined in Section 2 We remove settings where Ej E2 are below some predefined performance threshold Teval- The performance threshold Teval serves as a quantitative indicator of learned correlation strength.
In total, we generate 620 fine-tuning datasets Degal (100 synthetic; 520 real-world). We then fine-tune model M on each dataset with three random seeds, resulting in 1860 candidate evaluation settings. Finally, we filter out settings where model M does not consistently learn the spurious correlation; to this end, we only retain settings where both é and e2 exceed Teral = 10 across all three random seeds. We repeat this procedure across various pretrained VLMs M, resulting in 654 valid experimental settings. Additional implementation details are provided in Appendix
Implementation Details:

---

3.3 Results: RaVL Effectively Discovers Spurious Correlations
Comparisons to Prior Approaches: Given an evaluation setting with a predefined spurious correla- tion (eeval, , aavad), ) a fine-tuned VLM M, and an evaluation dataset Dy our goal is determine the extent to which RAVL can discover the correlation between e and aeval
To this end, we use the labeled zero-shot classification dataset Di which includes ground-truth region bounding boxes and associated region labels. We provide the ground-truth bounding boxes as input to RAVL. which returns a single top-ranked cluster of regions likely to include spurious features. We rank regions within the cluster based similarity to the cluster medoid, and we utilize the provided region-level labels in Degal to evaluate the proportion of top- regions that contain the desired spurious feature e ex In line with prior work [3], we report performance with Precision @ K metrics. We note that given an identified spurious feature e e the correlated textual attribute a can be detected by identifying the class label in Depal where the absence of feature e leads to degraded performance.
There are few existing approaches for performing automated detection of fine ane-grained spurious features learned by VLMs. Here, we compare RAVL with four peeviuusyyddevelope methods: Distilling Failures [22]. George [43, Domino [3], and Spurious-Aware Detection [56]. Distilling Failures, George, and Domino are state-of-the-ar approaches for automatic identification of model failures resulting from spurious correlations; although these methods operate at the global image level and are designed for unimodal settings, we adapt these approaches for our setting by utilizing regions and zero-shot classification scores as input. Spurious-Awe Aware Detection operates at the fine-grained region level by computing class-based performance gaps resulting from the presence absence of particular features. To enable a fair comparison with RAVL, we provide the same set of regions and associated embeddings as input to all baselines. We also compare RAVL with random baseline, where the ranked list of regions is shuffled randomly.
Table D summarizes mean Precision @ 10 metrics across all 654 evaluation settings. Results demon- strate that RAVL consistently outperforms prior approaches in discovering spurious correlations between image features and textual attributes, contributing to 191 % improvement over the closest baseline. In Table we evaluate the effects of learned spurious correlation strength by varying the er- ror threshold Tewol from 10 40 and reporting performance for the subset of valid evaluation settings. Results show that RAVL is particularly effective when VLM A learns a strong spurious correlation; as learned correlation strength increases, performance of RAVL increases by 47% whereas most baselines degrade in performance. We also observe that Domino, George, and Distilling Failures often achieve performance near below the random baseline across evaluation settings; this suggests that methods designed for detecting errors resulting from spurious correlations at the global lmage-level cannot be easily adapted for fine-grained region-level discovery. Figure 2 demonstrates that our findings hold for both synthetic and real-world data.
Ablations: Our ablation study evaluates the role of the cluster influence score He and the cluster performance gap metric Ge (Section 3.I) in enabling accurate discovery of spurious correlations between image features and textual attributes. We compare the following three metrics for ranking clusters: (1) an unweighted cluster performance gap metric where Wy is set to 1, (2) the cluster performance gap with wy computed in Section 3.1] and (3) a combination of the cluster performance gap and cluster influence metric as used in RAVL. As shown in Table 2 the metrics utilized by RAVL consistently demonstrate the best performance across various learned correlation strengths (Tewat). Our results suggest the utility of both the performance gap metric and the influence score in identifying fine-grained spurious correlations.
Evaluations in the Wild: In addition to our controlled evaluations, we evaluate the ability of RAVL to surface spurious correlations learned by 12 thf-the-shelf VLMs 122 20); this presents a realistic and uncontrolled evaluation setting. We consider two zero-shot classification tasks Dv: (1) a 397- class scene classification task on SUN397 [52) and (2) binary classification of cardiomegaly in chest X-rays from ObjectCXR 23 We use the cluster performance gap metric Ge, introduced in Section 3.1 quantify the degree of the learned spurious correlation.
Our results demonstrate that all evaluated models, which span a range of architecture, training data, and parameter counts, show evidence of having learned spurious correlations; this is demonstrated by nonzero values of the cluster performance gap metric G.- On average across the evaluated models, the top-ranked spurious feature cluster discovered by RAVL on SUN397 achieves cluster performance
7

---

Model
Classification Task
Spurious Features Identified by RaVL.
Zero-Shot Accuracy
RuMi
wibhstRatt idlntifedd losures
Montifeed fostures
Classifiation (SUN397)
(VIT-R/16)
Class kubelt fatt food
Iraga with latts
Irage without Rankk.
PubMedCLIP (RcsNct-50)
Chest
100
110
Classification (Object CXR)
Class sidiscandooneplly
Figure 3: RAVL surfaces spurious correlations off- the-shelf VLMs. RAVL identifies a spurious correlation learned by CLIP ViT-B/16 between the presence of text-based retail signage and the class label fast food restaurant correlation learned by PubMedCLII ResNet-50 between metal clips (found in clothing) and the class label y (a heart condition) on a chest X-ray classification task.
scene classification task. RAVL also surfaces a spurious
Table 3: RAVL effectively mitigates spurious correlations. Here, we report mean Image Overall, Image Worst- Vorst-Group (Img- WG), Region Overall, and Region Worst-Group (Reg. WG) metrics across our real-world evaluation settings. Since performance of mitigation methods is dependent on the results of Stage we report metrics across settings where Stage Precision 10> 0.6 and Stage Precision @ 10>> 0.8.
Method
Stage I Discovery Precision@ 10>>
0.6
Stage
Discovery Precision 10 0.8
Img. Overall
Img. WG
Reg. Overall
Reg. WG
Img. Overall
Img. WG
Reg, Overall
Reg. WG
Standard FT
64.0
31.4
72.0
46.9
64.6
31.0
72.9
47.4
Upsampled FT
66.6
37.8
74.3
52.2
66.7
37.7
74.7
52.8
VL-ERM
68.8
32.2
75.6
50.3
68.7
30.9
75.9
50.6
VL-GDRO
69.1
33.7
75.6
50.4
68.8
31.1
76.0
51.0
Sparious-Aware
69.8
33.6
76.5
50.6
69.2
30.7
76.8
50.5
RAVL (Ours)
39.1
57.8
70.2
40.8
79.5
58.5
gaps (G.) of 9.943.2 (minimum 5.1, maximum = 14.0). On ObjectCXR, the mean value of Go is 0.08 (minimum 0.04, maximum 0.12) Our results support findings from previous work suggesting that all models may learn spurious correlations [30].
In Figure 3 provide qualitative examples of discovered spurious features for the CLIP ViT-B/16 model evaluated on SUN397 and the PubmedCL.I ResNer-50 model evaluated on ObjectCXR. For the CLIP ViT-B/16 model, RAVL surfaces feature cluster consisting of text-based retail signage. We observe significant performance gaps between images containing the LAVL.iddentified feature and images that do not contain the feature. For instance, note 48.2 point difference in zero-shot classification accuracy for the class label fast food restauran ant, suggesting that CLIP ViT-B/16 model can better classify scene of fast food restaurant when text-based retail sign present. For the PubmedCLIP ResNet-50 model, RAVL discovers that the presence of metal clips (found in the patient's clothing) spuriously correlated with cardiomegaly. We observe that the presence of clips improves zero-shot classification accuracy for the class label cardi crrdaly by 15.3 points.
Our evaluations show that RAVL surface fine mne-grained spurious correlations in realistic settings. Additional implementation details and qualitative examples are provided in Appendix D
4 Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models
In this section, we present the second stage of RAVL, which aims to mitigate learned spurious correlations in VLMs. Section 4.1 discuss our methodology for mitigating fine-grained spurious correlations with novel region-aware loss function. In Section 4.2 we use the evaluation framework previously introduced in Section 3.2 to demonstrate that RAVL substantially outperforms prior approaches in mitigating spurious correlations between image features and textual attributes.
WW note that since the formula for G. involves summation over class labels. raw values of G. for our 2-class chest X X-ray classification task lower than those for 397-clas scene classification task.

---

4.1 Our Approach: Mitigating Spurious Correlations
As described in Section 3 Stage of RAVL discovers image features that VLM M has learned to spuriously correlate with textual attributes. We next aim to mitigate the spurious correlation. Motivated by prior work on fine-gratie une-grained VLMs [88] [6], our key insight is that utilizing region-level information during VLM training can enable models to focus on relevant image-text relationships and ignore spurious correlations.
Since dataset Dr exclusively consists of images and text, ground-truth subgroup and class labels are not available. As a result, we first assign plausible (i) region-level subgroup labels and (ii) image-level class labels to the vision-language fine-tuning dataset Dp. To assign subgroup labels, we decompose each image I; in dataset DF into set of candidate regions R4. We then fit the trained K-Medoids clustering model from Section 3.1 on R; and identify all spurious regions associated with the top ranked cluster. We represent the identified spurious regions as Ri and remaining non-spurious regions as RJ such that R3 UR; = R2. In order to assign plausible class labels, we parse the paired text T; associated with each image to identify samples that reference the class labels included in the zero-shot classification label set )/: we refer to the assigned class label for image 7; as Úi-
We now introduce novel region-aware contrastive loss function for training VLM Mnew- For batch B, we define Ri as the set of all spurious regions in the batch: Rg = Ur, RT For image 1; E B, the first loss component I. encourages high embedding similarity between non-spurious regions Ri and assigned class label ÿ when compared to other class labels.
Om (R,,R,,00 Ui) Ln = - llog Eg, EB T. Cm, 1T) P(R
(3)
Here, for region embedding function f and text embedding function 9. om (A, b) alp(maxaeaa(f(aa g(b)) /r)) with temperature т. The term P(Rg) is a penalty that enforces embedding-level dissimilarity between spurious regions and correlated class labels.
The second loss component L'A encourages high embedding similarity between non-spurious re- gions RJ and assigned class label ýi when compared to other regions. We define a(a, b) exp({S(a), gbb)) /T) with temperature
mm (R, 0.)
L' = log
(4)
Om (R 9.) + Σ |8|
شز=ر mm (Rj,, Ri) + Er,ERS (ry,
The final loss is expressed as L ALC + +1 (1- (4' + L').. and Les, takes the form of the original loss function used for training M: in our experiments, Lez is the CLIP objective 33). Extended formulations of our loss function are provided in Appendix C
Here, À is hyperparameter
4.2 Results: RAVL Effectively Mitigates Spurious Correlations
Comparisons to Prior Approaches: We use the evaluation framework previously introduced in Section 3.2 to compare RAVL with prior approaches. There are few existing approaches for mitigating spurious correlations in the setting of fine-tuned VLMs. Here, we compare RAVL with standard VLM fine-tuning, upsampled VLM fine-tuning, ERM, GDRO [39], and Spurious- Aware Mitigation [56]. Since ERM and GDRO are traditionally used in unimodal classification settings, we adapt these approaches for our setting by adding a contrastive vision-languan objective and using zero-shot classification scores during fine-tuning; we refer to these approaches as VL-EEMM and VL-GDDR respectively.
Table 3 summarizes mean shot classification results across our real-world evaluation settings. Since performance of mitigation methods is dependent on the accuracy of the discovered spurious correlations in Stage 1, Table displays results for two evaluation categories: (i) the 192 settings where RAVL Stage Precision @ 10 is greater than 0.6, and (ii) the 106 settings where RAVL Stage I Precision @ 10 is greater than 0.8. In line with prior works on robustness [39 [56), we report image overall performance and image worst-group performance. Additionally, in order to evaluate the extent to which the VLM understands fine-grained features, we introduce two new metrics: region overall performance and region worst-group performance. Region-level accuracies are computed by performing zero-shot classification with region embeddings and comparing predicted labels to the ground-truth region-level labels provided in the zero-shot classification dataset.

---

Results show that RAVL consistently outperforms prior approaches in mitigating spurious correlations. Across the two evaluation categories in T RAVL contributes an improvement of up to 8.2% on image worst-group performance and 10.8% on region worst-group performance over the nearest baseline. Improvements in region worst-group performance are partieularly notable, suggesting that RAVL can better interpret fine-grained features when compared to prior approaches. Additionally, as the accuracy of the discovered spurious correlations in Stage I increases, the performance of the RAVL mitigation approach increases proportionally. Our results demonstrate the efficacy of our fine-tuning procedure in mitigating spurious correlations when compared to prior approaches.
Conclusion
In this work, we introduced RAVL, fine- tineggrained region-aware approach for addressing spurious correlations in VLMs. We demonstrate through large-scale, controlled experiments as well as in-the- wild evaluations that RAVL can discover (191% improvement in identified correlations) and mitigate (8.2% improvement on worst-group performance) spurious correlations in VLMs. We hope that our work can help (i) diagnose and correct critical failure modes in VLMs prior to deployment and (ii) drive progress towards the development of fine- ine-grained approaches for model robustness.
Acknowledgments and Disclosure of Funding
We are thankful to Sophie Ostmeier, Eduardo Reis, and Ashwin Kumar for helpful discussions and feedback. MV is supported by graduate fellowship awards from the Department of Defense (NDSEG), the mnight-Hennesse Scholars program at Stanford University, and the Quad program. AC is supported by NIH grants RO1 HL167974, ROIHL RO1 AR077604, RO1 EB002524, ROI AR079431, P41 EB027060, AY2AX000045, and AXYSAX000024-001 and NIH contracts 75N92020C00008 and 75N92020C00021, CL is supported by NIH grants RO1 HL155410, RO1 HL157235, by AHRQ grant R R18HS026886, and by the Gordon and Betty Moore Foundation. JBD and CL are supported by the Medical Imaging and Data Resource Center (MIDRC), which is funded by the National Institute of Biomedical Imaging and Bioengineering (NIBIB) under contract 75N92020C00021 and through The Advanced Research Projects Agency for Health (ARPA-H).
10

---

References
[1] Dyah Adila, Changho Shin, Linrong Cai, and Frederic Sala. Zero-shot robustification of zero-shot models with foundation models, 2023.
[2] Amit Alfassy, Assaf Arbellc, Oshri Halimi, Sivan Harary, Roci Herzig, Eli Schwartz, Rameswar Panda, Michele Dolfi, Christoph Auer, Kate Sacnko, PeterW. J. Staar, Rogerio Feris, and Leonid Karlinsky, Feta: Towards specializing foundation models for expert task applications, 2022.
[3] Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[4] Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Mag- dalini Paschali, Zhihong Chen, Benoit Delbrouck, Eduardo Reis, Cesar Truyts, Christian Bluethgen, Malte Engmann Kjeldskov Jensen, Sophie Ostmeier, Maya Varma, Jeya Maria Jose Valanarasu, Zhongnan Fang, Zepeng Huo, Zaid Nabulsi, Diego Ardila, Wci-Hung Weng, Edson Amaro Junior, Neera Ahuja, Jason Fries, Nigam H. Shah, Andrew Johnston, Robert D. Boutin, Andrew Wentland, Curtis P. Langlotz, Jason Hom, Sergios Gatidis, and Akshay S. Chaudhari. Merlin: A vision language foundation model for 3d computed tomography, 2024.
[5] Zhihong Chen, Maya Varma, Jean-Benoit Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, Emily B. Tsai, Andrew Johnston, Cameron Olsen, Tanishq Mathew Abraham, Sergios Gatidis, Akshay S. Chaudhari, and Curtis Langlotz. Chexagent: Towards a foundation model for chest x-ray interpretation. 2024.
[6] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning, 2022.
[7] Patrick John Chia, Giuseppe Attanasio, Federico Bianchi, Silvia Terragni, Ana Rita Magalhaes, Diogo Goncalves, Ciro Greco, and Jacopo Tagliabue. Contrastive language and vision learning of general fashion concepts. Scientific Reports, 12(1), November 2022.
[8] Joseph Paul Cohen, Joseph D Viviano, Paul Bertin, Paul Morrison, Parsa Torabian, Matteo Guarrera, Matthew P Lungren, Akshay Chaudhari. Rupert Brooks, Mohammad Hashir. al. Torchxrayvision: library of chest x-ray datasets and models. In International Conference on Medical Imaging with Deep Learning, pages 23 2-9.. PMLR, 2022.
[9] Elliot Creager, Jöm-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In International Conference on Machine Learning, pages 1209. PMLR, 2021.
[10] Alex J DeGrave, Joseph D Janizek, and Su-In Lee. AI for radiographic COVID-19 detection selects shortcuts over signal, Nat. Mach. Intell., 3(7):610 619, May 2021.
[11] Li Deng. The mnist datahase of handwritten digit images for machine learning research. IEEF Signal Processing Magazine, 29(6):141-1422 2012.
[12] Sedighch Eslami, Christoph Meinel, and Gerard De Melo. Pubmedelip: How much docs clip benefit visual question answering in the medical domain? In Findings of the Association for Computational Linguistics: EACL 2023, pages 1151-1163, 2023.
[13] Sabri Eyuboglu, Maya Varma, Khaled Kamal Saab, Jean-Benoit Delbrouck, Christopher Lee-Messer, Jared Dunnmon, James Zou, and Christopher Re. Domino: Discovering systematic errors with cross-modal embeddings. In International Conference on Learning Representations, 2022.
[14] Alex Fang. Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and I. Ludwig Schmidt. Data determines distrihutional robustness in contrastive language image pre-training (clip), 2022.
[15] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference Learning Representations, 2021.
[16] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn, 2017.
[17] Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas Montine, and James Zou, A visual-language foundation model for pathology image analysis using medical twitter. Nat. Med., 29(9):2307-23) September 2023.

---

[18] Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing In Bernhard Schölkopf, Caroline Uhler, and Kun Zhang. edi- tors, Proceedings of the First Conference on Causal Learning and Reasoning, volume 177 of Proceedings of Machine Learning Research, pages 336-351, PMLR, 11-13 Apr 2022.
achieves competitive worst-group-aceurne
[19] Wisdom Oluchi Ikezogwo, Mehmet Saygin Seyfioglu. Fatemeh Ghezloo. Dylan Stefan Chan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt-Im: One million image-text pairs for histopathology. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[20] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongscok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. you use this software, please cite it below.
[21] Pavel Izmailov. Polina Kirichenko. Nate Gruver, and Andrew G Wilson. On feature learning in the presence of spurious correlations. Advances in Neural Information Processing Systems, 35:38516 15-38516--3532, 2022.
[22] Saachi Jain, Hannah Lawrence, Ankur Moitra, and Aleksander Madry. Distilling model failures as directions in latent space. The Eleventh International Conference on Learning Reprexentations, 2023.
[23] JFHealtheare. Object-cxr- - automatic detection of foreign objects on chest x-rays.
[24] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision- Asiounlanguuae representation learning with noisy text supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 4904-4916. PMLR, 18-24 Jul 2021.
[25] Nari Johnson. Ángel Alexander Cabrera. Gregory Plumb, and Ameet Talwalkar. Where does my model underperform? a human evaluation slice discovery algorithms, 2023.
[26] Weixin Liang and James Zou. Metashift: A dataset of datasets for evaluating contextual distribution shifts and training conflicts. In International Conference on Learning Representations, 2021.
[27] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects context. David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision - ECCV 2014, pages 740-755, Cham, 2014. Springer International Publishing.
[28] Evan Z Liu, Behzad Haghgoo. Annie S Chen. Aditi Raghunathan. Pang Wei Koh, Shiori Sagawa, Percy Liang. and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning. pages 6781-6792 PMLR, 2021.
[29] Mazda Moayeri, Phillip Pope. Yogesh Balaji, and Soheil Feizi. A comprehensive study of image classifica- tion model sensitivity foregrounds, backgrounds, and visual attributes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 19087-19097. 2022.
[30] Mazda Moayeri, Wenxiao Wang, Sahil Singla, and Soheil Feizi. Spuriosity rankings: Sorting data to measure and mitigate biases, 2023.
[31] Nicolas M. Müller, Simon Roschmann, Shahbaz Khan, Philip Sperl, and Konstantin Böttinger. Shortcut detection with variational autoencoders, 2023.
[32] Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. Advances in Neural Information Processing Systems, 33:20673-20684, 2020.
[33] Junhyun Nam, Jachyung Kim, Jaeho Lee, and Jinwoo Shin. Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation. In International Conference on Learning Repre- sentations, 2021.
[34] Luke Dakden-Rayner, Jared Dunmmon, Gustavo Carneiro, and Christopher Ré. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. September 2019.
[35] Obioma Pelka, Sven Koitka, Johannes Rückert, Felix Nensa, and Christoph M. Friedrich. Radiology objects in context (roco): A multimodal image dataset. In Danail Stoyanov, Zeike Taylor, Simone Balocco, Raphael Sznitman, Anne Martel, Lena Maier-Hein, Lue Duong, Guillaume Zahnd, Stefanie Demirci, Shadi Albarqouni, Su-Lin Lee, Stefano Moriconi, Veronika Cheplygina, Diana Mateus, Emanuele Trucco, Eric Granger, and Pierre Jannin, editors, Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, pages 180-189. Cham, 2018 Springer International Publishing.
12

---

[36] Alec Radford, Jong Wook Kim. Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors. Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748 8763. PMLR, 18 24 Jul 2021.
[37] Pranav Rajpurkar, Jeremy Irvin, Aart Bagul, Daisy Ding, Tony Duan, Hershel Mehta, Brandon Yang. Kaylie Zhu, Dillon Laird, Robyn I.. Ball, Curtis Langlotz, Katie Shpanskaya, Matthew P. Lungren, and Andrew Y. Ng. Mura: Large datasct for abnormality detection in musculoskeletal radiographs, 2018.
[38] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett editors, Advances in Neural Information Processing Systems, volume 28, pages 91-99. Curran Associates, Inc., 2015.
[39] Shiori Sagawa. Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Reprexentations, 2020.
[40] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kun- durthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b; An open large-scale dataset for training next generation image-text models, 2022.
[41] Sahil Singla and Soheil Feizi. Salient imagenet: How to discover spurious features in deep learning? In International Conference on Learning Representations, 2022.
[42] Sahil Singla, Besmira Nushi, Shital Shah, Eee Kamar, and Eric Horvitz. Understanding failures of deep networks via robust feature extraction. IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2021. Computer Vision Foundation / IEEE, 2021.
[43] Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse-grained classification problems. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 99339-19352. Curran Associates, Inc., 2020.
[44] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: the new data in multimedia research. Communications of the ACM, 59(2):64-73, January 2016.
[45] Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P Langlotz, Andrew Y Ng, and Pranav Rajpurkar. Expert-level detection of pathologies from unannotated chest x-ray images via supe-vupe learning, Nat. Biomed. Eng., 6(12):1399 1406, December 2022.
[46] Maya Varma. grained vision-language representation learning from real-world data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Delbrouck, Sarah Hooper, Akshay Chaudhari, and Curtis Langlotz. Villa: Fine-
[47] Maya Varma, Mandy Lu, Rachel Gardner, Jared Dunnmon, Nishith Khandwala, Pranav Rajpurkar, Jin Long, Christopher Beaulicu, Katie Shpanskaya, Li Fei-Fei, Matthew P. Lungren, and Bhavik N. Patel. Automated abnormality detection in lower extremity radiographs using deep learning. Nature Machine Intelligence, (112):578-583, December 2019.
[48] Julia K Winkler, Christine Fink, Ferdinand Toberer, Alexander Enk, Teresa Deinlein, Rainer Hofmann- Wellenhof, Lue Thomas, Aimilios Lallas, Andreas Blum, Wilhelm Stolz, et al. Association between surgical skin markings in dermoscopic images and diagnostic performance of a deep learning convolutional neural network for melanoma recognition. JAMA dermatology, 11350)11113-11::: 2019.
[49] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Korblithh, Rehecca Roelofs, Raphael Gontijo-Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong. and Ludwig Schmidt. Robust fine-tuning zero-shot models. arXiv preprint arXiv:2109.01900 01903, 2021. https://arxiv. org/ abs/2109. 01900
[50] Shirley Wu, Mert Yuksekgonul, Linjun Zhang, and James Zou. Discover and cure: Concept-awarc mitigation of spurious correlation. In ICML, 2023.
[51] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
13

---

[52] Jianxiong Xiao, James Hays, Krista A. Ehinger. Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to zoo. 2010 IEEE Computer Society Conference Computer Vision and Puttern Recognition, pages 3485-3492, 2010.
[53] Kai Yuanqing Xiao, Logan Engstrom, Andrew llyas, and Aleksander Madry. Noise signal: The role of image backgrounds in object recognition. International Conference on Learning Representations, 2020.
[54] Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, Yanbo Xu, Wei, Wenhui Wang, Shuming Ma. Furu Wei, Jianwei Yang. Chunyuan Li, Jianfeng Gao, Jaylen Rosemon, Tucker Bower, Soohee Lee, Roshanthi Weerasinghe, Bill Wright, Ari Robiesek, Brian Piening. Carlo Bifulco, Sheng Wang, and Hoifung Poon. A whole-slide foundation model for digital pathology from real-world data, Nature, 630(8015):811-188 June 2024.
[55] Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjic Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 6502-6509, 2020.
[56] Yu Yang, Besmira Nushi, Hamid Palangi, and Baharan Mirzasoleiman. Mitigating spurious correlations in multi-modal models during fine-tuning. In International Conference on Machine Learning, 2023.
[57] Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang. James Zou, Chelsea Finn. Improving out-of-distribution robustness selective augmentation. Proceeding of the Thirty-ninth International Conference on Machine Learning, 2022.
[58] Lewei Yao, Runhui Huang. Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training, 2022.
[59] Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? In International Conference on Learning Representations, 2020.
[60] Michael Zhang and Christopher Re, Contrastive adapters for foundation model group robustness. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[61] Michael Zhang, Nimit S. Sohoni, Hongyang R. Zhang, Chelsca Finn, and Christopher Ré. Correct-n- contrast: A contrastive approach for improving robustness to spurious correlations, 2022.
[62] Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Timn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Andrea Tupini, Yu Wang, Matt Mazzola, Swadheen Shukla, Lars Liden, Jianfeng Gao, Matthew P. Lungren, Tristan Naumann, Sheng Wang, and Hoifung Poon. Biomedclip: multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. 2024.
[63] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16793- 16803,66,,,.1111 June 2022.
14

---

Appendix
Contents
Related Work
15
B Extended Details on Evaluation Settings
16
Extended Details on RAVL Mitigation
17
D Extended Evaluations
18
D.I Extended Results for RAVL Stage (Discovery)
18
D.1.1 Extended Comparisons to Prior Approaches
18
D.1.2 Additional details for in-the-wild evaluations
18
D.2 Extended Results for RAVL Stage (Mitigation)
D.3 Computational Complexity Analysis
22
Extended Discussion
23
A
Related Work
Machine learning models often learn spurious correlations (also known shortcuts) between image features and class labels. For instance, models have been shown to rely on the presence chest tubes rather than disease features when identifying collapsed lungs in chest X-rays (34); surgical skin markings when detecting melanoma from skin lesions [48]; and environmental features when performing object recognition tasks (3). Models that learn spurious correlations will generalize poorly to real-world settings. Our work builds on several recent research directions for (i) discovering and (ii) mitigating spurious correlations.
Discovering Spurious Correlations. In the unimodal setting, prior works have developed automated methods for identifying systematic errors resulting from learned spurious correlations in vision models. Using labeled validation set, these approaches utilize clustering algorithms [3 43 or
22 31 to identify subgroups of images with high error rates; for instance, in the example in Figure T images containing butterflies without flowers may identified as one such subgroup. Given a of images in the identified subgroups,
lightweight models 42
then identify the common features and rectify the data or model. However, recent work has suggested that it is often challenging for humans to interpret identified subgroups and accurately determine the shared features resulting in model failure [25). Additionally, such methods often focus solely on identifying images with high error rates (e.g. butterflies without flowers) rather than identifying the specific class of features contributing to the error (e.g. flowers). related line of work has aimed to identify spurious features using human supervision [1 or external concept banks 500]
In the vision-language setting, Yang et al. use an external off-the-shelf object detector to annotate features [56]. Then, for each feature, the difference in zero-shot classification accuracy between images containing the feature and those without the fcature is measured; high performance gaps are used to signal spurious features. However, the efficacy of this approach is reliant the quality of the object detector and numan-in-the-looe is used to verify results; also, as we show in this work, performance gaps alone are not always sufficient for discovering spurious features.
Mitigating Spurious Correlations. There is line of work aiming to mitigate spurious correlations 9 28 33, 21]. These works explore strategies like data
in the context of deep learning [6] 32
augmentation [55 59 22
50) and instance upsampling R9 43 While these approaches have
been explored widely in unimodal tasks 29 53 mitigating spurious correlations in langonee
settings has not been extensively studied. Some previous works have studied this problem within the
context of pretrained VLMs [60] 49 I: however, their setting differs markedly from the fine-tuned
15

---

VLM setting, where datasets are composed of image-text pairs with no class or subgroup labels. In the fine-tuned VLM setting, existing works predominantly operate at the global image-level [56]]
which is unlikely to be sufficient for mitigating fine-grained spurious correlations.
B Extended Details on Evaluation Settings
We create 654 evaluation settings using data from two domains: (1) synthetic data (MNIST [ and PNSSionMNIST (51)) and (2) real-world data (COCO (27). Below, we provide implementation details for the four components included each evaluation setting:
Predefined spurious correlation: We define a spurious image feature and textual attribute pair (e " For MNIST and FashionMNIST, e represents red rectangle; a is generated from the set {zero, one, two, three, four five, six, seven, eight, nine} for MNIST and (t-shirt, trouser, pullover, dress, coat, sandal, shirt, sneaker, bag, ankle boot) for FNShionMNIST. For COCO, we sample e and a eval from the list of annotated attributes.
2. Fine-tuming dataser: Vision-language fine-tuning datasets De are sampled from the training sets of MNIST, FashionMNIST, and COCO such that the presence of feature e is correlated with the presence of text attribute avaa as measured by Cramer's V. For MNIST and FaShionMNIST, we synthetically generate text captions by randomly sampling from the following pre-defined prompt templates: THE IMAGE SHOWS A [CLASS LABEL], THE DIGIT APPEARS TO BE [CLASS LABEL], THERE IS AN IMAGE SHOWING A [CLASS LABEL], and THE NUMBER IS [CLASS LABEL]. In order to reflect real-world settings where spurious features (e.g. skin markings in dermoscopic images (8)) may not be annotated in text, text captions our synthetic settings solely refer to class labels and not describe the spurious feature. For COCO, we use the provided text captions.
3. Fine-tuned VLM: We fine-tune each model M on dataset Deral using a single NVIDIA A100 GPU with an initial learning rate of 5e-5, We use a batch size of 128 and train for 100 epochs with early stopping. We set the loss temperature as = 0.07. In line with
prior works that explore the benefits of locked image-text training 2 46 encoder and only learn weights for the image encoder.
we freeze the text
4. Evaluation dataset: We construct zero-shot classification datasets Dival from the test sets of MNIST, FaShionMNIST, and COCO. For MNIST and FashionMNIST, we generate region bounding boxes using equally-sized quadrants. For COCO, we use the ground-truth bounding boxes and associated labels. Evaluation datasets are sampled to ensure that a correlation between a avol and enaa does not exist. For MNIST, we perform prompt ensembling for zero-shot classification using the following prompts: PHOTO OF THE NUMBER [CLASS LABEL]; THE DIGIT [CLASS LABEL]; AN IMAGE OF A [CLASS LABEL]; [CLASS LABEL... For MNSHI we use the following prompts: A PHOTO OF A [CLASS LABEL]; THE [CLASS L.ABEI];; AN IMAGE OF A [CLASS LABEL.]; [CLASS LABEL]. For COCO, we use the following prompts: THERE IS A [CLASS LABEL];; A PHOTO OF THE [CLASS LABEL]]; A PHOTO OF A [CLASS LABEL]; [CLASS LABEL.]
Our 654 evaluation settings are summarized in Table Datasets are licensed under CC BY, CC BY-SA, CC BY-NC, or MIT licenses. In Figure we provide examples of both synthetic and real-world evaluation settings.
Table 4: Evaluation settings. We evaluate our approach on 654 settings, divided across 2 data domains and 2 model initializations.
Domain
Model M Initialization
CLIP-RN50
CLIP-RNIOI
Synthetic Data
129 171
162
Real-World Data
192
16

---

Langunge)
Image
Image
Image
Image
Image
Image
Image
Image
Samples
Samples with tevtual attriibute
Dataset: MNIST (Syttcciic)
amm
Text Theelian inggeshowng ng
Text the Rege
Text meiinage stowwa
Text there Image showding
Ciaes
Claen
Class Label
Cisen
appsan
restanglo
arliee
Textual Attribute (am-d)) nine
Samples without textuaal attribute 2010
Samples without tentbval attniibtte anma
Class Label
Class cisst
Class Label sovan
Class Label Iear
thilmaage
opprarsin
stowwa
Inagg doswing
Iwa
Image
Image
Image
Imsae
Image
Samples with textual attribute geda
Samples with textual attribute geal
Datasel: COCO (Real
Thot Amanty mg catonitopohi in
Text Amerotting coudh
Text Artimegoo oe cechccuu
Class Label
Class Label
Class Label
coudh
Spurious Feature (enve]
Image
Image
Image
Image
Image
Textual Attribute far-9 couch
Samples textual attribute
tevtual attr butte
grmel
aroul
Text Black Jumstpueuttt
Text Avsoode kithenn baachon
Banara Kewraskpsand paarud abiuephtte
Class Label
Class Label
Class Label
Figure 4: Example evaluation settings, Here, we provide examples of predefined spurious correlations, fine-tuning datasets, and evaluation datasets associated with a synthetic evaluation setting (top row) and real-world evaluation setting (bottom row). The example synthetic evaluation setting consists of predefined spurious correlation between red rectangle (spurious image feature evval) and nine (textual attribute zv This spurious correlation is visible in the vision-language fine-tuning dataset, where the presence of red rectangles and nines are strongly correlated, but in the evaluation dataset. Similarly, the example real-world evaluation setting consists of predefined spurious correlation between a person (spurious image feature ecaa) and couch (textual attribute =vai), Again, this spurious correlation is visible in the vision-language fine-tuning dataset, where the presence of people and couches are strongly correlated, but not in the evaluation dataset.
C Extended Details on RAVL Mitigation
In this section, we extend Section 4.1 by providing additional descriptions of our region- aware loss function.
For batch B, we define R2 as the set of spurious regions the batch: R2 UneB R; For image I; in batch B. the first component of our region-aware loss function Li is designed to maximize embedding similarity between non-spurious regions R. and assigned class label fi: simultaneously, Ln will minimize embedding similarity between non-spurious regions RV and other class labels in the batch. We formulate L as follows:
m (R 91
T log E d m (R; ú5) P(R
where P(R is penalty term that encourages dissimilarity between spurious features and correlated class labels as expressed below. Including this term in the denominator of L is meant to pull embeddings of spurious regions away from correlated class labels.
P(RE)
Σ max (ry, 266)
(6)
TER
The formula for T5 includes two similarity functions: and om. define a and o. as follows. Let represent region embedding function (associated with the image encoder of VLM M) and
17

---

let represent text embedding function (associated with the text encoder of VLM M). Then, for an arbitrary region a, the function f(a) will generate region embedding f(a) E Ee with embedding dimension d. For an arbitrary class label b, g(b) will generate text embedding g(b) R. Given this notation, we establish the following definitions for a and om:
aa,, b) = exp((f(a), g(b)) /T) /r))
(7)
Fm (A, cxp(max((f(()))))) aEA
In the loss term f. the function om (R,, (i)) will compute the maximum similarity between regions in R2 and class label V.. We specifically use the maximum operation in this computation since there are likely to be regions included in R2 that do not reflect the class label; for instance, in the example provided Figure L there may be non-spurious regions such as trees or leaves included in Ri which do not align with the animal class labels. The maximum operation ensures that the similarity between at least one region in R3 and the class label should be high.
The second component of our region- aware loss function L' is designed to maximize embedding similarity between non-spurious regions RT and assigned class label yı: simultancously, L'A will minimize embedding similarity between other regions in the batch and class label ft. We formulate L' as follows:
T (R,,, Uz) Σ |8| =v 0m (RJ, (.) Er,ERg d(j.
L'A log
(9)
(R,, (.)
D Extended Evaluations
D.1 Extended Results for RAVL Stage 1 (Discovery)
In this section, we extend the results provided in Section 3.3 with additional evaluations Stage I of RAVL. Our goal is to evaluate the ability of RAVL to discover fine-grained spurious correlations between image features and textual attributes.
D.1.1 Extended Comparisons to Prior Approaches
We implement RAVL according to the details provided in Section 3.1 RAVL includes clustering step that identifies groups of visually-similar regions. For all evaluation settings, we identify the optimal number of clusters by sweeping all cluster sizes ranging between | * 2 and [}'] select the optimal number of clusters using Silhouette scores. We select these bounds to be larger than the class label set size by several multiples in order to ensure that clusters adequately separate distinct features. Prior works [3] 43 have also utilized overclustering approaches for this objective. We note that users can adjust the bounds based on the composition of their dataset; for instance, complex datasets with diverse features may require a larger range, For MNIST and FashionMNIST. the size of the label set |/] is 10; For COCO, the size of the label set ranges between 2 and 5.
then
For all baselines, we utilize the official implementations provided by the authors. We adapt Domino, George, and Distilling Failures for our setting by providing region embeddings input rather than image embeddings.
In Figure 5] we provided an extended version of Figure 2 outperforms baselines across two domains (synthetic images and real images), two model initializa- tions (CLIP-RNSH) and CLIP-RNI0I), and four learned correlation strengths (measured by varying Teval E {10, 20, 30, 40)).
We demonstrate that RAVL consistently
D.1.2 Additional details for in-the-wild evaluations
Evaluations scene classification: Here, we provided extended details on in-the-wild evalua- tions performed on scene images (Section 3.3
We leverage ten off-the-shelf VLMs as our model M: CLIP-RN50, OpenCLIP-RN50, CLIP-RNI01, OpenCLLP-RNI01,, CI.PPVVTT/32, OpenCL.P-ViTTB/32 CLIP-VITB/16, OpenCLIP- CLIP-
ViTL/14, and OpenCLIP- O [36] 20). The four RN models utilize ResNet vision encoders
18

---

Spurtous-hware
Fisttling
Random
Synthetic
Domain: Real-Worid
CLIP-RN50
Predioon810
Predetond
Strength Correlation [trel
Strength of Spurious Correation [tow]
1.
CLIP-RN101
PredSaree0
Prediden
Strength of Learned Spurious Correlation fared
Strength of Lear ned Spurious Correlation [tow]
Average
Predsione00
0.6
Proditone
30
10
Strength of Leamed Spurtous Correlation (read
Strength Leaarhh Spurious Correlation [tow]
Figure 5: RAVL accurately identifies spurious correlations. Here, we provide an extended version of Figure? 2 which demonstrates that RAVL consistently outperforms prior methods in discovering learned spurious correlations between image features and textual attributes. Here, we provide Precision @ 10 metrics for CLIP-RN50 model fine-tuned on synthetic data (129 settings) and real- world data (171 settings); CLIP-RNIO) model fine-tuned on synthetic data (162 settings) and real-world data (192 settings); and an average across both model architectures.
and six ViT models utilize Vision Transformer backbones. The CLIP models were trained on proprietary dataset with 400M image-text pairs. OpenCLIP ResNet models were trained on YFCC15M 44 and OpenCLIP VIT models were trained on LAION2B 40
We select SUN397 as our zero-shot classification dataset Dy [52]. SUN397 consists of scene images from 397 classes. We use the test data from official partition number which consists 19,850 images. We then use an off-the-shelf region proposal network 63 to identify candidate regions.
For each VLM M, perform 397-class zero-shot scene classification SUN397. We prompt ensemble consisting of two prompt templates as provided by CLIP [36]. Due to the large size of the zero-shot classification dataset Dv, we perform clustering using the CLARA (Clustering for Large Applications) algorithm, which is an efficient implementation of K-Medoids, and fix the number of clusters | 2, which is 794 in this case.
Evaluations on chest X-ray classification: Here, we provided extended details on our in-the-wild
evaluations performed on medical images (Section 3.3). In recent years, range of vision 47
54 37
and vision-language 45 5 62] models have been proposed for learning diagnostic patterns in medical images, and there is critical need for methods capable of identifying spurious correlations in this domain. Our goal is to determine RAVL can effectively surface spurious correlations learned by real-world fine-tuned VLMs developed for medical image interpretation.
We leverage two off off-the-shelf variants of the PubMedCLIP model 12 as our VLM M: PubMedCLIP. model utilizes ResNet-50 vision
RN50 and PubMedCLIPP V The PubMedCLIP-RNG encoder and was fine-tuned from the CLIP-RN50 model. The PubMedCL.P-ViII Vision Transformer backbone for the vision encoder and was fine-tuned from the CLIP-ViTBAA
model utilizes
19

---

model. Both variants of PubMedCLIP are fine-tuned using ROCO, a large radiology dataset consisting of images and captions collected from PubMed [35]
We select Object-CXR as our zero-shot classification dataset Dy. Object-CXR a dataset of 10,000 frontal chest X-rays compiled from around 300 township hospitals in China 23 Twelve radiologists with 1-3 years of experience annotated the images, identifying foreign objects within the lung field using bounding boxes, ellipses, or masks, excluding support devices. We retain only bounding boxes and exclude chest X-rays without annotations, resulting in 8.726 object annotations across 4.372 images. For our evaluations, we use the Object-CXR dev split, which includes 974 object annotations across 489 images. We assign image-level labels to the dataset using torchxrayvision [8) a library that includes a variety of pretrained chest X-ray models. Specifically, we use the XRV. WINSENET12I-DENSEHE
1 pretrained model to produce multi-class labels for a variety of diseases, including Enlarged Cardiomediastinum, Cardiomegaly, Lung Opacity, Lung Lesion, Edema, Consolidation, Pneumonia, Atelectasis, Pneumothorax, Pleural Effusion, and Fracture. disease is identified as present if it meets confidence threshold of 0.60.
For each PubMedCLIP VLM M. we perform binary zero-shot classification of cardiomegaly in Object-CXR. Cardiomegaly medical condition characterized by the presence of an enlarged heart. After performing a manual search over the text prompt space, we identify CARDIOMEGALY and NORMAL as the prompts that lead the highest zero-shot classification accuracy for both model variants. The PubMedCLIP-RAN RN50 model achieves an overall zero-shot classification accuracy of 74.2, with an accuracy of 14.0 on the group of images with cardiomegaly and an accuracy of 91.9 on the group of images without cardiomegaly. The PubMedCLIPP VPVVBBB/32 achieves an overall zero-shot classification accuracy of 39.4, with an accuracy of 80.4 on the group of images with cardiomegaly and an accuracy of 28.0 on the group of images without cardiomegaly. Interestingly, and the RubMedCLIP-ViTB..
given the selected prompts, we note that the PubMedCLIP-RNY models exhibit inverse trends, with PubMedCLIP-RN500 achieving higher performance on the class of images without cardiomegaly and MEMMedCLIP-ViTB//3 achieving higher performance on the class of images with cardiomegaly.
Given VLM M and zero-shot classification dataset Dypal we apply RAVL in order surface learned spurious correlations. Similar to our controlled evaluations on synthetic datasets, we perform K- E-Medodds clustering with the number of clusters ranging from 20 to 50. The optimal number of clusters is selected using Silhouette distance; we use 24 clusters for PubMedCLIPP-NN5 and 20 clusters for PubMedCLIP-
The final cluster performance gap metric G. associated with the top-ranked spurious feature cluster is 0.041 and 0.119 for the PubMedCLIP-RND5 and PubMedCLIP-ViTBBA. models respectively.
Model
Classificttion Task
Spurious Features Identified by RaVL
Zero Shot Accuracy
woop Evii-
Irogee
OpenCLIP (VIT-L/14)
Scene Classifiation (SUN397)
Isbet oundeer chidion toop
identined featuuees
icentihed Inatures
CLIP (VIT-B/32)
Scene Classification (SUN397)
$36
39-3
lubelapub lindorr)
Insecwwtt with Ruvil slo-tifoal loater ov
Irageo Iragcwwishutteee RuV.
OpenCLIP
Scene Classification (SUN397)
300
Hicetika loaturto
100
Net-N111101)
Class lobel restaarantpati
Figure 6: RAVL surfaces spurious correlations in off-the-shelf VLMs. Here, extend Figure with additional examples of spurious correlations discovered by RAVL in off-the-shelff VLMs.
Extended Results: In Figure 6 we extend the qualitative results provided in Figure B with additional examples of spurious correlations surfaced by RAVL in off-the-shelf VLMs. We make the following observations:
For the OpenCLIP ViT-L/14 model, RaVL. surfaces a feature cluster consisting of green plants and fences. We observe performance gap of 18.3 points between images with class
20

---

label outdoor chicken coop that contain the RaVL-identified feature and those that do not contain the feature. This suggests that the OpenCLIP ViT-L/14 model can better classify an outdoor chicken coop scene when green plants and fences are present.
For the CLIP ViT-B/32 model, RaVL surfaces feature cluster consisting of people. We observe performance gap of 24.3 points between images with class label pub (indoor) that contain the RaVL--identifed feature and those that do not contain the feature. This suggests that the CLIP ViT-B/32 model can better classify pub (indoor) scene when people are present.
For the OpenCLIP ResNet-101 model, RaVL surfaces a feature cluster consisting of chairs. We observe a performance gap of 23.3 points between images with class label restaurant patio that contain the Ra V-iieeenti feature and those that do not contain the feature. This suggests that the OpenCLIP ResNet-101 model can better classify restaurant patio scenes when chairs are present.
D.2 Extended Results for RAVL Stage (Mitigation)
We train model MMeww using single NVIDIA A100 GPU with an initial learning rate of 5e-5. We use batch size of 128 and train for 100 epochs with early stopping. We set the loss temperature as 0.07 and use À 0.8 in loss function L. In line with prior works that utilize locked image-text fine-tuning [2 [6). we freeze the text encoder and solely learn weights for the image encoder. We generate candidate regions for the fine-tuning dataset Deyal using a region proposal network with identical settings to prior work [63].
Below, we provide additional implementation details for the five mitigation baselines we explore in this study. Since there are a limited number of existing approaches designed for mitigating spurious correlations in fine-tuned VLMs, we adapt several existing methods for our setting in order to train model Mnew>
Standard VLM Fine-Tuning: We perform standard VLM fine-tuning with the original loss function L.c. used train model M. In our experiments, Lcc. the CLIP objective (36
Upsampled VL.M Vine-Tuning: We perform VLM fine-tuning with the original loss function L.c used to train model M. In our experiments, Lec. is the CLIP objective [36]. We utilize weighted sampler to upsample minority groups during training; class and subgroup labels are derived from Stage I of RAVL as detailed in Section 4.]
VL-ERM: Since empirical risk minimization (ERM) is traditionally used in unimodal clas- sification settings, we adapt ERM for multimodal setting by incorporating extra contrastive vision-language objective function; this loss function is intended to ensure that VLM Mnew learns image-text relationships during training. Specifically, the final loss function during training is LEREM = XLCL + (1 L) Here, LeL takes the form of the original loss function used to train model M; in experiments Le is the CLIP objective [36]. We set À = 0.8. During training, we apply ERM to zero-shot classification logits computed using image embeddings and text embeddings of class labels. We utilize weighted sampler to upsample minority subgroups; class and subgroup labels are derived from Stage 1 of RAVL detailed in Section 4.1
VL-GDRO 339 Since GDRO is traditionally used in unimodal classification settings, we adapt GDRO for our multimodal setting by incorporating an extra contrastive vision- language objective function; this loss function is intended to ensure that VLM M learns image-text relationships during training. Specifically, the final loss function during training is LVLGDRO ALeL + (1 AGDDRO Here, LCL takes the form of the original loss function used to train model M; in our experiments LeL is the CLIP objective (36). We set
x = 0.8. During training, we apply GDRO zero-shot classification logits computed using image embeddings and text embeddings of class labels. In line with standard practice, we utilize weighted sampler to upsample minority subgroups: class and subgroup labels are derived from Stage of RAVI as detailed in Section 4.1
Spurious-Awar Miligution [56]: Spurious-aware mitigation aims to address spurious corre- lations in VLMSs using a combination of five loss functions: one CLIP objective function, two contrastive image objective functions meant to address spurious correlations in the image space, and two contrastive language objective functions meant to address spurious
21

---

Table 5: RAVL effectively mitigates spurious correlations across various model initializations. Here, we provide an extended version of Table 3 with a breakdown of results by model initialization (CLIP-RN50 CLIP-RNIOI.. Our results demonstrate that RAVL consistently outperforms prior methods mitigating spurious correlations. We report mean Image Overall (Img. Overall), Image Worst Group (Img. WG), Region Overall (Reg. Overall), and Region Worst Group (Reg. WG) metrics across our real-world evaluation settings.
Method
Discovery Precision @ 10> 0.6
Discovery Precision @ 10 0.8
Img. Overall
Img. WG
Reg. Overall
Reg. WG
Img. Overall
Img. WG
Reg. Overall
Reg. WG
CLIP-RNIO RLI0
Standard
64.2
35.8
36.0
74.3
Upsampled FT
65.2
36.7
73.7
51.0
65.2
38,0
74,3
52.3
VL-ERM
66.0
30.0
73.4
45.2
65.4
28.9
73.7
45.4
VL--GDRO
66.8
31.6
74.1
45.8
66.1
29.6
74.7
46.3
Spurious-Aware RAVL (Ours)
67.4
31.6
74.0
45.2
66.2
28.2
74.5
45,2
67.9
36.9
77.8
55.4
67.8
38.6
79.0
56.5
Standard FT
63.9
28.9
71.3
45.0
64.7
27.6
72.0
44,4
Upsampled FT
67.4
38.4
74.6
52.8
67.8
37.5
75.0
53,2
VI..RRM
70.5
33.5
77.0
53.4
70.8
32.2
77.4
54.0
VL-GDRO
70.5
34,9
76.5
53.1
70.5
32.1
76.8
54.1
Spurious-Aware
71.3
34.8
78.1
71.2
32.4
78.4
53.9
RAVL (Ours)
71.0
40.4
79.5
59.2
71.8
42.2
79.8
59.8
correlations in the text space. We note that Spurious-Aware Mitigation was explicitly de- signed for the fine-tuned VLM setting. We follow the implementation of Spurious- Aware Mitigation provided by [56]. In our work, since we solely fine-tuned the vision encoders of VLMMs M, we use version of Spurious-Aware Mitigation with the CLIP objective function and two contrastive image objective functions.
Prior works on model robustness predominantly evaluate model performance using image worst- group scores [39]. In addition to image worst-group accuracy, also report region overall and region worst-group accuracies, which evaluate the extent to which the VLM understands fine-grained features. Region-lever accuracies computed by performing zero-shot classification with region embeddings and comparing predicted labels to the ground-truth -ound-truth region-level labels provided in the zero-shot classification dataset.
In Table we provide an extended version of Table 3 with a breakdown of results by model initialization (CLIP-RN55 and CLIP-RN101). We demonstrate that RAVI. consistently outperforms prior methods across both model initializations. 3 with breakdown of results by the learned correlation strength of the original VLM M. RAVL consistently outperforms prior methods across four correlation strengths Teval {10, 20, 30, 40).
Table 6 we provide an extended version of Table
D.3 Computational Complexity Analysis
In this section, we provide an analysis of the computational complexity RAVL. RAVL is compu- tationally inexpensive; in particular, the RAVL discovery stage can be efficiently on CPU and the mitigation stage adds only small computational overhead. Below, we provide an analysis of computational complexity for each stage of RAVL.
Computational complexity analysis of RAVL Stage The discovery stage of RAVL is specifically designed to be run on labeled validation dataset Dv; in real-world settings, validation datasets are often relatively small in size due to the human effort needed for securing labels, rendering this stage as computationally inexpensive for diverse applications. Even if the validation dataset is large in size, R RAVL operates efficiently as follows:
First, RAVL preprocesses images by decomposing each image into candidate regions; there are variety of ways in which can decompose image into regions, such by using equal-sized segments (e.g. quadrants) or running inference with region proposal networks (RPNs). Both methods inexpensive and only need to be once in an offline manner. Similar approaches have been applied to large-scale datasets in prior work (3].
Then, embeddings need to generated for each region, which can done by utilizing VLM M for inference (forward passes only). Across set of 10 FashionMNIST and COCO
22

---

Table 6: RAVL effectively mittiattes spurious correlations across learned correlation strengths. Here, we provide an extended version of Table with a breakdown of results by the learned correlation strength (Teval E {10, 20, 30, 40) of the original model M. We use the subset of 106 evaluation settings where RAVL Stage Precision @ 10 is greater than 0.8. Our results demonstrate that RAVL consistently outperforms prior methods in mitigating spurious correlations across various correlation strengths and model initializations. We report mean Image Worst Group (Img. WG) and Region Worst Group (Reg. WG) metrics across our real-world evaluation settings. We note that there valid evaluation settings for CLIP-RNI01 when the learned correlation strength Teval of the original model M is 40.
Method
Tecal
10
Teval
20
Teval
30
Teral
40
Img. WG
Reg. WG
Img. WG
Reg. WG
Img. WG
Reg.
WG
Img. WG
Reg. WG
CLIP-RNI01 CLIP-RN50
Standard
36.0
51.8
43.8
46.6
39.5
Upsampled FT
38.0
52.3
27.7
43.5
33.4
47.3
33.4
46.5
VI--ERM
28.9
45.4
23.3
38.8
25.7
38.3
22.9
32.5
VL-GDRO
29.6
46.3
22.7
37.1
28.6
37.6
26.4
28.2
Spurious-Ars RAVL (Ours)
28.2
45.2
22.2
37.9
24.6
38.4
17.1
30.6
38.6
56.5
35.3
56.8
41.0
60.2
38.0
53.5
Standard FT
27.6
44.4
26.4
43.1
17.3
35.1
Upsampled FT
37.5
53.2
36.0
49.4
25.6
40.0
VL-ERM
32.2
54.0
32.1
51.5
18.1
42.5
VL-GDRO
32.1
54.1
52.8
45.3
Caurious-Aware
32.4
53.9
30.4
51.2
19.3
48.3
RAVL (Ours)
42.2
59.8
39.7
56,4
28.8
54,0
evaluation settings, we observe embedding generation to take a mean of 24.5 seconds on a single A100 GPU.
Finally, given candidate regions and corresponding embeddings, the remainder of the RaVL discovery procedure (clustering and computation of metries) can be run completely on CPU. Across set of 10 evaluation settings COCO and FashionMNIST. we observe that clustering and computation of metrics require mean of 3.4 seconds to run.
The mitigation stage of RAVL requires finetuning a VLM Mnew- Across a set of 10 evaluation settings on COCO and FashionMNIST, we observe that the inclusion of our fine-grained region-aware loss function this stage adds average of 0.15 seconds per training step (on a single 100 GPU) in comparison to the original fine-tuning procedure for M.
Computational complexity analysis of RAVL Stage 2:
E Extended Discussion
The goal of our work is to improve robustness of fine-tuned VLMs to spurious correlations. As VLMs become more commonplace in society, we hope that our approach can enable users to better detect and mitigate model failures prior to deployment. We also note that our work includes a series of evaluations on medical images; rigorous clinical testing is necessary before robustness approaches are deployed in healthcare settings.
Societal Impact:
Limitations: In line with prior works in vision-only and vision-language settings, our method is specifically designed to surface and mitigate local, fine-grained spurious features. There may be some sources of spurious signal that do not manifest this way; for instance, features like image brightness or gender can be considered global features, where the spurious signal is not localized to particular image region. Our approach not designed for these global spurious features. Rather, our problem setting is inspired by the many real-world, practical examples of region-level spurious features that have been demonstrated in literature affect model performance, such as lmage-level markings in dermoscopic images [48, medical devices in radiographs [34]. and text markers in medical images
10
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction describe the claims made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations Appendix Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
The paper does not include theoretical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided details for all experiments in Sections 3 and Sections 4 Hyperparameters and implementation details are discussed in detail in Appendix Sections B
C and D Code is available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
25

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code has been made available. All hyperparameters and details necessary for replicating our results are detailed in the main text and the appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
wublic/guides/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cotee//mdeeeSbmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided implementation details and hyperparameters for all models.
Details are included in Appendix Sections B and D
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Figures and include 95% confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
26

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details on compute resources and computational complexity are provided in
Appendix Sections B
and
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttp:://eeuriii http://wwwrrp..ccooo//occcccccaa
Answer: [Yes]
Justification: Our work conforms to the NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss societal impacts of our work in Appendix Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All existing data (MNIST, FashionMNIST, COCO, SUN397, ObjectCXR), models (CLIP VLMs, OpenCLIP VLMs, PubMedCLIP VLMs), and code used in this paper have been cited and are available under CC BY, CC BY-SA, CC BY-NC, or MIT licences.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
28

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: Our evaluation framework and algorithm are documented in Sections 3 and as well as Appendix Sections B C
and D Training and implementation details are also
documented in these sections.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "XJACqcvFnq", "reviewer": "Reviewer_LC5i", "summary": "This paper tackles spurious correlations between image features and textual attributes in fine-tuned VLMs. It proposes an approach to discover and mitigate spurious correlations using local image features (image regions rather than a whole image). Experiments are done in both controlled settings and realistic settings.", "questions": "Have you tried the approach on other settings, other than fine-tuned VLM?  For example, spurious correlations can happen in other vision classification models as well.", "limitations": "There are discussions on limitations in the supplementary material.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Discovering and mitigating spurious correlations is an interesting problem. The approach of looking into image regions is sound.\nThe paper is generally well written.", "weaknesses": "1) Looking into the regional features may make the approach computationally expensive. Computational complexity analysis is missing in the paper. \n2) A lot of experiments were done under controlled settings. More evaluations in the wild would make the results stronger.\n3) A small error in writing, line 259 says “are designed for unimodal settings” - ref [50] is designed for multi-modal settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer LC5I for reviewing our work and providing helpful feedback. \n\n**[Q1] Computational complexity. “Looking into the regional features may make the approach computationally expensive. Computational complexity analysis is missing in the paper.”**\n\nWe refer the reviewer to General Response [Q1], where we address this point.\n\n**[Q2] In-the-wild experiments. “A lot of experiments were done under controlled settings. More evaluations in the wild would make the results stronger.”**\n\nThank you for this suggestion, and we agree about the importance of real-world validation. However, we note that it is challenging to quantitatively evaluate the accuracy of discovery and mitigation approaches, since the ground-truth spurious correlations learned by a VLM are typically unknown. As a result, we opted to use controlled evaluations, which artificially induce spurious correlations and then quantitatively evaluate the extent to which the correlation can be detected and mitigated. Importantly, using controlled evaluations enables us to **perform large-scale, quantitative evaluations of RaVL without using humans in the loop**. We believe that this procedure is essential before scaling to unlabeled real-world datasets. \n\nIn addition to these controlled quantitative evaluations, our original manuscript provided qualitative results from several in-the-wild evaluations (Section 3.3 and Figure 3). In response to the reviewer's suggestion for expanding these analyses, we have extended this analysis with additional qualitative evaluations. We refer the reviewer to the PDF provided with the general response above. In Rebuttal Figure 1, we show the following:\n- For the OpenCLIP ViT-L/14 model, RaVL surfaces a feature cluster consisting of green plants and fences. We observe a performance gap of 18.3 points between images with class label *\"outdoor chicken coop\"* that contain the RaVL-identified feature and those that do not contain the feature. This suggests that the OpenCLIP ViT-L/14 model can better classify an *\"outdoor chicken coop\"* scene when green plants and fences are present. \n- For the CLIP ViT-B/32 model, RaVL surfaces a feature cluster consisting of people. We observe a performance gap of 24.3 points between images with class label *\"pub (indoor)\"* that contain the RaVL-identified feature and those that do not contain the feature. This suggests that the CLIP ViT-B/32 model can better classify a *\"pub (indoor)\"* scene when people are present. \n- For the OpenCLIP ResNet-101 model, RaVL surfaces a feature cluster consisting of chairs. We observe a performance gap of 23.3 points between images with class label *\"restaurant patio\"* that contain the RaVL-identified feature and those that do not contain the feature. This suggests that the OpenCLIP ResNet-101 model can better classify *\"restaurant patio\"* scenes when chairs are present. \n\n**[Q3] References. “A small error in writing, line 259 says “are designed for unimodal settings” - ref [50] is designed for multi-modal settings.”**\n\nThank you for raising this point. In Line 259, the phrase \"designed for unimodal settings\" is intended to refer solely to Distilling Failures, George, and Domino (referenced in Line 257). We agree with the reviewer that Spurious-Aware Detection (reference [50]) is not unimodal, and we will be sure to improve the clarity of this sentence in the final version of our manuscript.\n\n**[Q4] Extending beyond VLMs. “Have you tried the approach on other settings, other than fine-tuned VLM? For example, spurious correlations can happen in other vision classification models as well.”**\n\nThank you for this suggestion, and we agree that spurious correlations can occur in many types of models. In this work, we focus specifically on the setting of fine-tuned vision-language models because fine-tuned VLMs (i) are becoming increasingly commonplace, particularly in domain-specific applications like medicine, (ii) are likely to be trained on small domain-specific datasets, preventing models from gaining the robustness benefits associated with web-scale training and increasing the likelihood of learning spurious correlations, and (iii) are an underresearched yet important problem setting. In order to address spurious correlations in the fine-tuned VLM setting, RaVL assumes the existence of paired language; consequently, RaVL utilizes text embeddings for both discovery (RaVL Stage 1) and mitigation (RaVL Stage 2). \n\nIn the context of vision-only classification models where the language modality is not present, our discovery approach can be modified by computing the image score distribution vector $s_{I_i}$ and the region score distribution matrix $S_{R_i}$ using softmax-normalized class logits. The mitigation approach can be modified by framing the region-aware contrastive loss function as a cross entropy loss between region-level logits and class labels. We aim to explore these directions in future work. \n\nWe again thank Reviewer LC5I for their review of our manuscript and their positive overall assessment of our work. We hope that the above responses adequately address all concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iSTVqCoG4F", "reviewer": "Reviewer_aCX1", "summary": "Vision-language models (VLMs) tend to exhibit poor zero-shot performance when compared to task-specific models. However, fine-tuned VLMs may capture spurious correlations in domain-specific datasets which may be small in size. The paper proposes an automated spurious correlation detection and mitigation method for fine-tuned VLMs. The method first discovers spurious correlations by leveraging a region-level clustering approach to identify precise image features contributing to zero-shot classification errors. Then, it mitigates the identified spurious correlation with a region-aware loss function that enables the VLM to focus on relevant regions and ignore spurious relationships during fine-tuning. Experiments demonstrate the effectiveness of this method on general domain and medical-domain VLMs.", "questions": "1. How to obtain region-level labels when constructing the evaluation dataset (Line 220)?\n\n2. How to determine the hyperparameters used in the spurious correlation discovery and mitigation stages?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written. Key assumptions are clearly stated and the proposed method is easy to follow.\n\n- The paper proposes a useful method called RAVL to discover and mitigate fine-grained spurious correlations that can be easily interpreted by humans. \n\n- Experiments in the controlled and uncontrolled settings demonstrate that RAVL effectively discovers and mitigates spurious correlations between image features and textual attributes.", "weaknesses": "There is a lack of computational complexity analysis. To detect fine-grained spurious correlations, RAVL needs to segment images into multiple regions, cluster all the regions per class, and calculate the contributions of each cluster to mispredictions. Therefore, the computational complexity seems high. It would be better to analyze the time cost of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer ACX1 for reviewing our work and providing helpful feedback. \n\n**[Q1] Computational complexity. “There is a lack of computational complexity analysis. To detect fine-grained spurious correlations, RAVL needs to segment images into multiple regions, cluster all the regions per class, and calculate the contributions of each cluster to mispredictions. Therefore, the computational complexity seems high. It would be better to analyze the time cost of the proposed method.”**\n\nWe refer the reviewer to General Response [Q1], where we address this point.\n\n**[Q2] Region-level labels for evaluation. “How to obtain region-level labels when constructing the evaluation dataset (Line 220)?”**\n\nIn order to evaluate RaVL, we create 654 evaluation settings from two domains: (1) synthetic data (MNIST and FashionMNIST) and (2) real-world data (COCO). Each evaluation setting includes an *evaluation dataset* with images $I_i$, class labels $y_i$, region bounding boxes $R_i$, and region-level labels $L_i$. We obtain region-level labels as follows:\n- For our synthetic data settings, we construct each image $I_i$ to include four equally-sized quadrant regions $\\mathcal{R}_i$. We randomly select one quadrant to include the core object (i.e. digit or fashion item) associated with the class label $y_i$. In some images, we randomly select another quadrant to include the pre-defined spurious feature $e^{eval}$, which is a red rectangle in this case. Since these images are synthetically generated, we have a priori knowledge of the features in each quadrant, which we use to generate the region-level label set $L_i$. \n- For our real-world settings, the set $\\mathcal{R}_i$ consists of the ground-truth bounding boxes annotated in COCO and $L_i$ consists of the associated object labels. We note that recent advances in open-set object detection (e.g. Recognize Anything) can enable this procedure to be extended to other datasets even if ground-truth bounding boxes and labels are not available.\n\nWe additionally emphasize here that the RaVL discovery and mitigation approaches do not require region-level labels, and users who wish to use RaVL do not need access to region-level labels. Here, we exclusively use region-level labels within our evaluation framework in order to quantitatively assess performance. \n\n**[Q3] Hyperparameter selection. “How to determine the hyperparameters used in the spurious correlation discovery and mitigation stages?”**\n\nHyperparameters for the discovery stage (RaVL Stage 1) are selected as follows:\n- *Number of clusters*: RaVL includes a clustering step to identify groups of visually-similar regions. We do not manually set this hyperparameter; rather, we leverage an automated approach for selecting the optimal number of clusters. For each evaluation setting, we sweep all cluster sizes ranging between $|\\mathcal{Y}| * 2$ and $|\\mathcal{Y}| * 5$ where $|\\mathcal{Y}|$ represents the size of the class label set; we then select the optimal number of clusters using Silhouette scores. We select these bounds to be larger than the class label set size by several multiples in order to ensure that clusters adequately separate distinct features. Prior works have also utilized overclustering approaches for this objective [5,6]. Users can adjust the bounds based on their composition of their dataset; for instance, complex datasets with diverse features may require a larger range.\n- *Threshold for pruning cluster influence scores ($\\tau_l$)*: In order to identify candidate image features that directly contribute to classification errors, we prune all clusters with low cluster influence scores below a threshold $\\tau_l$. We set $\\tau_l$ to 0.25 in this work. This hyperparameter was determined empirically based on experiments on a small set of validation data.\n\nHyperparameters for the mitigation stage (RaVL Stage 2) are selected as follows:\n- *Contrastive loss temperature $\\tau$*: We set the contrastive loss temperature to a default value of 0.07 based on prior work [7]. \n- *Loss weighting factor $\\lambda$*: The loss weighting factor $\\lambda$ balances the tradeoff between the original CLIP loss function $L_{CL}$ and our novel region-aware loss function $L_{R} + L_{A}$. We set $\\lambda$ to 0.8 in this work, which upweights the CLIP loss function $L_{CL}$. This weighting factor ensures that the model $\\mathcal{M}_{new}$ learns accurate global image-text relationships without capturing fine-grained spurious correlations. We determined this hyperparameter based on experiments on a small set of validation data. We additionally note here that the \"Upsampled FT\" baseline in Table 3 is an ablation where $\\lambda$ is set to 1 (and the region-aware loss function is not utilized); this demonstrates the efficacy of our loss function.\n\nWe note that like all hyperparameters, these values could likely be further optimized; however, even without extensive hyperparameter tuning, we achieve competitive results. Additional details on hyperparameters are provided in Appendix B, C, and D. \n\nWe again thank Reviewer ACX1 for their review of our manuscript and their positive overall assessment of our work. We hope that the above responses adequately address all concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ED17JgIFI2", "reviewer": "Reviewer_onxK", "summary": "This paper introduces RAVL, a method designed to identify and address spurious correlations in Vision-Language Models (VLMs). The authors emphasize two key areas: (a) prioritizing local image features over global image-level features, and (b) concentrating on the fine-tuning phase rather than the pre-training phase. They implement the first idea through the following steps: 1. Utilizing the VLM embedding space to extract candidate image features from the validation set. 2. Filtering these candidate features to pinpoint those that directly contribute to classification errors. 3. Ranking the remaining image features based on the extent of their learned spurious correlations. To mitigate spurious correlations, the authors regularize the dissimilarity between the spurious regions and correlated labels while encouraging high embedding similarity.", "questions": "1. What if this framework, instead of using RoIAlign, considers feature maps as candidates and filters them channel-wise? If this approach is effective, it could provide a more general framework from the perspective of Weakness 1, as feature maps are activated differently, representing various features [1].\n\n2. Why did the authors use K-Medoids clustering? Additionally, I am curious about the authors' choice to use softmax-normalized embeddings. Feature maps might offer more diverse information for comparison during clustering.\n\n3. Is this framework robust on batch size?\n\n\n[1] Jeon, Myeongho, Myungjoo Kang, and Joonseok Lee. \"A Unified Framework for Robustness on Diverse Sampling Errors.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "limitations": "The authors discussed the limitations and potential negative societal impacts of their work in the Appendix. I fully agree with their assessment.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-structured and easy to follow.\n\n2. The proposed framework is sensible and, to the best of my knowledge, novel. I believe its contribution is slightly above the acceptance borderline.\n\n3. The enhancement of performance quantitatively demonstrates the proposed method. It is notable.", "weaknesses": "1. This framework is limited to mitigating geometric spurious correlations by focusing on regional candidates. While some studies such as [1] highlight texture and background information as significant contributors to spurious correlations, it is more desirable to address all potential sources of spurious correlations.ss all the potential spurious correlations.\n\n2. Although they used RoIAlign, there are no qualitative visualization results to show which regions affect which classes. I believe this would be important for understanding the model's behavior from a human vision perspective.\n\n[1] Geirhos, Robert, et al. \"ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.\" arXiv preprint arXiv:1811.12231 (2018).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer ONXK for reviewing our work and providing helpful feedback. \n\n**[Q1] Assumption of region-level spurious correlations.**\n\nWe thank the reviewer for raising this point. In line with prior works in vision-only and vision-language settings [1,2,3], RaVL is specifically designed to surface and mitigate fine-grained, region-level spurious features. As we acknowledge in the Appendix (Lines 779-787), we agree with the reviewer that there are some sources of spurious signal that do not manifest in this way (e.g. global, image-level features like texture or background information as mentioned in Geirhos et al.). However, we focus explicitly on region-level spurious features due to the fact that such spurious correlations have been shown to negatively affect model performance in many real-world, practical settings; some notable examples include image-level markings in dermoscopic images [8], medical devices in radiographs [9], and text markers in medical images [10]. As a result, we believe our work effectively complements work done by Geirhos et al. and others by targeting an important yet underresearched category of spurious correlations (namely, region-level features) that have been shown to contribute to significant model performance gaps.\n\n**[Q2] Visualizations.**\n\nThank you for this suggestion. We refer the reviewer to the PDF provided with the general response. In Rebuttal Figure 2, we demonstrate the utility of our mitigation approach by providing qualitative visualizations using GradCAM. Visualizations show that after the RaVL mitigation procedure, the VLM $\\mathcal{M}_{new}$ more precisely relies on core features. \n\nWe also note that in order to quantitatively evaluate the extent to which the VLM understands fine-grained region-level information, we introduced the Region Overall and Region Worst Group metrics in Section 4.2. Results in Table 3 show that RaVL contributes to improvements on these metrics, suggesting better understanding of region-level information.\n\n**[Q3] Use of feature maps.**\n\nWe agree that feature maps serve as a rich source of signal and can be leveraged for identifying and addressing spurious correlations. Indeed, several works have explored this direction by identifying neural features that influence classification decisions and then using humans in the loop to annotate feature maps as core or spurious [2,11]. Although this approach was shown to work well, a key consideration is the need for human supervision in order to interpret feature maps and distinguish between core and spurious features. \n\nIn our work, we used ROIAlign and region-level embeddings rather than feature maps for the following reasons: \n- Enable automated analysis: Our approach is designed to operate without the need for human supervision, enabling generalizability to datasets from diverse domains (e.g. medical image data). Using region-level embeddings enables us to perform automated detection and mitigation of spurious correlations via our novel clustering approach and our region-aware loss function. On the other hand, using feature maps will likely require human supervision, as shown by [2] and [11].\n- Perform dimensionality reduction: Whereas feature maps may be large in size with many channels, region-level embeddings exhibit low dimensionality while still capturing relevant semantic information. With regards to the reviewer's point about the clustering stage, using softmax-normalized region-level embeddings can enable more efficient and accurate clustering in comparison with using large feature-maps that encompass diverse features.\n- Target fine-grained spurious correlations: As discussed in [Q1], our goal is to specifically detect and mitigate fine-grained, region-level spurious correlations. Using local region-level embeddings rather than global feature maps helps us accomplish this goal. \n\n**[Q4] Clustering approach.**\n\nWe used K-medoids clustering for the following reasons:\n- Robustness to outliers: When compared to a standard K-Means clustering approach, K-Medoids is less sensitive to the presence of outliers due to the fact that cluster centroids are medians rather than means. Robustness to outliers is important in our setting, since region-level embeddings represent visually-diverse features and there is likely to be significant variation across samples.\n- Ability to operate with custom distance functions: Whereas K-Means traditionally minimizes the Euclidean distance between the centroid and the samples in a cluster, K-Medoids can be implemented with custom distance functions. Due to the fact that the original VLM $M$ is trained with respect to cosine distance, we implement K-Medoids with a cosine distance metric in this work in order to effectively model the patterns learned by $M$. \n\n**[Q5] Batch Size.**\n\nIn order to train model $M_{new}$, we utilize a combination of the CLIP loss $L_{CL}$ and our region-aware loss $L_R + L_A$. Both loss functions are contrastive in nature and require a large set of negative samples in a batch in order to learn useful representations. As a result, our framework is sensitive to batch size; this is expected behavior [12] and is consistent with a long, existing line of work on contrastive learning methods (e.g. [7]). We note that for all results reported in Table 3, we keep batch size consistent across methods in order to enable fair comparisons. Below, we provide a brief batch size ablation on a single COCO evaluation setting. As expected, significant reductions in the batch size reduce efficacy of contrastive learning. \n\n| Batch Size | Method | Image Worst-Group | \n|----------------------|------------------|------------------|\n| 128 | Standard FT    | 51.5 | \n| 128 | RaVL (Ours) | **62.1** |\n| 32 | Standard FT | 39.4 | \n| 32 | RaVL (Ours) | **48.3** |\n\nWe again thank Reviewer ONXK for their review of our manuscript and their positive overall assessment of our work. We hope that the above responses adequately address all concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful review of our manuscript. We were encouraged to see that all reviewers rated our work as a "technically solid, moderate-to-high impact paper". Reviewers also found the paper to be "well-written" and "well-structured" (Reviewers LC5I, ACX1, ONXK); the problem setting to be "interesting" (Reviewer LC5I); and our proposed approach to be "sound", "novel", and "easy to follow" with "notable" performance improvements (Reviewers LC5I, ONXK, ACX1). 

In response to feedback, we provide a general response here to points raised by multiple reviewers, individual responses below to address each reviewer’s concerns, and an attached one-page PDF with new figures.

**[Q1] Reviewers LC5I and ACX1 asked for additional analysis on the computational complexity of RaVL.**

We designed our approach to be computationally inexpensive; in particular, the discovery stage can be run efficiently on CPU and the mitigation stage adds only a small computational overhead. Although RaVL does involve multiple stages as noted by reviewer ACX1, our procedure delivers the significant advantage of enabling **fully-automated analysis** of spurious correlations; in contrast, several recent works on fine-grained robustness (both in the vision-only and vision-language settings) have leveraged humans-in-the-loop [1,2,3]. Below, we provide an analysis of computational complexity for each stage of RaVL. 

*Computational complexity analysis of RaVL Stage 1*: In response to points from Reviewer ACX1 about the discovery stage (RaVL Stage 1), we note that our approach is specifically designed to be run on a labeled validation dataset $\mathcal{D_V}$; in real-world settings, validation datasets are often relatively small in size due to the human effort needed for securing labels, rendering this stage as computationally inexpensive for diverse applications. Even if the validation dataset is large in size, RaVL operates efficiently as follows:
- First, RaVL preprocesses images by decomposing each image into candidate regions; there are a variety of ways in which a user can decompose an image into regions, such as by using equal-sized segments (e.g. quadrants) or running inference with region proposal networks (RPNs). Both methods are inexpensive and only need to be run once in an offline manner. Similar approaches have been applied to large-scale datasets in prior work [4]. 
- Then, embeddings need to be generated for each region, which can be done by utilizing VLM $\mathcal{M}$ for inference (forward passes only). Across a set of 10 FashionMNIST and COCO evaluation settings, we observe embedding generation to take a mean of 24.5 seconds on a single A100 GPU. 
- Finally, given candidate regions and corresponding embeddings, the remainder of the RaVL discovery procedure (clustering and computation of metrics) can be run completely on CPU. Across a set of 10 evaluation settings on COCO and FashionMNIST, we observe that clustering and computation of metrics require a mean of 3.4 seconds to run on a single A100 GPU. 

*Computational complexity analysis of RaVL Stage 2*: The mitigation stage (RaVL Stage 2) requires finetuning a VLM $M_{new}$. Across a set of 10 evaluation settings on COCO and FashionMNIST, we observe that the inclusion of our fine-grained region-aware loss function at this stage adds an average of 0.15 seconds per training step (on a single A100 GPU) in comparison to the original fine-tuning procedure for $M$. 

We will be sure to update our manuscript with this computational complexity analysis. 

We would again like to thank all reviewers for their time and feedback, and we hope that our responses adequately address all concerns.

**References:**

[1] Yang, Y., et al. Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning. ICML, 2023. 

[2] Singla, S. et al. Salient ImageNet: How to discover spurious features in Deep Learning? ICLR, 2022. 

[3] Moayeri, M. et al. Hard ImageNet: Segmentations for Objects with Strong Spurious Cues. NeurIps, 2022. 

[4] Zhong, Y. et al. RegionCLIP: Region-based Language-Image Pretraining. CVPR, 2022. 

[5] Eyuboglu et al. “Domino: Discovering Systematic Errors with Cross-Modal Embeddings.” ICLR 2022. 

[6] Sohoni et al. “No subclass left behind: Fine-grained robustness in coarse-grained classification problems.” NeurIps 2020. 

[7] Radford et al. “Learning Transferable Visual Models From Natural Language Supervision.” ICML 2021.

[8] Winkler, J. K., et al. Association between surgical skin markings in dermoscopic images and diagnostic performance of a deep learning convolutional neural network for melanoma recognition. JAMA dermatology, 2019. 

[9] Oakden-Rayner, L., et al. Hidden stratification causes clinically meaningful failures
in machine learning for medical imaging. ACM Conference on Health, Inference, and Learning (CHIL), 2020. 

[10] DeGrave et al. AI for radiographic COVID-19 detection selects shortcuts over signal. Nature Machine Intelligence, 2021. 

[11] Moayeri et al. Spuriosity Rankings: Sorting Data to Measure and Mitigate Biases. NeurIps, 2023. 

[12] Chen et al. “Why do We Need Large Batchsizes in Contrastive Learning? A Gradient-Bias Perspective.” NeurIps 2022.

Meta-review: RaVL is a method to detect and mitigate spurious correlations in VLMs. The method tries to figure out what features contribute to zero-shot classification errors and then try to remedy those. The reviewers agree this is an important problem, the paper is well written and presents a contribution to the literature. There was some discussion with one of the reviewers about the impact of batch size on performance, but I do not think this changes the main findings of the paper. Overall the method seems solid, experimental evidence seems satisfactory. For these reasons I vote to accept this paper.
Decision: Accept (poster)