Mitigating Spurious Correlations in Zero-Shot Multimodal Models
Authors: Shenyu Lu
Junyi Chai
Xiaoqian Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Spurious correlation
Zero shot
Multimodal models
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose a method that mitigates spurious correlations in multimodal models in the zero-shot setting.

Abstract: Multimodal models or Vision Language Models (VLMs) have reshaped the paradigm in machine learning, offering zero-shot capabilities that require no additional training when adapted to new classification tasks. However, despite their advancements, spurious correlations still exist in VLMs. Existing approaches to tackle this issue often require target label annotations, contradicting the principle of zero-shot classification, or they primarily focus on a single modality, risking misalignment between text and image modalities. Others rely on extensive domain knowledge or large language models (LLMs) to characterize spurious features, making the performance sensitive to the generated prompts and undermining zero-shot capability. In response, we propose a new solution that tackles spurious correlations in VLMs within the zero-shot setting. Our approach utilizes a translation operation that preserves the latent space distribution to address issues of spurious correlations. In particular, our method is grounded in and inspired by a theoretical analysis, which identifies that the optimal translation directions are along the spurious vector. As VLMs unify two modalities, we compute spurious vectors from the text prompts and guide the translation for image embeddings, aligning the requirements for the fusion of different modalities in VLMs. We conducted experiments on benchmark datasets, which have shown significant improvements in worst-group accuracy. Additionally, our visualizations of VLMs further demonstrate the effectiveness of this intervention.

Published as a conference paper at ICLR 2025
MITIGATING SPURIOUS CORRELATIONS IN ZERO- SHOT MULTIMODAL MODELS
Shenyu Lu, Junyi Chai & Xiaoqian Wang"
Elmore Family School of Electrical and Computer Engineering Purdue University West Lafayette, IN 47906, USA {lu876, chai28, joywang) Spurdue edu
ABSTRACT
Multimodal models or Vision Language Models (VLMs) have reshaped the paradigm in machine learning, offering zero-shot capabilities that require ad- ditional training when adapted to new classification tasks. However, despite their advancements, spurious correlations still exist in VLMs. Existing approaches tackle this issue often require target label annotations, contradicting the principle of zero-shot classification, or they primarily focus on a single modality, risking misalignment between text and image modalities. Others rely on extensive do- main knowledge or large language models (LLMss) to characterize spurious fea- tures, making the performance sensitive to the generated prompts and undermin- ing zero-shot capability. response, we propose a new solution that tackles spu- rious correlations in VLMs within the zero-shot setting. Our approach utilizes a translation operation that preserves the latent space distribution to address issues of spurious correlations. In particular, our method is grounded in and inspired by a theoretical analysis, which identifies that the optimal translation directions are along the spurious vector. As VLMs unify two modalities, we compute spurious vectors from the text prompts and guide the translation for image embeddings, aligning the requirements for the fusion of different modalities VLMs. We conducted experiments on benchmark datasets, which have shown significant im- provements in worst-group accuracy. Additionally, our visualizations of VLMs further demonstrate the effectiveness of this intervention.
INTRODUCTION
Vision Language Models (VLMs) have sigmificantly enhanced the capabilities of machine leamn- ing systems. Contrastive Language-Imen Pretraining (CLIP) (Radford et al.] [202T), which bridges the fields of computer vision and natural language processing, has profoundly transformed the land- scape. One of the fascinating capabilities of VLMs is their zero-shot functionality (Guo et al.] 2023). This functionality enables models to infer the most probable answer from a set of potential responses provided by the user, even without training on the specific dataset.
Despite the power of VLMs, these models still suffer from spurious correlations (Zheng al. 2024 Dehdashtian et al.) 2024 Wortsman et al.) 2022), phenomenon where predictions are based on irclevant features, leading to detrimental performance for certain groups (Sagawa et al. 2019). Spurious correlations pose significant risks in high-stakes settings such medical diagnostics. For instance, in diagnosing skin cancer, if a color patch is spuriously correlated with benign samples, the model may erroneously base its predictions the presence of this color patch (Yan al. 2023
Nauta et al. 2021 (See Figure ISIC Dataset (Codella et al. 2019)).
Addressing spurious correlations in VLMs is increasingly imperative. Efforts such as (Yang al.) 2023 Pang et al. 2024 Goyal et al., 2023 Zhang & Rd 2022] Wang et al. (023) have aimed to mitigate spurious correlations issues within VLMs. However, these methods rely on target labels, practice that contradicts the label-free requirements of zero-shot classification.
"Corresponding author.

---

Published as a conference paper at ICLR 2025
ISIC Dataset
Waterbirds Dataset
CLIP zero-shot
Ours
CLIP zero-shot
Ours
Figure 1: Heatmap visualization for zero-shot classification. The benign lesion class in the ISIC dataset is spuriously correlated with the presence of color patches, leading to predictions of benign lesions being dangerously dependent on this feature in the biomedical setting. Similarly, in the Waterbirds dataset, there a spurious correlation between waterbirds and water backgrounds. Our approach effectively decorrelates these spurious relationships without requiring training process, promoting group robustness in the zero-shot setting.
A key characteristic of VLMs is the integration of an image encoder and text encoder, which pro- cess image and text inputs, respectively. These inputs are transformed into image embeddings and text embeddings. Many studies (An et al. 2024 Chuang et al.) 2023: Trager et al. 2023) have con- centrated on mitigating spurious correlations via text embeddings. However, these methods present several challenges. Firstly, they concentrate exclusively tial risk of misalignment between modalities. This contradicts the principle of matching different modalities in VLMs. Secondly, these methods require strong domain expertise access gen- erative tools such as Large Language Models (LLMs) to generate descriptions of the concepts of spurious features substantial exemplars of such features. However, the responses from generative tools are not reliable. ZZhang et al. (2023b); : Xu et al. (2024) indicate the existence of hallucinations in LLMs. This unreliability substantially diminishes the effectiveness of methods designed to mit- igate spurious correlations through text- based modalities. Moreover, An et al. (2024); Adila et al. 2024 observe performance disparities when using different L.LMs.
single modality, posing substan-
A recent study, ROBOSHOT (Adila et al. 2024), has been proposed to address spurious correla- tion issues by considering both image and text modalities. ROBOSHOT employs LLMs generate sufficient insights for spurious features and then applies linear projection to map image embed- dings onto a neutralization hyperplane for these spurious features. This approach presents several challenges. First, the spurious insights generated by LLMs are inherently less reliable. Second, the projection operation distorts the distribution of image embeddings and significantly reduces their diversity. Third, this method lacks theoretical analysis of the optimality of the projection direction, a factor that critically influences the performance of group robustness.
To up, existing methods be categorized into three types, each with specific concerns. First, some methods require target labels, violating the zero-shot classification requirements. Second, methods that focus solely on one modality face risks of misalignment when integrating different modalities. Third, approaches using linear projection distort the distribution of image embeddings. Additionally, reliance LLMs introduces concerns regarding reliability.
To robustify zero-shot VLMs effectively, the main requirements are no training, no label require- ment, no reliance on LLMs. To address these challenges, we propose novel approach TIE, a framework that utilizes text prompt guidance to reduce spurious features in image embeddings. Contrary to the linear transformation techniques introduced in (Adila et al.] 2024] CCuang et al., 2023), we adopted translation operation in the latent space, which preserves the distribution of image embeddings. Our method is grounded theoretical analysis that identifies the optimal pa- rameter for translating image embeddings. Unlike methods that focus on single modality, we incorporate text prompts to guide the translation operation in the image space, thereby preserving alignment across both modalities.
In practice, when spurious labels are inaccessible, we develop TIE+. TIE* leverages a zero-shot manner to infer spurious features and utilizes pseudo-spurious labels to enhance the group robustness of VLMs, without relying on manual annotations. Throughout this process, our method does not require training any parameters in VLMs, thus enhancing efficiency.
We conducted extensive experiments on real-world datasets, including high-stakes biomedical set- tings. The results show that method significantly outperforms existing approaches. Additionally,
2

---

Published as a conference paper at ICLR 2025
provide visualizations to demonstrate that the proposed method effectively mitigates spurious correlations.
We summarize contribution follows:
We propose a theoretically inspired method that is simple and effective in mitigating spuri- ous correlation issues in VLMS for zero-shot classification.
The proposed algorithm operates without the need for LLMss labeled data, and does not require access to the internal parameters of VLMs.
We empirically validate the effectiveness of the proposed method, including visualizations across both image and text modalities.
RELATED WORKS
2.1
GROUP ROBUSTNESS
Many methods have been proposed to enhance group robustness and address issues spurious cor-
relations (Sagawa et al. 2019 Arjovsky al. 2019 drisssi et al. 2022 Kirichenko al., 2022 Liu
al. 2021] Yao et 2022 Krueger et al., [221]]
L Lu et 22024) These approaches predominantly
utilize reweighting techniques adjust the weights of samples in the training These methods designed for single-modality classification and involve training either or a subset of the model's parameters. In contrast, approach significantly differs from these conventional methods it requires no adjustments to the parameters in the backbone during the robustification process.
2.2 MITIGATING SPURIOUS CORRELATION VLMs
To mitigate spurious correlations in VLMs, many approaches focus on fine-tuning using labeled datasets. Specifically, Goyal et al. 2023) employ target labels derived from text descriptions and fine-tunes the model using a contrastive loss. Yang et al. (2033) propose method that detects spu- rious attributes and fine-tunes VLMs using contrastive loss both within and across different modali- ties. Petryk et al. (2022) propose a framework that uses VLMs to integrate textual information with images and generate saliency map. This map is then used supervise the training of classifier. Zhang Ré 2022) propose an adapter that connects to the embedding layer and utilizes contrastive loss fine-tune the adapter. Dehdashtian al. (024) propose method that employs the Hilbert- Schmidt Independence Criterion (HSIC) to debias both image and text embeddings. Pang et al. (202)) introduce method for distributional robustness via language that maximizes the entropy of predictions on spurious attributes. Distinet from the existing methods mentioned above, our method operates without any labeled data, thus fulfilling the requirements for zero-shot classification.
2.3 GROUP ROBUSTNESS IN ZERO-SHOT CLASSIFICATION
Another line of research addresses spurious correlation issues in VLMs in a zero-shot manner. Trager et al. (2023) propose method that combines target prompt with spurious prompts and averages them to generate an 'Ideal words' prompt. An et al. (2024) employs a two-step inference method that first identifies spurious features and then augments the text prompt with these identified features. Chuang et al.] (2023) propose method that projects text embeddings onto space or- thogonal to the spurious attribute space. [G et (2023) aim to enhance text prompt robustness by focusing on label augmentation. AAdila et al. (2024) propose a method that uses the Gran-Schmidt process to project representations onto space orthogonal to spurious features. In contrast, our method does not depend on augmenting the prompt, which simplifies usage and reduces concerns about the hallucination problem in LLMs. Additionally, our approach aims to mitigate spurious correlations from multimodal perspective.
METHODS
3.1 PRELIMINARIES
Setting. This work focuses on the group robustness setting (Sagaaa et al. 2019) in the zero-shot classification task. Denote E X' as the input image, y E y the target label, and A as the spurious feature. Define group 9y.a E G considering the combination of target label y and spurious feature a. To mitigate the impact of spurious correlations prediction, approach
3

---

Published as a conference paper at ICLR 2025
follows the established practices (Sagawa et al. 2019 Liu al. 2021 Kirichenko al. 2022)
aimed at enhancing the accuracy the worst groups while preserving overall accuracy.
Relationship between vanilla classification and zero-shot classification. We first bridge these two tasks for the subsequent theoretical discussion. Denote dy(+) as the image encoder, Ør(-) as the text encoder, by T as the text prompt, with each text prompt corresponding to target label y. For example, in waterbirds dataset (Sagawa et al. (201)), for = Waterbird, t, = "a photo a waterbird, rd, photo of landbird, }.
of waterbird", T {a photo where [7] K, corresponding to K classes of text prompts. For zero-shot classification, the VLMs model serves as score function that maps X' x T R:
i arg [ (0r(x), @r((+)).
Equation D shows the zero-shot paradigm that predicts the class the one with the highest inner product between the image embedding and the text prompt embedding.
Vanilla classification: Denote h E R the representation learned from neural network, which processed by an image encoder Φ iee. h = (x)). W = [We] E Rdx as a linear
classifier. The vanilla classification task:
arg maxke[K] W h = arg ( W).
(2)
Comparing Equation with Equation it can be concluded that the zero-shot classification rep- resents a specialized form of vanilla classification, where the linear classifier composed of text embeddings. For simplicity in the following analysis, we use h to denote Φy (x) and w to represent Ør (ty), based on their equivalence.
3.2 THEORETICAL ANALYSIS
Spurious correlation modcling. adopt a common setting in modeling spurious correlation
(Sagawa et al. 2020 Idrissi et al. 2022 Yao et al. [022] Wang & Wang 2024). Concretely, denote a spurious feature a {-1, I and a label y E 1, Each (y, a) group denoted as 9y.a has its
own distribution over the image embedding
has hoores Anoise] E Rd where
hale ~ N((,o)))) lccorly ~ Nyy o deore)s
hgoise (O,
(3)
The data model assumption is for the simplicity of the following analysis. Without loss of generality, the dimensions of core features and spurious features can be arbitrary. We investigate the problem of improving the group robustness of VLMs in zero-shot setting by adjusting h given fixed target text prompts. By modeling each group with equal weight, the goal is to maximize each group-wise utility:
A (beggaa w) max
Σ A(by... w;y).
(4)
h
9y.a EG
where A(-) is the accuracy function, hg... corresponds to the image embeddings from group Jy.a- We introduce Lemma 1, which establishes that the accuracy for each group be derived in an analytical form.
Lemma Under the above data model assumption, the group-wise accuracy can be derived as
w Pa
2 Zerfc(-
), if =
/2w 'Egg,
A A(b w;y) =
(5)
1 +
- erfl
if = -1,
2
//w Egg.a
where pgy.. and Egy.. represent the mean and covariance matrix of the image embedding hgg..
The proof is presented in Appendix A This lemma quantifies the accuracy of each (y, a) group given a fixed classifier w. According to Lemma I, adjusting either fu or Σ impacts the group-wise accuracy. The solution proposed by (Adila al. 2024) involves changing Σ, which changes the
4

---

Published as a conference paper at ICLR 2025
distribution of the image embeddings in the latent space. This change necessitates highly precise decision boundary for spurious features, as the accuracy of the porft-ner group is extremely sensitive to the accuracy of this boundary. If the boundary is not accurately defined, the worst- performing group's accuracy will significantly deteriorate. We discuss this phenomenon further and provide theoretical comparison along with experimental validation of our approach in Section 3.3 and Appendix C.I
Objective. We propose translation operator that preserves the distribution of image embeddings. In particular, our objective function is to find the optimal translation vectors Va to maximize the following objective function:
Acco bog.a. W) max Σ A(bg,, + Va. w;y),
(6)
Ox.EE9
Va is the translation vectors based on the label of spurious features. In Theorem 1, we establish the optimal vector for translation within the complete set of feasible directions. We leave the detailed proof Appendix
Theorem Given the objective function and the data model, the maximizer of the objective is ob- tained by
E Phal,
0 0 0
where P Rdxd is an elementary matrix, P
Theorem states that the optimal translate vector Va can be computed by Va = which the negative direction of the spurious feature vector. However, estimating the spurious feature vector presents challenge. Wu et al. (2023) proposed first training a classifier to classify the spurious feature and then using the vector orthogonal to the decision hyperplane the spurious feature vector. We argue that this method significantly compromises efficiency as the need for training and risks misalignment combining both text and image embeddings is crucial. Therefore, we propose using spurious text embeddings to guide image embeddings toward an optimal state.
TE[ aa O,.... 0].
the text embedding space. the realm of VLMs, effectively
3.3
TIE: TEXT PROMPT BASED IMAGE EMBEDDING TRANSLATION
Target: classify
O vs.
Spurious: color
Spraoou prompts
Deternine spurions veetors
procepts
photo with water and bockground"
Text encoder
photo of waterbird/ lanobira"
Text emcoder
Infer pscudo label.
Zenedant
Translato along spasious vectors
Image
TI8
Figure 2: TIE overview. First, we utilize spurious prompts to compute the spurious vectors. We then employ the CLIP model to infer the spurious label for each sample. Subsequently, translate the image embeddings along the spurious vector based on the pseudo-spurious label. Finally, we use these translated embeddings to perform the zero-shot classification task.
We now present our method to mitigate spurious correlations in the VLMs, an overview is shown in Figure 2 Based on the analysis in Section 3.2. we first compute the spurious feature vector. Next, translate the image embeddings along the opposite of this direction, and then use the adjusted image embeddings to perform zero-shot classification.
Computation on Spurious feature vector: Given set of spurious text prompts To (e.g. photo with a water background, a photo with a land background), TIE computes the spurious vector v @r(tai a), s.t. to To. TIE normalizes by its La norm: Va T

---

Published as a conference paper at ICLR 2025
Translate the image embeddings. Given image, TIE first computes its image embedding using the image encoder, i.c., h2 = (x;xa)). Then, TIE computes the magnitude of the translation by A, = [ val, which is the average projection length on the direction of Va. Next, TIE translates image embedding by
ha ha da Va-
The zero-shot classification task employs h. and target prompts for execution.
Without spurious feature label. One constraint on TIE is its dependency on access to labels for spurious features, with samples bearing various spurious labels moving in different directions to achieve an optimal state. To address this, we propose + that eliminates the need for any labeled data within the dataset.
An et al. (2024) empirically demonstrated that spurious features can be effectively inferred using VLMs. Building upon this insight, leverage VLMs infer the spurious labels for each sample in the dataset. Concretely, we assign a Pseudo-spurious label in the zero-shot classification setting:
à arg m (or (x), dr(t)))
(9)
where à is the pseudo-spurious label for the sample. In equation 9 the pseudo-labeling procedure requires of all possible spurious text prompts. We utilize these pscudo-labeled to implement the corresponding translation operation as introduced the previous section. We summarize our method in Algorithm[]
We conduct experiments under two scenarios: the first, where the labeled spurious feature is available, we apply the true spurious label to implement TIE. In the second scenario, where the labeled spurious feature is unavailable, we execute the complete algorithm as outlined in Algorithm denoted as TIE Additionally, we investigate a method applicable when partially labeled data is available. The detailed discussion of this method is deferred to Section
3.4 THEORETICAL COMPARISON BETWEEN TIE AND ROBOSHOT
TIE and ROBOSHOT are methods designed to address spurious correlations by leveraging both image and text modalities. We provide a detailed comparison of the worst group accuracy between two methods under different spurious text prompts and label prompts. To quantify the effects of spurious text prompts and target label text prompts, as discussed in 3.I these prompts form two classifiers: W. for spurious prompts and for label prompts. We define w. = [1, o., 0) and [1, 0), 8 E IR+ A smaller a indicates more accurate spurious decision boundary, while a larger indicates more accurate task boundary. Utilizing these definitions, have the analytical forms for the worst group accuracy (WG) for both ROBOSHOT and TIE:
ROBOSHOT: WGns(a, 3) minf erfe(- a²
3)a + + ) V2(1 + a2) + t08)
(10)
1
a² (B 1)a
erf(
2
2
V2(1 + a2]++a)((1+ (1)) a.))
(1 3)V 1 a² aB
TIE: = =min( erfe(
2
v 2(1 (2) 02)
(11)
(1 33v1 a² + a3 -1 + + 1
2 erf(
/2 (2)+++aaa++))
We defer the derivation of equations T0 and in Appendix C We present a plot of the theoretical worst group accuracy with respect to a and @ in Figure We observe that ROBOSHOT only achieves 0, representing the perfect spurious classifier. Otherwise, ROBOSHOT's
higher WG when performance drops rapidly when the spurious classifier is inaccurately approximated, showing a sig- nificant margin compared TIE. other words, the performance of TIE shows better robustness across different text prompts. We further substantiate this analysis with empirical validation on real-world dataset, as detailed in Appendix C.1
EXPERIMENT
4.1 SETUP

---

Published as a conference paper at ICLR 2025
Datasets. We study five well-established benchmark datasets for spurious correlation re-
search: Waterbirds (Koh et al., 2021; Sagawa
eet al. 2019), CelebA (Liu et al.
2015), ), ISIC
(Codella 2019), COVID-19 (Chhen
2020). FMOW (Christie et al., 2018). Please
refer appendix E
for detailed information.
18
12
10
Backbones. Existing research indicates that different visual backbones produce varied re-
Figure 3: Theoretical comparison of worst group accuracy between TIE and ROBOSHOT.
sults. Following established protocols (Adila
al. 2024), for the Waterbirds and ISIC datasets, we examine CLIP models with vision backbone of ViT-B/32, ViT-L/14, and RN50 (Ilharco et al. 2021 Cherti et al. 2023 Radford al.
2021 For the ISIC and COVID-19 datasets, utilize Biomed CLIP (Zhang et al. 2023a as the vision backbone. For the FMoW dataset, we employ the ViT-L/14 model due to the dataset's complex nature.
We compare our method against two baselines and existing state-of-the of--oh--tttetttt methods in robust zero-shot classification. Concretely, two baselines are vanilla zero-shot classification (ZS), Zero-shot with group information (Group prompt). Existing SOTA methods including Ideal Prompt (Trager al. 2023). Orth-Cali (Chuang et al. 2023), Perception CLIP (An al., 2024). RO- BOSHOT (Adila et al. 2024). We leave the details of baselines in Appendix
Baselines.
Text Prompts for Reproducibility. Zero-shot classification employs two types of prompts: label prompts and spurious prompts. To ensure a fair comparison, all methods utilize the same label prompts. For example, the label prompts for the Waterbirds dataset are [a photo landbird, photo of wate raterba erbirr]. For spurious prompts, we use the prompts provided by the authors if the method is tested on specific dataset. Otherwise, we generate spurious prompts using generative AI tools like ChatGPT (OpenAT] 2023), following the guidelines specified in the original papers. For reproducibility prompts used our experiments are provided in Appendix G
Metrics. Following the protocol established by robust learning studies (Sagawa et al.) 2019 Adila jet al. (024)), we report three metrics: worst group accuracy (WG), average accuracy (Avg), and the gap between these two metrics (Gap). We highlight the best result in bold and underline the second-ber best result.
4.2 MAIN RESULTS.
Waterbirds. Table D summarizes results on the Waterbirds dataset. TIE achieves significant im- provement over comparative methods by a relatively large margin, especially for the ViT-L14 vision backbone, where the worst group accuracy reaches 78.82%, surpassing the previous method by 14.65%. TIE* achieves a comparable performance in the ViT backbones. However, performance varies with different backbone models. For ResNet-50, Orth-Cali outperforms other methods.
Table 1: Zero Shot classification results on Waterbirds
Method
CLIP (ViT-B32)
CLIP (ViT-L14)
CLIP (ResNet-50)
WG 1
Avg 1 68.48
Gap 4
WG
Avg 1
Gap |
WG t
Avg t
Gap
ZS
41.37
27.11
31.93
83.72
51.79
35.36
80.64
45.28
Group Prompt
43.46
66.79
23.33
10.44
56.12
45.68
49.84
70.96
21.12
Ideal words
60.28
79.20
18.92
64.17
87.67
23.50
39.09
79.48
40.39
Orth-Cali
54.99
69.19
14.20
58.56
86.31
27.75
64.80
84.47
19.67
Perception CLIP
59.78
82.50
22.72
54.12
86.74
32.62
48.21
91.51
43.30
ROBOSHOT
54.41
71.92
17.51
45.17
64.43
19.26
26.61
69.06
42.45
TIE (Ours)
71.35
79.82
8.47
78.82
84.12
5.30
52.96
83.62
30.66
TIE* (Ours*)
61.24
76.91
15.67
61.60
78.98
17.38
34.11
81.19
47.08
Coode
httppaa://gihhbb.ccmm/

---

Published as a conference paper at ICLR 2025
CelebA. Table presents results for the CelebA dataset. Similar to the Waterbirds dataset, TIE sistently outperiorms comparison baselines and achieves the smallest gap in ViT backbone models. The performance of TIE+ is comparable to that of TIE. For the ResNet backbone, Perception CLIP outperforms other methods.
Table 2: Zero Shot classification results on CelebA
Method
CLIP (ViT-B32)
CLIP (ViT-L14)
CLIP (ResNet-50)
WG f
Avg
Gap 4
WG f
Avg
Gap
WG f
Avg
Gap |
ZS
78.89
84.27
5.38
73.35
81.20
7.85
69.69
81.58
11.89
Group Prompt
74.90
80.38
5.48
68.94
77.86
8.92
70.59
79.48
8.89
Ideal words
78.12
80.96
2.84
76.67
89.15
12.48
65.65
76.27
10.62
Orth-Cali
77.92
82.31
4.39
77.69
81.39
3.70
69.13
76.47
7.34
Perception CLIP
76.46
80.32
3.86
78.70
81.41
2.71
80.22
85.17
4.95
ROBOSHOT
80.52
84.77
4.25
82.61
85.54
2.93
73.96
80.90
6.94
TIE (Ours) 82.63
85.11
2.48
84.60
86.17
1.57
75.32
81.71
6.39
TIE* (Ours*)
82.61
85.10
2.49
81.98
84.27
2.29
75.30
81.70
6.40
ISC and COVID-19. Our experiments extend to specialty datasets within high-stakes settings, specifically deploying VLM models in the medical domain. Table shows the results for the ISIC and COVID-19 datasets where our method outperforms baseline methods in worst-group accuracy and achieves comparable average accuracy.
Table 3: Zero Shot classification results on ISIC and Covid-19 19 datasets
Method
ISIC (Biomed CLIP)
COVID-19 (Biomed CLIP)
WG f
Avg t
Gap |
WG f
Avg t
Gap |
ZS
42.21
70.21
28.00
44.83
61.81
16.98
Group Prompt
12.13
30.05
17.92
27.58
48.27
20.69
Ideal words
41.42
53.07
11.65
23.53
56.84
33.31
Orth-Cali
21.43
72.54
51.11
44.83
51.72
6.89
Perception CLIP
41.55
52.74
11.19
48.84
56.87
8.03
ROBOSHOT
53.30
59.84
6.54
32.75
53.10
20.35
TIE (Ours)
65.87
69.90
4.03
52.17
62.50
10.33
TIE* x (Ours*)
61.11
71.68
10.57
50.22
61.08
10.86
FMOW. We extend our experiments to multiclasses and multigroup settings. The FMOW dataset includes 62 classes and is organized into 5 spurious groups. Table shows the results for FMOW. TIE achieves the highest accuracy in the worst-perforning group, TIE* shows comparable per- formance on the worst group accuracy and has the highest overall accuracy. These results further validate the effectiveness of our methods in mitigating spurious correlations in the zero-shot setting.
Table 4: Top-1 Accuracy and Worst Group accuracy on FMOW dataset.
WG f
Avg t
Gap |
ZS
18.06
26.02
7.96
Group Prompt
8.75
14.69
5.94
Ideal words
11.14
20.21
9.07
Orth-Cali
19.45
26.11
6.66
Perception CLIP
12.61
17.70
5.09
ROBOSHOT
10.88
19.79
8.91
TIE
20.19
26.62
6.43
TIE*
19.84
26.65
6.81
Discussion. From Table 4 a comparable result but still has a performance gap, which will be discussed the following sec- tion. We found TIE shows relative suboptimal performance using ResNet-50 the Waterbirds
TIE consistently achieves the best second-best WG, TIE* achieves
8

---

Published as a conference paper at ICLR 2025
dataset. Note that all text encoders are transformer-basee based models, while the vision backbones vary. We hypothesize that this suboptimality primarily arises from a misalignment between the direction of the spurious vector in the text space and the image space. This misalignment stems from the structure and scales of the encoders, which echoes the finding that different CLIP structures show significantly different zero-shot classification results (Radford et al., 2021]. Methods like Orth-Cale or Perception CLIP, which only focus on debiasing text embeddings, introduce randomness into zero-shot classification. This randomness can occasionally enhance performance. However, adjust- ing text embeddings without considering image embeddings can result in misalignment, leading to significant drop performance. For example, Orth- Cali shows suboptimal performance on the ISIC dataset. Conversely, our method mitigates this randomness by integrating both image and text modalities, thereby enhancing the stability of zero-shot classification outcomes.
4.3 GROUP ROBUST TEXT PROMPT
In this section, we demonstrate that our method is compatible with other methods focused on mitigating spurious correlations in the text modality. [n et al.] (2024) highlight that providing additional context enhances the performance of VLM models. Inspired by this insight, we employed group-robust prompts to identify spurious directions. Specifically, we utilize GPT-4 (OpenAl, 2023) to generate five sentences that serve as synonyms for spurious features. The prompt for the GPT4 is Please generate 5 synenyms of [Spurious feature]. For instance, the robustified spurious prompts for the Waterbirds dataset include: for a land background, [A photo with land background. A photo of forest background. A photo of a mountain background. A photo of a Terrain background. photo of Ground background]; and for water background, [A photo with water background. A photo of an ocean background. sea background. background. We computed the average text embedding from these spurious prompts and used to update the image embedding. The results are shown in Table We observe that the robustified prompt helps find a more robust direction for the spurious features, leading to improved WG and Avg metries with ViT-B32 and ResNet-50 models.
A photo of a photo of Lake background. A photo of River
Table Group robustify prompting
Method
ViT-B-32
ViT-L-114
ResNet-50
WG f 61.24
Avg f
Gap +
WG + 61.60
Avg t 78.98
Gap 4 17.38 17.00
WG f
Avg t 81.19
Gap 4 47.08
TIE+
76.91
15.67
34.11
TIE* Robust
64.96
78.63
13.67
61.46
78.46
38.63
82.22
43.59
4.4 LIMITED ACCESS TO LABELS OF THE SPURIOUS FEATURES
Treveass a performance disparity between TIE and TIE+ +, suggesting that accurate estimation of the spurious label enhances performance. Wang & Wang 2024) theoretically demonstrates that feature separability directly influences performance, especially when spurious features are more separable than core features. Based on this, accurately predicting labels of the spurious features necessitates significantly fewer training samples. Therefore, we propose using a partially spurious feature labeled dataset infer the spurious labels of the entire dataset, and subsequently apply our algorithm based on the pseudo labels of the spurious feature. We tested this approach on the Waterbirds dataset with training sample sizes ranging from 100 to 1000. To optimize efficiency, we employed a smaller-scale architecture, ResNet-18 (He et al.) 2016). to predict the pseudo-spurious feature labels. The model was trained using an SGD optimizer with a learning rate of 10-1, a weight decay of 10-3 and momentum of 0.9, over 200 epochs. The VLM model is tested using ViTB-32.
Figure Aeportss the outcomes utilizing different sample sizes within the training set. Observations indicate that increasing the amount of labeled data enhances the worst group accuracy of the CLIP model. Specifically, using 1000 samples, performance nearly matches that of our method when attribute a is known. Additionally. the figure demonstrates nearly linear improvement in worst group accuracy as the accuracy of predictions on spurious feature labels increases the CLIP model.

---

Published as a conference paper at ICLR 2025
0.80
Zero shot
0.75
Zero-shot group
0.66
0.70
Zero shot Avg
0.65
Without
0.62
200
400
600
800
1000
0.82
0.84
0.86
0.88
The number of samples in the training set
Accuracy of predicting spurious label
Figure 4: Performance on the Waterbirds dataset using partially labeled spurious features.
4.5 VISUALIZATION
In addition to the superior performance of our method, we further investigate its capacity to ensure that predictions are correct for the right reasons. This can be verified through visual explanation maps, as illustrated in Figure 5 We employed the explainability method from (Chefer et al. 2021) generate heatmaps for both image features and text prompts. Our method significantly reduces reliance on spurious features in zero-shot setting. In the ISIC dataset, it specifically minimizes attention irrelevant color patches. For samples of malignant lesions, our approach enhances focus on the lesion itself rather than the other skin part. For the Waterbirds dataset, even in the vanilla zero-shot where the focus might incorrectly shift the background, our method effectively redirects attention towards the core features of the subject. Interestingly, after implementing our method, the text prompts also show increased attention to specific objects, such as bird rdd and ma gnant.
Original
ES
CLIP
Zan
Bell
Chis
shoto Vittr
Aphoto of
phoct Birt
Figure Attention based explanations (Chefer et 2021 for ISIC and Waterbirds datasets.
5 CONCLUSION
Addressing spurious correlations presents critical challenge in the realm of zero-shot VLMs. This study draws inspiration from rigorous theoretical analysis to examine optimal strategies for translat- ing image embeddings. To address the spurious correlations effectively, we have designed the TIE algorithm, which guides the translation of image embeddings based on the text prompt. Extensive experiments conducted on real-world datasets demonstrate that our method only significantly improves the worst-group accuracy across all datasets but also achieves comparable overall accu- racy. Additionally, we visualize results from both modalities to confirm that the predictions are based on valid reasons.
Failure case discussion and Future direction. Although our proposed method demonstrates sig- nificant robustness, TIE. may encounter failures when pseudo-spuriout labels are incorrectly as- signed. We present a comprehensive analysis of these failure cases and propose solutions in Ap- pendix K Additionally. TIE faces limitations when processing images with artifacts. We discuss these issues in detail in Appendix ! Identifying such artifacts could be a promising direction for future research enhance zero-shot classification performance.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was partially supported by the EMBRIO Institute, contract #2120200, a National Sci- ence Foundation (NSF) Biology Integration Institute, and NSF IIS #1955890, IIS #2146091, IIS #2345235.
REFERENCES
Dyah Adila, Changho Shin, Linrong Cai, and Frederic Sala. Zero-shot robustification of zero-shot models. In The Twelfth International Conference on Learning Representations, 2024.
Bang An, Sicheng Zhu, Michael-Andrei Panaitescu-Liess, Chaithanya Kumar Mummadi, and Furong Huang. More context, less distraction: Zero-shot visual classification by inferring and conditioning on contextual attributes. In The Tweifth International Conference on Learning Rep- resentations, 2024.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez- Pazz Invariant risk minimization. arXiv preprint 1XXX::1907002893 2019.
Hila Chefer, Shir Gur. and Lior Wolf. Generic attention-modi explainability for interpreting bi- modal and encoder- transformers. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV), pp. 397-406, October 2021.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive iogguag--mmage learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-2829, 2023.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6172- 6180, 2018.
Ching- Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. Debias- ing vision-language models via biased prompts. arXiv preprint arXiv:2302.00070 2023.
Noel Codella, Veronica Rotemberg, Philipp Tschandl, Emre Celebi, Stephen Dusza, David Gut- man, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.0334 2019.
Joseph Paul Cohen, Paul Morrison, Lan Dao, Karsten Roth, Tim Duong, Marzyeh Ghassem, et al. Covid-19 image data collection: Prospective predictions the future. Machine Learning for Biomedical Imaging, 1(December 2020 issue):1-38, 2020.
Sepehr Dehdashtian, Lan Wang, and Vishmu Naresh Boddeti. Fairerclip: Debiasing clip's zero-shot predictions using functions in rkhss. arXiv preprint arXiv:2403. 15593, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
Sabit Ekin. Prompt engineering for chatgpt: quick guide to techniques, tips. and best practices. Authorea Preprints, 2023.
Christiane Fellbaum. Wordnet: An electronic lexical database. MIT Press google schola, 2:678-688 -666 1998.
Daniel Fischer. change, 2013. www.:itpss/maahhttaaccccnnn..ooo
Gaussian integrals over half-space.
Mathematics Stack Ex-
URL attps: ///ath. st stackeccange.e.. cttpt/eeccc//////0////0//
(version: 2013-11-25).

---

Published as a conference paper at ICLR 2025
Yunhao Ge, Jie Ren, Andrew Gallagher, Yuxiao Wang. Ming Hsuan Yang, Hartwig Adam, Laurent Itti, Balaji Lakshminarayanan, and Jiaping Zhao. Improving zero-shot generalization and robust- ness of multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Viston and Puttern Recognition, pp. 11093-11101. 2023.
Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19338-19347, 2023.
Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, and Bin Cui. Calip: Zero-shot enhancement of clip with parameter-free attention. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 746-754, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive corst-group-accuree In Conference on Causal Learning and Reasoning, pp. 336-351- PMLR, 2022.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL hatps::/ zenodo. 5 5143773 If you use this software, please cite it as below.
5281/
Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew G Wilson. On feature learning in the presence of spurious correlations. Advances in Neural Information Processing Systems, 35: 38516-38532, 2022.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference machine learning, pp. 4904-4916. PMLR, 2021.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. In The Eleventh International Conference on Learning Representations, 2022.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu. Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: benchmark of wn-the-wild distribution shifts. In International Conference on Machine Learning (ICML), 2021.
David Krueger, Ethan Caballero, Heeri- Jacobsen, Amy Zhang. Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribation generalization via risk extrapo- lation (rex). In International Conference Machine Learning, pp. 5815-588 5826.. PMLR, 2021.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Shenyu Lu, Junyi Chai, and Xiaoqian Wang. Neural collapse inspired debiased representation leamn ing for min max fairesss. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2048 2059, August 2024.
12

---

Published as a conference paper at ICLR 2025
Meike Nauta, Ricky Walsh, Adam Dubowski, and Christin Seifert. Uncovering and correcting shortcut learning in machine learning models for skin cancer diagnosis. Diagnostics, 12(1):40, 2021.
OpenAI. Chatgpt.
2023. Accessed: 2024-05-12 12.
Yijiang Pang, Hoang Bao, and Jiayu Zhou. Cross-modality debiasing: using language to mitigate sub-population shifts in imaging. arXiv preprint arXiv:2403.078888 2024.
Suzanne Petryk. Lisa Dunlap, Keyan Nasseri, Joseph Gonzalez, Trevor Darrell, and Anna Rohrbach. On guiding visual attention with language specification. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 18092-18102, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-876 PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Puttern Recognition (CVPR), pp. 10684-10695, June 2022.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. arXiv preprint arXiv: 1.08731, 2019.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterizatioo exacerbates spurious correlations. In International Conference on Machine PMLR, 2020.
Learning, pp.
Matthew Trager, Pramuditha Perera, Luca Zancato, Alessandro Achille, Parminder Bhatia, and Ste- fano Soatto. Linear spaces of meanings: compositional structures in vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15395-15404, 2023.
Yipei Wang and Xiaoqian Wang. On the effect of key factors in spurious correlation: A theoretical perspective. International Conference on Artificial Intelligence and Statistics, pp. 3745-3753. PMLR, 2024.
Zhengbo Wang, Jian Liang, Ran He, Nan Xu, Zilei Wang, and Tieniu Tan. Improving zero-shot generalization for clip with synthesized prompts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3032-3042, 2023.
WorldSEnder. Validated. hww.litpp//stttssucccccca...oo/p
Linear transformation of gaussian random variable.
Cross
URL
sttps://stata. eestatsssstcceeeee.... www.kstchange..e..eoee cww.e.coo/coooo www/////////// 387 (version: 2020-08-04),
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi. Ali Farhadi, Hongseok Namkoong, al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7959-7971, 2022.
Shirley Wu, Mert Yuksckgonul, Linjun Zhang, and James Zou. Discover and cure: mitigation of spurious correlation. In International Conference Machine Learning, pp. 37765- 37786. PMLR, 2023.
Concept-aware
Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817. 2024.
Siyuan Yan, Zhen Yu, Xuelin Zhang, Dwarikanath Mahapatra, Shekhar S. Chandra, Monika Janda, Peter Soyer, and Zongyuan Ge. Towards trustable skin cancer diagnosis via rewriting model's 's de- cision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11568-18 11577, June 2023.
13

---

Published as a conference paper at ICLR 2025
Yu Yang, Besmira Nushi, Hamid Palangi, and Baharan Mirzasoleiman. Mitigating spurious relations in modti--oddal models during fine-tumin mne-tuning. In International Conference on Machine Learning. pp.
PMLR, 2023.
Huaxiu Yao. Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Im- proving out-of-distribution robustness via selective augmentation. In International Conference on Machine Learning. pp. 25407-25437. PMLR, 2022.
Michael Zhang and Christopher Ré. Contrastive adapters for foundation model group robustness. Advances in Neural Information Processing Systems, 35:21682-21697 2022.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Pre- ston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915, 2023a.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, al. Siren's song in the ocean: a survey on hallucination in large language models. arXiv preprint 2rXiv:230900111, 2023b.
Guangtao Zheng, Wenqian Yc, and Aidong Zhang. Learning robust classifiers with self-guided spurious correlation mitigation. arXiv preprint arXiv:2405.03649 2024.
14

---

Published as a conference paper at ICLR 2025
A PROOF OF LEMMA
Lemma Under the above data model, the group-wise accuracy can be derived as
w Mayl
2 eerfe(-
if ==
/ Eon.
A(uau. wiy)
(12)
wi Magia
erfi
) + if = -1
/2w Dgv.a
where Pgo.o and Egg,. represent the mean and covariance matrix of the image embedding h gg...
Denote the linear classifier as w E R To simplify the notation, we drop the subscript of 9y.a. The hyperplane is defined as two spaces::
Ω = [hl w h > 0}.
(13)
Ω_ [h|w h 0}.
The probability density function can be written as:
(h ue Σ) (2x)d/2/dett) exp( (h 11) =- 1t).
(14)
We first consider y For computing the group accuracy, we integrate fH μe, Σ) over the region of Q4. In the following proof, we omit the input of A(-) for simplicity:
(
A
Sa [H( ne E)dh.
(15)
Transform h to reduce the mean term, define h' = h ur 92 = (h'lw Th' + w > 0} Fischer 2013):
exp( 1 h
(16)
(2T) det
2
Σ is positive definite matrix, we have E =
Q
Q2 where Q is an orthogonal matrix, and E' is
diagonal matrix. We solve E- Q -TE LQ
Q
A=
exp( x
(17)
(2m)
EetE 0
Denote h" = -Qhh, 02- = (h" w
TQQTh"
+
w
> 0}, then Equation 17 becomes
1
1
A =
exp(
h" Σ'-11 'hn detQ dh",
(2m) detE
n
2
(18)
1
exp(
h1" ET'--11 "'T''''"'dhh" hh"dhh.
(2m)d//
nz
2
Eliminate the covariance term by defining h" V---hh h, S23 0}.
[h" w VV''"" - w
Then Equation 18 becomes:
d
A = (2m)) d/2 /dets.
exp(
1 d"" -
ng
2
(19)
1
exp( 1 h T h dh"
(2T)
The space Ω2 (h" w/ wTTh"
h + w 1 0}, where w' = V..Q..
15

---

Published as a conference paper at ICLR 2025
Define an orthogonal matrix U s.t. Uw' |
Define h"" Uh", Q4
{h""
l|w'lle "w'leeT" h" w n > 0}. ||w'll = V (WVE'Qw) (APTQ@]][VVVV ('QQ) vw Ew. We have
00
1 exp( - 1op(---1"'dh h2'dh
A A( Bo w: =
V2x
2
Vi 2i
(20)
w
erfe(
) if y 1.
2
22v w
Σw
Similarly, for y -1, consider integration over the region of Ω
h Dy.a?
w:
Vw Ew
expl
h2)dh
h
2T
2
-00
(21)
w
erf(
+
if
1.
2
V2VW w
Σw
2
Thus prove the statement D.
B PROOF OF THEOREM
Theorem 1 Given the objective function and the data model, the maximizer of the objective is obtained by
Va E Phal
(22)
1
0
0 0
0
where P Rddd an elementary matrix, P
We rewrite the objective function to ensure the completeness of the proof. We first solve for the stationary point, then verify the stationary point is local maximum point.
Cec h Bo.a. w)
max Σ
Agu.e (hgg.a + Vas W:
(23)
To maximize the objective function, the stationary point can be computed by Vv, Lace 0:
Vv. LAce = Σ A(bgg, + va) va)=0. =0
(24)
Pv.cEl
With Lemma 1, we have
w T ((ug, + +Va),
w T 1.0 + vaa ) 1 0. (25)
V LAce Vv.
2 Lerfe(-
+ 2 -erf(-
V2w T Ew
V2w TWw
Decompose Equation
25
based a, first compute V1:
V.i 2 erfe(- w TH + v1),
w T (( + V2) )
2 erf(
V2w T Ew
V2w T Eww
(26)
(exp( w (Mgn, + v1) (2) exp(p((
wT (Mg-1, + v1) ) 0.
w
V T Ew
V2w Ew
/2w' Ew
As - W Σw + 0 equation 26 can be rewritten
(M + vi) = exp(-( W (Mg i. + v1) )2).
exp(
(27)
22w Ew
V2w 1 Ew
16

---

Published as a conference paper at ICLR 2025
Taking the natural log of both sides implies:
(Mm.1 + v1))3 = --( w (Ho-11 + +v1), )
(28)
V2w Ew
V2w T Ew
or equivalently
w (Mor, + v1) = =w T (#g_L, + v1).
(29)
can be solved as:
vÎ
Σ Mgu.1
(30)
vE{-1.13
Then, compute V-I+
1
w ('(M + v
w (M +V-1)
+ 2
V
2 Zerfel-
erf(
V2w Eww
V2w Ew
(31)
(exp(-1 wT. (Hgr..-) + +V-1l )2) exp(-((
ww TT(He (Mo-..-- + +-=1) |3]]
w
0,
V2Tw Ew
V2w Σw
V2w Tww
and similarly,
v Σ Mg
(32)
vE(1,-1} 1}
Substitute the data assumption in Equation 30 and 32 we have
a- O, OJ"
(33)
We rewrite Equation 33 into a matrix product form:
v² = PPEh = Ph],
(34)
... 0
0
where P
We next verify local maximality. We compute the second derivative of LAcc W.r.t. V.- Because the objective depend on va via the scalar projection, we define
w
(35)
Rewrite the objective function of Sa
8a + w
T Mgna
1
8a + w Mg-1.a
((sa) = erfel
+ erf(
(36)
V2w Ew
V2w Ew
To simplify notation, let
(1())) 8a + w Mgn.
1=1 Sa + w T 1g-11
(37)
22w Ew
/2w Ew
The second derivative w.r.t. as
2
L (" (2 Ew) At the stationary point, we have
(21(5a) exp((22(ss)) - 2 1==((s))expp (sa)exp((=
(38)
21(5)) =-2-168%)
(39)
17

---

Published as a conference paper at ICLR 2025
Substitute back in equation 38 obtain
-2z1 (2)) exp((=2 (5 Vi(w Σw)
L" (s
(40)
The sign of equation 40 is determined by 21 (82). We plug Va in equation 37 we have
1 w (Mgi.a Mg-1.a) 21 (+
(41)
V2w Tww
where w is the classifier for the positive sample, therefore 21 (8%) > 0. Thus (*( < 0. This indicates va is local maximum.
Next, we compare the value of the objective [s**) with its value at the boundaries of the domain.
(i As +20, then 21 (8a) + +oo and 2 (8a) + 400. Using the limits erfc(-() + 2 and erf( -t) 1) + - as 4 +00. We have ((8a) = 1.
-o,, then (5) - erf( crf(-t) + 1 as - -c. We have [(sa) =
(ii) As
and z z-1 (8) - -00 Using the limits erfe(-t) 0 and
(iii) At S. s. ((sa) 1+ (x((z))))))) where z (5)) > follows that
((s))
(42)
Thus the value of the objective function at s is strictly greater than its value as 8a to0. The unique stationary point sy is the global maximizer, which concludes the statement D.
C DERIVATION OF EQUATIONS 10 AND
Modeling ROBOSHOT. ROBOSHOT is a method that linearly projects the image embedding onto the hyperplane associated with spurious features. Denote the spurious hyperplane as follows:
w 0.
(43)
The projected point can be written as:
W x ||wll Wa-
Xproj
(44)
Based the spurious modeling h follows Gaussian mixture model. According to the relationship defined in Equation each component in the Gaussian mixture model Xproj ~ N (pprroj Epros) WorldSEnder) where
W W
Mproj 11
Wa
(45)
Eprof BΣB T
where I
[w. fe E(x]. With Lemma
w. w.
the analytical expression for ROBOSHOT is:
w.T H [wal
w (1x
Wa)
2 erfc(
), if =
V2w BEB w
(RoBOSHOT w, waig)
(46)
w T (μ
Iw. Wa)
w
crf((
) + 2.
=f==-1
V2w BEB
where B I
W. w [[w.]]
18

---

Published as a conference paper at ICLR 2025
Modeling TIE. TIE is method that translates the image embedding along the negative direction and equation
of the spurious vectors. With Lemma 1
the analytical expression for TIE is
erfe( w w. H.W.),
if =
V2w Ew
Arre(h, w, Waiy)
(47)
w T (m Wa (wwa)
erf(
+ 2' if -
V2w Ew
Next, plug the spurious feature classifier w. ox, 0] and the label classifier w [1,3,0], and
spurious data model in equation 46 and equation 47 have
1
a² -1+
crfel
if
2
V2 a2]
(a,
(48)
1
a² -B-
erf(
1+ if 11,
(1+ 31V13 0² ag
2 erfe(
) if =
22 (2) a²)
ATIE B:
(49)
(1-B)V1 a² + +
2 erf(
if 11
22(1+ (2) a²)
2
The worst group accuracy takes the min value equation 48 and equation 49
C.1 EXPERIMENT VALIDATION
we further experimentally investigate the impact of various spurious classifiers on the worst group accuracy TIE and ROBOSHOT. We generate synonymous spurious text prompts using GPT (OpenAl 2023) for land features and 6 for water features, shown in Table 6 We test individual spurious text prompts, yielding 36 combinations (6 from water features, from land features). The results are presented in Figure Furthermore, we examine all possible combinations two text prompts within the same spurious feature to expand the search range of spurious prompts, resulting in 225 combinations, These results are shown in Figure
Building on the theoretical analysis in Section 3.3
Table 6: Spurious Prompt used in experiments comparing ROBOSHOT and TIE.
Spurious Template
Land Attributes
Water Attributes
"A photo with a/an {a} background"
[land, field, hill, {water, ocean, river, desert, forest, moun- lake, sea, pond) tain]
19

---

Published as a conference paper at ICLR 2025
072
1071
6.3
06
623
024
02
Lieel rostor car prosges
022
Teanenn
Figure 6: Experimental comparison between ROBOSHOT and TIE across different spurious text prompts, using single spurious text prompt for each test.
0.28
6.4
0.5
0.1
2210
Lind bathum pramzes
1790
10
Resture
Figure Experimental comparison between ROBOSHOT and TIE on different spurious text prompts, using multiple spurious text prompts for each test.
From Figure 6 and we observe significant performance gap between TIE and ROBOSHOT. This suggests that TIE more robust, and less dependent on the accuracy spurious text prompts compared to ROBOSHOT.
D ALGORITHM FOR TIE*
Algorithm TIEE
Input: Input x, Image encoder (2).), Text encoder dr(+), Spurious text prompts Ta Target text prompts T.
Output: Predicted label fj
1: for t. E TT do
2:
Va dt,(t)) Va
> Computing the spurious vector > Normalize
3:
v. T
4: end for
5: arg rigmax,es EA (x(x)),
Psuedo labeling on spurious feature Image embedding A> Estimate the scale coefficient > Translate image embedding & Zero shot classfication
6: ha δ)(x;ā)
7: A E[(h va)l
8: h + ha daVa
9: I = arg max vey < ha, 0r(t))>
10: return
20

---

Published as a conference paper at ICLR 2025
D.1 ESTIMATION OF THE SCALE COEFFICIENT X
Based on Equations 30 and 32 the theoretical optimal scale coefficient can be computed as
Aa
Σ MEDI
E( vayyl,
(50)
y
where [ denotes the number of classes. However, directly computing equation 50 is intractable since the component Va.y requires label information, which violates the zero-shot setting. To address this issue, we analyze the distributions the two groups: hg Va.yla 1, y = and h Vo.g a 1, 1. We perform a two-sample f-test under the null hypothesis Ho that the means of these two distributions are equal. Experimental results the Waterbirds dataset yield p 0.78 for ViT/B and p 0.40 for ViT/L, both of which are greater than the significance level of 0.05. Hence, we do have sufficient evidence to reject Ho. In other words, the mean values of the two distributions are equal.
Figure illustrates the density distributions of the scale coefficients. This statistical test indicates that the mean scale coefficient is consistent across groups. Therefore, we can estimate the scale coefficient as
A = [hg va)l.
(51)
We further validate our estimated scale coefficient on the Waterbirds dataset using both the ViT/B and ViT/L backbones. We iteratively adjust the scale coefficient and plot the worst group accuracy for each scale value. Figure ? shows that the estimated scale is nearly identical to the theoretical optimal scale. Moreover, the worst group accuracy obtained using the estimated scale demonstrates robust performance.
40
hyvayly=
17.5
hv=yly==
30
h'Vayly=1
15.0
h'v,,ly=1
12.5
Density
Density
20
7.5
10
5.0
2.5
0.0
0.16
0.18
0.20
0.22
0.24
0.26
0.10
0.20
Figure 8: The distribution of h Vay across different target groups. Left: Results from testing with VTT/B. Right: Results from testing with ViT/L.
E DATASET
We evaluate our method and all comparison methods on the following datasets:
Waterbirds (Koh et al. 2021 Sagawa et al. 2019): The primary task of the Waterbirds dataset is classify bird types, specifically,
[Landbird. Waterbird). The spurious confounder in this dataset is the background, = {Land background, Water background }. includes four groups: [Landbird with Land background, Landbird with a Water background, Waterbird with Land background, Waterbird with Water background).
CelebA (Liu et al. 2015): The CelebA dataset comprises over 200K celebrity faces. Fol- lowing the protocol by (Sagawa et al. 2019), the task is to identify hair color with target labels y (dark hair, blonde hair). The spurious correlation label is gender, (female, malc}. This dataset is segmented into four groups: {a female with dark hair, female with blonde hair, male with dark hair, male with blonde hair).
21

---

Published as a conference paper at ICLR 2025
75
70
70
Worst Group Acc
65
worst Group
60
55
Worst Acc
Worst Acc
Estimated
40
Estimated
Theoretical optimal
Theoretical optimal
45
30
0.1
0.2
0.3
0.4
0.5
0.6
0.0
0.1
0.2
0.3
A1 ViT-Base
ViT-Large
Figure 9: Relationship between worst-group accuracy and the scale coefficient. The estimated scale coefficient closely aligns with the theoretical optimal value and approaches the maximum achievable worst-group accuracy.
ISIC (Codella et al. 2019): The ISIC dataset is utilized for skin cancer diagnosis. Follow- ing the task from (Wu et al. 2023), the task is predict the type of skin cancer, denoted as [Benign, Malignant] The spurious correlation feature this dataset is a = {with color patch, without color patch}. encompasses three groups: {Benign cancer with color patch, Benign cancer without color patch, Malignant cancer without a color patch).
COVID-19 (Cohen et al. 2020): The COVID-19 dataset is used to diagnose from X-ray images, with the classification task defined as y {no Pneumonia, pneumonia]. The spurious confounder in this dataset is gender, groups: {a male with pneumonia, a male without pneumonia, a female with pneumonia, female without pneumonia).
{male, female). It consists of four
FMOW (Christie al. 2018): The Functional Map of the World (FMOW) is a large- scale satellite image dataset comprising 62 classes. We follow the protocol outlined in (Wul et al.] 2023 Izmailov al. 2022) to define groups based on geographical regions: Africa, the Americas, Occania, Asia, and Europe.
F BASELINES
We compare TIE against several state-of-the-art methods for zero-shot classification.
Group Prompt a method that includes spurious correlation labels text prompts. For example, in the waterbirds dataset, the text prompts for Group Prompt specify the background along with the bird type, [a photo of landbird rd with land landbird with a water background, a photo of waterbird with a land background, a photo of a waterbird with a water background].
Group Prompt:
background, photo
Ideal words (Trager et al., 2023): The ideal prompt is start by adding prompts related to target labels before integrating those associated with spurious correlation attributes. Sub- sequently, the ideal method averages across the spurious correlation prompts.
Orth-Call (Chuang al. 2023): The Orth-Cali method is designed to debias text prompts by making the text embeddings invariant to spurious features. This approach introduces projection matrix that projects the text into the null space defined by the span of spurious text prompts. It then employs regularization to ensure that these projected prompts are closely mapped within the text embedding space.
Perception CLIP (An et al. 2024): Perception CLIP is a method inspired by empirical findings that suggest that including contextual attributes in text prompts enhances zero-shot classification performance and mitigates the effects spurious correlations. To improve the group robustness, Perception CL.P incorporates information about spurious features.
22

---

Published as a conference paper at ICLR 2025
ROBOSHOT (Adila et al. 2024): Roboshot is method that utilizes LLMs to identify spurious insights. then removes these spurious features from the image embeddings using the process, which projects the image embeddings onto space orthogonal to that of the spurious insights. Subsequently, Roboshot enhances the image embeddings by projecting them along vectors representing helpful insights.
G IMPLEMENTATION
We conducted all experiments on an Nvidia RTX 3090 GPU with 24 GB of memory, using frozen CLIP models across various datasets. Specifically, for the Waterbirds and CelebA datasets, the vi- sion encoder backbones included ViT-B-32 (Dosovitskiy et al. 2020). ViT-L-14 (Dosovitskiy al,
2020). and ResNet 50 (He et al. 2016). Model construction and pre-trained weights are sourced
from Open CLIP (Ilharco et al.] 2021p.
For specialized datasets, including ISIC and COVID-19, we employed the Biomed CLIP backbone (Zang et al. 2023a), acknowledging that the training set from general CLIP significantly diverges from the biomedical context, leading to substantial shifts in test performance. With ViT-L-32, we observed 0 % worst-group accuracy. Hence, excluded results using the general backbone for these specialized datasets.
As training
conducted for all methods, the results deterministic.
To facilitate the reproduction of our results, we have detailed both the label prompts and spurious prompts in Table 7 Note that the nature of CLIP sensitive to prompts; our spurious prompts are created through simple adaptations of the label prompts. We incorporate our label prompts and spurious prompts all comparison methods except for vanilla shot ensure a fair comparison.
Table 7: Prompts details
Dataset
Label prompts
Spurious prompts
Waterbirds
(a photo of a landbird, a photo of waterbird]
[a photo with a water background, photo with land background] [female, male]
CelebA
la photo of celebrity with dark hair, a photo of celebrity with blonde hair]
ISIC
[This is a benign lesion, This is a malignant lesion]
[There exists no color patch, There exists color patch]
COVID-19
[An X-ray image of chest without Pneumonia, x-ray image of chest with Pneumonia]
[An X-ray image from female, X-ray image from male]
H ABLATION STUDY
H.1 DIFFERENT SPURIOUS TEXT PROMPT TEMPLATES
Beyond the textual description of spurious features, the format of spurious text prompt templates also impacts the performance. To further validate the effectiveness of all methods, we conducted experiments using various text templates, including '(spurious feature label)' and 'A photo with spurious feature, {spurious feature label], in the waterbirds dataset. The results are presented in T
23

---

Published as a conference paper at ICLR 2025
Table 8: FMOW Prompt details
Class- Template
Spurious Template
Class
Group
Over {g}
(airport, airport hangar, airport termi- nal, amusement park, aquaculture, ar- chacological site, barn, border check- point, burial site, car dealership, construc- tion site, crop field, dam, debris rub- ble, educational institution, electric sub- station, factory powerplant, fire station, flooded road, fountain, gas station, golf course, ground transportation station, heli- pad, hospital, impoverished settlement, in- terchange, lake or pond, lighthouse, mili- tary facility, multi-unit residential, nuclear- powerplant, office building, oil or gas fa- cility, park, parking lot or garage, place of worship, police station, port, prison, race track, railway bridge, recreational facility, road bridge, runway, shipyard, shopping mall, single-unit residential, smokestack, solar farm, space facility, stadium, storage tank, surface mine, swimming pool, toll booth, tower, tunnel opening, waste dis- posal, water treatment facility, wind farm, zoo}
(Europe, Asia,
satellite image of a/an (y).
Americas,
Africa, Oceania]
Table 9: Zero-shot classification results on the Waterbirds dataset with different spurious prompt templates. T1: {Spurious feature label), T2: photo with spurious feature, {Spurious feature label]. (CLIP ViT-B/32)
T1 Spurious Template
T2 Spurious Template
WG f
Avg
Gap
WG f
Avg t
Gap
ZS
41.37
68.48
27.11
41.37
68.48
27.11
Group Prompt
43.46
66.79
23.33
43.46
66.79
23.33
Ideal words
61.99
78.87
16.88
60.44
79.82
19.38
Orth-Cali
64.08
73.74
9.66
67.14
76.58
9.44
Perception CLIP
23.37
61.54
38.17
46.20
73.37
27.17
ROBOSHOT
44.35
69.03
24.68
45.99
69.67
23.68
TIE
71.04
80.11
9.07
69.63
82.02
12.39
TIE*
56.14
75.00
18.86
67.60
79.84
12.24
H.2 MORE BACKBONE RESULTS.
Our paper focuses CLIP it serves foundational model widely applied across various do- mains, like in stable diffusion (Rombach et al. 2022). Beyond the CLIP family models, we have expanded experiments to incorporate various backbone models. We utilize ALIGN (Jia et al. 2021 backbones on the Waterbirds dataset, with results shown Table 10
we observe that TIE demonstrates robust performance across various spurious prompt templates and different backbones, indicating significant potential for real-world applica- tions.
From Table
24

---

Published as a conference paper at ICLR 2025
Table 10: Zero Shot classification results on the Waterbirds dataset with the ALIGN backbone
WG f
Avg f
Gap 4
ZS
47.50
69.83
22.33
Group Prompt
5.81
72.55
66.74
Ideal words
51.71
67.17
15.46
Orth-Cali
28.35
58.73
30.38
Perception CLIP
31.60
54.39
22.79
ROBOSHOT
41.02
50.95
9.93
TIE
56.07
69.54
13.47
TIE*
52.49
64.27
11.78
I DISCUSSION ON TEXT PROMPTS
The effectiveness of VLMs depends on the quality of text prompts. The guidelines for selecting text prompts represent critical area for deeper exploration. To address this, show our insights through experiments designed to identify an effective and generalizable approach for creating opti- mal prompts in practice.
We investigate this issue by decomposing text prompt into a template and an object.
T1: "A photo with [Object]"
T2: "A photo with a spurious feature, [Object]"
T3: "[Object]"
For the object, Ge et al. (2023) shows that labels exhibit a hierarchical structure in "WordNet" Fell- baum (1998). For example, the hierarchical progression of the word 'strawberry' includes 'berry "edible fruit', 'food', each level becoming more general Ge et al. (2023). In our experiments, test three labeling strategies: using the level directly above to represent more generalized category, the spurious feature itself, and average of the top five most specific terms the bottom of the hierarchy for greater specificity. We provide details of the object candidates in Table [T] The aim determine the most effective level of generality or specificity for descriptions. We conducted experiments on the Waterbirds dataset using T TIE (ViT-L14), The results are shown in Table 12
Table 11: Object candidates
Water background prompts
Land background prompts
O1 (hypernyms)
Fluid
Ground Land
02 (self)
Water
03 (hyponyms)
Sea, Lake, River, Stream, Creek
Arable Land, Farmland, Forest
Land, Grassland, Desert
Insights: We note that using proper object description is important. We suggest using a specific description of the spurious feature or their hyponyms, as this can improve the worst group accu- racy (WG) in the zero-shot classification. In contrast, using overly general descriptions such as hypernyms significantly degrades performance. This observation aligns with recommendations for specificity and clarity in text prompt engincering for language models Ekin 2023).
In terms templates, we found that giving a portion of contextual information, such as the prefix "a photo with" or "a photo with a spurious feature," helps the WG. Templates lacking a prefix demonstrate poor performance, a finding that aligns with the observations presented in Radford et al. (2021) For practical purposes in ViT-based CLIP models, encourage users to adopt templates that include a prefix, with the object description utilizing the spurious feature itself, balancing case of use and performance.
25

---

Published as a conference paper at ICLR 2025
Performance evaluation of CLIP-ViTL14 for TIE*, We have highlighted in bold the results that surpass the WG in T
Table 12:
Text prompts
WG f
Avg 1
Gap |
T1+01
53.97
76.49
22.52
T1+02
61.60
78.98
17.38
TI+03
65.26
80.20
14.94
T2+01
46.48
72.69
26.21
T2+02
63.77
80.35
16.58
T3+03
63.19
79.06
15.87
T3+01
45.90
73.19
27.29
T3+02
60.62
78.84
18.22
T3+03
59.56
77.91
18.35
J FUTURE DIRECTION DISCUSSION
We introduce TIE to mitigate the effect of spurious correlations, which are vital in prediction tasks. While our approach demonstrates strong performance, it faces challenges redirecting attention the object in the presence of pronounced artifacts (c.g., watermarks) without appropriate text prompts. Figure [Ojuuurtttllsttttess rare case where the dominant feature is watermark. To evaluate our method's capability in redirecting attention, we provide the following text prompts:
Text prompt (TP1): photo with water background,
Text prompt (TP2): A photo with watermark
zero-shot
Ours:
Ours:TP2
Figure 10: Attention-based explanations in image with strong artificial landmark in the Wa- terbirds dataset. TP1: photo with water background, TP2: A photo with watermark.
From Figure TO we observe that when using TP1, a text prompt representing common spurious feature in the dataset, the attention fails to redirect back the correct core feature (the bird in the image). Interestingly, when providing a corresponding text prompt (TP2), the attention successfully shifts from the watermark to the bird. This highlights the potential of our proposed method to address misclassifications caused by factors beyond spurious correlations, offering a promising direction for further research.
K FAILURE CASE ANALYSIS FOR TIE*
TTE* is a method free from using any annotations and requires the spurious text prompt for infer- ence of the spurious label in the dataset. We analyzed TIE* failure cases, which can be broadly categorized into two scenarios: (1) inaccuracies in the pseudo-spurious labels and (2) images con- taining artifacts (e.g., watermarks).
For (1): The majority of failures in TIE* occur when zero-shot classification incorrectly assigns a spurious label. This misassignment causes samples to be translated in the opposite direction, leading to incorrect classifications. sification and the accuracy of pseudo-spurious labels. Our analysis reveals that the pseudo-spurious
Section 4.4 we examine the worst-group accuracy in zero-shot clas-
26

---

Published as a conference paper at ICLR 2025
labels assigned by TIE* have a direct impact the worst-group accuracy in zero-shot classification: higher accuracy in assigning these labels corresponds to improved worst-group accuracy.
To potentially improve TIE+'s 's performance, propose three practical strategies: utilizing group- robustified spurious text prompts (Section (.3). employing a small subset spurious-labeles data (Section 4.4), and following the guidelines for effective text prompts (Section to achieve better performance.
For (2): discussed this scenario in Section This is case where the artifact (e.g., watermark) becomes the dominant feature. While using TIE or TIE reduce dependency on spurious features (such background information), it cannot eliminate the effect of the artifact. This limitation can lead to the failure of our algorithm. Interestingly, we found that TIE has the potential to remove unwanted features when provided with appropriate text prompts. However, the identification of these incorrect features remains an open area for further investigation.
L BROADER IMPACTS
Our work aims to mitigate spurious correlations in VLM models, a crucial endeavor for the machine learning community. Beyond enhancing group robustness, the positive impacts of our work extend to domains such as fairness, trustworthiness, and generalization. This is particularly significant when deploying machine learning algorithms in high-stakes domains.
27


Reviews:
[{"review_id": "QQlkp2w2g3", "reviewer": "Reviewer_y3Qc", "summary": "In this paper, the authors proposed a method to address spurious correlations in Vision Language Models while preserving their zero-shot capabilities. Drawing on theoretical analysis to explore optimal strategies for translating image embeddings, they introduce TIE—a simple algorithm that guides the translation of image embeddings based on the text prompt. The core idea is  to apply a translation transformation in the latent space using \"spurious vectors\" derived from text prompts to better align text and image embeddings. Experimental results on real-world datasets demonstrate that this approach not only improves the worst-group accuracy across all datasets but also achieves comparable overall accuracy, enhancing the robustness of VLMs across modalities.", "questions": "- Have you conducted an evaluation of the quality of the predicted pseudo-spurious labels on a validation set in order to analyze and bridge the accuracy gap between TIE and TIE*?\n\n- In large-scale datasets and tasks, how valid is the assumption of a single spurious feature for the entire dataset? How can spurious features be efficiently identified on a per-data-point basis?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The important task of mitigating spurious correlations in VLMs\n- The interesting idea of applying a translation transformation in the latent space for cross-modal alignment\n- Rigorous theoretical analysis to examine optimal strategies for translating image embeddings\n- Quantitative and qualitative results validating the effectiveness of the proposed method", "weaknesses": "- More in-depth discussion of the method is necessary (Why does it work? When does it fail? etc.)\n- English can be improved", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QfsEIAPpWs", "reviewer": "Reviewer_RFvW", "summary": "- Paper proposes TIE, a method to mitigate spurious correlations in VLMs.\n- TIE uses a translation operation in the latent space, guided by spurious text prompts, to shift image embeddings away from spurious features.\n- The approach preserves the embedding distribution, unlike projection-based methods.\n- TIE*, a variant, uses the VLM itself to infer spurious labels when explicit labels are unavailable.\n- Experiments on several benchmarks show improvements over other methods.", "questions": "1. How sensitive is TIE to the choice of spurious prompts? Are there any guidelines for selecting effective prompts?\n2. What are the failure cases of TIE* and how can automatic inference of spurious labels be improved?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- TIE/TIE*'s translation-based approach is a novel contribution to the field as far as I know.\n- The method has strong theoretical analysis\n- It effectively addresses spurious correlations without requiring training or labeled data.\n- The experimental results are good.", "weaknesses": "- TIE relies on accurate spurious prompts. The authors propose TIE* but its performance is not as good as TIE. Further investigation into the limitations of TIE* and potential improvements would help.\n- The experiments focus on CLIP-based models. Exploring TIE/TIE* on other VLMs would be helpful.\n- nitpick: the capitalization of the term \"TIE\" implies its an acronym. If it is an acronym, it's unclear what it stands for.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VfqT4p3Thd", "reviewer": "Reviewer_wW81", "summary": "The paper proposes a way to robustify VLMs like CLIP against spurious correlation. The proposed method (TIE) does not require any extra data and fine-tuning; and operate across modalities (as opposed to previous method that only operates in single modalities). In order to preserve the image embedding distribution, the proposed method is based on a translation operation in the embedding space, and find the optimal translation vector based on theoretical analysis. In practice, this translating vector is found using spurious text prompts.\n\nTheoretically, the authors show how this method compare to another zero-shot rebustification method, Roboshot; and show the major drawback of Roboshot that is tackled by TIE. Empirically, the method outperforms almost all existing baselines.", "questions": "1. What is $h_a$ in Theorem 1? is it image embeddings w/ spurious features $a$ regardless of the label?\n2. What is the difference between $t_{spu}$ and $t_a$? (Equation 9)\n3. For TIE*, does it mean we need access to all possible spurious features descriptions? Not necessarily a drawback as mostly in datasets we know this beforehand", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Very nice theoretical analysis that backs the proposed methods. First the authors derives what is the optimal translation vector based on the definition of zero-shot classification and group robustness problems.\n- Very nice comparison with RoboShot, which strongly shows why the proposed method is necessary to improve upon RoboShot.\n- Strong empirical results with strong baselines.\n- Nice visualization results in 4.5, strongly backs the claim why CLIP robustification is necessary in the first place.", "weaknesses": "1. Some notations need to be clarified (check Questions).\n2. I think the TIE* procedure needs to be clarified more, do we need to know possible values of $t_a$ beforehand? this need to be made clearer in the early in the writing when TIE* is introduced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1woZ7MlKAU", "reviewer": "Reviewer_gGse", "summary": "To address the spurious correlations existing in VLMs, the authors propose a new solution called TIE  that tackles spurious correlations within the zero-shot setting without requiring target label annotations, which would contradict the zero-shot principle. TIE utilizes a translation operation in the latent space that preserves the distribution of image embeddings, guided by text prompts to address spurious correlations. This approach is theoretically grounded, identifying optimal translation directions along the spurious vector.", "questions": "1. When RESNET-50 is used as the backbone, the proposed method performs poorly. Is there a way to solve the misalignment between text and image?\n\n2. How do we ensure the accuracy of the textual hints provided when using artificially identified fake features? Is there an evaluation mechanism to verify the effectiveness of these hints?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The experiments in this paper are comprehensive and have been verified on datasets from many different fields (four different image domains), proving that the method is universal and has good generalization properties.\n\n2. The structure of the paper is clear, and the methods proposed are easy to follow.", "weaknesses": "1. The performance improvements of the proposed method are not particularly obvious.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)