Mitigating Spurious Correlations in Zero-Shot Multimodal Models

Authors: Shenyu Lu, Junyi Chai, Xiaoqian Wang
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Spurious correlation, Zero shot, Multimodal models

Abstract:
Multimodal models or Vision Language Models (VLMs) have reshaped the paradigm in machine learning, offering zero-shot capabilities that require no additional training when adapted to new classification tasks. However, despite their advancements, spurious correlations still exist in VLMs. Existing approaches to tackle this issue often require target label annotations, contradicting the principle of zero-shot classification, or they primarily focus on a single modality, risking misalignment between text and image modalities. Others rely on extensive domain knowledge or large language models (LLMs) to characterize spurious features, making the performance sensitive to the generated prompts and undermining zero-shot capability. In response, we propose a new solution that tackles spurious correlations in VLMs within the zero-shot setting. Our approach utilizes a translation operation that preserves the latent space distribution to address issues of spurious correlations. In particular, our method is grounded in and inspired by a theoretical analysis, which identifies that the optimal translation directions are along the spurious vector. As VLMs unify two modalities, we compute spurious vectors from the text prompts and guide the translation for image embeddings, aligning the requirements for the fusion of different modalities in VLMs. We conducted experiments on benchmark datasets, which have shown significant improvements in worst-group accuracy. Additionally, our visualizations of VLMs further demonstrate the effectiveness of this intervention.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730287922279

thread_type:
official_review

<thread_object>
creator: reviewer_ggse

<content>
summary:
To address the spurious correlations existing in VLMs, the authors propose a new solution called TIE  that tackles spurious correlations within the zero-shot setting without requiring target label annotations, which would contradict the zero-shot principle. TIE utilizes a translation operation in the latent space that preserves the distribution of image embeddings, guided by text prompts to address spurious correlations. This approach is theoretically grounded, identifying optimal translation directions along the spurious vector.

strengths:
1. The experiments in this paper are comprehensive and have been verified on datasets from many different fields (four different image domains), proving that the method is universal and has good generalization properties.

2. The structure of the paper is clear, and the methods proposed are easy to follow.

weaknesses:
1. The performance improvements of the proposed method are not particularly obvious.

questions:
1. When RESNET-50 is used as the backbone, the proposed method performs poorly. Is there a way to solve the misalignment between text and image?

2. How do we ensure the accuracy of the textual hints provided when using artificially identified fake features? Is there an evaluation mechanism to verify the effectiveness of these hints?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730302845625

thread_type:
official_review

<thread_object>
creator: reviewer_ww81

<content>
summary:
The paper proposes a way to robustify VLMs like CLIP against spurious correlation. The proposed method (TIE) does not require any extra data and fine-tuning; and operate across modalities (as opposed to previous method that only operates in single modalities). In order to preserve the image embedding distribution, the proposed method is based on a translation operation in the embedding space, and find the optimal translation vector based on theoretical analysis. In practice, this translating vector is found using spurious text prompts.

Theoretically, the authors show how this method compare to another zero-shot rebustification method, Roboshot; and show the major drawback of Roboshot that is tackled by TIE. Empirically, the method outperforms almost all existing baselines.

strengths:
- Very nice theoretical analysis that backs the proposed methods. First the authors derives what is the optimal translation vector based on the definition of zero-shot classification and group robustness problems.
- Very nice comparison with RoboShot, which strongly shows why the proposed method is necessary to improve upon RoboShot.
- Strong empirical results with strong baselines.
- Nice visualization results in 4.5, strongly backs the claim why CLIP robustification is necessary in the first place.

weaknesses:
1. Some notations need to be clarified (check Questions).
2. I think the TIE* procedure needs to be clarified more, do we need to know possible values of $t_a$ beforehand? this need to be made clearer in the early in the writing when TIE* is introduced.

questions:
1. What is $h_a$ in Theorem 1? is it image embeddings w/ spurious features $a$ regardless of the label?
2. What is the difference between $t_{spu}$ and $t_a$? (Equation 9)
3. For TIE*, does it mean we need access to all possible spurious features descriptions? Not necessarily a drawback as mostly in datasets we know this beforehand
</content>
</thread_object>

</thread>
<thread>
creation date:
1730503684361

thread_type:
official_review

<thread_object>
creator: reviewer_rfvw

<content>
summary:
- Paper proposes TIE, a method to mitigate spurious correlations in VLMs.
- TIE uses a translation operation in the latent space, guided by spurious text prompts, to shift image embeddings away from spurious features.
- The approach preserves the embedding distribution, unlike projection-based methods.
- TIE*, a variant, uses the VLM itself to infer spurious labels when explicit labels are unavailable.
- Experiments on several benchmarks show improvements over other methods.

strengths:
- TIE/TIE*'s translation-based approach is a novel contribution to the field as far as I know.
- The method has strong theoretical analysis
- It effectively addresses spurious correlations without requiring training or labeled data.
- The experimental results are good.

weaknesses:
- TIE relies on accurate spurious prompts. The authors propose TIE* but its performance is not as good as TIE. Further investigation into the limitations of TIE* and potential improvements would help.
- The experiments focus on CLIP-based models. Exploring TIE/TIE* on other VLMs would be helpful.
- nitpick: the capitalization of the term "TIE" implies its an acronym. If it is an acronym, it's unclear what it stands for.

questions:
1. How sensitive is TIE to the choice of spurious prompts? Are there any guidelines for selecting effective prompts?
2. What are the failure cases of TIE* and how can automatic inference of spurious labels be improved?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730816088711

thread_type:
official_review

<thread_object>
creator: reviewer_y3qc

<content>
summary:
In this paper, the authors proposed a method to address spurious correlations in Vision Language Models while preserving their zero-shot capabilities. Drawing on theoretical analysis to explore optimal strategies for translating image embeddings, they introduce TIE—a simple algorithm that guides the translation of image embeddings based on the text prompt. The core idea is  to apply a translation transformation in the latent space using "spurious vectors" derived from text prompts to better align text and image embeddings. Experimental results on real-world datasets demonstrate that this approach not only improves the worst-group accuracy across all datasets but also achieves comparable overall accuracy, enhancing the robustness of VLMs across modalities.

strengths:
- The important task of mitigating spurious correlations in VLMs
- The interesting idea of applying a translation transformation in the latent space for cross-modal alignment
- Rigorous theoretical analysis to examine optimal strategies for translating image embeddings
- Quantitative and qualitative results validating the effectiveness of the proposed method

weaknesses:
- More in-depth discussion of the method is necessary (Why does it work? When does it fail? etc.)
- English can be improved

questions:
- Have you conducted an evaluation of the quality of the predicted pseudo-spurious labels on a validation set in order to analyze and bridge the accuracy gap between TIE and TIE*?

- In large-scale datasets and tasks, how valid is the assumption of a single spurious feature for the entire dataset? How can spurious features be efficiently identified on a per-data-point basis?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732152412132

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response part 1

comment:
We sincerely thank all the reviewers for their inspiring and constructive feedback! We deeply appreciate the time and effort spent in providing valuable advice, suggestions, and questions to help refine our submission. We would like to take this opportunity to address the common questions raised.

Reviews **RFvW** and **gGse** share a similar question regarding the guidelines for selecting spurious text prompts: 
> "How sensitive is TIE to the choice of spurious prompts? Are there any guidelines for selecting effective prompts? "
> "How do we ensure the accuracy of the textual hints provided when using artificially identified fake features?"


**A1**: One of the major advantages of our proposed method is its robustness. Our method is less sensitive to differences in spurious text prompts compared to existing state-of-the-art method. This robustness is theoretically analyzed in Section 3.4. In addition, we tested 261 different spurious prompts, with the results detailed in Appendix C.1 , see Figures 6 and 7. We also test on different spurious text prompts templates, the results are shown in Table 9. Our observations confirm that our approach is markedly more robust, consistently achieving high worst group accuracy.

Nevertheless, the effectiveness of VLMs still depends on the quality of text prompts. The guidelines for selecting text prompts represent a critical area for deeper exploration. To address this, we show our insights through experiments designed to identify an effective and generalizable approach for creating optimal text prompts in practice.

We investigate this issue by decomposing a text prompt into a *template* and an *object*. Templates include phrases like 'A photo with [Object]', 'A photo with a spurious feature, [Object]', and simply '[Object]'. For the object, [1] shows that labels exhibit a hierarchical structure in WordNet [2].  For example, the hierarchical progression of the word 'strawberry' includes 'berry', 'edible fruit', 'food', each level becoming more general [2]. In our experiments, we test three labeling strategies: using the level directly above to represent a more generalized category, the spurious feature itself, and an average of the top five most specific terms at the bottom of the hierarchy for greater specificity. The aim is to determine the most effective level of generality or specificity for descriptions. Our experiments on the Waterbirds dataset using TIE* (ViT-L14) explore these concepts, with detailed paired text prompts provided below. We selected ViT-L14 to potentially achieve the best performance.

Table 1. Template candidates
||Template|
|-|-|
|T1| A photo with [Object]  | 
|T2 |A photo with a spurious feature, [Object] |
|T3| [Object] | 


Table 2. Object candidates
||water background |land background | 
|-|-|-|
|O1 (hypernyms)| Fluid  | Ground  | 
|O2 (self)| Water |  Land  |
|O3 (hyponyms)| Sea, Lake, River , Stream, Creek | Arable Land,  Farmland,  Forest Land, Grassland , Desert | 

Table 3. Performance evaluation of CLIP-ViTL14 for TIE*, We have highlighted in bold the results that surpass the TIE* in our submission.
||WGA |ACC | Gap|
|-|-|-|-|
|T1+O1| 53.97 | 76.49  | 22.52 |
|T1+O2| 61.60 | 78.98   | 17.38 |
|T1+O3| **65.26** | 80.20 | 14.94|
|T2+O1| 46.48 |  72.69 | 26.21|
|T2+O2| **63.77** | 80.35   |16.58 |
|T2+O3| **63.19** | 79.06 | 15.87 |
|T3+O1| 45.90 | 73.19  | 27.29 |
|T3+O2| 60.62 | 78.84   | 18.22|
|T3+O3| 59.56 |  77.91 | 18.35 |

**Insights:** We note that using a proper *object* description is important. We suggest using a specific description of the spurious feature or their hyponyms, as this can improve WGA in the zero-shot classification. In contrast, using overly general descriptions such as hypernyms significantly degrades performance. This observation aligns with recommendations for specificity and clarity in text prompt engineering for language models [3]. 

In terms of templates, we found that giving a portion of contextual information, such as the prefix "a photo with" or "a photo with a spurious feature," helps the WGA. Templates lacking a prefix demonstrate poor performance, a finding that aligns with the observations presented in [4]. For practical purposes in ViT-based CLIP models, we encourage users to adopt templates that include a prefix, with the object description utilizing the spurious feature itself, balancing ease of use and performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732152650250

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response part 2

comment:
Reviewer **y3Qc**, **RFvW** share a similar question regarding the failure case discussion.
> When does it fail?
> What are the failure cases of TIE* and how can automatic inference of spurious labels be improved?

**A2** TIE* is a method free from using any annotations and requires the spurious text prompt for inference of the spurious label in the dataset. We analyzed TIE* failure cases, which can be broadly categorized into two scenarios: (1) inaccuracies in the pseudo-spurious labels and (2) images containing artifacts (e.g., watermarks).



For (1): The majority of failures of TIE* arise when zero-shot classification incorrectly assigns the spurious label. This misassignment leads samples to be translated in the opposite direction, resulting in incorrect classifications. The pseudo-spurious labels assigned by TIE* have a direct effect on the worst group accuracy in zero-shot classification. We substantiate this finding in Section 4.4, as well as in our response Q1 to Reviewer Y3Qc, to which we kindly refer our reviewers for further details. In summary, accurately assigning pseudo-spurious labels enhances TIE*'s performance on the worst group accuracy in zero-shot classification. 

To potentially improve TIE*'s performance, we propose three practical strategies: utilizing group-robustified spurious text prompts (as discussed in Section 4.3), employing a small subset of spurious-labeled data (as discussed in Section 4.4), and following the guidelines for effective text prompts (as detailed in A1) to achieve optimal performance. 

For (2),  we discussed this scenario in Section I of the appendix. This is a case where the artifact (e.g., a watermark) becomes the dominant feature instead of the spurious feature. While using TIE or TIE* may reduce dependency on spurious features (such as background information), it cannot eliminate the effect of the artifact. This limitation can lead to the failure of our algorithm. Interestingly, we found that the TIE can potentially remove such unwanted features if provides proper text prompts. However, the identification of these incorrect features remains an area under investigation.

[1]Ge, Yunhao, et al. "Improving zero-shot generalization and robustness of multi-modal models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

[2]Fellbaum, Christiane. "WordNet: An electronic lexical database." MIT Press google schola 2 (1998): 678-686.

[3] Ekin, Sabit. "Prompt engineering for ChatGPT: a quick guide to techniques, tips, and best practices." Authorea Preprints (2023).

[4] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732552613647

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised version released

comment:
We sincerely thank the reviewers for their constructive suggestions, which have significantly helped improve our work. We have revised the manuscript accordingly, with the major revisions highlighted in blue. A summary of the key revisions is as follows:

(1) Based on Reviewer **wW81**'s feedback, we have revised the notations throughout the manuscript to ensure consistency.

(2) As recommended by Reviewers **RFvW** and **gGse**, we have added a section in Appendix I to provide guidelines for selecting text prompts.

(3) Following the suggestions from Reviewers **y3Qc** and **RFvW**, we have included a section in Appendix K to thoughtfully discuss the failure cases of our methods.

(4) Following Reviewer **wW81**'s suggestion, we explicitly state the requirements for implementing TIE*.


We hope these revisions address the suggestions and questions raised by the reviewers. If there are any further inquiries or clarifications needed,   please do not hesitate to let us know, and we will be happy to provide additional details. Thank you once again for your valuable feedback and effort!
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors propose a new approach to addressing spurious correlations in VLMs in a zero-shot setting, which offers a theoretical grounding and an efficient solution using a translation operation. The method demonstrates robust performance across four image domains, and the authors provide theoretical analysis and visualizations, supporting its generalizability. The paper is well-structured and clearly presented, with relevant insights into the potential improvements for VLMs.

During the rebuttal period, the authors actively engaged in discussions with the reviewers, and the final scores were (8, 6, 6, 6). All reviewers rated the paper positively.  Overall, this paper makes a valuable contribution to addressing spurious correlations in VLMs, and I recommend it for acceptance at ICLR. The authors are encouraged to polish the writing, revise the notations, and discuss more about the guidelines for selecting text prompts and failure cases in the final version.