Cross-modal Mitigation of Spurious Correlation for Prompt-tuning in VLMs with Causally Motivated Logic Alignment

Authors: Xueyang Tang, Song Guo, Xiaosong Ma, Haoxi Li, Jie ZHANG, Yue Yu
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Vision-Language Models, Prompt Tuning, Spurious Correlations, Out-of-Distribution Generalization, Causality, Probability of Necessity and Sufficiency

Abstract:
Recent studies have shown that pre-trained vision-language models can effectively adapt to diverse downstream tasks through parameter-efficient prompt tuning. Unfortunately, the tuned models can exploit spurious correlations during prediction, resulting in a failure to generalize to out-of-distribution test data, especially when the tuning dataset exhibits bias. How to achieve cross-modal mitigation of spurious correlations during prompt tuning of vision-language models remains an open question. In this paper, the challenging problem is tackled by leveraging the stable relationship between necessary and sufficient causal features and the corresponding label. On the one hand, we constrain the learning process of prompt by reinforcing the necessary and sufficient connection between the textual labels and textual features. On the other hand, the probability of necessity and sufficiency between the textual features and the filtered visual features is measured and maximized to enhance cross-modal feature alignment. By iteratively optimizing these two objectives, we can achieve cross-modal mitigation of spurious correlations because the logic equivalence between textual labels and visual features is bolstered. The theoretical analysis on generalization error indicates that our method can achieve a tighter generalization error bound than existing approaches. We evaluate the proposed method on several commonly adopted out-of-distribution datasets, and the empirical results demonstrate the superiority of our method over the state-of-the-art competitors.

Decision: reject
Reviewer scores: 5, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1730265963674

thread_type:
official_review

<thread_object>
creator: reviewer_rh7y

<content>
summary:
This article introduces the concept of logical alignment to address the cross-modal mitigation problem of spurious correlation for prompt adjustment in visual language models. To achieve this, the authors maximize the probability of necessity and sufficiency corresponding to cross-modal and textual logical alignment. Theoretical analysis proves that the proposed method has a tighter generalization error bound compared to existing approaches. Performance is analyzed across a few different test data distributions, and components of the method are ablated.

strengths:
The generalization error bond of the presented method is provided with theory analysis in Appendix A. 

The novelty of the method is how to integrate the probability of necessity and sufficiency in multi-modal learning.

weaknesses:
The paper is not easy to follow. This is due to the symbols being confused, e.g., $\Phi$ denotes the filter in Cross-modal logic alignment but visual representation space in Textual logic alignment. 
While the first half of the paper explains the idea and motivation well, creating a rightful sense of expectation of the result, the section on the results somewhat comes short of delivering the findings with a bang.  After reading the first half I was excited to read the next pages to find "Where are those indeed integrated areas for boosting the expected performance ", and tingling with an expectation of learning something new. But then, for some reason, the Overall Performance and Ablation Study sections are very timid and just present dry numbers for each of the tests that were planned.  

1. It would be helpful to include a better explanation of what the "spurious correlation in vision-language models" is exactly. Maybe a picture. 
2.The paper borrowed too much content from the existing papers and it could be removed by referring these papers. Even so, the motivation is not clear since the authors employ PNS without explanation.

3. Page 6, Section 4.1: The NSC feature shown Figure 1 is not explained by the authors themselves how to make use and take advantage of this. 

4. Compared to CoOPood,  it seems that the proposed method exploits PNS terms instead of mutual information to align cross-modal representations. I am wondering how effective the PNS term is in cross-modal mitigation of spurious correlation.  
5. Why was it necessary to do textual logic alignment?

questions:
I would like to hear the authors' discussion regarding the three weakness that I highlikeed above:

1. Usefulness of the PNS term. 

2. Usefulness of the textual logic alignment. 

3. How to extractive NSC features in textual and visual modality.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730275925427

thread_type:
official_review

<thread_object>
creator: reviewer_zmpk

<content>
summary:
This paper introduces the novel LogicAI-PT framework to mitigate learning of spurious correlations in prompt tuning of CLIPs. It models the PNS (probability of necessity and sufficiency) by introducing intervention $\bar{Q}$ and $\bar{\Phi}$. The author provide extensive introduction of the methodology and background and the experimental results are significant.

strengths:
The proposed method is quite simple and effective regarding the significant improvement of multiple benchmarks. The idea of tackling spurious correlation problem from a sufficiency-necessity view is intuitive and is implemented based on thorough proof.

weaknesses:
1. The proposed method seems to be universally applicable to many tasks rather than only prompting of VLM as classifier. Causal Representation Learning baselines adapting from other tasks can largely consolidate the motivation of this paper.
2. Ablation study of the proposed is not enough. What is the effect of different $\alpha, \beta$? #419 introduces the author chooses different value combination for different benchmarks. The robustness regarding different hyper-parameters can largely affect the applicability of this method.

questions:
1. What kind of knowledge does the $\bar{Q}$ and $\bar{\Phi}$ learnt durning the training process? I am curious about more quantitative and qualitative results about these additionally introduced intervention modules. 
2. Could this method transfer to other VLMs such as EVA-CLIP and single tower VLMs such as BEiT?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714055852

thread_type:
official_review

<thread_object>
creator: reviewer_zvzb

<content>
summary:
This paper presents Logical-pt, a framework for mitigating spurious correlations in vision-language models. It uses causally motivated logic alignment to align visual and textual features during prompt tuning. The method is backed by a tighter generalization error bound and empirically validated on several datasets, outperforming existing methods in out-of-distribution generalization.

strengths:
* the results reported in this paper demonstrates good performance on multiple benchmarks
* this paper did extensive evaluations and experiments to validate the method's effectiveness

weaknesses:
* The proposed method shows superior performance compared with other benchmarks. What is the computational efficiency compare to simpler methods?

questions:
same as above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721787991

thread_type:
official_review

<thread_object>
creator: reviewer_fk2q

<content>
summary:
The paper presents a novel framework, LogicAl-PT, that addresses the challenge of cross-modal mitigation of spurious correlations in prompt tuning of vision-language models. The authors introduce a new concept, logic alignment, which integrates the mitigation of spurious correlations with cross-modal alignment of representations, and demonstrates its effectiveness through theoretical analysis and empirical results on various out-of-distribution datasets. LogicAI-PT earns competitive performance compared with traditional prompt-tuning methods for CLIP model.

strengths:
Extensive theoretical analysis is provided to verify the proposed concept, PNS and PNS risk modeling.

weaknesses:
- In contrast to the detailed theoretical analyses, the empirical verifications are fairly absent in this paper. Take the most recent competitor Coopood as an example, this paper presents much fewer empirical analyses, i.e. only 2 tables in the experiment section for verification. More ablation studies about the hyper-parameter chosen, visual results about the improvement on spurious correlations should be provided.

-Some more recent prompt tuning methods[a][b] should be discussed and compared with.

- Experiments on more architectures like ViT except for ResNet-50 should be done as well.

[a] Self-regulating Prompts: Foundational Model Adaptation without Forgetting, https://arxiv.org/abs/2307.06948)
[b] DePT: Decoupled Prompt Tuning, https://arxiv.org/abs/2309.07439

questions:
Please see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732797268386

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Responses

comment:
We sincerely thank all the reviewers for their efforts and valuable feedback, which have greatly contributed to improving the quality of our work. In particular, we appreciate the reviewers' recognition of our:

**(1) novel and intuitive methodology design** (by reviewer **Rh7Y**, **zmpK**); 

**(2) extensive and thorough theoretical analysis** (by Reviewer **fk2Q**, **zmpK**, **Rh7Y**); 

**(3) significant performance improvement** on multiple benchmarks (by Reviewer **ZvzB**, **zmpK**).

To address the reviewers' concerns and questions, we have provided detailed explanations for unclear content and conducted extensive experiments to further validate the superiority of our method. Specifically, **we summarize the updated content included in the revised version of the paper as follows:**

>**Clarifications:**

**(1)** Updated illustrations and descriptions of the overall framework in **Figure 1, on page 5**.

**(2)** Added a detailed illustration to analyze the necessity and superiority of logic alignment in VLMs in **Appendix A, on page 14**.

>**Evaluations:**

**(1) Visualization experiments and analysis** that demonstrate the proposed LogicAl-PT effectively mitigates cross-modal spurious correlations and enhances logical alignment between visual representation and text label, in **Section 5.3 (on page 8-9)**.

**(2)** Added two state-of-the-art prompt tuning methods as **additional baselines** (i.e., PromptSRC and DePT), in **Section 5.2 (on page 8)**.

**(3)** Experiments on **another backbone model** (i.e., ViT-B/32), in **Section 5.2 (on page 8)**.

**(4)** **Visual explanation** to illustrate the necessity of the proposed textual logic alignment, in **Section 5.4 (on page 9-10)**.

**(5)** **Ablation study** on the sensitivity of hyper-parameters, in **Section 5.4 (on page 10)**.

**(6)** Experimental comparison with **causal representation learning baselines** adapted from single-modal scenarios, **on line 1126-1151, page 21-22, in Appendix D**.

**(7)** Evaluation of computational overhead to verify the **computational efficiency** of our method, **on line 1107-1124, page 21, in Appendix D**.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733174639051

thread_type:
official_comment

<thread_object>
creator: area_chair_tvng

<content>
title:
Dec 2 - last day for reviewers' questions

comment:
Dear Reviewer fk2Q and Rh7Y,

This is a kind reminder that December 2 is the last day for reviewers to ask questions to authors. As the paper received diverse ratings and your initial ratings are negative, could you check the authors' responses by today and see whether the responses addressed your concerns? Your constructive and timely communications are strong contributions to the reviewing process.

Thank you,

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper aims to tackle the issue of cross-modal spurious correlation for parameter-efficient prompt tuning of VLMs. This work pointed out that cross-modal mitigation of spurious correlations during prompt tuning of vision-language models remains an open question, and further proposed the logic of logic alignment and a practical framework to calculate the probability of necessity and sufficiency (PNS) between the textual label and textual representations.

This paper recevied diverse ratings, i.e., 6, 6, 5, 3. The AC has read reviewers' comments, authors' responses, and the revised version.  The idea of calculating the probability of necessity and sufficiency and analyzing the cross-modal spurious correlation for the aspect of causal inference is novel. The main reasons for reject are that (1) the paper has been revised significantly to include more experimental results and quantitive analysis compared to the original version, which indicates that the submission is not fully ready for publication, (2) although the performance comparisons with PromptSRC and DePT have been included in the revision, the performance gap between the proposed method, PromptSRC and DePT are marginal, and the reasons on why the gap is marginal is not discussed. (3) The presentation of the paper is not good, which is not easy to follow the idea and contributions. Therefore, the AC does not recommend the current submission as accept.