ERICT: Enhancing Robustness by Identifying Concept Tokens in Zero-Shot Vision Language Models
Authors: Xinpeng Dong
Min Zhang
Didi Zhu
Ye Jun Jian
Zhang Keli
Aimin Zhou
Fei Wu
Kun Kuang
Venue: ICML 2025 poster (2025)
Keywords: Robustness
Spurious correlation
VLM
Zero shot
Primary area: deep_learning->robustness

Abstract: Pre-trained vision-language models (VLMs) have revolutionized the field of machine learning, demonstrating exceptional performance across a wide range of tasks. However, their robustness remains vulnerable to the spurious-correlation problem. Existing works often involve fine-tuning the model with labeled data or relying on large language models (LLMs) to generate more complex prompts. Although effective to some extent, these methods introduce new challenges, including additional computational costs and dependence on the quality of prompts without fully utilizing the vision modality. To address these limitations, we propose a novel method named ERICT to Enhance model Robustness by Identifying Concept Tokens. ERICT mitigates spurious correlation directly in the inference stage and comprises two key steps: (1) Identify concept tokens capturing invariant features through auxiliary prompts to generate a token-level mask. (2) Apply the mask to the attention weights of the CLS token in the vision encoder to help the model focus on the relevant image region. Extensive experiments show that ERICT significantly improves the overall performance including that of the worst group, and achieves new state-of-the-art results.

ERICT:
Enhancing Robustness by Identifying Concept Tokens in Zero-Shot Vision Language Models
Xinpeng Dong" "
Min Zhang" 2
Didi Zhu'
Junjian Ye
Keli Zhang Aimin Zhou 2 Fei Wu
Kun Kuang"
Abstract
Pre-trained vision-language models (VLMs) have revolutionized the field of machine learning. demonstrating exceptional performance across a wide range of tasks. However, their robustness re- mains vulnerable to the apurious-correlation prob- lem. Existing works often involve fine-tuning the model with labeled data or relying on large language models (LL.Ms) to generate plex prompts. Although effective to some ex- tent, these methods introduce new challenges, in- cluding additional computational costs and de- pendence on the quality of prompts without fully utilizing the vision modality. To address these lim- itations, we propose a novel method named ER- ICT Enhance model Robustness by Identifying Concept Tokens. ERICT mitigates spurious cor- relation directly in the inference stage and com- prises two key steps: (I) Identify concept to- kens capturing invariant features through auxiliary prompts to generate a token-level mask. (2) Apply the mask to the attention weights of the CLS token in the vision encoder help the model focus on the relevant image region. Extensive experiments show that ERICT significantly improves the over- all performance, including that of the worst group, and achieves new state-of-the-ar results.
Tree-related token
Astenioon heatmap
Token-Rooe Rocus image
Bird
loken
Attention healmap
Token-Focus image
Raw image
token
Attention heatmap
Token image
Token attention visualization. We selected three tokens with distinct semantics from the original image and visualized their attention heatmaps, highlighting their cor- responding regions in the image. The experiment reveals that the attention of tokens effectively captures semantic information related to specific areas within the image.
Figure 1:
(CLIP) (Radford al., 2021) similar strategies, leverag- ing vast amounts of web-scraped data for pretraining. By mapping images and text into unified representation space, VLMs effectively break the modality barrier between the two. These models can infer the most probable answers by comparing similarities between text and images, demon- strating exceptional zero-shot generalization capabilities. The success of VLMMs has introduced groundbreaking paradigm for modern vision-language model research.
1. Introduction
Vision language models (VLMs) have achieved remarkable success across various multimodal downstream tasks (Rad- ford et al., 2021; Dehdashtian et al., 2024; Lin et al., 2024; An et al., 2023; Ge et al., 2023). These models are typically built on contrastive language-image pretraining
While VLMs have introduced a groundbreaking research paradigm. recent studies (Izmailov et al., 2022: Zheng et al., 2024; Ye et al., 2024a) have revealed their vulnerability to spurious correlations (Sagawa et al., 2019). Specifically, these models may rely on spurious features (e.g., back- grounds) for predictions, leading to performance degrada- tion samples where spurious information changes. In recent years, few studies have attempted to address the issue of spurious correlations in VLMs (Yang et al., 2023; Dehdashtian et al., 2024; Zhang & Ré, 2022; Phan et al., 2024). These approaches mitigate spurious correlation by fine-tuning pre-trained VLMs using group-labeled data, sig-
'Equal contribution 'Department of Computer Science and Technology, Zhejiang University. Hangzhou, China "East China Normal University 'Huawei Noah's Ark Lab, Correspondence to: Kun Kuang <kunkuang www.@zzjj..cccco zju.edu.cn>>
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ERICT: Enhancing Robustness by Identifying Concept Tokens in Zero-Shot Vision Language Models
nificantly improving robustness. However, obtaining group labels typically requires expert human knowledge and fine- tuning introduces significant computational costs. To ad- dress the limitations of fine-tuning VLMs, some methods have been proposed to mitigate spurious problems during the inference phase, without relying on additional train- ing or group labels (An et al., 2023; Chuang et al., 2023; Adila al., 2024). These methods design more reasonable prompts use large language models (LLMs) to generate additional spurious attributes, applying linear projections to debias visual embeddings. However, these methods either mainly focus on the text modality failing to utilize the infor- mation from the vision modality, or the attributes generated by LLMs are inherently unreliable, with linear projection potentially distorting the distribution of visual embeddings.
zero-shot inference phase by identifying invariant vision tokens without relying on the assistance of L.MSs or group labels. (2) We present theorem that explains why our approach can effectively handle spurious correlation. (3) We conduct extensive experiments and validate the superiority of our methods via quantitative and visualization results.
2. Related Work
Mitigating spurious correlation in VLMs. There many methods try to mitigate spurious correlation in VLMSs These methods can be categorized into two approaches: fine- tuning and zero-shot generalization. Most of them focus on fine-tuning using training data with group label (Yang et al., 2023; Varma et al., 2024; Kim et al., 2023; Eastwood et al., 2024; Zhang et al., 2023; 2024). For example, Fair- erCLIP (Dehdashtian al., 2024) appends two adapters and employs the Hilbert Schmidt Independence Criterion (HSIC) to learn spurious features and invariant features sep- arately. CPT (Phan et al., 2024) minimizes the entropy of loss distribution across different groups combining prompt tuning to mitigate spurious correlation. These methods re- quire data with group information for training. However, such data is difficult obtain and typically requires human expert knowledge for annotation.
To achieve better modality synergy which effectively lever- ages information from both text and vision modalities, we draw inspiration from patch tokens within the vision en- coder and are surprised to find that, although these tokens were not explicitly optimized during the pre-training phase, their representations can capture semantic information cor- responding to the relevant region in the image. We visualize the attention weight of different tokens in Figure 1. We can observe that the representations of different image to- kens focus on the regions of the image that are relevant to themselves. For example, in Figure 1, the token in the last line is related to the concept of tourist, and its attention is concentrated on tourists in the image. Based on this finding, we leverage text prompts to identify which tokens contain task-relevant features and use these features debias. Un- like methods that focus on a single modality, we incorporate text prompts to guide the vision encoder to focus spe- cific tokens and get better vision embedding which preserve alignment across text and image modalities.
Due to the superior performance and convenience of VLMS zero-shot generalization, some researchers focus on mitigat- ing spurious associations during the inference phase with- out relying on training. Hierarchy- -CIIP (Ge et al., 2023) aims to enhance the robustness of prompts by focusing on the semantic parents and children of label categories. Per- ceptionCLIP (An et al., 2023) utilizes prior knowledge to identify spurious features and then employs these identified features to enhance prompts. Chuang et al. (2023) train a projection using the constructed text pairs and project the text embeddings into a space orthogonal to the spurious attribute space. ROBOSHOT (Adila et al., 2024) generates insight for spurious features with LLMs and applies linear projection to map image embeddings to neutralization hyperplane for spurious features. These works either only consider the textual modality without considering the vision modality, or rely on the help of LLMs. Our method com- bines information from two modalities and does not require any training data or LLMs assistance.
In this paper, we propose ERICT which enhances the model's robustness during the inference phase without rely- ing on training, assistance of LLMs or group labels. For an image, we first use an auxiliary prompt get the score matrix of all tokens and perform a sparsification operation to generate a token-level mask that reserves the tokens con- taining task-relevant features. Then we apply the mask to the attention weights of the CLS token in the vision encoder during the inference stage, which can effectively help the model focus on the relevant image region. Furthermore, to address the challenge of acquiring auxiliary prompt words, we propose the ERICT-C. The detailed implementation of the method is provided in Section 4. We conduct extensive experiments on spurious correlation datasets. The results and visualization figures show that our approach effectively mitigates the spurious correlation.
Difference from token pruning. Currently, many ap- proaches solving vision-language model tasks emphasize manipulating tokens and attention weights, and token prun- ing is one of the most popular applications. The main goal of cxisting token pruning methods is to improve model effi- ciency (He et al., 2021: Tang et al., 2023; Wang et al., 2023; Ye et al., 2024b). Generally, token pruning can be achieved through two primary methods: attention mask pruning (Rao et al., 2021) and activation mask pruning (Kim et al., 2022).
We summarize our contributions as follows: (1) We propose new approach to mitigate spurious correlation in VLMs
2

---

ERICT: Enhancing Robustness by Identifying Concept Tokens in Zero-Shot Vision Language Models
h
Vision Encoder
MarMil
Feature
Spore Bisaiiathie
SorMMa
Attention Block
Intammn
Atterrion Black
BRICT
Text Encoder
Text Encoder
Ausilary hhmpp
Mariitl
FRICT-C
Text Encoder
Tipik
Narm
Clas Procpees
placo of
CLASS
Eimbodded Patches
Step I: Identify invariant tokens
Step 2: Mitigate during inference
Figure 2: Overall framework. Our framework mainly includes two steps: In Step 1, we construct an auxiliary embedding =t to identify tokens containing invariant information, obtaining a token-level mask. In Step 2. we apply the mask within the attention mechanism of the vision encoder, making tokens containing spurious information invisible to the [CLS] token.
Our method is partly inspired by attention mask pruning, but different from it. Token pruning involves removing to- kens that earry few information or are redundant during the attention computation process. In contrast, our approach aims to direct the model's attention specific regions of the image. In terms of method implementation, we don't 't remove the tokens completely and all tokens undergo the full attention computation process. We unilaterally reduce the attention of the CLS token on certain tokens. In this way, we enable the model to produce representations with more invariant features rather than spurious features.
the input image 3y. E RR3x to a h-dimennional vision embedding 2n = fo( E Rt, where W and H denote the width and height of the image. The textual encoder pro- cesses the corresponding textual information 2, generate its embedding = (xx),, where 2 RR with L rep- resenting the text length and B representing the dimension. Taking the CLIP model (Radford et al., 202 as an example, a representative approach VLMs, the specific form of its optimization loss function is presented as follows:
exp(d (1)/7)
LOLIP = -E log ΣØ= exp d(2),, ((<)),2)/TT)
(I)
3. Preliminary
where d(.)) denotes the cosine similarity. is learnable temperature parameter. N means the number of samples.
In this section, we first give the problem definition, and then provide a detailed description of dision-language models.
Specifically, taking the classification downstream task as an example, the zero-shot inference stage, hard prompts are used as text to generate a zero-shot classifier, such "A photo of (CLASS)". After obtaining vision and textual beddings, CLIP is classified by calculating their similarity:
Problem definition. This paper primarily focuses on ad- dressing the problem of group robustness (Sagawa al., 2019). Specifically, let E X represent the input image, y Y the target label, and A the spurious feature. A dataset with spurious correlations (y, a) is annotated with a group label E G, where G = Y x A denotes the set of all possible combinations of class labels and spurious attributes. To mitigate the impact spurious correlations on predictions, our method follows the basic goal of existing works (Liu al., 2021; Asgari et al., 2022; Le al., 2024; Arefin et al., 2024), aiming to improve the accuracy of the worst group and narrow the gap between the worst group performance and average performance.
argmax; d( (u, =(),
(2)
where represents the class label predicted by model and N, represents the number of classes
4. Methodology
In this section, we propose ERICT and ERICT-C to mitigate the spurious correlation during zero-shot inference of VLMs. Our methods don't 't require any training data group labels, making it both efficient and broadly applicable. There are two main steps: (1) identify invariant vision tokens in Sec- tion 4.1 and (2) mitigate spurious correlation by leveraging the discovered invariant information Section 4.2.
-laguagl models (VLMs). VLMs contain two types of encoders: vision encoder S parameterized by and a textual encoder f parameterized by Φ. For simplicity, we omit and é in the following. The vision encoder maps
3

---

ERICT: Enhancing Robustness by Identifying Concept Tokens in Zero-Shot Vision Language Models
(a) Prompt = "bird"
(b) Prompt = "tourist"
(c) Prompt = "stone"
(d) Prompt = "road"
(e) Prompt = "tree"
Figure 3: Score maps with different prompts. We use different prompts to compute score matrices with the representations of tokens. The heatmaps show that scores relevant tokens are significantly lower than other tokens.
4.1. Step 1: Identify Invariant Vision Tokens
the following conditions:
For every image, we try to identify which tokens focus on invariant information and which tokens focus on spurious information, is achieved based on our finding.
k max lc E [p) 11 11+////T> E,5e,,/Th
(4)
where r is the temperature parameter which we will analyze in the ablation study. Then we calculate the threshold
Finding. As mentioned in Section 1, in vanilla CLIP in- ference process, the image patch token representations are discarded, even though these representations contain rich semantic information. We discovered that this semantic information can be captured with specific prompts. Specif- ically, we calculate the similarity score matrix between a prompt and the representations of all the image tokens. Dif- ferent from the CLS token, we observed that tokens whose semantics align more closely with the prompt tend to have lower similarity scores, as shown in Figures 3. Based on this discovery, the difference in scores allows us to identify the tokens focusing on invariant features within the image, and we achieve this via a token-level mask.
1=
(5)
k
The mask M determined based on the relationship be- tween the score and the threshold.
0
if ifs 31
M
(6)
if
Auxiliary embedding. The auxiliary embedding =f is pretty important in this process. For ERICT, 2 = f+(x)) Ideally, the auxiliary prompt æ should be the ground- category name or description relevant features corresponding to the image, but is not feasible during the zero-shot infer- ence stage. So we adopt the superclass of the task category with a template "(} in photo". For example, on the CelebA dataset, we use "hair in photo" as an auxiliary prompt. Fur- thermore, when difficult to determine the superclass, we propose ERICT-C whose auxiliary embedding can be obtained through aggregating class prompt embeddings. For datasets with small number categories (eg., Waterbirds), it is sufficient to directly aggregate the representations of all categories.
The overall framework is shown in Figure 2. For ERICT, we auxiliary prompt =t for every task and get the auxil- iary text feature by the text encoder. The specific method for selecting auxiliary prompts will be introduced later.
Then, given an image token embedding sequence X [Xets X1...., xp." for attention blocks Vision Trans- former(ViT) where Yek represents the added global class token, {zzli 1,2....,pf p} denote local patch tokens and p represents the number of patch, score matrix S E IR? between the final token embeddings X patch = X } and the auxiliary prompt:
compute the
- fe(z)).
where C' denotes the set of classes and 28 represents embed- ding of auxiliary prompt, which corresponds to f( in the previous equation.
S = d(X f((z)))
(3)
According to the obtained score matrix, we generate the corresponding mask M. A common method to set a percentage threshold, but the proportion of related attributes varies in different images. So we use a method similar to SparseMax (Martins Astudillo, 2016) to generate specific mask for every image.
When the dataset contains a large number of class (e.g.. ImageNet), ERICT-C adopt top-K strategy. Specifically, in Step 1, we first compute the model's top-K outputs, and then merge the corresponding k textual representations. Step already involves a inference, so the top-k strategy does not require an additional inference.
Let s' = Sort(-1 S), which means S
S² 2 ... Sy. we firstly find the max index & that satisfies
z K f(()))
(8)

---

ERICT: Enhancing Robustness by Identifying Concept Tokens in Zero-Shot Vision Language Models
We present the results of ERICT-C on the ImageNet series datasets in the Appendix D.1.
ground truth yr- The ground truth y is uniformly drawn from {-1,1), and the spurious attribute drawn from {-1,1} with the probability Pspu Pr(a = y). where > Pepu > 1/2. The variance o² denotes the variation of the corresponding variable.
4.2. Step 2: Mitigate Spurious During Inference
In this step we perform the actual debiasing and inference. Existing works (Chen et al., 2022; Li et al., 2023a; Lin et al., 2024) have demonstrated that attention weights can capture the relationships between different patches, thereby reflecting the regions that the model focuses on.
For the architecture, we consider two simple linear encoders fo Rdd - R for the image modality and ft R - R for the text modality, implemented as fo( = Was, and f(()) = WrT, with W1 E R and Wr e Rhxdr respectively. In the vanilla CLIP zero-shot inference phase, the decision is made by computing similarity score, which takes the following form:
Based on the token-level mask obtained in Step 1, we en- force the model focus on the regions containing invariant information. Specifically, we apply the mask the attention weight of CLS token in vision encoder. In this case, the CLS token's 's perception of certain tokens is blocked, and these tokens correspond to the parts containing spurious in- formation. Only the tokens containing invariant information are retained for further perception. The attention process with mask becomes as follow:
T W
(13)
Theorem 5.2. Given task defined in 5.1 and an over- parameterized CLIP model where n = w(1), di = Qmn) and dr = Q(n). A distinguishable mask M ~ Bernoull( aims to remove spurious components from the vision modality. If the gap between the vari- (9) ances of the core and spurious features significant: iine = (1)), Fan 2 and o spu 0( MTogn Then with high probability at af least 1 - 0( poly(n)) = 1 1 o(1), the
(M (M QoskT
A(ras::) = softmax
vdk )
A[Tpatchess:) :) = softmax( Spatches KT
(10)
vdk
CLIP model achieves large error in zero-shof accuracy in the out-of- -itiof--dittibttionn test data where $ y:
X = x Proj([A(Faas: :), A paptcchss :)] V),
(11)
where Q, K,V V are query matrix, key matrix, value matrix in attention calculation process respectively and Proj is re- lated projection module in the vision encoder. dx is the dimension of K. A represents the Q-K attention weight.
((k) - o(1) S &Erraty S ((k) + o()),
where ks = V(I+a Pa (2papu 1) o spu
Prmnak (2pap 1)-11)1-1111 1- Gsny
and &b de-
notes the CDF of a standard normal distribution.
Then, we construct prompts using a template and class name like vanilla CLIP. Image classification performed using the similarity between image representation and text repre- sentation. For all C classes:
Notice that 0 S Pmask S and 1 Pmask is the actual mask ratio ( [mmskk- = 1 represents the vanilla error bound without mask) ). As the proportion of the masked spurious components increases, the model error bound gradually de- creases. We defer the proof of Theorem 5.2 to Appendix A, where a detailed derivation provided.
i argmarieN, w...ddd(ffaaa f)
(12)
where o is single class prompt simply constructed with "a photo of a (CLASS)" template.
6. Experiments
6.1. Experiment Setting
5. Theory Analysis
Datasets. We evaluate our approach on three widely used spurious correlation datasets, including Waterbirds (Sagawa et al., 2019), CelebA (Liu et al., 2015), and Urbancars (Li et al., 2023b). For Waterbirds and CelebA, we follow the setting of previous works (Sarridis al., 2024; Yang et al., 2024; You et al., 2024). Waterbirds dataset defines a task to identify whether bird is a land bird or water bird, and the spurious attribute is background. CelebA dataset defines a task to identify whether person's 's hair is blonde or not and the spurious attribute is gender. Urbancars is an artificially generated dataset to identify car types, and the spurious attribute are background and co-occurrence objects.
To better understand why the binary mask can effectively mitigate spurious correlation, in this section, we present an error probability bound of the model under the masking condition and demonstrate that using a binary mask can effectively reduce the error bound. Firstly, we define the task in the context of vision-language settings:
Definition 5.1. Consider a classification task with n pairs of {x " Every image zi, is generated from the un- derlying latent factor Z = Zin:: zopul which is composed of the invariant factor Zinu ~ Ny oin)) and spurious fac- tor spm ~ N(a, 0 p. Prompt ot is generated from the
5

---

ERICT: Enhancing Robustness by Identifying Concept Tokens in Zero-Shot Vision Language Models
Table 1: Experiments on Waterbirds, We highlight the best results in bold, the second-best results with an underline.
ViT-B/32
VIT-B/16
ViT-L/14
WG (1)
AVG (t)
Gap (L)
WG (t)
AVG (f)
Gap (L)
WG (1)
AVG (t)
Gap (L)
ZSCLIP
39.33
67.34
28.01
22.74
79.23
56.49
35.20
84.17
48.97
Group Prompt
64.17
79.91
15.74
16.82
81.51
64.69
29.43
85.22
55.79
PerceptionCLIP
66.07
89.8
23.73
16.07
82.98
66,91
44.94
86.44
41.51
PerceptionCLIP4
60.33
78.6
18.27
41.07
85.8
44.73
61.12
87.74
26.62
ROBOSHOT
54.4
82.0
27.6
45.2
79.9
34.7
ERICT
71.44
78.67
7.23
56.23
83.81
27.58
56.39
88.66
32.27
ERICT-C
57.2
72.52
15.23
57.32
82.46
25.14
62.31
88.09
25.78
Table 2: Experiments on CelebA. We highlight the best results in bold, the second-best results with underline.
ViT-B/32
ViT-B/16
ViT-L/14
WG (f)
AVG (t)
Gap (L)
WG (1)
AVG (1)
Gap (L)
WG (1)
AVG (1)
Gap (L)
7SCLIP
61.11
90.48
29.37
68.88
88.35
19.47
72.77
87.68
14.91
Group Prompt
45.32
88.46
43.14
67.09
86.13
19.04
75.00
89.09
14.09
PerceptionCLIP
76.70
79.89
3.19
65.13
75.27
10.14
74.31
80.30
5.99
PerceptionCLIP+
75.94
82.02
6,08
69.18
77.17
7.99
80.84
83.04
2.20
ROBOSHOT
80.5
84.8
4.3
82.6
85.5
2.9
ERICT 83.46
86.53
3.07
80.00
86.49
6.49
85.00
86.43
1,43
ERICT-C
83.62
86.25
2.63
81.98
84.30
2.32
85.44
86.53
1.09
Baselines. We compare our method with state-of-the- art methods, including vanilla zero-shot CLIP (ZSCLIP), ZSCLIP with group information (Group Prompt), Percep- tionCLIP (An et al., 2023) and ROBOSHOT (Adila et al., 2024). Group Prompt assumes access to spurious attributes and includes them the label prompts. For instance, the la- bel prompts Waterbirds dataset become "a photo of water- bird with water background, photo of waterbird with land background, photo of landbird with water background, photo of landbird with land background". For Perception- CLIP, it uses variety of different combinations of assists. We show of them, PerceptionC represents using the standard spurious attribute (e.g., gender for CelebA dataset), and PerceptionCLIPe represents using extended attributes (e.g., gender, age, and race for CelebA dataset).
case group accuracy of 57.32%, surpassing the current best- performing method by margin of 16.25 percentage points.
Table 2 presents the results on the CelebA dataset. Our methods perform outstandingly on the CelebA dataset, with ERICT-C and ERICT achieving the best and second best performance across backbones, respectively. In particu- lar, with the VIT-B/16 backbone. our methods achieve an improvement of more than 10 percentage points the worst group compared to the previous baseline methods. Addi- tionally, we significantly reduced the gap between the per- formance of the worst group and the average performance, narrowing the gap to 1.09 on the ViT-L/14 backbone.
We also conduct experiments on the Urbancars dataset in Table 3, which poses a significant challenge for CLIP. Nev- ertheless, method remains effective: the ViT-L/14 backbone, it improves the accuracy of the worst group per- formance from 12.4% to 49.6%. Importantly, this improve- ment does not come at the cost of average performance; in fact, we observe a notable enhancement in average accuracy.
6.2. Main Results
We present results on three widely used backbones, in- cluding ViT-B/16, ViT-B/32 and ViT-L/14. Generally, our method has significant improvement on the worst group performance and narrows the gap between the worst group and the average performance among all backbones.
Overall, as shown in Tables 3, ERICT and ERICT-C consistently outperform all other methods across different datasets and backbones, setting mance. However, we observed that ERICT does not show significant improvement in some cases, and ERICT-C out- performs ERICT in many cases. The phenomenon points to the suboptimality impact the auxiliary prompt, which we will discuss in following sections.
tfte--of-hhe-r perfor-
Table shows the results on the Waterbirds dataset. Our methods perform well across all backbones, especially on the base backbone, where it shows significant improvement compared to baseline methods. On the ViT-B/32 backbone, ERICT achieves worst-case group accuracy of 71.44%. And on the ViT-B/16 backbone, ERICT-C achieves a worst-
6

---

ERICT: Enhancing Robustness by Identifying Concept Tokens in Zero-Shot Vision Language Models
Table 3: Experiments on Urbancars. We highlight the best results in bold, the second-best results with an underline.
ViT-B/32
ViT-B/16
ViT-L/14
WG (t)
AVG (1)
Gap (4)
WG (1)
AVG (1)
Gap (L)
WG (1)
AVG (t)
Gap (L)
ZSCLIP
14.80
51.60
36.80
0.00
52.00
52.00
12.40
54.00
41.60
Group Prompt
33.20
53.50
20.30
7.60
52.70
45.10
25.20
51.30
26.10
ERICT
38.80
53.30
14.50
9.20
51.40
42.20
49.60
63,30
13.70
ERICT-C
29.60
53.00
23.40
8.40
51.70
43.30
48.00
62.60
14.60
20
60
Water Background
Land Background
-00
Waterbird
Lanobiro
Land Backgroundd Watte Beckgraund
-00
Landbird Waterbird
(a) CL.I
(b) ERICT
Figure 4: t-SNE comparison diagram on the Waterbirds dataset. The output representations of the ERICT method in the t-SNE visualization exhibit limited distinction with respect to background information, indicating that the model's decision- making does not rely on background information (i.e., spurious attribute).
Original image
Score heatmap
Original CAM
FRICT CAM
final layer in the model before and after ERICT application. As illustrated Figure 5, can be observed that, in com- parison to the vanilla zero-shot CLIP, ERICT effectively guides the model'ss attention toward the invariant features, thereby mitigating the impact of spurious attributes in- ference outcomes. For instance, in the Waterbirds dataset, the vanilla zero-shot CLIP erroneously focuses on back- ground elements, whereas our method successfully redirects attention to the invariant features, specifically the bird.
Considering that our method directly adjusts the attention mechanism, CAM images may not accurately reflect the de- biasing capability. Therefore, we also visualize the model's output representations using t-SNE (Van der Maaten & Hin- ton, 2008), with separate divisions based on ground-truth label and spurious attribute. We present the t-SNE results for the Waterbirds dataset in Figure The vanilla zero-shot CLIP separates the two types of background distinetly in the feature space, indicating that it considers background features important discriminative factor. The t-SNE visualization of our method shows that after applying ER- ICT, the model no longer focuses on the background, which is a spurious attribute. All the images were generated using the default t-SNE parameters from the seikit-learn package.
Top FOJ
TO
FU!
Figure 5: CAM comparison diagram. We visualized some images from the Waterbirds and CelebA datasets, showing that our approach successfully focuses the desired region.
6.3. Visualization Results
6.4. Ablation Study
Beyond the quantitative analysis of the datasets, fur- ther provide detailed demonstration of ERICT's debiasing capability by presenting visualized results.
In this section, we present the results of several ablation studies aiming to identify key factors that influence the per- formance of our proposed method. Specifically, we examine the impact of the temperature parameter, the effect of mask placement, and the role of auxiliary prompt words.
Considering the widespread use of CAM (Zhou et al., 2016; Jung& & Oh, 2021; Vilas et al., 2024) images for interpreting what model focus on, we compared the CAM images of the

---

ERICT: Enhancing Robustness by Identifying Concept Tokens in Zero-Shot Vision Language Models
Impact of temperature. In this section, we investigate the impact of the temperature parameter 7. The temperature parameter controls the sharpness of the similarity score ma- trix distribution, thereby influencing the mask ratio during the inference phase. A lower temperature value results in a higher mask ratio. The optimal temperature parameter varies depending on specific dataset. In general, the lower the proportion of invariant features in an image, the smaller the temperature parameter. For example, the temperature parameter for the Waterbirds dataset is lower than that for the CelebA dataset, as the proportion of bird features in the Waterbirds dataset is smaller, whereas the proportion of hair features is larger in the CelebA dataset.
Table 4: Effect of mask location with ViTL14 backbone on CelebA dataset. In our method, the mask is applied to the CLS weights of the last two layers.
WG (f)
AVG (t)
Gap (4)
ZSCLIP
72.77
87.68
14.91
Mask image
81.67
90.30
8.63
Last I layer
83.33
85.48
2.15
Last 2 layers
85.44
86.53
1.09
Last 3 layers
84.44
87.37
2.93
Last half backbone
80.00
88.61
8.61
All layers
79.44
88.18
8.74
Token purning
83.89
87.91
4.02
90-
90
80-
50
Impact of the auxiliary prompt. In our approach, auxiliary prompt words play critical role, as they determine the key information that the model focuses on. In this paper, we propose ERICT, which computes auxiliary representations using parent class prompts, and ERICT-C, which employs clustering based on class prompt representations as aux- iliary representations. Due to the limitations of model's capabilities, the current recognition stage remains relatively simplistic, capturing primarily basic semanties. The aux- iliary information used by ERICT and ERICT-C offers a viable solution, though it is not the optimal one. We hope that future work can address this challenge.
70-
70-
60
Wiet Group
60
Wort Group
50
0.5
1.0
1.5
2.0
2.5
3,0
50
0.5
1.0
1.5
2.0
2.5
3.0
(a) ViT-B/16
(b) ViT-L/14
Figure 6: The impact of different temperature parameters on the CelebA. As the temperature gradually increases, the performance of the worst group exhibits an initial upward trend followed by a subsequent decline.
We illustrate the impact of different temperature parame- ters in Figure 6. As the temperature parameter decreases, the performance of the worst-performing group initially im- proves, followed by a decline. This can be explained by the following reasoning: as the mask ratio increases, the model's 's robustness improves. However, when the mask ratio becomes excessively high, some invariant features are also masked, resulting in performance degradation. The optimal results are presented in the main experiments.
7. Conclusion
In this paper, we aim to enhance the robustness of vision language models during zero-shot inference without requir- ing any additional training external model assistance. To achieve this, we proposed ERICT and ERICT-C, which identify concept tokens that capture invariant features and leverage this invariant knowledge to effectively alleviate spurious correlation during inference. Through experiments and visualizations, we demonstrate the effectiveness of our approach, which significantly reduces the impact of spu- rious correlations and achieves notable performance im- provements compared to existing baseline methods across multiple datasets and backbones. Ablation studies on vari- ous experimental settings explain the impact of temperature parameter and mask location on performance. Additionally, for CLIP-based VLMs, we provided new theory support- ing the effectiveness of masks to enhance robustness, pro- viding strong theoretical support for our method. Although our proposed method demonstrates significant performance, it still has some limitations. The current model relies on specific auxiliary embedding, which don't 't guarantee the optimality. Future research could explore the following directions: investigating how to obtain more effective auxil- iary text representation and optimizing mask strategies.
Application location of mask. In this section, we analyze the impact of the applied mask location. Previous stud- ies (Kirichenko et al., 2022; Gandelsman et al., 2023) have emphasized the critical role of the final layers, particularly the last two layers, in enhancing the model's robustness. Therefore, we apply the mask to the last two layers of the vi- sion encoder. To assess its effect, we conduct a comparative experiment where the mask is applied to various locations: the original image, the last 1-3 layers in vision encoder, the latter half of the vision encoder, and all layers in vision encoder. The detailed results are presented in Table
As shown in Table 4, we observe that, compared the vanilla CLIP, various masking strategies contribute to im- proved robustness. Among these strategies, applying the mask to the CLS weight of the last two layers achieves the best performance on the worst porff-performing group. The re- sults presented in our main experiments correspond to those obtained by masking the last two layers.
8

---

ERICT: Enhancing Robustness by Identifying Concept Tokens in Zero-Shot Vision Language Models
Acknowledgements
Eastwood, C., Singh, S., Nicolicioiu, A. L., Vlastel- ica Pogančić, M., von Kügelgen, J., and Schölkopf, B. Spuriosity didn't "tdn't kill the classifier: Using invariant pre- dictions to harness spurious features. Advances in Neural Information Processing Systems, 36, 2024.
This work was supported part by the National Key Research and Development Program of China (2024YFE0203700), National Natural Science Foundation of China (62376243, 62441605), and "Pioneer" and "Lead- ing Goose" R&D Program of Zhejiang (2025C02037). All opinions in this paper are those of the authors and donot necessarily reflect the views of the funding agencies.
Gandelsman, Y., Efros, A. A., and Steinhardt, Interpreting clip's s image representation via text-based decomposition. 2023.
Ge. Y., Ren, J., Gallagher, A., Wang, Y., Yang, M.-H., Adam, H., Itti, L.,, Laksshinnarayanan, B., and Zhao, J. Improving zero-shot generalization and robustness of multi-modal modelssercceptionccip.. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 11093-11100) 2023.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be
specifically highlighted here.
He, X., Keivanloo, I., Xu, Y., He, X., Zeng, B., Rajagopalan, S., and Chilimbi, T. Magic pyramid: Accelerating infer- ence with early exiting and token pruning. arXiv preprint arXiv:21 10.022/11.00230,, 2021.
References
Adila, D., Shin, C., Cai, L., and Sala, F. Zero-shot robustifi- cation of zero-shot models. In The Twelfth International Conference on Learning Representations, 2024.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F,, Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of dutto--distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8340-8349, 2021a.
An, B., Zhu, S., Panaitescu-Liess, M.-A., Mummadi, C. K., and Huang, F. Perceptionclip: Visual classification by inferring and conditioning on contexts. In The Twelfth International Conference Learning Representations, 2023.
Arefin, M. R., Zhang, Y., Baratin, A., Locatello, F., Rish, I., Liu, D., and Kawaguchi, K. Unsupervised concept discovery mitigates spurious correlations. arXiv preprint 1eXiv:2402.13366 2024.
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 15262-15271, 2021b.
Asgari, S., Khani, A., Khani, F., Gholami, A., Tran, L., Mahdavi Amiri, A., and Hamarneh, G. Masktune: Mit- igating spurious correlations by forcing to explore. Ad- vances in Neural Information Processing Systems, 35: 23284-23296, 2022.
Izmailov, P., Kirichenko, P., Gruver, N., and Wilson, A. G. On feature learning in the presence of spurious correla- tions. Advances in Neural Information Processing Sys- tems, 01338516-35533 2022.
Jung, H. and Oh, Y. Towards better explanations of class activation mapping. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 1336- 1344, 2021.
Chen, P., Li, Q. Biaz, S., Bui, T., and Nguyen, A. gscore- cam: What objects is clip looking at? In Proceedings of the Asian Conference on Computer Vision, pp. 1959- 1975, 2022.
Kim, J. M., Koepke, A., Schmid, C.,, and Akata, Z. Expos- ing and mitigating spurious correlations for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2585- 2595, 2023.
Chuang, C.-Y., Jampani, V., Li, Y., Torralba, A., and Jegelka, S. Debiasing languale models via biased prompts. arXiv preprint wxXiv23322.00070 2023.
Dehdashtian, S., Wang, L., and Boddeti, V. N. Fairerclip: Debiasing zero-shot predictions of clip in rkhss. In Inter- national Conference on Learning Representations, ICLR, 2024.
Kim, S., Shen, S., Thorsley, D., Gholami, A., Kwon, W., Hassoun, J., and Keutzer, K. Learned token pruning for transformers. pp. 784-794, 2022.
Deng. J., Dong, W., Socher, L.-.J K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.
Kirichenko, P., Izmailov, P., and Wilson, A. G. Last layer re-training is sufficient for robustness to spurious correla- tions. arXiv preprint urXiv:2204.02997 2022.

---

ERICT: Enhancing Robustness by Identifying Concept Tokens in Zero-Shot Vision Language Models
Le, Q.. Schlötterer, J., and Seifert, C. Out of spuriousity: Improving robustness to spurious correlations without group annotations. arXiv preprint arXiv:240..14974, 2024.
Tang, Q., Zhang, B., Liu, J., Liu, F,, and Liu, Y. Dynamic token pruning in plain vision transformers for semantic segmentation. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision. pp. 777-786, 2023.
Li, R., Mai, Z., Zhang, Z., Jang, J., and Sanner, S. Transcam: Transformer crcetioo-bbsedd cam refinement for weakly supervised semantic segmentation. Journal of Visual Communication and Image Representation, 92: 10:100 2023a.
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(1 1), 2008,
Varma, M., Delbrouck, J.-B., Chen, Z., Chaudhari. A., and Langiotz, C. Ravl: Discovering and mitigating spurious correlations in fine-tuned vision- llanuallan models. arXiv preprint urXiv:241 1204411114997 2024.
Li, Z.,, Evtimov, I., Gordo, A., Hazirbas, C., Hassner, T., Ferrer, C. C., Xu, C., and Ibrahim, M. A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others. pp. 20071-20088 2023b.
Vilas, M. G., Schaumlöffel, T., and Roig. G. Analyzing vi- sion transformers for image classification class embed- ding space. Advances in neural information processing systems, 36, 2024.
Lin, Y.. Chen, M., Zhang, K., Li, H., Li, M., Yang, Z., Lv, D., Lin, B., Liu, H., and Cai, D. Tagelip: A local-to-glote golobbl framework to enhance open-vocabulary multi-label clas- sification of clip without training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 3513-3521, 2024.
Wang. Z., Chen, J., Zhou, W. Zhu, H., Liang, J., Shan, L., Liu, M., Xu, D., Yang, Q. and Qin, B. Smarttrim: Adaptive tokens and attention pruning for efficient vision- language models. arXiv preprint arXiv:2305.15033 2023.
Liu, E. Z., Haghgoo, B., Chen, S., Raghunathan, A., Koh, W., Sagawa, S., Liang, P., and Finn, C. Just train twice: Improving group robustness without training group information. pp. 6781-6792, 2021.
Xue, Y., Joshi, S., Nguyen, D., and Mirzasoleiman, B. Un- derstanding the robustness of multi-modal contrastive learning to distribution shift, 2024. URL https:// arxiv. axxiv.org/ai wwg. abs///////// 2310 04971.
Liu, Z., Luo, P., Wang. X., and Tang, X. Deep learning face attributes in the wild. pp. 3730-3738, 2015.
Yang, Y., Nushi, B., Palangi, H., and Mirzasoleiman, B. Mitigating spurious correlations in multi-modal models during fine-tuning. In International Conference on Ma- chine Learning, ICML, pp. 39365-39379, 2023.
Martins, A. and Astudillo, R. From softmax to sparsemax: sparse model of attention and multi-label classification. pp. 1614-1623, 2016.
Phan, H., Wilson, A. G., and Lei, Q. Controllable prompt tuning for balancing group distributional robustness. arXiv preprint arXiv:2403.02690 2024.
Yang. Y., Jiang, C., Lin, Z., Xiao, J., Zhang, J., and Sang, J. Debiasing vison-languu soon-languaag models with text-only training. arXiv preprint arXiv:2410.0995
2024.
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. pp. 8748-8766 2021.
Ye, W., Zheng, G., Cao, X., Ma. Y. and Zhang, Spurious correlations in machine learning: A survey, arXiv preprint 1277: 2024a.
Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., and Hsieh, C.-J. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34: 14:13937-13949, 2021.
Ye, X., Gan, Y., Ge, Y., Zhang, X.PP., and Tang, Y. Atp-Ilava: Adaptive token pruning for large vision language models. arXiv preprint arXiv:2412.00444 2024b.
You, C.. Min, Y., Dai, W., Sekhon, J. S., Staib, L., and Duncan, J. S. Calibrating multi-modal representations: A pursuit of group robustness without annotations. IEEE conference on computer vision and pattern recognition, CVPR, 2024.
Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case gener- alization. arXiv preprint
2019.
Sarridis, I., Koutlis, C., Papadopoulos, S., and Diou, C. Mavias: Mitigate any visual bias. arXiv preprint arXiv:24/2.06632. 2024.
Zhang, M. and Ré, C. Contrastive adapters for foundation model group robustness. Advances in Neural Information Processing Systems, NeurIPS, 02221682221697 2022.
10

---

ERICT: Enhancing Robustness by Identifying Concept Tokens in Zero-Shot Vision Language Models
Zhang, M., Yuan, J., He, Y., Li, W., Chen, Z., and Kuang, K. Map: Towards balanced generalization of iid and ood through model-agnostic adapters. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1993,, 2023.
Zhang, M., Li, H., Wu, F., and Kuang, K. Metacoco: A few-shot classification benchmark with spurious correla- tion. arXiv preprint arXiv:2404. 19644, 2024.
Zheng, G., Ye, W., and Zhang, A. Learning robust classifiers with self-guided spurious correlation mitigation. arXiv preprint ar wrXiv:2405.03644 2024.
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. Learning deep features for discriminative localization. pp. 2921-2929, 2016.

---

ERICT: Enhancing Robustness by Identifying Concept Tokens in Zero-Shot Vision Language Models
A. Proof of Theory
Definition A.1. Consider a classification task with n pairs {- "iji=1 Every image xi E Rdr is generated from the underlying latent factor Z [Zinos zspu] which is composed of the invariant factor Zine Ny. (ne) and spurious factor Zine ~ (a,
o(,, App)). Text zi E Rd generated from the ground truth yı. The ground truth y is uniformly drawn from
{-1,1); 1), and the spurious attribute drawn from {-1, 1} with the probability Pr(a y) Papus where 1 > Pspa 1/2.
The variance o² denotes the variation of the corresponding variable.
With the definition, we can write y & 32_
where & (O, m. NO, (pp) are two Gaussian variables in the
y definition. The 2v generated via F, D, text 24 is generated via y = Dr 0 with D2 Rdxx and Dy E R The matrix D, and Dr is matrix with orthonormal columns which can be considered as dictionary matrix. And the distinguishable mask M ~ Bernouli(PPmas)) removes spurious components from the vision modality.
In clip architecture, there are two encoders fo R - IR for the image modality and ft R - R for the text modality, implemented as fo (x,,) = WIZ, and [x(x)) Tr with W E Rg and WT E Rixdr respectively. the vanilla CLIP zero-shot inference phase, the decision is made by computing a similarity score, which takes the following form:
T W T
(14)
Before the formal proof, we firstly introduce useful lemma: Lemma A.2 ((Xue et al., 2024)). The minimizer of linearized CLJP loss Wii W satisfies the following with a probabilit of at least 1 O (oly(n)) such that,
W T W
1
D1
1 o inn 2p-pa -1
2pspu - 1]
DJ 12 S 1 O(VE))
(15)
1 + 0²
Jap
0
where 6D = O(v logn
Then, we have
WiW 1
-I p
1
Dr
1 o
2pppu - 11
Diaalla S ||.. - s Olveellogn)
||..
(16)
2popu
+ 2pm
p
In Lemma A.2, notice that
1++ an [2Pppu
2pppu 1) + 2pp
2 Dj
D. ((v + )n + (a (2) 1).
(17)
For inference, the model discriminates based on the similarity between the two modalities. Consider the scenario where the model encounters samples with spurious correlations (y = a,a= ), and we apply the mask M modality. When CLIP makes an incorrect prediction, we have
[
on vision
M
Mr Wi 7 8-1 < M W
T W W 7ot
(18)
y + E1 Pmnas + {2) + (1 Pmask Eu
where Eo can be considered as (0 0). The entire spurious part can be considered as following Gaussian mixture distribution.
Note that M follows a Bernoulli distribution, so Mr.-
By substituting Mæ, for 2. in Eq. 16, still obtain:
+ o 2p -1
2pspu - -1
D Mr. T ;
DT i= + 0 (
+ o2
apu
(19)
Mr. D
1 0
2pspu
D. j- - ((Vellog))
D=
2Papa -1
1 + o2
spu
12

---

ERICT: Enhancing Robustness by Identifying Concept Tokens in Zero-Shot Vision Language Models
with Eq. 17 plugged in, denote 61 OO we further have
2 (1 +1) + (mo) + Pmask p2) -) 9
(20)
Since (1 + (nn)) + E2(2pspu 1)Pmask a Gaussian variable follows the distribution of
(1 + 2nn) + E2(2pspu - 1) N/O,, (1 + ) + (2popu 1)2 Pmast Pppu):
then, we have
Pr(2 [1 + (1) + (n + Pmaakk -1 (2) 1) - 61] < 0)
Pmask (2Popu 1) 1 Osn + c1
(1 + ? inv + (2pspu 1)2pmask spu 2o?
(21)
Pmask (2psu 1) Jine + C1
o2 ) sav + (2popn 1)2 mmack
20
spa
where & is the CDF of the standard Gaussian distribution. Then, suffices to know that the ErTy=1.o=-1 lower bounded by ((k) (1) and upper bounded by ((k) + o(1) where k the case y - 1, a
which also applies to Pmmao (2papu 1) -ins
(1+a Fo (2popu 1y Pimest pp
Next demonstrate the impact of Pinask-
dk
(2papu +) Pa. +(1 1 (2p 1) PpuPousk Considering
dPmask
(..o)..)00 +2p "uuuuemt
1/2 Papu and Pmask S 1, & increases monotonically p increases. Notice that Pmask is the actual mask ratio and Pmask represents the vanilla error bound without mask. As the proportion of the masked spurious components increases, the model's 's error bound gradually decreases.
B. Experimental Details
All of our experiments are conducted on a single NVIDIA GeForce RTX 4090 GPU, The class prompts and auxiliary prompts for ERICT used in experiments are shown in Table 5.
Table 5: Summary of prompts in main experiments.
Dataset
Class Prompts
Auxiliary Prompt for ERICT
a photo of landbird
Waterbirds
bird in photo"
a photo of a waterbird "
a photo of person with non-blonde hair a photo of person with blonde hair photo of an urban car".
CelebA
hair in photo
Urbancars
in photo
photo of country car"
13

---

ERICT: Enhancing Robustness by Identifying Concept Tokens in Zero-Shot Vision Language Models
C. Pseudocode
Algorithm 1:
Step 1 of ERICT-C
Input
:Image Tes Image encoder fu. Text encoder fi, Class
prompts dt,, Temperature parameter т, Top-K strategy
parameter k, Patch number p.
// Auxiliary Prompt
zels
fu(x.)(els)
[If.(z.)]cte]]]
for x E xz do
fe(w])
zi [[(()
zele o
end
- sort descending):
S₁ S[1 : k]
7, corresponding
2 +- >E= =
1/ Get Mask
2p patches f.(r.) (patch]
TT ]atchh]]
for 1 to p do
S. zte spatches
x
end
S + sort( -1 1 x S, descending)
S - sort(S. descending)
P + max i:S. > Eim S//T):
1 (E S:/T 1)
for = 1 to pddo do
if S; > then
M4 < 1;
end
else
M, + 0;
end
end
Output: M
14

---

ERICT: Enhancing Robustness by Identifying Concept Tokens in Zero-Shot Vision Language Models
Algorithm 2: Step 2 of ERICT-C
Input :Image Token sequence X, Image encoder fu. Text encoder fI. Class prompts Tt, Corresponding mask M.
for BLK E fol: -2] do
X t- (X)
end
for BLK E f.l-2:1 do K, V + Projoxv(X))
Ads = softmax ( Qut R
vikk
Apatehes = softmax Opatches KT
Vdv
X + Proj([ Aotas Apatchee] V)
X + FFN(X):
end
2. (xoj(X) for
r/ E 2i do
f((w))
2 [1f((=))1
end
i argmax
dt (2o x 21)
Output:
D. More results
D.I. Imagenet Series Datasets
In these section, present the results of ERICT-C on multi-category datasets. Imagenet (Deng et al., 2009) is a widely used large-scale vision dataset containing 2021b) is a subset of ImageNet consisting of hard samples that were misclassified by the ResNet model, which contains 200 categories. ImageNet-R (Hendrycks et al., 202 la) is constructed by stylizing some images in the ImageNet in various ways, aiming to test the robustness of the model when facing style-transfert images. It contains 200 categories.
than 14 million images covering 1,000 categories. ImageNet-A (Hendrycks et al.,
We present the performance of ERICT-C with the top-3 strategy as an example across three datasets Table As shown in table 6, our method outperforms vanilla CLIP all these datasets, with particularly significant improvements observed on ViT-L14.
Table 6: Experimental results of ERIT on Imagenet dataset with top-3 strategy.
Imagenet
Imagenet-A
Imagenet-R
VIT-B/32
VIT-B/16
ViT-L/14
ViT-B/32
VIT-B/16
VIT-L/14
ViT-B/32
ViT-B/16
ViT-L/14
ZSCLIP 59.31 64.01
70.21
29.07
46.32
66.75
66.54
73.58
85.19
ERICT-C 59.45 64.0210.01
7..15
31.1912.12
48.19
70.03
67.2510.71
75.23/1.66
87.1912.00
D.2. Visualization Experiments
15

---

ERICT: Enhancing Robustness by Identifying Concept Tokens in Zero-Shot Vision Language Models
Figure 7: CAM comparison diagram. From left to right, the sequence is as follows: the original image, the score heatmap
visualization ( to identify the desired tokens), the original CAM image, and the CAM image with ERICT.
16


Reviews:
[{"review_id": "h8gObeNy93", "reviewer": "Reviewer_ZMVi", "summary": "This paper proposes ERICT, a zero-shot method to improve robustness in vision-language models by identifying “concept tokens” that represent invariant image features. ERICT uses auxiliary prompts to generate masks applied to attention weights, aiming to reduce spurious correlations. The authors evaluate ERICT on standard benchmarks (Waterbirds, CelebA, Urbancars), claiming substantial improvements, especially on worst-group accuracy.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on our work. We **sincerely appreciate your recognition of our contributions and the significance of the research problems we address**. Your support further strengthens our confidence in the proposed approach. If you have any additional questions or suggestions, we would be happy to discuss them.", "claims_and_evidence": "The claims made in the paper are largely supported by thorough experimental results on multiple benchmarks (Waterbirds, CelebA, Urbancars), showing clear improvements in worst-group accuracy. Visualizations provided further illustrate that ERICT effectively shifts attention away from spurious features.", "methods_and_evaluation_criteria": "Yes, the proposed method directly addresses the spurious correlation issue in zero-shot VLMs, and the benchmarks used (Waterbirds, CelebA, Urbancars) are appropriate, clearly capturing improvements in robustness and worst-group accuracy.", "theoretical_claims": "Theorem 5.2 establishes the theoretical error bound after applying the proposed mask. The theorem itself is well-formulated, and the intuition (reducing spurious correlations lowers the error probability) is reasonable.", "experimental_designs_or_analyses": "The experimental design is sound and clearly demonstrates improved robustness.", "relation_to_broader_scientific_literature": "The paper builds on existing literature addressing spurious correlations in VLMs, offering a zero-shot solution distinct from previous fine-tuning or LLM-based methods.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bt3TXRFYej", "reviewer": "Reviewer_USPN", "summary": "The paper introduces ERICT and ERICT-C, mitigate spurious correlations in vision-language models (VLMs) during zero-shot inference. These approaches aim to enhance model robustness by identifying invariant features within image tokens and focusing the model's attention on relevant regions through token masking. The methods are theoretically grounded and demonstrate significant improvements in performance.\n\n## update after rebuttal\n\nThe rebuttal resolves my concerns, I am inclined to accept this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1: However, the distinction between ERICT and ERICT-C needs clarification**\n\n**A1**: The key difference between ERICT and ERICT-C lies in the way they obtain the auxiliary embeddings. As shown in Step 1 of Figure 2, ERICT uses auxiliary prompts (e.g., \"bird in photo\") as input to the text encoder to obtain auxiliary embeddings. In contrast, ERICT-C constructs prompts using class names (e.g., \"a photo of a landbird\") as input to the text encoder. Then, ERICT-C ranks the embeddings output by the class names based on similarity and aggregates the top-K most similar embeddings to obtain the auxiliary embedding, where K is a hyperparameter less than or equal to the maximum number of categories in the dataset.\n\n> **Q2: The underperformance of ERICT-C in some scenarios requires further investigation.**\n\n**A2**: ERICT and ERICT-C each have their strengths and weaknesses under different settings, primarily due to their use of two different strategies (as shown in Q1) for obtaining auxiliary embeddings. The difference in how these embeddings are obtained leads to varying performance in the task.\n\n\n> **Q3: The choice of the top-3 strategy in ERICT-C lacks justification and the limitations mentioned in the text are not adequately addressed in subsequent sections.**\n\n**A3**: We apologize for any confusion caused to the reviewer. We have clarified the Top-K strategy. In the experiments, the value of k should be less than or equal to the number of categories in the dataset. For example, for the binary classification task on the Waterbirds dataset, K=2. Similarly, we have provided experiments with different values of K on three distorted versions of the Imagenet dataset, as shown below.\n\nThe limitation mentioned in the paper mainly refers to the suboptimality of the auxiliary prompt. In fact, we discuss this limitation in the \"Impact of the auxiliary prompt\" section of the ablation study (lines 400-412). How to obtain an auxiliary embedding with the best guiding significance is indeed a meaningful topic for further exploration. However, the main contribution of this paper lies in enhancing the model's robustness during the inference phase without relying on training, the assistance of LLMs, or group labels. We hope to address this limitation for future work.\n\n|  ViT-L/14 |Imagenet-1k |Imagenet-A|Imagenet-R|\n| :-: | :-: | :-: | :-: |\n|K=2|72.32|69.95|87.30|\n|K=3|72.15| 70.03| 87.19|\n|K=5|72.03| 69.76| 86.99|\n\n> **Q4: The paper could benefit from experiments on more data such as MetaShift, and Living-17 datasets, as well as referring to more debiasing algorithms.**\n\n**A4**: As suggested by the reviewer, we add extensive experiments comparing the MetaShift and Living-17 datasets. The experimental results are shown below. These experiments demonstrate the robustness of our method across different datasets. Please note that the code for \"Debiased Fine-Tuning for Vision-Language Models by Prompt Regularization\" mentioned has not been open-sourced. The experiments for CLIPood and additional baseline results are presented in Q5.\n\n|    ViT-B/16     |              | MetaShift |             |             | Living-17 |             |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n|  model  |      wg      |    avg    |     gap     |     wg      |    avg    | gap         |\n| ZSCLIP  |    87.56     |   94.96   |     7.4     |    31.5     |   93.7    | 62.2        |\n|  ERICT  |  **91.89**   |   95.38   |  **3.49**   | 38.2 |   94.1    | 55.9 |\n| ERICT-C | 90.27 |   95.33   | 5.06 |  **38.5**   |   94.2    | **55.7**    |\n\n> **Q5: The paper does not compare with ERM, AFR and CFR algorithms.**\n\n**A5**: We thank the reviewer for pointing out this issue. We have added detailed experiments comparing our method with extensive baseline methods. It is worth noting that although these baseline methods are training-based, our approach is training-free. However, in certain settings, our method outperforms these training-based methods, further demonstrating the robustness of our approach.\n\n|   ViT-L/14      |          |       | Waterbirds |       |       | CelebA |      |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n|         | Training | Wg    |    Avg     | Gap   | Wg    | Avg    | Gap  |\n| ERM     | Yes      | 57.9  |    97.6    | 39.7  | 30.4  | 94.6   | 64.2 |\n| AFR     | Yes      | 73.4  |    88.2    | 14.8  | 70.0  | 85.2   | 15.2 |\n| CFR     | Yes      | 88.2  |    96.8    | 8.6   | 84.8  | 87.8   | 3.0  |\n| CLIPood |   Yes    | 79.6 |    94.1    | 14.5 | 31.1 |  95.3  | 64.2 |\n|  ERICT  |    NO    | 61.2 |    74.1    | 12.9 | 85.0 |  88.6  | 3.6  |\n| ERICT-C | NO | 64.8 | 79.4 | 14.6 | 83.3 | 89.0 | 5.7 |", "claims_and_evidence": "The authors' claims regarding improved robustness and reduced spurious correlations are supported by experimental results. However, the distinction between ERICT and ERICT-C needs clarification, and the underperformance of ERICT-C in some scenarios requires further investigation.", "methods_and_evaluation_criteria": "The proposed methods aim to address spurious correlations by leveraging invariant features in image tokens. However, the distinction between ERICT and ERICT-C is not clearly articulated. The evaluation criteria (WG, AVG, Gap) are appropriate for assessing robustness.", "theoretical_claims": "The error probability bound theorem supports the effectiveness of the approach.", "experimental_designs_or_analyses": "The experimental designs are robust, with evaluations on multiple datasets and backbones. However, the choice of the top-3 strategy in ERICT-C lacks justification, and the limitations mentioned in the text are not adequately addressed in subsequent sections.", "relation_to_broader_scientific_literature": "The work contributes to the field of robust vision-language models by proposing a novel approach to mitigate spurious correlations without requiring additional training data or group labels.", "essential_references_not_discussed": "The paper could benefit from experiments on more data such as MetaShift, and Living-17 datasets, as well as refering more debiasing algorithm like [1][2]\n\n[1] Debiased Fine-Tuning for Vision-language Models by Prompt Regularization\n[2] CLIPood: Generalizing CLIP to Out-of-Distributions", "other_strengths_and_weaknesses": "Strengths:\n•\tThe approach is efficient as it does not require additional training or group labels.\n•\tThe paper provides visualizations and ablation studies that help understand the method's effectiveness.\nWeaknesses:\n•\tThe distinction between ERICT and ERICT-C is not clearly explained.\n•\tERICT-C underperforms ERICT in some cases as shown in Table 1, 2, 3, suggesting limitations in the auxiliary prompt strategy.\n•\tThe choice of the top-3 strategy in ERICT-C lacks experimental justification.\n•\tThe paper does not compare with ERM[1], AFR[2], and CFR[3] algorithms.\n[1]Principles of risk minimization for learning theory.\n[2]Simple and fast group robustness by automatic feature reweighting. \n[3]Calibrating multimodal representations: A pursuit of group robustness without annotations.", "other_comments_or_suggestions": "The paper presents a novel and effective approach to enhancing robustness in VLMs. To strengthen the work, the authors should:\n1.\tClarify the distinction between ERICT and ERICT-C.\n2.\tInvestigate why ERICT-C underperforms in certain scenarios.\n3.\tProvide justification for the top-3 strategy choice.\n4.\tAddress the limitations mentioned in the text.\n5.\tInclude results on additional datasets and compare with more algorithms.", "questions_for_authors": "see weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jtGwVeVKKZ", "reviewer": "Reviewer_89KE", "summary": "This paper presents ERICT, a novel method to enhance the robustness of vision-language models (VLMs) by mitigating spurious correlations at the inference stage. The approach identifies concept tokens to create a token-level mask, which is then applied to the vision encoder’s attention mechanism. Experimental results demonstrate that ERICT improves overall and worst-case performance, achieving state-of-the-art results.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1: Section 5 lacks a detailed discussion on disentangling invariant and spurious factors, requiring further exploration.**\n\n**A1**: Consistent with previous works[1-2], our theoretical analysis adopts the classic data assumption, which disentangles spurious datasets into invariant and spurious parts. In this paper, our method identifies concept tokens to effectively extract both invariant and spurious information.\n\n[1] Robust Learning with Progressive Data Expansion Against Spurious Correlation.NeurIPS 2023.\n\n[2] A Sober Look at the Robustness of CLIPs to Spurious Features. NeurIPS 2024.\n\n\n> **Q2: The experiments on different prompts were not presented.**\n\n**A2**: We thank the reviewer for pointing out this issue. The experiments in our paper used the commonly adopted prompt template \"a photo of a {}\". Additionally, we have conducted extensive experiments on CelebA to evaluate the performance of ERICT under different prompt templates, where  ``Prompt 80`` represents the average of 80 templates designed by OpenAI for CLIP.\n\n|ViT-L/14|Prompt template|WG|AVG|Gap|\n|:-:|:-:|:-:|:-:|:-:|\n|ERICT|a photo of a {CLASS}|85.00|86.43|1.43|\n|ERICT|Prompt 80|84.51|86.36|1.85|\n|ERICT|a {CLASS}|84.89|86.80|1.91|\n|ERICT-C|a photo of a {CLASS}|85.44|86.53|1.09|\n|ERICT-C|Prompt 80|84.48|86.16|1.69|\n|ERICT-C|a {CLASS}|85.00|87.81|2.81|\n\n> **Q3: The authors introduce a threshold l in Eq (5), but don't analyze its impact or the effect of different threshold methods.**\n\n**A3**: We further clarify the threshold ``l``. In this paper, ``l`` is dynamically calculated based on the sparsification process, considering that the number of invariant information tokens varies across different images. Each image in the task has a different threshold depending on the variation in spurious information (see Equations (4) and (5)). The key parameter influencing the sparsification process is the temperature coefficient, which we have already analyzed in the ablation study section (lines 385-420). To further demonstrate the effectiveness of this process, we provide comparative experiments on different threshold settings using the CelebA.\n\n|ViT-L/14||WG|AVG|Gap|\n|:-:|:-:|:-:|:-:|:-:|\n|ERICT|sparse|**85.00**|86.43|**1.43**|\n|ERICT|50%|76.67|88.01|11.34|\n|ERICT|80%|79.44|88.82|9.38|\n|ERICT-C|sparse|**85.44**|86.53|**1.09**|\n|ERICT-C|50%|78.33|87.76|9.43|\n|ERICT-C|80%|84.44|86.24|1.80|\n\n> **Q4: The discussion about the auxiliary prompt. Other worthy prompt forms.**\n\n**A4**: We sincerely apologize for not clearly expressing this point in the submitted paper. What we actually want to convey is that the auxiliary embedding is very important as it guides model inference. Specifically, ERICT utilizes superclass information, while ERICT-C employs a Top-K strategy to aggregate class prompt embeddings. Since they obtain auxiliary embeddings in different ways, their performance varies across different tasks. We do not want to emphasize that the \"superclass of the task category is better than the ground-truth category\", but to point out the limitations of the auxiliary prompt (which requires certain priors), and then introduce ERICT-C.  We will clarify this issue in the final version.\n\n> **Q5: The caption of Figure 2.**\n\n**A5**: Thanks for the reviewer's suggestion. Figure 2 mainly includes two steps: In Step 1, we construct an auxiliary embedding $z_t^a$ to identify tokens containing invariant information, obtaining a token-level mask. In Step 2, we apply this token-level mask within the attention mechanism of the vision encoder, making tokens containing spurious information \"invisible\" to the [CLS] token.\n\n> **Q6: The authors should clarify how the binary mask mitigates spurious correlations, particularly its impact on feature selection, information filtering, and decision-making.**\n\n**A6**: Thanks for the interesting question raised by the reviewer. According to our findings (line 178), the non-CLS tokens in the visual encoder capture local information. The original visual representation is obtained through the attention interaction mechanism between the CLS token and other tokens. By using a binary mask, we \"shield\" the tokens that focus on spurious information from the CLS token, so that the final representation no longer attends to spurious-related information (as demonstrated clearly in our T-SNE visualization). This prevents the model from making decisions based on spurious information.\n\n> **Q7: What are the underlying reasons for spurious correlation in VLMs?**\n\n**A7**: Thanks for your interesting question. In our view, although pre-trained VLMs use large-scale image-text pairs, they still fundamentally follow a supervised learning paradigm, which cannot fully avoid the influence of fundamental learning issues, a point supported by related work [3]. Unfortunately, real-world data are full of statistical biases.\n\n[3] Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning. ICML 2023", "claims_and_evidence": "In this paper, the authors argue that while fine-tuning methods are somewhat effective in mitigating the spurious correlation problem, they come with additional computational costs and rely heavily on the quality of prompts, without fully leveraging the vision modality. To address these limitations, the authors propose the ERICT method, which enhances robustness without requiring additional training, assistance from LLMs, or access to group labels. To validate the effectiveness of the proposed approach, the authors employ an error probability bound of the model. \n\nThe paper lacks an in-depth discussion on the disentanglement of invariant and spurious factors in Section 5. While it is intuitively feasible to decompose image samples into invariant and spurious factors, a more detailed exploration of the disentanglement approach is necessary.", "methods_and_evaluation_criteria": "The proposed method is generally reasonable and has been validated for effectiveness from both average performance and worst-case perspectives. Additionally, the authors have theoretically demonstrated that the method can reduce the model’s probabilistic error bound. However, there are still some aspects of the method that warrant further discussion.", "theoretical_claims": "The authors validate the effectiveness of the proposed binary mask approach by analyzing the model’s error probability bound. A thorough examination of the theoretical analysis in Section 5 and the corresponding appendix confirms its correctness.", "experimental_designs_or_analyses": "I have reviewed the authors’ experimental design. They conducted experiments on three widely used datasets with spurious correlations and compared their method against several state-of-the-art approaches from multiple perspectives. Additionally, visualization experiments were designed to further demonstrate the robustness of the proposed method. However, the experiments on different prompts, as shown in Table 5, were not presented.", "relation_to_broader_scientific_literature": "The key contribution of this paper lies in proposing a novel method to mitigate the spurious correlation problem in vision-language models. By directly applying a concept token mask during the inference stage, this method enhances the model’s robustness and generalization ability, overcoming the limitations of existing methods that rely on large-scale language models and labeled data. Therefore, it represents a novel contribution to the literature.", "essential_references_not_discussed": "To the best of my knowledge, the paper includes imporant related works.", "other_strengths_and_weaknesses": "Strengths\n\n- The proposed ERICT method mitigates the spurious correlation problem without the need for fine-tuning or additional complex prompts, reducing computational costs and dependency on prompt quality. \n\n- Extensive experiments demonstrate that ERICT improves overall model performance, including that of the worst-performing group, and achieves new state-of-the-art results, highlighting its effectiveness in addressing the robustness challenges of vision-language models.\n\nWeaknesses\n\n- The authors introduce a threshold $l$ in Equation (5), but do not further analyze it in the experimental section. It is recommended that the authors include two parts in the ablation study: (1) the impact of different threshold values on the experimental results; (2) the effect of different search threshold methods on performance.\n\n- In line 189, the authors mention that the auxiliary prompt is very important, but they do not further analyze why the superclass of the task category is better than the ground-truth category. Additionally, are there other forms of prompts worth further discussion and investigation?", "other_comments_or_suggestions": "The caption of Figure 2 should describe the specific functions of the two steps to help readers quickly understand the content of the figure.", "questions_for_authors": "- The authors should further clarify the rationale behind using a binary mask to mitigate spurious correlation issues in VLMs. Specifically, they should elaborate on how the binary mask influences feature selection, information filtering, and the model’s final decision-making process to enhance the understanding and persuasiveness of the proposed approach.\n\n- Pre-trained VLMs typically exhibit strong generalization capabilities but can still be affected by spurious correlations. What are the underlying reasons for this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nzD09vNgkK", "reviewer": "Reviewer_Cu15", "summary": "This paper introduces ERICT, a novel method to enhance model robustness by identifying concept tokens and mitigating spurious correlations at the inference stage. ERICT operates in two key steps: (1) identifying invariant concept tokens using auxiliary prompts to generate a token-level mask and (2) applying the mask to the CLS token's attention weights in the vision encoder, ensuring the model focuses on relevant image regions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1: The paper observes that “tokens whose semantics align more closely with the prompt tend to have lower similarity scores,” but provides only limited examples to support this claim. Given that this observation appears counterintuitive, stronger empirical validation is necessary.**\n\n**A1**: Thanks for the reviewer’s suggestion. We have provided additional image results (https://anonymous.4open.science/r/rebuttal-6618/more-image.pdf) from different datasets to further support this point. In each row, the leftmost image is the original image, followed by the score heatmaps and score maps for three different concepts in the image. These results further validate our findings.\n\n> **Q2: Based on this observation, the method should ideally select the top-k lowest similarity token embeddings. However, it seems to do the opposite, which requires further explanation.**\n\n**A2**: We thank the reviewer for pointing out this error. In Equation (6), $S_i$ should actually be $S_{i}^{'}$, with $S^{'} = Sort(-1 \\times S)$. In the implementation, consistent with the finding in line 178, we use sparsification to select tokens with lower scores to obtain the final visual representation. We will correct this error in the revised version. Additionally, we would like to re-explain the top-k strategy, which is used in ERICT-C to select potential class prompts for aggregation as auxiliary embeddings. In this process, we sort the inference results from vanilla CLIP and select the class prompts that are most similar to the image.\n\n> **Q3: The selection of auxiliary prompts appears to require domain knowledge, and ERICT-C is only evaluated using the top-3 version. Both aspects require a more in-depth investigation.**\n\n**A3**: ERICT uses the superclass of the task category as prior knowledge to compute the auxiliary embedding. This prior does not require domain experts and is not complex. It is also used in the baseline method, PerceptionCLIP.\n\nERICT-C, on the other hand, does not rely on any prior knowledge. It computes embeddings for each class prompt in the classification task and aggregates the top-k embeddings with the highest similarity to obtain the auxiliary embedding. To demonstrate the effectiveness of the top-k strategy, we presented results with top-3 in the paper on multi-category datasets such as ImageNet-R. Additionally, to further illustrate this, we have provided more results for different values of k.\n\n|   ViT-L/14   | Imagenet-1k | Imagenet-A | Imagenet-R|\n| :-: | :-: | :-: | :-: |\n| K=2  |    72.32    |   69.95    |   87.30 |\n| K=3  |    72.15    |   70.03    |   87.19    |\n| K=5  |    72.03    |   69.76    |   86.99    |\n\n> **Q4 Compared with existing works, this work focuses on zero-shot learning without the help of extra LLMs and labeled data. However, expert knowledge to design the auxiliary prompt is needed, which might not be trivial for all tasks. The alternative method, ERICT-C, is not well evaluated.**\n\n**A4**: ERICT does require the introduction of certain priors to use auxiliary prompts, but these priors are similar to the simple assumption of \"bird in photo\" in the waterbirds dataset, which is also used in the widely discussed work PerceptionCLIP in this field. At the same time, ERICT-C does not rely on any priors, addressing this limitation and demonstrating superior performance. We have also provided additional experiments (e.g., Tables 1, 2, 3, 6 in our manuscript and the table in Q5) to validate the effectiveness of our method.\n\n> **Q5: Compared with baseline method, ROBOSHOT, the method missed experiments on other datasets, e.g., PACS, VLCS and CXR14.**\n\n**A5**: We sincerely clarify that our paper primarily focuses on the issue of spurious correlations, while PACS, VLCS, and CXR14 are domain generalization (DG) benchmarks, not spurious correlation problems. Following the reviewer's suggestion and the experimental setup of ROBOSHOT, we conducted a comprehensive comparison on PACS, VLCS, and CXR14. The experimental results show that our method still demonstrates strong robustness on these DG benchmarks.\n\n|ViT-L/14||PACS|||VLCS|||CXR14(BiomedCLIP)||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Model|WG|AVG|Gap|WG|AVG|Gap|WG|AVG|Gap|\n|ZSCLIP|79.8| 98.1 | 18.3 | 4.2| 72.6 | 68.4 | 28.9 | 55.3  |26.4|\n|ROBOSHOT|83.9 | 98.1 | 14.2 | 12.6 | 71.1 | 58.5 | 41.6 | 56.2  | 14.6 |\n|ERICT|83.1| 96.4 | 13.3 | 39.8 | 73.2 | 33.4 | 42.3 | 56.7  | 14.4 |\n|ERICT-C|83.3| 97.2 | 13.9 | 44.3 | 78.9 | 34.6 | 45.6 | 57.2  | 11.6 |", "claims_and_evidence": "No, there are several unclear aspects that need further clarification: \n\n(i) The paper observes that “tokens whose semantics align more closely with the prompt tend to have lower similarity scores,” but provides only limited examples to support this claim. Given that this observation appears counterintuitive, stronger empirical validation is necessary. \n\n(ii) Based on this observation, the method should ideally select the top-k lowest similarity token embeddings. However, it seems to do the opposite, which requires further explanation. \n\n(iii) The selection of auxiliary prompts appears to require domain knowledge, and ERICT-C is only evaluated using the top-3 version. Both aspects require a more in-depth investigation.", "methods_and_evaluation_criteria": "Yes. The proposed method is pretty straightforward and should be easily used to enhance the zero-shot performance of VLMs.", "theoretical_claims": "Yes, there is a simple proof to support the assumption, which is correct.", "experimental_designs_or_analyses": "Yes. The main experiments should be fine. However, some ablation studies are missed. For example, how to select a better auxiliary prompt and ERICT-C with diverse top-k.", "relation_to_broader_scientific_literature": "Compared with existing works, this work focuses on zero-shot learning without the help of extra LLMs and labeled data. However, expert knowledge to design the auxiliary prompt is needed, which might not be trivial for all tasks. The alternative method, ERICT-C, is not well evaluated.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths: \n\n+The proposed method is simple and elegant. \n\n+The experiments on selected datasets show the effectiveness of the method. \n\n \n\nWeaknesses: \n\n- Several unclear aspects exist in this work as mentioned before. \n\n- Compared with baseline method, ROBOSHOT, the method missed experiments on other datasets, e.g., PACS, VLCS and CXR14.", "other_comments_or_suggestions": "No.", "questions_for_authors": "My questions have been mainly mentioned above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received consistently positive reviews. The initial concerns were primarily related to the clarity of the experimental setup and implementation details. The rebuttal effectively addressed these issues.

As a result, the AC concurs with the reviewers and recommend acceptance. It is encouraged to incorporate the clarifications provided in the rebuttal into the camera-ready version to further strengthen the paper.
Decision: Accept (poster)