Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition
Authors: Xinyu Tian
Shu Zou
Zhaoyuan Yang
Mengqi He
Jing Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Few-shot Adaptation
Prompt Learning
Vision-Language Models
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Few-shot adaptation for Vision-Language Models (VLMs) presents a dilemma: balancing in-distribution accuracy with out-of-distribution generalization. Recent research has utilized low-level concepts such as visual attributes to enhance generalization. However, this study reveals that VLMs overly rely on a small subset of attributes on decision-making, which co-occur with the category but are not inherently part of it, termed spuriously correlated attributes. This biased nature of VLMs results in poor generalization. To address this, 1) we first propose Spurious Attribute Probing (SAP), identifying and filtering out these problematic attributes to significantly enhance the generalization of existing attribute-based methods; 2) We introduce Spurious Attribute Shielding (SAS), a plug-and-play module that mitigates the influence of these attributes on prediction, seamlessly integrating into various Parameter-Efficient Fine-Tuning (PEFT) methods. In experiments, SAP and SAS significantly enhance accuracy on distribution shifts across 11 datasets and 3 generalization tasks without compromising downstream performance, establishing a new state-of-the-art benchmark.

Published as a conference paper at ICLR 2025
BLACK SHEEP IN THE HERD: PLAYING WITH SPURIOUSLY CORRELATED ATTRIBUTES FOR VISION-LANGUAGE RECOGNITION
Xinyu Tian', Shu Zou', Zhaoyuan Yang", Mengqi Jing Zhang' Australian National University, 2GE Research
ABSTRACT
Few-shot adaptation for Vision-Language Models (VLMs) presents dilemma: balancing in-distribution accuracy with duitt--diitributtin generalization. Recent research has utilized low-level concepts such visual attributes enhance gen- eralization. However, this study reveals that VLMs overly rely on a small subset of attributes on srecision-making which co-occur with the category but are not inherently part of it, termed spuriously correlated attributes. This biased nature of VLMs results in poor generalization. To address this, 1) we first propose SPURI- OUS ATTRIBUTE PROBING (SAP), identifying and filtering out these problematic attributes to significantly enhance the generalization of existing attribute-baxee methods; 2) We introduce SPURIOUS ATTRIBUTE SHELLDIGG (SAS), plug- and-play module that mitigates the influence these attributes, seamlessly inte- grating into various Parameter- Effficiiett Fine-Tume Tuming (PEFT) methods. In experi- ments, SAP and SAS significantly enhance accuracy on distribution shifts across 11 datasets and 3 generalization tasks while preserving downstream performance, establishing a new state-of-the-arr benchmark. The code will be available here
INTRODUCTION
The emergence of large-scale pre-trained Vision-Language Models (VLMs) (Radford et al. 2021
Li al., (202a)) bridges the gap between images and texts. However, conventional fine-tuning of these models entails significant computational burdens, leading to Parameter-Efficient
Tuning (PEFT), such as prompt tuning (Khattak et al., 2023a Zhou et al. 2022a adapters (Sung et al.,
2022
Gao et al., 2024) and LoRA (Hu et al.) 2021 Dettmers et al.
2024). With PEFT, requir-
ing approximately 1% of model parameters, one may adeptly adapt to downstream tasks, achieving
comparable or even superior performance to full fine-tuning (Liu al. 2021). Yet, recent studies have revealed that in few-shot scenarios where observed samples are limited, PEFT struggles to generalize to out-of-distributiin datasets and may compromise the VLMs' strong zero-shot capa- bility (Zhou et al.) [202b]] Yao et al., 22023 Bulat & Tzimiropoulos 2023). This creates a trade-off where individuals aim for strong performance downstream tasks while endeavoring to maintain the ability of VLMS to handle distribution shifts.
In response the above-mentioned challenges, various strategies have been proposed such cat- egory conditioning (Zhou et al.) 2022b] Yao et al. 2024), prompt regularization (Yao et al. 2023 Bulat & Tzimiropoulos [2023] Khattak al. (023b) and training-free adaptation (Udandarao al. 2022 Zhang et al.) 2022b). Recently, it has been discovered that incorporating descriptors, also
known as visual attributes during training can significantly improve the accuracy of adapted mod- ules on out-of-distributiin
datasets (Zhang et al., 2024c Liao et al., 2024, 'Tian et al. 2024: Ma al. 2023 Liu et al., 2024b). The motivation behind these works is attributes, as lower-level concepts, are more likely to establish connections to unseen categories compared to the high-level names (Zhang et al.) 2024cc Tian et al.) 2024). These methods can be divided into two types: one involves generating visual attributes for the target category using Large Language Models (LLMs) (Tian et al. 2024 Ma et al. 2023 Liu et al. 2024b), while the other entails searching for optimal attributes from a pre-defined vocabulary that maximizes semantic similarity (Zhang et al.) 2024ch or training accuracy (Liao et al. 2024). Yet, a commonality among them their reliance on the set of generated attributes, i.e., the attribute pool.

---

We rank attribute weights Published as a conference paper at ICLR 2025
E52
Figure 1: The phenomenon of Black Sheep in the Herd.
0.3
rechud
0.70
072
500
6.7
ander
675
0.35
on VLM predictions using CBMs, with yellow and pur- ple bars to denote spurious and core attributes respec- tively. In (b), we observe that for vanilla VLMs, 2 out of the top-3 are spurious attributes, heavily influencing decisions. In (c), SAS mitigates this by suppressing the influence of spurious attributes.
snarre
6.41
bdsrr
0.23
1.5
Betk 1//
fireboat
0.2
0.4
0.6
0.6
0.2
0.4
0.6
0.0
7004
0.25
0.00
0.24
0.7
075
guaard
2.24
See
pun
Usk
0.44
park bench
02
0.4
08
08
(a) prediction
(b) CLIP
CLIP + SAS
While promising, the limitations of this line of work have been underexplored. Initial suspicions emerge from Roth et al. (2023), which find that in certain cases, replacing attributes with random sequences does not lead to notable performance decline. Subsequently, An et al. (2023) discover that VLMs sometimes disregard the presence of attributes, leading to minimal gains. This prompts us to inquire: Are attributes truly dependable? If so, then whence do these failure cases arise?
To tackle the aforementioned inquiries, we conduct a manual examination of the attribute pool gen- erated by existing methods. We stumble upon an often overlooked fact: while most attributes accu- rately depict the intrinsic characteristics of the target category, there exists small subset of attributes that co-occur with the category but are not part of it, leading to strong spurious correlations. For instance, when querying LLM with what does a mountain bike look like we receive attributes like wheels, handle, and basket, yet unexpected attributes such as trees and road also emerge. This phenomenon is also observed in vocabulary- -base methods, where attributes are chosen based on in-distribution samples. Inspired by (Singla & Feizi (2021). we refer to the former core attributes and the latter as spuriously correlated altribute
Building upon the motivation to enhance generalization, natural idea is to strive for "pure" at- tribute pool that accurately reflects the true characteristics of the category. Therefore, we conduct a simple experimental study based on existing methods (Zhang et al. 2024c] Tian al., 2024) where we manually identify attributes that might lead to spurious correlations with the target category and remove them. Despite the small proportion of these attributes (< 7%), we observe a significant im- provement in attributes affect VLMs, we employ concept bottleneck models (CBMs) (Koh et al. 2020; Yang et al.) 2023), well-established method for interpreting and ranking attribute weights in decision- processes. Our analysis reveals that despite their small representation in the overall pool, spurious attributes exert a significant influence, particularly among the top-3 attributes on decision recision-making as depicted in Fig. [ We term this phenomenon Black Sheep in the Herd since 1) spurious attributes act the "black sheep" within the pool, constituting small fraction; 2) nevertheless, this small fraction significantly impacts the generalization ability of VLMs.
generalization accuracy. To gain deeper insights into how spurious
We could utilize the aforementioned manual inspection to assist existing attribute-based methods. However, this remains a mere fancy dream since manually identifying spurious attributes in the pool is prohibitively expensive. This motivates us to devise new method for generating pure pool, one that contains only those core attributes helonging to the category. Hence, we propose SPURI- OUS ATTRIBUTE PROBING (SAP), an approach derive an attribute pool where core and spurious attributes are clearly separated. SAP integrates Multi-modal Large Language Models (MLLMs) and Concept Bottleneck Models (CBMs) to tackle this challenge. Leveraging MLLMs, SAP ini- tially distinguishes core attributes from non-core counterparts, and then CBMs prioritize the latter by selecting those with significant impact on model decisions as spurious attributes. SAP com- plements existing attribute-based methods and, to the best of our knowledge, is the first approach to identifying spurious attributes in open- world settings without explicit human supervision.
Despite the effectiveness of SAP, it faces limitations: It may prevent the presence of spurious at- tributes in the language branch, yet it cannot stop the model from learning spurious features. This ex-
We refer spuriously correlated attributes to spurious attributes for brevity.
2

---

Published as a conference paper at ICLR 2025
tends the scope beyond suribute-based' methods, and will result in poor generalization across PEFT family. Therefore, we propose SPURIOUS ATTRIBUTE SHIELDING (SAS). a module to mitigate the influence of spurious features which can be seamlessly integrated into arbitrary PEFT methods. Specifically, SAS introduces subsidiary task by creating a set of pseudo categories defined by spu- rious attributes alongside the real ones, allowing VLMS to distinguish between them. For instance, if streetlight is considered a spurious attribute for the target category vehicle, we establish a separate pseudo category exclusively for sireetlight and discern between the two, thus decreasing the de- pendency on strectlight for identifying vehicle. The experiments show that by combining SAS into existing PEFT approaches, the accuracy under distribution shifts is significantly improved, reaching new state of the art.
In summary, our main contributions are as follows:
Despite the promise of visual attributes in various applications, discover group of black sheep, ie., spurious attributes, on which VLMs inherently heavily rely, thereby lead- ing to poor generalization and robustness.
We introduce SPURIOUS ATTRIBUTE PROBING (SAP), aiming identify and eliminate these problematic attributes, thereby substantially improving the generalization of current attribute based methods.
We present SPURIOUS ATTRIBUTE SHEELDING (SAS), a plug-and-play module seamlessly integrating into various PEFT methods to mitigate the influence of spurious attributes on predictions.
2 RELATED WORK
Vision-Language Models. Recently, it has been discovered that associating text and images for pre-training, instead of using images alone, enables powerful zero-shot capability, leading to VLMs. Initially, simple dual-tower structures are employed, where the representations of the two modali- ties are modeled by separate encoders and connected via contrastive learning, i.e., CLIP (Radford itt al. (021). Subsequently, more works have been built upon this foundation. For instance, Li et al. (2022a) bridge two encoders by fusion for better cross-modality interactions, [Li et al. (2023b) em- play masked image modeling to achieve a trade-off between accuracy and training time, and Li et al. (2022b) incorporate visual detection and grounding in pre-training for object-level reasoning. For more information, we refer to /Zhag et al. 2024a) for detailed survey of recent VI.Ms.
Parfinter-Efffcient Fine-Tuning. As pre-trained models grow larger, traditional fine-tuning de- mands significant resources, leading to PEFT (Hu et al.] [001]] Liu et al., [021 Lester et al., 2021 Houlsby al. 2019). However, PEFT is double-edged sword: while adeptly adjusts to down- stream tasks, also brings poor generalization to the open world, inspiring various current remedies.
For instance, category conditioning (Zhou et al. 2022b Yao et al., 2024) infuses category-aware knowledge for discriminative and generalizable learning, prompt regularization (Yao et al.) 2023 Bulat & Tzimiropoulos 2023) confines learnable prompts to corresponding textual features, and training-free adaptation (Udandarao et al., 2022 iZhang et al., 2022b) eschews gradient- opti- mization to prevent overfitting. Recently, another line of work leveraging visual attributes has shown promising results, achieving state-of-the-art performance in various generalization tasks.
Visual Attributes for Recognition, The initial exploration of attributes for recognition begins in zero-shot settings (Menon & Vondrick 2003] Pratt et al.) (2023), where individuals utilize attributes generated by LLMs to offer more expressive and accurate descriptions. Subsequently. it observed that training VLMs to grasp fundamental concepts like visual attributes aids in generalizing to un- seen data, prompting surge in attribute- brusbbd methods. For instance, TTian et al. (2024) appends attributes category names, [Lioo et al. (2024) initializes learnable tokens as attribute embeddings, Ma et al.) (2023) adopts a more aggressive approach by replacing category names entirely with attributes. Additionally, Wei et al. (2019) utilize adversarial training to learn attribute-object com- position, while Huang et al. (2024) and Wang et al. (2015) improve the model's fine-grained under- standing by building multi-gramt granularity and hierarchical attributes. Nonetheless, recent studies have noted decline in attribute effectiveness in certain scenarios (An al., 2023). sometimes reducing to a mere ensembling effect (Roth et al., (023). This paper delves into the issue, attributing it to spurious attributes, and proposes two plug-and-plas approaches complement existing methods.
3

---

Published as a conference paper at ICLR 2025
Spurious attributes arise from model debiasing (Seth et al. 2023| Chuang et al. 2023; Berg et al. 2022), defined as those likely to co-occur with the object but not part of (Singla a & Feizi 2021]. Although well-known term, it remains underexplored due
Spurious Attribute Identification.
to the difficulty in identification. The initial endeavor by |innlla & Feizi 2021) involves manually
labeling to identify spurious attributes. Similarly, WWong et al. (2021) integrates human supervision with sparse linear layers to mitigate labor expenses. Others identity spurious attributes by analyzing their properties. For instance, [Wu al.] (2023) observe that spurious attributes exhibit instability across data environments and introduce concept sensitivity for identification. Conversely, [Teotia al. (2022) train an attribute probing network to predict spurious attributes. Recently, the work most akin to ours, Adila et al.] (2023), utilize LLMs to derive harmful insight representations by comparing differences between concepts. However; the inference complexity of this method esca- lates exponentially with the number of concepts, restricting its application to small-scale datasets. In contrast, our proposed SAP 1) necessitates neither human labeling nor a training process, rendering it extremely cost-effective; and 2) is scalable to any large-scale dataset, e.g., ImageNet.
Spurious Correlation Mitigation. Current spurious mitigation methods can be mainly categorized into two types. The first assumes that spurious attributes within the dataset are either unknown or complex, employing various proxies to mitigate spurious correlations. For instance, Xu et al. (2020); Yao al.] (2022); Han al. (2022) introduce augmentation via domain mix-up learn invariant features, while Li et al. (2022c); Zhang et al.] (2022a); [Utama et al.] (2020) advocate for instance reweighting to emphasize hard samples. Others calibrate biased representation through contrastive learning YYou et al.] 2024| Zhang & Ré. (022). The second type explicitly assumes that spurious correlations arise from known attributes (Chuang et al. 2023 Berg al., (022). For
instance. Wu et al. (2023) balance training data by swapping spurious concepts among categories, whereas Adila et al. (2023) calibrate embeddings by removing spurious representations. In contrast, SAS belongs to the latter, where the attribute prior is known, and thanks to the effectiveness of SAP, may accurately mitigate spurious correlations caused by identified spurious attributes. For further details, quantitative comparison between SAS and related works is provided in Supp. Mat. B
3 METHOD
3.1 PROBLEM SETUP
We assume the training set of pairs D c),, where x X and C represent the image and ground truth label, respectively. The attribute-based methods aim to construct a category-wise prompt to = f(c, P) such that the conditional distribution of the prediction y given a is modeled as exp(>(ór(x), (1,))//
P(yl=) Σ cxp(s(()(()) eEc
(1)
where Φi and Φr represent the vision and language encoder, respectively, s(---) indicates the simi- larity function, and 7 is the temperature scaler. P = [Ac c E C} is an attribute pool generated by A. = =H((c) such that A. = {ac: az.... ai Depending on previous work, 24 could be a LLM, thus H(c) is a set of LLM prompts incorporating the category name of U could also be large vocabulary, such that H(c) becomes key to search for the semantically related attributes. There- fore, .( is denoted as concatenation function to integrate the category name and corresponding attributes together. The optimization objective is typically cross-entropy loss Lee.
It's important to mention that in this work, we refrain from specifying particular learnable parame- ters; they could encompass learnable prompts, adapters, or LoRA. Our goal is to ensure the versatil- ity of our hlug-and-play method across various PEFT approaches.
3.2 MOTIVATION
We present the motivation of this work by revealing an overlooked fact: the attribute pool in cur- rent methods are not purely aligned with the intrinsic semantics of categories. Some attributes are spurious, co-occurring with categories but not inherently linked to them. To investigate the impact of these "black sheep", we conduct simple experimental study. Specifically, we manually traverse the attribute pool P across various methods and identify spurious attributes within. We use a con- ventional method following (Singla & Feizi (202T) with simplistic version. Given the category e,
4

---

Published as a conference paper at ICLR 2025
FGVCAircraft
SUN397
1lowers102
DTD
Average
Method
Base
New
SR
Base
New
SR
Base
New
SR
Base
New
SR
Basc
New
SR
CPL
42.27 38.85 5.43 81.88 79.65 6.61 98.07 80.43 5.71 80.92 62.27 5.13 75.79 65.30 5.72
CPL. SA
42.62 41.33
82,14 82.36
98.35 82.16
81.62 64.77
76.18 67.66
ArGue
41.29
38.80
5.13
81.89 80.48
6.45 98.62
77.96
6.69 80.33
67.03
5.97
75.53
66.07 6.06
ArGue*
41.34
39.34
4.86
81.97 80,62 6.11
98.58
78.11 6,44
80.41 67.26 5.62
75.58
66.33 5.76
ArGue
SA
41.55
41.60
82.33 81.94 98.73 78.75 80.79 68.47
75.85
67.69
Table 1: The results on base-to-new generalization before and after removing spurious tributes (SA) from the pool. We report accuracy on base and new categories, and spurious rate (SR), which refers to the proportion of spurious attributes to the entire pool.
we randomly sample 5 images from the shots and visualize the heatmap. For specific attribute at. we determine whether it is a part of the main object, or separate objects in the background based on the sampled images with the heatmap activations.
Upon identifying these spurious attributes, we remove them from the pool and compare the changes in their generalization capability before and after elimination. The experiment is evaluated on base-to-new generalization, following the outlined settings in Section 4 As baselines, we select and LLM-
CPL (Zhang et al., 2024c) and ArGue (Tian et al., 2024), representing vocabulary- assisted methods, respectively. Additionally, we consider variant, ArGue where modify the LLM prompts to reduce the likelihood of spurious attribute occurrence. For instance, we append an additional instruction focus mountain bike itself to the original prompt. Further details are provided in Supp. Mat. A
Removing spurious attributes significantly enhances generalization. While most attributes contribute positively to generalization, spurious attributes are exceptions to this trend. Remov- ing these exceptions leads to notable increase in accuracy on the new category on average (65.30% + 67.66% for CPL and 66.07% + 67. 67.69% for ArGue), without compromising accu- racy on the base category. This phenomenon is aptly described as Black Sheep in the Herd since 1) spurious attributes constitute only a small portion of the pool (< 7%); 2) yet this small portion significantly impacts the generalization ability of VLMs.
VLMs heavily rely on spurious attributes for predictions. To deepen our understanding of this phenomenon, we use concept bottleneck models (CBMs) (Koh et al., 2020) determine attribute weights in model decision-making. In Fig. T attributes are ranked by weight from high to low. Among the top-3 attributes influencing predictions, spurious attributes occupy two positions. For instance, in predicting fireboat, VLMs heavily rely sea and lake as crucial concepts, while for park bench, grass and path are primary indicators. This implics that 1) VLMs may exhibit insensitivity to the presence of core attributes; 2) directly adapting to downstream tasks may heavily rely on spurious features for predictions. In fact, this also aligns with findings from concurrent work (Wang et al., 2024), which indicates that VLMs are more susceptible spurious features compared to unimodal architectures.
In addition the above observations, this prompts us consider several questions.
Where do spurious attributes come from? In the case of LLMs, this phenomenon may be at- tributed statistical bias in their large-scale training data. In practical scenarios. when describing a complex object, there may be a tendency to focus more on its accompanying scenes and asso- ciated elements rather than its core components. However, these accompanying elements may not contribute to a VLM's generalizable understanding of specific category. Conversely, regarding vocabulary- based methods, their attribute selection heavily relies on in in-distribttion samples, and this preference for attributes may be also detrimental to generalization.
Is there better way to identify spurious attributes? While manually purifying the attribute pool may enhance generalization, it faces two primary challenges: 1) scalability issues as the dataset size grows, and 2) it is a simple solution lacking quantitative assessment of the spurious correlation of each attribute, potentially leading to false positives, i.e., attributes that co-occur with the cate- gory merely by chance. Hence, we also experiment with an LLM- assisted variant called ArGue*, which adjusts the LLM prompts encourage stronger focus the category itself. However, as demonstrated empirically in Table | the reduction in the spurious rate is modest (6.06% + 5.76%), resulting in only marginal gains (66.07% 66.33%).
5

---

Published as a conference paper at ICLR 2025
3.3 SPURIOUS ATTRIBUTE PROBING
Motivated by the above considerations, we introduce SPURIOUS ATTRIBUTE PROBING (SAP), an approach to creating a comprehensive attribute pool where spurious and core attributes are distinctly separated. Initially, SAP utilizes Multi-modal Large Language Models (MLLMs) to differentiate attributes belonging to the target category, distinguishing core attributes from non-core counterparts. To determine if the coexistence of the latter with the category is coincidental or correlated, Concept Bottleneck Models (CBMs) gauge their impact on VLMss decision- makiing, with high-influence ones being identified as spurious attributes. By leveraging SAP, pure and robust attribute pool is achieved, significantly improving the generalization of existing attribute-based methods.
Prompting MLLMs. Here we assume U a MILM, differing from prior methods that only accept textual prompts. concurrently processes both prompts and images as input such that A = u(X,, H(c)). where X. represents training images labeled with and H(c) is set of chain-of- thought prompts probing two aspects: core attributes and non-core counterparts. Specifically, we design three question formats:
Q1: List all the visual cues you see in the photo:
Q2: Are the objects you list part of _
Q3: Describe - in the photo in details:
Combining QI and Q2 helps identify non-core attributes in the images, while Q3 provides detailed core attributes belonging to the category. Empirically, we'll use multiple templates for each question to ensure thoroughness. Upon reformulation, we derive A.- = A U At.. where A. denotes non- core attributes, and At represents the core ones.
Finding spurious attributes. Given non-core attributes A,, use their weights on model predic- tions as a proxy to indicate the extent of spurious correlations. We use a CBM (Koh et al., (020) achieve this goal. Specifically, we construct a bottleneck embedding
g. against the at- tribute pool P. where N = |C] x J indicates the total number of attributes in the current pool, each row & Rd indicates the feature of the corresponding attribute, and d is the feature dimension. In other words, & is a feature matrix that concatenates attributes across all the categories together. The procedure of CBMs is to combine two functions to make the prediction:
h(g(0r (x), E)), where g: Rd x IR Nxd + N measures the score between the image feature and every attribute in the bottleneck, h: RN + produces the final prediction based on the score vector. Following Yang al. al. (023),
set the score vector g the dot product of the features between two modalities g(or(z), 8) = oe (x)- - and h the linear projection with a learnable weight matrix W E IR CC|x such that h(g; W) = so roftmar(g- WT). Intuitively, Wiy indicates the impact factor of jth attribute, i.e,, af on the prediction i. To learn the weights of attributes on predictions, cross-entropy loss is typically employed.
For non-core attributes, a high weight indicates a strong correlation with the category, whereas a low weight suggests that its presence might be coincidental. Thus, natural idea to confirm spurious attributes is by thresholding 7. Formally, for specific prediction i, the spurious attributes are defined as
A = {a ĀJ Wiy 2
There is trade-off in choosing 7. If y is too large, Conversely, if it is too small, a large number of false positives may be introduced. Additionally, we observe significant variability in attribute weight distributions among different categories, posing challenges in identifying spurious attributes with a uniform threshold. Creating manual threshold for each category is prohibitively expensive. Henee, we introduce adaptive strategy. Given prediction c, we select The as the lowest weight of A+ such that non-core attributes with weights higher than any of the core attributes are considered spurious. This ensures flexible selection of spurious attributes, greatly aiding SAS introduced in Section 3.4 to be discussed next.
spurious attributes may be overlooked.
3.4 SPURIOUS ATTRIBUTE SHIELDING
SAP complements existing attribute- based methods by screening out spurious attributes from the pool, while it does not prevent the PEFT family from learning spurious features in the training im-

---

Published as a conference paper at ICLR 2025
Steel
beiret
bealtt
Visua
Visual
Adopt
mountain bike
threshalding
found attributes
(a) spurious
CLIP's Testual
CLIPs Textual Enteder
photo (spurious
The photo a Espurious
OR
target categentes
pawudl catagoriea
objective
secondary sbjectivo
(b) psoudo catagory construction
images
(c) subsidiary task
Figure 2: The overview of SAS. In (a), we generate and identify spurious attributes with SAP, In (b), we construct pseudo categories by synthetic data (SD) retrieval (LAION). In (c), apart from the main objective (i), e.g., cross-entropy loss, we introduce an auxiliary subsidiary task (ii) for learning robust features.
ages. Hence, we propose SPURIOUS ATTRIBUTE SHIELDING (SAS), plug-and-play module to be seamlessly integrated into arbitrary PEFT methods by mitigating the influence spurious features. Building upon the spurious attributes detected by SAP, SAS introduces subsidiary task by structing a set of pseudo categories alongside the real one and let VLMS differentiate among them. This auxiliary learning objective effectively prompts VLMs to learn robust features rather than ones referred by these spurious attributes. For instance, if strectlight. is a spurious attribute for the cat- egory vehicle impacting decision-making significantly, we introduce pseudo category specifically for streetlight and differentiate between the two, thereby reducing the reliance of streetlight when identifying vehicle. Fig. demonstrates the overall pipeline of SAS.
Formally, given category we establish set of pseudo categories To with constructed images {&²s, | E J.). Thus we define a subsidiary dataset D. = {(z, v) x E & U.. and Te U {c}}. We aim to optimize the following
exp(
Cpse Σ
E log E exp(s()r(z), dz (t,))//
(3)
CEC (x.
JEJ.U(e)
That is, we introduce additional cross-entropy loss for classifying between each target category and its corresponding pseudo categories, which are defined by spurious attributes. This can be viewed as subsidiary task, aimed at reducing reliance on spurious attributes while achieving correct classification in the downstream task. When integrated with existing methods, we introduce scaler À to balance the importance of Lpsc: Liot Lee + Xppee-
A natural question ask is: how construct & such that the adapted modules may effectively distinguish spurious attributes from the target categories? In this work, we introduce two approaches.
Synthetic Generation. We create pseudo categories using synthetic data by leveraging the text- to-image model Stable Diffusion (SD) (Rombach et al.) 2022). We consider two key factors: 1) diversity: Our goal is for pseudo categories fully represent the features of spurious attributes. To achieve this, we use LLMs to generate various prompts, which are then used inputs to SD to produce range of images. 2) purity: the constructed images contain not only spurious attributes but also unexpected elements, i.e., noise attributes, these noise attributes may create new shortcuts, affecting the effectiveness SAS. Empirically, selecting the top-k images that are most similar to the corresponding spurious attribute can help reduce the presence of noise attributes. Further details are Supp. Mat. A
Pretraining Retrieval An alternative way is to gather image samples from pre-training data such as LAION-5B (Schuhmann et al. 2022), a publicly available subset CLIP's s pre-training datasets. We captions proxy to efficiently determine semantic similarity between pre- mre-training images and spurious attributes. Finally, we select the top-k matches to the spurious attributes to create the pseudo categories.

---

Published as a conference paper at ICLR 2025
CoCoOp
$2.5
KyCoOp
PeompiiCC
MaPLe
0X0 ac0eraa
LASP
CNL
MAP
CLIP-Adaper
Tip-Adaperr
Zaro-chat
Raatiie:: SAP
(a) base-to-new generalization
cross-dataset transfer
(6) domain genemlization
Baotire
Figure 3: The average results of three generalization tasks over 11 datasets. The x-axis and y- axis represent in-distribution/bas accuracy and cul-of-distrrbuttionneww accuracy, respectively. We present the out-of- out-of-distribution accuracy of vanilla CLIP horizontal bar represent the zero-shot capability. The detailed numerical results are provided Supp. Mat.
ImageNet
OxfordPets
FGVCAireraft
Method
Test
Counter
Test
Counter
Test
Counter
PSRC
72,46
63.06
87.51
74.08
39.72
21.35
PSRC + SAS
73.82
68.55
88.27
77.17
39.54
26.43
TCP
71.54
60.17
89.11
72.84
38.30
23.61
SAS
72.81
64.50
89.86
76.14
39.23
28.44
CPL
69.08
64.75
90.32
79.91
40.43
27.65
CPI. SAS
70.15
67.98
90.12
83.45
40.64
32.12
(a) trinnnn
(b) test
(c) counter
Figure 4: Example samples from test set and counter group. The samples from counter group do not contain spu- rious attributes, e.g., ice or sky.
Table The results for standard few-shot classifi- cation on test set and counter group, respectively. Essentiallyy counter group is subset of test set where spurious attributes are removed.
EXPERIMENT
Task Setting. Following previous work (Khattak et al. 2023a Zhou et al. 2022b; Khattak et al., 2023b), the experiment is conducted on base-to-new generalization, cross-dataset transfer and do- main generalization. For base-to-new generalization, the datasets are equally divided into base and new categories, where the model trained on base categories and evaluated on unseen ones. For cross-dataset transfer, the model will be trained large-scale dataset, and generalized across var- ious other datasets. For domain generalization, the model will be transferred from an in-distribution dataset several variants.
Datasets, For base-to-new new generalization, we employ 11 datasets, including ImageNet (Deng et al, 2009). 1a (Fei-Fei al. 2004), OxfordPets (Parkhi et al.; 2012). StanfordCars (Krause et al. 2013), Flowers 102 (Nilsback & Zisserman 2008), Food101 (Bossard et al. 2014)) FGVCAir- craft MMaji al. 2013), SUN397 (Xiao al. 2010). UCF101 (Soomro al. 2012). DTD (Cimpoi ct al. 2014) and Eu (Helber et al. 2019). For cross-dataset transfer, we train models on Im- ageNet (Deng al. 2009), and evaluate on the remaining datasets mentioned above. For domain generalization, we designate ImageNet as the in in-distribution dataset, with four out-of tiatof-dditributton variants encompassing ImageNetV2 (Recht et al.) 2019). ImageNet-Sketed. (Wang al.) 2019), ImageNet-A ( (Hendrycks et al. (021b) and ImageNet-Ri (Hendrycks et al. (22 The experiments are carried out in the few-shot setting, where we randomly sample shots for each category to compose the training set.
Baselines. We consider various PEFT approaches. Specifically, for prompt tuning, we consider category conditioning including CoCoOp (Zhou et al. 2022b) and TCP (Yao et al. (0024). regular- ization techniques encompassing KgCoOp Yao et al. 2023). LASP (Bulat & Tzimiropoulos 2023) and PromptSRC (Khattak al., 2023b), attribute-based methods such as CPL (Zhang et al. (224c), ArGue (Tian et al. (0244) and MAP (Liu al. 2024b). We also consider multi--mdal prompt tuning, Le.. MaPLe (Khattak et al. 2023a). Besides, CI.IP-Adapeer (Gao al. 2024) and its training-free
8

---

Published as a conference paper at ICLR 2025
Original
CLIP
SAA
Original
CLIP
+SAS
Original
CLIP
+SAS
(a)
chocolate cake
(b) personal laptop
(c) street sign
Figure 5: The saliency map of VLMs with and without SAS. From left to right we show three
example categories, including chocolate cake, personal laptop, and street sign.
#p
0.0 0.2
0.4
0.6
0.8
1.0
Ada.
Base 83.10 83.47 83.41
83.59
83.64
83.51
83.68
Base 82.95 83.16 83.53
83.43
83.28
83.23 83.64
New 75,05 75.89 76.91 77.19
77.36
77.41 77.23
New 76.21 76.60
76.41
76.01
75.30
74.82
77.36
HM 78.87 79.50 80.03 80.26 80.38 80.34 80.32
HM 79.44 79.75 79.81 79.55 79.09
78.80
80.38
Table 3: Varying number of SD prompts on base-to-new generalization. The results are av- eraged across 11 datasets and 11 baselines.
Table 4: Varying 7 on base-to-new generaliza- tion. We experiment with fixed values, juxtapos- ing them against the suggested adaptive strategy.
Method
Epoch
Time
Accuracy
Gain
Table 5: The efficiency of SAS with and without selective trick. We eval- uate in terms training time and ac- curacy gains given the same number of epochs. We opt for two time-intensive baselines, i.e., CoCoOp and Prompt- SRC. and train both on ImageNet un- der base-to-new generalization task.
ZSCLIP
70.22
CoCoOp
4h37m
73.10
SSAS
10
6h18m
74.21
+1.11
selective trick
4h51m
74.02
+0.92
PromptSRC
50
3h29m
74.01
SAS
50
4h56m
75.46
+1.45
selective trick
3h38m
75.20
+1.19
version Tip-Adapter (Zhang 2022b) are involved. All results are averaged over three runs with distinct initialization.
Implementation Details. Unless specified otherwise, we use pre-trained CLIP (Radford et al. 2021) and ViT-B 66 as the visual backbone for fair comparison. Since our proposed method is a plug- and-play module, we strictly adhere to the settings of existing works, including optimizers, batch size, learning rate, and other strategies. This indicates that for different baselines, we may use dis- tinct hyperparameters specified their respective papers. For SAP, we use GPT-4V Turbo (Achiam) al. 2023) as the MILM, with temperature scaler of 0.7 and image understanding level set to high. For SAS, by default, we use ChatGPT to generate prompts for each spurious attribute, which then fed into Stable Diffusion (Rombach et al. 2022) to create pseudo categories. More details, such as the effect choices of MLLMs and comparison between pseudo category construction with synthesized and pre-training data, are provided in Supp. Mat. B Each pseudo category contains 16 shots, matching the number the target category. All experiments are conducted on a single NVIDIA 4090 GPU.
4.1
MAIN RESULTS
Our method is complementary to PEFT approaches. Fig. depicts the results of baselines and their integration with SAP and SAS. We observe an upward trend in accuracy, indicating improve- ments in out-of distriiiitiittionnn accuracy without compromising downstream task performance. For conventional methods, e.g., CoCoOp. SAS helps achieve zero-shot capability on distribution shifts. For strong baselines, e.g., CPL, the incorporation of SAP and SAS enables them to reach new state-of-the-ard benchmark. Overall, applying our method leads an average improvement over 2% in most baselines. These promising results align with the observation of the biased nature of VLMs, as demonstrated in Section 3.2
Our method is effective counter test samples. To further highlight the effectiveness of SAS, we conduct standard few-shot classification in an adversarial evaluation manner. This involves se- lecting subset from the original test set to create counter group for evaluating the VLM. For each category, we filter out images from the test set that bear high semantic similarity to the identified spurious attributes, retaining only images free of such attributes. This counter group is significantly

---

Published as a conference paper at ICLR 2025
more challenging to predict using spurious attributes compared to the entire test set. Fig. + displays example images from the test set and counter group. Table 2 preeeets the improvement in accuracy of SAS over baselines, both for the test set and counter group. We notice that 1) for the counter group, its accuracy is much lower than that of the test set; 2) SAS effectively bridges this gap, with improvement on the counter group far exceeding that on the test set, up to approximately 6%.
4.2 ABLATION STUDY
Diverse construction is beneficial for learning robust features. During training, aim for the constructed pseudo-categories to possess similar semantics to their target counterparts, thereby cre- ating strong contrast, while also maintaining high diversity to comprehensively represent spurious features. This trade-off is achieved by varying the number of SD prompts. As shown Table 3 the effectiveness of SAS assisting the baselines becomes more evident with an increasing num- ber of prompts. This underscores the importance of the quality of pseudo-categories, which should thoroughly reflect the corresponding spurious attributes.
Selecting appropriate spurious attributes matters. The core principle of SAS is to introduce auxiliary categories be trained alongside the main task, preventing the model from achieving high accuracy through spurious features. A natural concern is whether the model's gains are due to the introduction of additional data rather than an increase in robustness. In other words, does the model genuinely learn to distinguish spurious attributes from pseudo categories? We investigate this by adjusting the threshold y to control the presence of spurious attributes in pseudo categories. A higher indicates shortage of identified spurious attributes, while a lower 7 may introduce false positives. In Table we observe that performance significantly drops when y is either too high too low. This indicates that 1) spurious attributes play a crucial role in the contribution of SAS, and 2) the introduction of noisy attributes actually impairs the model's robustness. Additionally, the suggested adaptive strategy, which allows for flexible selection of spurious attributes, outperforms the pre-defined 7,
4.3 FURTHER ANALYSIS
SAS corrects the preference of VLMs on spurious attributes. To qualitatively assess SAS's impact VLMs, we present the saliency maps of VLMs with and without SAS in Fig. [5] Common spurious correlations can be observed, such as (a) utensils appearing alongside chocolate cake, and (b) mouse typically appearing with laptop. In critical applications, such autonomous driving. (c) road tends to act as confounders for street sign. SAS can effectively shift attention from these spurious attributes to the corresponding main objects. While we revisit Fig. Fig. this also aligns with the interpretation of CBMs that SAS suppresses the influence of spurious attributes on predictions.
Balancing the trade-off between efficiency and effeetiveness. One potential concern with SAS is its impact on training efficiency. Applying distinct loss to each category can be computationally demanding. To address this, we introduce selective optimization trick. Rather than targeting all categories, we only optimize ones that heavily rely on spurious attributes for predictions. Details of this approach are outlined in Supp. Mat. C In Table we demonstrate the effectiveness of this selective strategy by optimizing only 10% of the categories, showing the training time and accuracy. This approach significantly reduces SA.'s training time while preserving most of its accuracy gains.
CONCLUSION
This paper is motivated by an often-overlooked fact: VLMs tend to favor spurious attributes in their predictions, leading to decreased accuracy on out-of-distribution datasets. To tackle this issue, we first introduce SPURIOUS ATTRIBUTE PROBING (SAP), which identifies and filters out these problematic attributes, significantly improving the generalization of existing attribute-based meth- ods. Furthermore, to alleviate the biased nature of VLMs, we introduce SPURIOUS ATTRIBUTE SHIELDING (SAS), a plug-and-plaa module that reduces the influence of these attributes on predic- tions and complements various PEFT approaches. Both solutions significantly enhance accuracy in handling distribution shifts without compromising performance on downstream tasks, achieving a new oate-offthheart level.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This research was, in part, funded by the U.S. Government DARPA TIAMAT HR001 12490421. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, the U.S. Government.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint 3xxx:23333007772 2023.
Dyah Adila, Changho Shin, Linrong Cai, and Frederic Sala. Zero-shot robustification of zero-shot models with foundation models. In JCLR, 2023.
Bang An, Sicheng Zhu, Michael-Andrei Panaitescu-Liess, Chaithanya Kumar Mummadi, and Furong Huang. More context, less distraction: Improving zero-shot inference of clip by infer- ring and describing spurious features. In Workshop on Efficient Systems for Foundation Models@ ICML2023, 2023.
Hugo Berg, Siobhan Hall, Yash Bhalgat, Hannah Kirk, Aleksandar Shtedritski, and Max Bain. A prompt array keeps the bias away: Debiasing vision-language models with adversarial learning. In Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang (eds.), ACL, 2022.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative com- ponents with random forests. In ECCV (6), volume 8694 of Lecture Notes in Computer Science, pp. 446-461. Springer, 2014.
Adrian Bulat and Georgios Tzimiropoulos. LASP: text-to-text optimization for language-aware soft prompting of vision& & language models. In CVPR, pp. 23232-23244 IEBE, 2023.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- lusuul-inguustic tasks. In CVPR, pp. 24185-24198, 2024.
Ching-Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. Debias- ing vision-language models via biased prompts. arXiv preprint arXiv:2302.000070 2023.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, pp. 3606-3613, IEEE Computer Society, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255. IEEE Computer Society, 2009.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. In NeurIPS, volume 36, 2024.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training ex- amples: An incremental bayesian approach tested on 101 object categories. In CVPR Workshops, pp. 178. IEEE Computer Society, 2004.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vasion-language models with feature adapters. In IJCV, 2024.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, pp. 5842-5850, 2017.
Zongbo Han, Zhipeng Liang, Fan Yang, Liu Liu, Lanqing Li, Yatao Bian, Peilin Zhao, Bingzhe Wu, Changqing Zhang, and Jianhua Yao. Umix: Improving importance weighting for subpopulation shift via uncertainty-aware mixup. In NeurIPS, volume 35, pp. 37704-37718, 2022.

---

Published as a conference paper at ICLR 2025
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens., 12(7):2217-2226, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of dfdstiiibbttti generalization. In ICCV, pp. 8320-8329. IEEE, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, pp. 15262-15271. Computer Vision Foundation IEEE, 2021b.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In ICML, 2019.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2021.
Jiaxing Huang, Jingyi Zhang. Kai Jiang, and Shijian Lu. Open-vocabulary object detection via language hierarchy. In NeurIPS, 2024.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06955 2017.
Muhammad Uzair Khattak, Hanoona Abdul Rasheed, Muhammad Maaz, Salman H. Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, pp. 19113-19122. IEEE, 2023a.
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming- Hsuan Yang, and Fahad Shahbaz Khan. Self-regul prompts: Foundational model adaptation without forgetting. ICCV, pp. 15190-152000 2023b.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In CVPR, pp. 4015-4026, 2023.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 5338-5348. PMLR, 2020.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV Workshops, pp. 554-561. IEEE Computer Society, 2013.
Alex Krizhevsky, Geoffrey Hinton, Learning multiple layers of features from tiny images. NA, 2009.
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In ICCV, pp. 2556-2563. IEEE, 201
Brian Lester, Rami AI-Rfou, and Noah Constant, The power of scale for parameter-efficient prompt tuning. In EMNLP (1), pp. 3045-3059. Association for Computational Linguistics, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified lingunelanguage understanding and generation. In ICML, 2022a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ICMI. pp. 19730-19742. PMLR, 2023a.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chumyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded aunuage-image pre- training. In CVPR, pp. (0965-10975 2022b.
12

---

Published as a conference paper at ICLR 2025
Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scaling law for clip training. In NeurIPS, volume 36, 2024.
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking, In CVPR, pp. 23390-23400, 2023b.
Zhiheng Li, Anthony Hoogs, and Chenliang Xu. Discover and mitigate unknown biases with debi- asing alternate networks. In ECCV, pp. 270-288. Springer, 2022c.
Christopher Liao, Theodoros Tsiligkaridis, and Brian Kulis. Descriptor and word soups: Overcom- ing the parameter efficiency accuracy tradeoff for out-of-distribution few-shot learning. In CVPR, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, volume 36, 2024a.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P- tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. In ACL, 2021.
Xin Liu, Jiamin Wu, and Tianzhu Zhang. Multi-modal attribute prompting for vision-language models. arXiv preprint arXiv:2403.002/9 2024b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, (5(2018):11, 2018.
Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel. Retrieval augmented classification for long-tail visual recognition. In CVPR, pp. 6959-6969, 2022.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and L Repaint Van Gool. Inpainting using denoising diffusion probabilistic models. In CVPR, pp. 11461-11471, 2022.
Chaofan Ma, Yuhuan Yang, Chen Ju, Fei Zhang, Ya Zhang, and Yanfeng Wang. Attrseg: Open- vocabulary semantic segmentation via attribute eecompostioonaaggrggatin. NeuriPS, 2023.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. CoRR, abs/ 1b0/5536,,,,,, 2013.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR. OpenReview.net, 2023.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification large number of classes. In ICVGIP, pp. 722-729. IEEE Computer Society, 2008
Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caver- lee, and Shu Kong. The neglected tails of vision-language models. In CVPR, 2024.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar, Cats and dogs. CVPR, pp. 3498-3505, IEEE Computer Society, 2012.
Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does platypus look like? generating customized prompts for zero-shot image classification. In ICCV, pp. 15691-15701, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8766 PMLR, 2021.
Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Fine-tuned clip models are efficient video learners. In CVPR, pp. 6545-6554 2023.
13

---

Published as a conference paper at ICLR 2025
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In JCML, volume 97 of Proceedings of Machine Learning Research, pp- 5389-5400. PMLR, 2019.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. resolution image synthesis with latent diffusion models. In CVPR, 2022.
High-
Karsten Roth, Jae Myung Kim, A. Sophia Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. In ICCV, pp. 15746-15757, October 2023.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. arXiv preprint arXiv:/91 19117371,, 2019.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. In ICLR, 2020.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In NeurlPS, 2022.
Ashish Seth, Mayur Hemani, and Chirag Agarwal. Dear: Debiasing vision-language models with additive residuals. In CVPR, 2023.
Julio Rodriguzz Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. A closer look at the few-shot adaptation of large vision-language models. In CVPR, pp. 23681-23690 2024.
Sahil Singla and Soheil Feizi. Salient imagenet: How to discover spurious features deep learning? In ICLR, 2021.
Soomro. Ucf101: A dataset of 101 human actions classes from videos the wild. arXiv preprint arXiv:1212.0402. 2012.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from videos in the wild. Center for Research in Computer Vision, 2(11), 2012.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.153899 2023.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. VI-adapter: Parameter--fffiient transfer learning for vision-and-languuge tasks. In CVPR, pp. 5227-5237, 2022.
Revant Teotia, Chengzhi Mao, and Carl Vondrick. Finding spuriously correlated visual attributes. In ICML 2022: Workshop on Spurious Correlations, Invariance and Stability, 2022.
Xinyu Tian, Shu Zou, Zhaoyuan Yang, and Jing Zhang. Argue: Attribute-gided prompt tuning for vision-language models. In CVPR, 2024.
Vishaal Udandarao, Ankush Gupta, and Samuel Albanie. Sus-x: Training-free name-only transfer of vision-language models. In ICCV, 2022.
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. Towards debiasing nlu models from unknown biases. EMNLP. 2020.
Dequan Wang, Zhiqiang Shen, Jie Shao, Wei Zhang, Xiangyang Xue, and Zheng Zhang. Multiple granularity descriptors for iine-grained categorization. In ICCV, pp. 2399-2406. 2015.
Haohan Wang, Songwei Ge, Zachary C. Lipton, and Eric P. Xing. Learning robust global represen- tations by penalizing local predictive power. In NeurIPS, pp. 10506-10518, 2019.
Qizhou Wang, Yong Lin, Yongqiang Chen, Ludwig Schmidt, Bo Han, and Tong Zhang. Do clips always generalize better than imagenet models? arXiv preprint arXiv:2403.1 /497, 2024.
14

---

Published as a conference paper at ICLR 2025
Kun Wei, Muli Yang, Hao Wang, Cheng Deng, and Xianglong Liu. Adversarial fine- ain-gnained com- position learning for unseen attribute-objec recognition. In ICCV, pp. 3741-3749, 2019.
Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debug- gable deep networks. In ICML, pp. 11205-11210 2021.
Shirley Wu, Mert Yuksekgonul, Linjun Zhang, and James Zou. Discover and cure: Concept-aware mitigation of spurious correlation. In ICML, pp. 37765-37786. PMLR, 2023.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In CVPR, pp. 3485-3492 IEEE Computer Society, 2010.
Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. AAAI, volume 34, pp. 6502-6509, 2020.
Lingxiao Yang, Ru- Yuan Zhang. Yanchen Wang, and Xiaohua Xie. Mma: Multi-modal adapter for vision-language models. In CVPR, pp. 23826-2388 23837, 2024.
Yue Yang. Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in bottle: Language model guided concept bottlenecks for interpretable image classification. In CVPR, pp. 19187-19197. IEEE, 2023.
Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge- guided context optimization. In CVPR, pp. 6757-6767, 2023.
Hantao Yao, Rui Zhang, and Changsheng Xu. Tep: Textual-based class-aware prompt tuning for visual-language model. In CVPR, 2024.
Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Im- proving out-o--distribution robustness via selective augmentation. In ICML, pp. 25407-25437. PMLR, 2022.
Chenyu You, Yifei Mint, Weicheng Dai, Jasjeet S Sckhon, Lawrence Staib, and James Duncan. Calibrating multi-modal representations: A pursuit of group robustness without annotations. In CVPR, pp. 26140-26150. IEEE, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, pp. 11975-11986, 2023.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. TPAMI, 2024a.
Michael Zhang and Christopher Ré. Contrastive adapters for foundation model group robustness. In NeurlPS, volume 35, pp. 21682-21697, 2022.
Michael Zhang, Nimit S Sohoni, Hongyang R Zhang, Chelsea Finn, and Christopher Ré. Correct- n-contrast: A contrastive approach for improving robustness to spurious correlations. In ICML, 2022a.
Renrui Zhang, Rongyao Fang. Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training- frre clip-adapter for better vasion-language modeling. In ECCV, 2022b.
Yabin Zhang. Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou and Lei Zhang. Dual memory networks: A versatile adaptation approach for vision-language models. In CVPR, pp. 28718- 28728, 2024b.
Yi Zhang, Ce Zhang, Ke Yu, Yushun Tang, and Zhihai He. Concept-guided prompt learning for generalization in vision-language models. In AAAI, 2024c.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. IJCV, 13009):2337-2348 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, pp. 16795-16804. IEEE, 2022b.
15

---

Published as a conference paper at ICLR 2025
CONTENTS
IMPLEMENTATION DETAILS
17
FINDING SPURIOUS ATTRIBUTES
17
A.2
PROMPTING LLMS
17
A.3
QUERYING MILMS
17
A.4
CONSTRUCTING PSEUDO CATEGORIES
17
MORE EVALUATION
18
EXAMPLE SPURIOUS ATTRIBUTES
18
SAP VS HUMAN SUPERVISION
B.3
QUERYING WITH SAP AT SCALE
18
FFFECT OF CHOICES OF MILMS
19
B 5
SYNTHETIC GENERATION VS PRE-TRAINE TRAIIIG RETRIEVAL
19
B
6
BALANCING THE EFFECT OF SAS
20
7
QUANTITATIVE COMPARISON WITH RELATED WORK
20
B.8
GENERALIZATION UNDER LIMITED SHOTS
21
9
STANDARD FEW-SHOT CLASSIFICATION
DISCUSSION OF HYPERPARAMETER SENSITIVITY
ABLATION ON PERFORMANCE GAINS
22
MORE VISUALIZATION EXAMPLES
22
13
MORE LISION-LANGUAAE MODELS
23
14
COMPUTATIONAL EFFICIENCY AND COST
23
B. 15
MORE MODALITIES AND TASKS
................................
24
B 16
EVALUATION ON MORE BASELINES
................................
25
B. 17
ABLATION DIFFUSION STEPS
25
FURTHER EXPLORATION
26
SPURIOUS ATTRIBUTES FOR ZERO-SHOT RECOGNITION
26
C.2
SELECTIVE OPTIMIZATION TRICK
26
C.3
VARIANTS OF SAS
27
C,4
NON-SEMANTIC SPURIOUS ATTRIBUTE
27
C.5
LIMITATION AND FAILURE CASES
28
C.6
MORE RELATED WORKS
28
BROADER SOCIETAL IMPACTS
SUPPLEMENTARY RESULTS
29
CONSTRUCTED IMAGES
29
E.2
SPURIOUS ATTRIBUTE STATISTICS
29
MOTIVATIONAL RESULTS
29
NUMERICAL MAIN RESULTS
29
16

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
A.1 FINDING SPURIOUS ATTRIBUTES
We delve into manual identification process for spurious attributes as described in Section 3.2. Following the approach outlined (Singla Feizi 2021), we present a simplified version. For each category, we randomly select 5 images from the training set and generate the corresponding heatmap. We also reference external sources like Wikipedia and seek advice from ChatGPT. Using this information, assess whether attribute helongs to the main object separate background element, with options: "Yes", "No", or "Unsure". Finally, attributes categorized as "No" are deemed spurious attributes. It important to mention that unlike (Singla Feizi 2021 do conduct crowd studies. All supervision tasks are performed by the authors.
A.2 PROMPTING LLMS
We conduct a naive attempt to modify the prompting technique of LLMs to avoid generating spurious attributes in Section 3.2. We try three variant prompt templates by appending inserting additional instructions as follows:
T1: Only focus on itself.
T2: Imagine you are an expert of
T3: Do not describe other than
For each instruction, position it at either the beginning or the end, yielding six combinations. Then, we employ existing attribute-based methods, e.g., ArGue (Tian et al. 2024), to derive results, averaging them across all combinations.
A.3 QUERYING MLLMS
In addition to the techniques and parameters introduced in the main paper, we believe a crucial step in dealing with MILMS is managing their outputs. Given specified temperature, the output vari- ance of an MLLM, particularly GPT-4V, for the same input can be significant. The responses may range from single word complete paragraph, and the model may fail to follow the demon- strated formats or refuse to respond. Similar challenges have been noted in recent studies, such as DCLIP (Menon Vondrick 2023) and CuPL (Pratt et al., 2023). when using MLLMs or LLMs to generate attributes. In this work, we employ a simple regular expression to retain responses of suitable length and exclude those that are not formatted with bullet points. Additionally, we filter out duplicate or similar attributes. For example, between ice surface and glacier we typically randomly select only one.
A.4 CONSTRUCTING PSEUDO CATEGORIES
Here, describe the process of constructing images targeting spurious attributes using SD (Rom- bach et al., 2022) or LAION-5B (Schuhmann et al.) 2022). For the former, following Sus-X (Udan- darao al. 2022), we use the common checkpoint stable-diffusion-vl 7.0. The diffusion step is set to 100, with a fixed output resolution of 512x512. Additionally, to ensure the diversity of the images, we use ChatGPT generate multiple SD prompts. Specifically, we provide vanilla prompt an example, e.g., a photo of mouse, and then ask GPT to rephrase the prompt different formats, Some example generated prompts are displayed below.
with a guidance scale of
P1: 3D realistic photo of a P2: high-quality natural image of P3: intriguing portray of
worth noting that the prompts mentioned above are also applicable for pre- pre- trainining retrieval. For LAION-5B, select the matches with the highest average semantic similarity to these GPT- generated prompts to construct pseudo categories. This approach ensures the diversity of the re- trieved images while also enhancing the reliability of semantic matching.
17

---

Published as a conference paper at ICLR 2025
08.20
CPL
Our our-of-ditriluution acc.
06.58
65.81
06.05
88.11
CPL Man.
63.40
03.61
CPL SAP
79.58
76.51
76.91
75
personal laptop
freight truck
mountain bike
apple pic
average
Figure 6: The per-category out-of- -otoo--distribution accuracy on domain generalization. In this set- ting, based on the strong baseline CPL,, we remove spurious attributes identified by manual inspec- tion (Man.) or SAP for specific category and compare the accuracy change on the category in the out-of-distribution datasets. All results are averaged over ImageNet variants.
Category name
Spurious attributes
Average weights
Personal laptop
mouse, coffee, charger, worktable road, traffic light, trees, street frees, mad, mountain, swamp fork, platex, dining tublecloth
77.34% / 46.73%
Freight truck
82.16% / 54.69%
Mountain bike
74.81% 43.002%
Apple pie
67.29% 37.44%
The spurious attributes identified by SAP. For each example category, we pinpoint its spurious attributes and determine the average attribute weights on model predictions using CBMs across identified spurious attributes (Left) and all generated attributes (Right).
Table 6:
B
MORE EVALUATION
B. EXAMPLE SPURIOUS ATTRIBUTES
SAP quantifies the identification of spurious attributes without human supervision, offering more precise measure of spurious correlation. This aids in effectively pinpointing attributes favored by VLMs. Table 6 showccsess typical spurious attributes found by SAP, including instances like mouse frequently appearing with laptop, or fork being closely associated with apple pie. Additionally, we assess their weights on model predictions, along with the average weights of all generated attributes for reference. Notably, spurious attributes carry substantially higher weights in model decision- making compared overall attributes, further underscoring the biased nature of VLMs.
B.2
SAP VS HUMAN SUPERVISION
Finding spurious attributes through human supervision (Singla Feizi 2021 Wong et al. 202 l). while comprehensive, has significant drawbacks: 1) incurs extremely high labor costs; 2) its strong subjectivity easily introduces false positives, where identified attributes are only present by chance. Here, compare the performance of the proposed automatic identification method, SAP, with human supervision. We adopt domain generalization as the task and select CPL (Zhang et al.) (024c) the baseline. For better interpretation, one at a time and record the change in per-category accuracy on out-of-distribution datasets. Fig, 6 depicts the results in CPL, as well as the results after removing spurious attributes through the two identification approaches. It can be seen that SAP's performance is comparable or even outperforms human supervision.
remove spurious attributes from individual categories
B.3 QUERYING WITH SAP AT SCALE
In the main paper, we address a challenging setting, specifically few-shot scenarios where training data is limited. This leads to pertinent question: is small number of images truly adequate for SAP to identify spurious attributes within categories? In other words, would querying more images further enhance SAP's performance? To investigate the potential of scaling up, expand
18

---

Published as a conference paper at ICLR 2025
Query Images
16
CoCoOp
75.62
76.06
76.84
77.56
78.
78.24
78.28
MaPLe
81.98
82.55
83.47
84.01
84.49
84.60
84.53
PromptSRC
82.59
83.35 84.13 84.72 85.46 85.63 85.68
Table 7: The evaluation on base-to-new generalization while querying different number of images per-category. The results are averaged across 11 datasets.
MLLM
BLIP-2
LLaVA
InterVV.
GPT-4V
CoCoOp
72.28
72.79
73.14
73.50
MaPLe
76.20
76.83
77.25
77.69
PromptSRC
76.43
77.01
77.37
77.88
Table 8: The evaluation on base-to-new generalization with various MLLMs.
the training dataset from 16-shot 256-shot and have GPT-4V query 2. 4, 16. 32, and 64 randomly selected images from the training shots. We evaluate the average new category accuracy on base-to-new generalization tasks across 11 datasets, comparing three typical baselines: CoCoOp, MaPLee and PromptSRC. As shown in Table 7 despite the availability of additional shots during training, the results tend to plateau when querying with 16 images. This indicates that even with an expanded training dataset, MLLMs require only around 16 query images to capture sufficient and effective spurious attributes.
B.4 EFFECT OF CHOICES OF MLLMS
In previous experiments, we default our MLLM to GPT-4V. Here, we attempt use more open- sourced MLLMs to comprehensively evaluate the robustness of our proposed method. We con- sider three recently popular MLLMs: BLIP-2 (Li al. 2023a LLaVA (Liu al. 2024a), and InternVL. (Chen et al. 2024 Table & presents the performance of these different MLLMs on base- to-new generalization tasks with 16- 16-shot. As expected, GPT-4V, the proprietary model, achieves the best results. The next best performance is from VL. Conversely, BLIP-2 shows the poorest performance, which we attribute to its tendency to produce limited vocabulary that results overly broad core and spurious attributes.
B.5 SYNTHETIC GENERATION PRE-TRAINING RETRIEVAL
In previous experiments, our default approach is to utilize Stable Diffusion for constructing pseudo categories. Here, we explore an alternative method: retrieving image samples from the pre-training dataset. Table / illustrates the results of both approaches across several basclines. Notably, Stable Diffusion consistently outperforms retrieval from LAION-5B. This unexpected result is intriguing.
(A) prediction
(B) SD
LAION
Figure 7: Constructed images.
19

---

Published as a conference paper at ICLR 2025
CoCoOp
KgCoOp
MaPLe PromptSRC
CPL
SD
76.93
78.51
80.06
80.57
82.87
LAION
76.46
78.45
79.32
80.85
82.35
Table 9: The comparison between synthetic generation and pre-training dataset retrieval. We select strong baselines. The results are harmonic mean of accuracy on base and new categories for base-to-new generalization.
2
5
10
20
Base 83.23 83.73
83.64
83.15
82.41
81.53
New 74.82 76.00
77.36
77.73
76.60
76.89
HM 78.80 79.68 80.38
80.35 79.40
79.14
Table 10: The effect of Lyse with different À on base-to-new generalization.
considering that pre-training images predominantly consist of real data, which would expect to better match the style the target category. However, the results suggest otherwise. We spec- ulate that the complexity of real image distributions, coupled with noise attributes, may contribute to this disparity. For instance, in Fig. 7 when associating the spurious attribute snowforest. with snowmobile, the top-1 match retrieved using LAION-5B includes elements such as tent and bag. These noise attributes could potentially introduce new shortcuts, complicating the model's ability to differentiate spurious attributes from the target category.
B.6 BALANCING THE EFFECT OF SAS
Table 10 examines the balancing effect between Syse and primary learning objectives in existing work terms of A. The best trade-off is observed at around À 2. As À increases further, begins to neglect the primary objectives of the baselines, leading to decline in base accuracy. Notably, these results are averaged across multiple baselines. fact, for distinct baselines. we suggest exploring optimal values individually due to their respective learning characteristics.
B.7 QUANTITATIVE COMPARISON WITH RELATED WORK
In the main text, primarily demonstrate the effectiveness of SAS complementing existing PEFT methods. Here, we further substantiate the advantages of SAS by comparing it with other state-of- the-art spurious correlation mitigation approaches. We evaluate a typical property of VLMs, group robustness, which indicates the invariance of VLMs under different associations between labels and attributes. For the baselines, we consider C-Adapter (Zhang Ré (022) and CFR YYou al. 2024).
Method
Waterbirds
CelehA
BREEDS
CIFAR-10.02
Accuracy (%)
WG
Avg
Gap
WG
Avg
Gap
WG
Avg
Gap
WG
Avg
Gap
Zero-shot RoboShot
25.7
87.3
61.6
62.1
71.9
9.8
4.0
86.6
82.6
72.0
93.2
21.2
45.2
79.9
34.7
82.6
85.5
2.9
56.4
80.3
23.9
79.1
95.6
16.5
Linear Probe
65.9
97.6
31.7
28.3
94.7
66.4
84.0
98.6
14.6
87.5
96.1
8.6
C-Adapter
86.9
9.3
84.6
90.4
80.0
17.5
82.2
96.1
13.9
DISC
88.7
93.8
5.1
82.0
92.5
10.5
86.3
95.8
9.5
84.7
94.3
9.6
CFR
88.2
96.7
8.5
84.7
87.8
3.1
85.0
96.1
11.1
89.1
92.5
3.4
SAS
89.7
96.3
6.6
87.4
91.1
3.7
87.8
96.4
8.6
88.5
95.2
6.7
Table 11: The group robustness evaluation of SAS and other spurious correlation mitigation methods. We report worst-group accuracy (WG), average-group accuracy (Avg) and the gap be- tween. Note that RoboShot is a zero-shot calibration method, while other approaches are training- required.
20

---

Published as a conference paper at ICLR 2025
AV ARC
CPL
CPI.
TCP
TCP 545
TCP SAS PSRC
PARC
"gm PMRC
5AS
PSRC 5A5
aShete
new generalization
(B) standard classification
Figure 8: The results varying shots on base-to-new generalization and few-shot classifcation.
where spurious attributes are assumed be unknown. We also include RoboShot (Adila 2023 and DISC (Wu et al. 2023). where, similar to our approach, spurious concepts are identified and used for precise mitigation. By default, we configure SAS optimize only the learnable textual prompt, i.e., CoOp Zhou et al., (022a). It is worth noting that RoboShot (Adila et al. (2023) is a zero-shot approach that calibrates pre-trained embeddings. Following Zhang & Ré (2022), we consider four datasets with group annotations: Waterbirds (Sagawa et al. 2019), CelebA (Liu al.) 2018), BREEDS Living-17 (Santurkar et al. 2020). and CIFAR-IO.00 (Krizhessky et al.) 2009). In Table I1 wverage-groupp accuracy, worst-group accuracy, and their gap reported. It can be observed that SAS achieves a new state-of-the-art in worst-group accuracy across most datasets without excessively compromising peeaage-gouup accuracy.
B.8 GENERALIZATION UNDER LIMITED SHOTS
We consider generalization capability in extreme cases, where the shots are further limited, ie., 1/2/4/8 shots. is noteworthy that this scenario, limitations arise from both the insufficient amount of data and the impact on SAP's precision identify spurious attributes, further affecting SAS performance. We select three strong baselines, encompassing PromptSRC (Khattak et al, 2023b), TCP Yao et al. 2024) and CPL (Zhang et al. 2024c). Fig. g (A) shows the results of combining SAS on base-to-new generalization across different shot settings. It can be seen that the results consistently outperform the original baselines, even only shot is given.
B.9 STANDARD FEW-SHOT CLASSIFICATION
We consider the standard scenario where test and training samples originate from the same dataset distribution. Fig. 8 (B) illustrates the results in standard few-shot classification. Notably, integrat- ing SAS does not compromise in-distribution accuracy; instead, shows slight and consistent improvement.
B.10 DISCUSSION OF HYPERPARAMETER SENSITIVITY
Although we observe the state-of-the-art performance of SAS in the main context, an important aspect. hyperparameter sensitivity, still requires discussion. For the newly introduced hyperparam- eters in SAS, such as À and y, their impact on the results has been examined in previous ablation experiments. These experiments reveal that while optimal value is preferred, SAS is not overly sensitive to these hyperparameters and consistently provides stable improvements within a certain range.
Regarding training hyperparameters such as learning rate and batch size, recent studies (Silaa-) Rodriguez et al.) 2024) have found that some adaptation methods heavily rely these hyperpa- rameters in few-shot scenarios, complicating practical deployment. In contrast, as shown in Fig. 3 of the main paper, although SAS uses different training hyperparameters for different baselines as specified in the original papers, it consistently achieves gains, demonstrating its robustness to hyperparameters.
21

---

Published as a conference paper at ICLR 2025
Training Data
CoCoOp
MaPLe
PrompiSRC
Average
16-shot main
70.05
75.39
73.78
72.94
32-shot main
71.12
76.37
75.52
74.34
16-shot main 16-shot pscudo main
70.34
75.80
74.46
73.53
16-shot main 16-shot pseudo spurious (ours)
73.50
77.69
77.88
76.36
Table 12: The ablation study on the performance gains. We introduce two baselines, where the first incorporates additional data from the training set (32-shot main), and the second involves vanilla constructed data from pseudo categories mirroring the main categories (16-shot main + shot pseudo main). In contrast, the pseudo categories of our method feature spurious attributes (16-shot main + 16-shot pseudo spurious)
B.11 ABLATION ON PERFORMANCE GAINS
In the main paper. verify the effectiveness of SAS and explore the contribution of spurious at- tributes to its performance. To further confirm that the performance gains are primarily due to the model's enhanced robustness spurious attributes rather than additional data, here conduct simple ablation study. Specifically, in addition to the proposed method, we design two baselines. In the first baseline, consider additional data directly from the original dataset featuring the main objects, where we extend the training data from 16 shots to 32 shots (32-shot main). In the second baseline, we involve additional data generated by pseudo categories. where instead featuring spu- rious attributes, these pseudo categories are the same as the main categories, i.e., vanilla constructed data (16-shot main 16-shot pseudo main). In contrast to the first two baselines. our approach cre- ates pseudo categories based on spurious attributes (16-shot main + 16-shot pseudo spurious). For fairness, we ensure that the amount of training data identical between the two baselines and our approach. We select three typical methods for comparison, including CoCoOp (Zhou et al. 2022b), MaPLe (Khattak et al. 2023a), and PromptSRC (Khattak et al. 2023b), and evaluate them on the base-to-new generalization task. All results are averaged across datasets.
As shown in Table 12 generating additional data using spurious attributes significantly outper- forms vanilla constructed data for main categories (76.36% vs 73.53%). Furthermore, proposed method even exceeds the performance of the 32-shot main (76.36% vs 74.34%). is important to note that this comparison is not entirely fair for our method, the latter relies more labeled data from the original training set. This further suggests that the performance gains are primarily driven by the model's enhanced robustness spurious attributes, rather than merely the increased training data.
B.12
MORE VISUALIZATION EXAMPLES
Fig we present the saliency maps for some typical categories with and without SAS. For com- pleteness, we provide more examples here. As shown in Fig. SAS consistently reduces VLMs' bias towards spurious cues across various categories, enabling a greater focus on the main objects.
Original
CLIP
+SAS
Original
CLIP
+SAS
Original
CLIP
+SAS
(d) tree frog
(e) airliner
(f) polar bear
Original
CLIP
SAA
Original
CLIP
SAS
Original
CLIP
SAS
(g) mountain bike
(h) water bird
(i) football
Figure More saliency map visualization with and without SAS.
22

---

Published as a conference paper at ICLR 2025
Model
CoCoOp
MaPLe
PromptSRC
Average
BLIP
68.81
72.32
72.62
70.58
BLIP + SAS
70.54
74.35
73.97
72.95
CLIPA-v2
70.28
73.40
74.52
72.73
CLIPA-v2 SAS
72.42
74.88
77.08
74.79
EVA-CLIP
72.75
77.58
76.13
75.49
EVA-CLIP SAS
74.60
77.92
77.82
76.78
SigLIP
74.99
73.78
78.64
75.80
S SAS
76.41
75.26
79.87
77.18
Table 13: The evaluation results of SAS on other VLMs. We consider four representative VLMs including BLIP, CLIPA-v2, EVA-CLIP and SigLIP.
Method
Flowers102
Food101
FGVCAireraft
StanfordCars
Average Time
CoCoOp
10m48s
18m34s
8m02s
12m46s
12m32s
+ SAS
13m14s
24m07s
13m36s
17m08s
17m01s
selective trick
11m03s
20m45s
10m23s
14m55s
14m 16s
PromptSRC
6m25s
15m09s
5m44s
9m36s
9m13s
SAS
8m26s
18m21s
7mlls
12m04s
11m30s
selective trick
6m58s
16m22s
6m35s
10m20s
10m03s
Table 14: The training efficiency of SAS and selective optimization other datasets.
For example, for the tree frog, SAS reduces VLMs' reliance on tree branches, while for the airliner, the typical spurious attributes are sky or clouds, and the application of SAS alleviates the model's bias towards these elements.
B.13 MORE VISION-LANGUAA MODELS
For completeness, here extend the evaluation of SAS additional VLMs other than CLIP Specifically, we select four typical VLMs encompassing BLIP (Li et al. 2022a), CLIPA-v2 (Li al. 2024). EVA-CLIP (Sun et al., 2023) and SigLIP (Zhai al. | [202)). We record the results on base-to-ae base-to-new generalization, where the setting is consistent with the main paper. As demonstrated in Table 13 our proposed method, SAS, consistently yields performance gains across range of VLMs, extending beyond just CLIP.
B. 14
COMPUTATIONAL EFFICIENCY AND COST
In this section, we present the computational and time costs of the proposed method, accompanied by a thorough analysis.
The cost of training. In the main paper, present the training time of SAS and the proposed selective optimization trick on ImageNet. Here, furthermore, we provide time statistics for other datasets. The time is measured the runtime of the training script based on the implementation of CoOp Zhou et al. 2022a). As shown in Table [4] for most datasets, the integration of SAS only increases the training time by approximately 3 minutes, while selective optimization further reduces this time to negligible amount. In fact, the selective optimization trick is proposed to address large-scale datasets, such as ImageNet, which contains 1000 categories. For regular datasets (~ 100 categories), the time consumption of SAS is fully acceptable.
The cost diffusion generation. Here, we provide the estimated inference time required to con- struct pseudo categories through Stable Diffusion for each dataset. As shown in Table [5 the total inference time is proportional to the size of the dataset, particularly the number of categories in- volved. For most datasets, the inference time is under half hour, and the entire inference process
23

---

Published as a conference paper at ICLR 2025
Dataset
Caltech
Pets
Cars
Flowers
Food
Aircraft
SUN
DTD
EuroSAT
UCF
INet
Time 25min
10min
45min
30min
25min
30min
90min
15min
5min
25min
3h50min
Table 15: The diffusion inference time for each dataset.
Dataset
Caltech
Pets
Cars
Flowers
Food
Aircraft
SUN
DTD
EuroSAT
UCF
INet
Time
10min
10min
25min
10min
10min
10min
35min
5min
3min
10min
1h30min
Table 16: The GPT prompting time for each dataset.
can be completed within half a day. is important to note that this is a one-time operation, and no additional inference is needed during subsequent training.
The cost of GPT prompting. In our method, key step is identifying the spurious attributes within each category, which accomplish by prompting MLLMs, i.e., GPT. Here provide the time cost of this process along with a thorough analysis. Specifically, to enhance efficiency, we employ batch inference as implemented in Menon & Vondrick (2023), where multiple queries can be processed concurrently, which significantly reduces the inference time for GPT. As shown in Table [6 the GPT inference time for most datasets is under 10 minutes. The complete inference process takes approximately three hours, which is also one-time operation that does not need to be repeated thereafter. is worth noting that upon obtaining the responses, we need to perform post-processing such as filtering and selection to determine valid attributes, as detailed in Section A.3 which may require additional time.
B.15 MORE MODALITIES AND TASKS
To assess the transferability our method to other modalities or tasks, we explore video recog- nition and leave
tasks, such language reasoning, for future work. Specifically, choose ViFi-CLIP (Rasheed et al.] 2023). a fully fine-tuned CLIP model tailored for video understanding. ViFi-CLIP employs training framework similar to CLIP, incorporating temporal pooling layer to derive video representations from multiple frames. Following the base-to-new generalization setting in Rasheed al. (2023). we evaluate video-level generalization performance on four video datasets: K-400 (Kay et al. (017), HMDB-51 (Kuehne et al., 2011). UCF-101 (Soomro) 2012), and SSv2 (Goyal al. (017). As in the main paper, we select three representative baseline meth- ods: CoCoOp (Zhou et al., 20226), MaPLe (Khattak et al., 2023a). and PromptSRC (Khattak et al. 2023b). Since ViFi-CLIP shares its architecture with CLIP, these methods can be easily transferred to VIFi-CLIP, which has been implemented by Khattak et al. (2023b). We incorporate the proposed method, SAS, into these baselines to verify its effectiveness by contrasting spurious attributes with each frame of the video. We record the category accuracy for each dataset which directly flects the generalization performance on unseen categories. As shown in Table 17 despite the input modalities shifting from images videos, SAS consistently delivers performance gains across all
Method
K-400
HMDB-51
UCF-101
SSv2
ViFiCLIP
61.10
53.30
67.70
12.10
CoCoOp
64.70
54.41
68.21
14.24
CoCoOp + SAS
66.39
56.64
70.40
16.01
MaPLe
64,52
58.23
70.73
14.74
MaPLe SAS
66.42
59.32
72.66
16.40
PromptSRC
68.31
62.38
76.79
17.22
PromptSRC SAS
70.23
64.70
79.31
18.95
Table 17: The evaluation results of SAS on four video datasets. CLIP, fully fine-tuned CLIP model for video reasoning.
The training is based on ViFi-
24

---

Published as a conference paper at ICLR 2025
Method
ImageNet
Flowers102
SUN397
FGVCAircraft StanfordCars
MMA
71.00
75.93
78.57
36.33
73.10
MMA + SAS
72.61
77.27
80.19
37.85
75,46
DMN
72.28
78.49
77.32
32.60
74.22
DMN + SAS
73.34
80.17
79.74
35,38
76.30
Table 18:
The evaluation of SAS on other baselines. proaches, including MMA and DMN.
We include two recently proposed ap-
Flowers102
Food101
FGVCAireraft
StanfordCars
Average
Step
Acc
Time
Acc
Time
Acc
Time
Ace
Time
Acc
Time
25
72.24
10min
91.18
6min
27.23
8min
73.55
13min
66.05
9min
50
72.85
15min 91.96
11min
28.41
14min
74.67
23min
66.97
16min
75
72.81
22min
92.28
18min
28.19
24min
74.82
31min
67.02
24min
100
72.99
28min
92.12
26min 28.30
31min
74.96
40min 67.09
31min
Table 19: The performance of SAS and diffusion time with different number of diffusion steps.
datasets, proving it be an effective plug-and-pe and-play method that can be generalized to complex modalities and tasks.
B.16 EVALUATION ON MORE BASELINES
For completeness, here evaluate our method on the two recently proposed works. Specifically, we select MMA (Yang et al. 2024) and DMN (Zhang al. 2024b).. For the former, we train the newly introduced adapters in the deep layers that bridge the text and image representations, follow- ing their setting and implementation. For the latter, we optimize its memory projection functions and incorporate both the static and dynamic memory networks, which is the strongest variant cording to their paper. We select the base-to-new generalization task, as illustrated in Section 4, and record the new category accuracy, which directly reflects the generalization performance. As shown in Table [8] SAS consistently improves performance on both methods, demonstrating its complementarity.
B.17 ABLATION ON DIFFUSION STEPS
Considering the computations introduced by diffusion in generating images, here we perform an ablation study on the efficiency of diffusion inference. Specifically, we vary the number of diffusion steps, which is the key hyperparameter influencing the inference time. Intuitively, fewer steps are more efficient yet yield lower image quality, while more steps ensure image fidelity but require more computation. We select CoCoOp as the baseline and record the new category accuracy on base-to- new generalization.
shown in Table 19 by default, we use 100 steps throughout the paper as described in Section A.4, which requires an average of 31 minutes to generate images per dataset. Here we try fewer steps, such as 50, and observe that the time required for diffusion nearly halves (31min - 16min) with minimal degradation in performance (67.09 66.97). However, while the number of steps is further reduced to 25, there is a dramatic performance drop (66.97 - 66.05), possibly due the decline in image quality. This suggests may safely adjust the number of steps from 100 to 50, which halves the required time with minimal accuracy loss, significantly improving the efficiency of 5AS.
25

---

Published as a conference paper at ICLR 2025
Method
ImageNet
Caltech101 OxfordPets
StanfordCars
DTD
EuroSAT
ImageNet-A
Average
CLIP
66.54
94.62
90.41
64.69
44.84
47.50
49.32
65.42
DCLIP
68.52
95.48
91.88
65.70
45.52
49.08
49,88
66.58
DCLIP- SA
67.67
94.76
91.31
64.79
45.06
47.92
49,30
65.83
CuPL
69.99
96.51
92.62
66.91
47.32
50.33
50.14
67.69
CuPL SA
68.70
95.89
92.38
65.66
46,06
49.58
49.28
66.79
Table 20: The zero-shot accuracy before and after removing spurious attributes. The model is evaluated on generic datasets (ImageNet (Deng et al., 2009), Caltech101 (Fei-Fei et al. (004]] ).
2 fine-grained datasets (OxfordPets (Parkhi et al. (012). StanfordCars (Krause et al.] 2013)),
specialized datasets (DTD (Cimpoi et al. 2014). EuroSAT (Helber al. 2019)) and adversarial dataset (ImageNet-A (Hendrycks et al. 2021bb).
C FURTHER EXPLORATION
C.I SPURIOUS ATTRIBUTES FOR ZERO-SHOT RECOGNITION
The primary takeaway of this paper is the unbalanced treatment of various semantic attributes by VLMs, which extends beyond the generalization task and suggests that the language encoder of VLMs may allocate distinct attention to different tokens. We examine typical example: zero-shot recognition, where attributes are directly utilized to make predictions without training. We consider three baselines: CLIP (Radford al., (021), DCLIP (Menon & Vondrick 2023), and CuPI. (Pratt et al. 2023). The latter two employ LLMs to generate attributes and enhance zero-shot accuracy. In a manner similar the previous motivational study, we remove identified spurious attributes from the existing baselines and record the accuracy before and after this intervention. Table 20 presents the results before and after removal. We observe significant drop in accuracy for the baselines (from 63.23% to 62.49% for DCLIP and from 64.34% to 63 13.45% for CuPL). with DCLIP almost reverting to the performance of vanilla CLIP (62.49% vs. 62.07%). This indicates that 1) similar to the generalization task, zero-shot recognition is also dominated by spurious attributes, nearly ignoring the presence of other generated attributes; and 2) spurious attributes, in a sense, improve zero-shot performance natural datasets by scaling up the model's 's inherent bias.
C.2 SELECTIVE OPTIMIZATION TRICK
SAS introduces a subsidiary task that includes constructed pseudo categories and auxiliary learning objectives. With an increasing number of spurious attributes, a large number of pseudo categories are introduced, significantly increasing computational costs. To tackle this challenge, we introduce strategy that selectively optimizes partial target categories with a heavy bias towards spurious at- tributes. In other words, we only mitigate the influence of spurious attributes on categories that overly rely them. To identify these categories, we propose Spurious Correlation Ratio (SCR). SCR is calculated as the ratio of the average weights of spurious attributes to the average weights of all attributes, as exemplified in the rightmost column of Tablee higher SCR indicates that the pre- diction of the corresponding category relies more on spurious attributes. In implementing this trick, we empirically select only the top 10% of categories ranked by SCR for optimization. To verify the trick, we choose two time-intensive basclines, CoCoOp (Zhou et al. 20226) and PromptSRC (Khat-) instance-conditioned mech-
tak et al. 2023b), for comparison. CoCoOp's training slow due to anism, while PromptSRC adds three extra learning objectives to the original cross-entropy loss. To emphasize the results, we conduct evaluation on the base-to-new generalization task using Ima- geNet (Deng et al. 2009) and record both training time and harmonic mean accuracy. Table 5 in the main paper illustrates the trade-off between effectiveness and efficiency with SAS and the proposed trick. It is evident that integrated with selective optimization, the required time is significantly duced compared to the original SAS. For instance, on PromptSRC, it only adds minutes training time while preserving most of the performance gains.
26

---

Published as a conference paper at ICLR 2025
Original
Masking
Inpainting
Figure 10: The constructed categories with masking and inpainting. For the former, we directly mask the primary object with SAM (Kirillov et al.) 2023), whereas for the latter, we further use RePaint (Lugmayr al., 2022) fill in the missing parts. We compare the performance between the pseudo categories constructed with masking, inpainting and our synthesis method.
C.3 VARIANTS OF SAS
In the main paper, SAS primarily constructs pseudo categories using synthetic or pre-trained data, which has proven effective. Here, we consider two simple yet direct variants: 1) instead of utilizing spurious attributes to create new data, we directly mask the main object in the original images and use these as the corresponding pseudo categories, termed SAS MAs-maskingg 2) upon masking, fill the masked area through in-painting, termed SAS-inpaai innpinniannnn..... Fig. I0 displays some example images of pseudo categories by these two variants. The motivation here is to enhance VLMs' awareness of core attributes by contrasting target categories with their corresponding images that lack main objects. We refer to the original approach as SAS-synt sesythhhsss,... where pseudo categories constructed with SD-synthesized images. Table 21 presents the performance of the three methods, showing that both variants perform worse than sesssyythessie s. We speculate that this is because masking or in-painting significantly reduces image fidelity: and 2) this approach introduces excessive noise attributes, thereby forming new set of spurious attributes for VLMs to learn.
C.4 NON-SEMANTIC SPURIOUS ATTRIBUTE
In the evaluated datasets, including previous work measuring group robustness ZZang & Ré 2022 Adila et al., 2023), most spurious attributes are semantically related, wherein the attribute and label exhibit natural association, egg., water and water bird. In this study, extend our exploration to non-semantic attributes, where the association between the attribute and label is ar- tificially constructed. We implement straightforward color-shifting experiment using ColoredM- NIST. This dataset comprises 10 classes, each representing digit; however, instead of the stan- dard black background MNIST, each digit class features a distinctly colored background. Each color demonstrates strong spurious correlation with its corresponding digit, effectively serving as a spurious attribute. Fig. [T illlstraats examples from ColoredMNIST. We employ GPT-4V iden- tify these non-semantic spurious attributes, resulting in descriptors such as green background and pure yellow background. We evaluate SAS on the test set of ColoredMNIST, where the color back- grounds randomized across labels. As shown in Table 22
SAS significantly enhances VLMs'
Method
CoCoOp MaPLe PromptSRC
SAS-masking
71.08
74.53
75.39
SAS-inpainting
72.95
76.81
77.04
SAS-synthesis
73.50
77.69
77.88
Table 21: The evaluation base-to-new generalization with two SAS variants.
27

---

Published as a conference paper at ICLR 2025
Method
CoCoOp
MaPLe
PromptSRC
w/o SA5
72.47
75.89
74.08
w/ SAS
84.70
88.27
87,48
Table 22: The evaluation on ColoredMNIST with and without SAS.
robustness to color shifting, indicating that MLLMs may capture non-semantic attributes in images, and SAS effectively leverages these attributes to improve generalization.
C.5 LIMITATION AND FAILURE CASES
In SAP, the primary limitation stems from the necessity of having available images, Previous ap- proaches to generating visual attributes only require textual information, e.g., category names. The underlying assumption is that the generated attributes would be dataset-agnostic. For example, at- tributes like headlights, doors, or wheels for the category vehicle are assumed to be consistent across datasets. However, spurious attributes do not adhere to this assumption; they are contin- gent on the specific characteristies of the dataset. For instance, vehicle images in different datasets might be taken on highway in parking lot, resulting in vastly different spurious attributes. This highlights the need for visual information from the dataset itself to accurately identify spurious attributes.
For SAS, the main concern still lies in efficiency. While the use of synthetic or pre-training images has been employed to address data scarcity in many recent works, such as SuS-X (Odandarao al., 2022) and Real-Prompt (Parashar et al., (022).. these methods inevitably introduce additional com- putational overhead. The inference of Stable Diffusion (Rombach et al. 2022), relative to its large data requirements, is not particularly fast, and retrieval requires finding top-k matches from huge pre-training dataset (Schuhmann al., 2022), both of which have efficiency bottlenecks. While selective optimization tricks can minimize computational burdens as much as possible, they come at the cost of accuracy.
C.6 MORE RELATED WORKS
Aetrieval-Augmenter Generation. RAG is proposed essentially address the insufficiency or lack of desired data. For example, Long et al. (002)) improves long-tail recognition performance by retrieving text representations for tail classes. Similarly, Parasharr et al. (2024) enhances VLMs' tail accuracy by identifying and retrieving high-frequency text synonyms corresponding to tail names from the training set. Furthermore, [ddandaroo et al. (2022) mitigates data sparsity issues by retriev- ing external images through class names for data augmentation. Sharing motivations with previous work, we construct pseudo categories featuring spurious attributes through retrieval, thereby enhanc- ing the model's 's robustness to these attributes. Nevertheless, beyond retrieval, we also explore data synthesis. In Section B.5 we compare the performance our method using synthesized and re- trieved data, empirically concluding that synthesized data yields greater accuracy gains. Compared to retrieval, synthesis can offer more tailored and precise scenarios and objects, which may be more suitable for our method given the diverse identified attributes.
}
234
5
6
7
3
9
Figure 11: ColOredMNIST..
28

---

Published as a conference paper at ICLR 2025
D BROADER SOCIETAL IMPACTS
Our work has positive societal impacts. As illustrated in Fig. 5 of the main paper, VLMs may exhibit bias by associating harmful spurious attributes with target categories. For instance, when recognizing street sign, VLMs often rely excessively on concepts like street and road. This non- robust visual perception may lead severe consequences in real-world applications, particularly in autonomous driving. The introduction of SAP can effectively identify such harmful attributes and even create spurious attribute pool for specific applications. helping to determine situations where performance is compromised. Meanwhile, SAS provides effective approach to suppress the influence of spurious attributes VLMs, significantly enhancing the model's robustness against these attributes, including protected ones such as gender and race. Currently, we have not identified negative societal impacts of this work. However, due to objective factors, such as the availability of datasets and baselines' code, this will need to be further discussed in the future.
E SUPPLEMENTARY RESULTS
E.1
CONSTRUCTED IMAGES
In Fig. 12 we provide more constructed images by SAS, with Stable Diffusion and retrieval from LAION-5B, respectively.
E.2
SPURIOUS ATTRIBUTE STATISTICS
Here, present the spurious attribute statistics for the evaluated datasets. Specifically, we report proportion images containing one or more spurious attributes identified by SAS across datasets, shown in Table 23 The data reveals that for most datasets, over 50% of images contain spurious attributes, highlighting the biased nature of these datasets and the consequent spurious correlations learned by VLMs.
E.3
MOTIVATIONAL RESULTS
Given the enhanced generalization performance of VLMs before and after removing spurious at- tributes in Table of the main paper, further illustrate the impact spurious attributes, here we present the improvement of the models on the counter group in Table 24 It can be observed that the accuracy of VLMS on the counter group shows more significant improvement, up 9% on the unseen categories.
E.4 NUMERICAL MAIN RESULTS
Here we quantitatively demonstrate the main results as depicted Fig. 3 of the main paper. Table and Table 2oblee6]preeentt the numerical results of base-to-new generalization, cross-datasct transfer, and domain generalization, respectively.
29

---

Published as a conference paper at ICLR 2025
Original
SD
LAION
Figure 12: More examples of generated and retrieved images with Stable Diffusion (SD) and LAION, respectively. The category above is polar bear, with the primary spurious attribute being snow-covered ground. The category below is chocolate cake, where one of the spurious attributes the dinner plate.
30

---

Published as a conference paper at ICLR 2025
Dataset
Images with spurious attributes (%)
ImageNet
62.48
Caltech101
58.22
OxfordPets
73.54
StanfordCars
69.92
Flowers
63.50
Food101
54.59
FGVCAireraft
47.97
SUN397
52.20
DTD
42.68
EuroSAT
47.90
UCFI01
71.57
Table 23: The proportion of images containing one or more spurious attributes of 11 datasets.
31

---

Published as a conference paper at ICLR 2025
FGVCAircraft
SUN397
Flowers102
DTD
Average
Method
Base
New
SR
Base
New
SR
Base
New
SR
Base
New
SR
Base
New
SR
CPL
29.25 24.80 5.43 63.47 54.74
76.98 60.77 5.71 68.93 43.14 5.13 59.66 45.86 5.72
CPL SA
32.42
31.34
64.81
60.35
78.35 69.80
71.26 52.65
61.71
53.54
ArGue
27.24 21.92
5.13
65.23
57.44
6.45
72.26
59.32 6.69
70.77
40.12
5.97
58.87
44.70 6.06
ArGue*
27.92
23.83 4.86
65.99
58.21
6.11
72.87 60.49 6.44
71.29
41.28
5.62
61.27
45.95 5.76
ArGue SA 30.71 29.20 67.90 63.68
74.68 66.45 73.70 50.19 61.75 52.38
Table 24: The results on the counter group in base-to-new generalization before and after removing spurious attributes (SA) from the pool. We extract the counter group for both the base and new categories where spurious cues are removed. It can be observed that the accuracy of VLMs improves after removing spurious attributes in this context.
32

---

Published as a conference paper at ICLR 2025
ImageNet
Caltech
OxfordPess
Flowers
Aircraft
DTD
EonSSAT
UCF
Method
Base
New
Base
Base
New
Base
New
Base
New
Base
CLIP
72:43
91.17 97.26 63.37 74,89 72.08
91.22
27.19 36.29 69.36 15.35
59,90
56.48
77.50
CoCoUp
75.98
70.43
93.81
95.20
97,09
70.49
73.59
94.87
90.70
91.29
33:41
23.11
79.74
76.86
77,01
56.00
57.49
00.04
82.33
73.45
SAB
76.40
71.38
98,62
95.73
97.80
70.70
74.96
95.17
90.50
92.12
80.40
79.67
57.75
53.77
63.61
75.66
KyCo0pp
7538
69.96
97.72
94.30
94.65
97.76
71.76
75.04
95.00
90.50
91.70
33.55
80.29
76.53 77.55
64.34
$2.99
76.67
BAB
75.78
97.96 95.40
95.43
98.67
71,46
75.93 95.51
90.60
92.03
36.30
80.50
79.0
19.02 77.80
78
73.73 83.38
78.04
MaPl
76/66
70.54 97.74 94.36 95.43
97.76
72.94
74.00
95.92
90.71
92.05
35.61
80.83 78
80.36
59.18
94.07
73.23
78.66
76.69
70.82 97.03 95.35 95.88
73.16
95.93
76.74
91.41
92:47
37.87
39.68
81.30
80
00.72. 80.78 63.21
38
78.45
82.89
80.04
PrompiSCC
77.60 70.73 98.10 94:03 95.33
97.30
78.27
74.97
98.07
76.50
90.67
91.53
42.73
37,87
82.67
78.4
83.37
62.97
92.90
73,90
87.10
78.50
888
77.48 71.48 98.52 95.20
95.92
98.50
78.62
75.24
98.45
79.11 90.99
92:43
42
40.16
83.23
80.00
83.94
63.46
90.35
76.68
87.66
81.55
LASP
06:25 71.17 98.17 94.33 95.73
97.87
75.23
71.77
97.17
73.53
91,20
91.90
38.05
33:20
80.70
79.3
$1.10
62.57
95.00
83.37
83.53 78:20
888
76.62
72.45 98.44
95.23
96.00
98.58
76.29
72.85
96.98
75.30
92.01
92.65
38.58
33.97
80.99
80.42
81,45
64.11
95.61
83.69
85.70
79.19
TCP
77:27
69.57 98.23
94.67
94.67
97.20
80.90
74.13
97.73
73.57
90..7
91.37
41.97
34.43
82.03
78.20
82.77
58.07
91.63
74.73
55.13 80.77
77.89 70.53 98.42 95.40
95.48
98.23
80.68
75.
80 98.43
75.48
91.18
92.70
42.58
35.10
83.41
79.27
83.10
60.67
92.01
73.34
87.50
81.62
CLIP-Adapter
77.18
70.25 97.52 93.48
95.18
96.43
77.43
72.64
96.83
71.75 90.99
90.55
41.89
33.10
N0.73
77.98
H2.17
58.72
93.94
71.84
86.40
77.38
888
77.09 71.98 98.10 95.55
95.72
97.70
77.63
75.46
97.59
13.13
90.74
91.38
41.5
36.50
81:42
80.15
82.84
61.27
90.99
73.87
86.87
78:51
Tip-Adlpperr
78.04
71.96
98 68 94.17
96.21
98.57
80.79
73.66
08.
74.36
92.62
91.01
43.34
35.73
81.77
79.27
84.58
59.91
94.82
74.81
86.78 78.04
8AB
77.89 72.58 98.89 95.72
96.65
VI.43
80.84
75.42
98.81
76.73
92.84
92.81
43:27
38.31
N2.96
80.13
84.77
03.38
95.51
77.54
85.42 79.37
ArCie
76.95 71.56 98.63 94.70
96.
98.59
75.06
74.18
98.62
77.96
91.42
92:40
41.29
38,50
$1.89
80.45
80.33
67.03
95.10
90.68
56.00
79:43
BAP
77.32 72.04 98.57 95.12 96.
99.86
75.72
74.90
98.66
77878
91.54
92.63
41
39.65
N2.43
81.78
80.87
68.36
95
46
91.51
86.59
80.28
77.59 72.36 98.69 95.88 96.52
78.75
76.24
75.51 08.
79.65
91.81
93.42
42.39
40.84
82.71
H2.21
81.35
69.73
95.41
92.47
87.05
81.73
MAP
76,60 70:60 98. 30 93.80 05.43
96.90
76.70
73.75 97.57 75.23 90. 30
89. 30
41.63
36.43
02.53
76.30
82.63
66.23
92.13
76.10
86.67
78:77
+SAP
76.73
71.17 98.21 94.32 95.70 98.00
98.09 77.34
74. 10 97.
85 77.55 90.60
90.76
42.05
37.73
82.15 78.05
80.77
67.61
93.53
77.22
87.09
70.42
8AB
76.79 72:25 98.53 94.52 96.19 M8.N5 72.70
80 97.79 79999 90089 91.5 0. 42.33
39.92
M2.74 78.47 M2.87 08.44
93.30
78.21
87.44
80.8
CPL
78:74
72.00 98.35 95.13 95
98.21
79.31 76.65 98.07 90.43 91.92 93.87
42.27
38.65
81.88 79.65 80,92
62.27
94.18
$1.05
86.73
80.17
-SAP
78.76
72.64 98.67 95.72 96.3 98.87 79.24
78.12 08.37 82.51 92. 19 94
42.15 40.92
82.16
81. #2.21
65.72
94.47
83.55
86.51
80.91
+.5
78,82
73.49 98. 59 95.98 96.76 98.82 79.77 30.35 08.71 83.46 93.76 95.45 42.61 41.72 82.11 83.17 83.00
67.89
94.75
87.07
87.21
X2.22
Table 25: The numerical results on base-to-new generalization.
33

---

Published as a conference paper at ICLR 2025
Source
Cress-dataset Transfer Target
Domain Generalization Target
ImageeNe--t V
ImageNot-A
ImageNot- -R
ImageNer
Caltech
Flowers
Airera
EurOSAT
ImageNote
Peta
Cars
Food
SUN
DTD
UCF
CLIP
66.54
94.62
90.41
64.69
70.30
85.63
23.73
66.12
44,84
47,50
67.42
63.20
48.35
49.32
76.57
CoCoOp
71.02
94.43
90.14
65.32
71.88
86.06
22.94
67.36
45.73
45.37
68.21
64.07
48.75
50.63
76,18
+888
71.35
95.59
66.76
72.47
86.34
23.81
68.99
47.94
46,90
70.84
64.97
49.56
51.61
77.31
KgCoOp
70.66
93.92
89.83
65.41
70.01
86.36
22.51
66.16
46.35
46.04
68.50
64.10
48.97
50.69
76.70
+335
70.90
94.33
89.68
67.82
71.13
88.91
24.60
67.4
47.47
47.72
48.22
68.52
64.53
49.72
51.70
77.22
MaPLe
70.72
65.57
46.49
48.06
+SAS
71.21
93.61
91.76 67.53
73.60
87.58
24.54
67.69
47.98
48.17
71.96
63.98
50.74
51.57
77.25
PromptSRC
71.27
93.60
90.25
65.70
70.25
85.15
23.90
67.10
46.87
45.50
68.75
64.35
49.55
50.90
77.80
+388
71.53
93.25
92.60 60 66.44
70.13
88,
25.05
67.87
47,22
45.50
68.99
64.07
50.40
51,52
78.98
LASP
71.34
93.65
91.83 67.29
70.82
88.54
28.60
65.75
54.83
43.65
69.23
64.04
47.93
49.11
75.36
+SA8
94.62
55.74
71.63
47.91
TCP
71.40
93.97
91.25 64.69
71.21
86.69
23.45
67.15
44.35
51.45
68.73
64.60
49.50
51.20
76.73
+SA8
71.73
94.73
92.60
66.54
71.44
87.81
24.80
68.94
45.15
52.93
70.31
65.62
50.79
52.94
78.82
CLIP-Adapter
72.35
93.06
90.76 63.17
69.23
85.13
20.54
65.57
43.27
49.64
66.33
62.91
49.15
51.74
76.81
+SAS
72.53
93.12
91.72 66.65
69.18
88.10
22.27
66.60
45.69
50.38
69.80
64.50
49.70
52.39
77.75
Tip-Adapter
72.53
95.71
66.61
89.22
23.63
68.32
47.31
53.40
68.15
63.30
49.26
50.18
+SAS
72.81
95.49
94.88 67.80 68.46
91.77
25.00
69.46
49.55
54.33
68.94
64.21
50.34
50.89
77.93
ArGue
71.84
94.20
92.66 70.70 71.29
91.64
28.28
70.51
53.37
45.76
71.97
65.02
49.25
51.47
76.96
+SAP
72.14
95.2
93.75 71.80 72.48
91.87
28.53
70.88
56.54
46.86
72.96
65.47
49.94
52.48
71.38
+SAS
72.28
95.67
94.29 72.72 74.63 92.53 29.10
71.96
$7.40
48.22
73.82
66.12
49.90
52.85
77.90
71.60
93.93
90.9 63.00 68.40
86.07
24.87
68.10
51.87
42.63
68.73
64.47
49.07
51.07
77.37
+SAP
71.93
95.
92.63 64.50 68.13 87.18 26.80
69.99
51.35
44.10
70.50
65.06
49.88
51.64
77.34
+335
72.21
95.82 93.73 66.69 68.46 88, 28.62
70.29
51.91
45.73
71.59
66.14
50.78
52,19
77.70
CPL
73.53
95 91.64 66.17
87.68
27.36
68.24
48.96
51.25
70.52
63.24
50.84
52.10
76.76
+SAP
73.75
95.83 92.92 66.69 74.32 88.33 29.58
69.64
49.81
52.72
71.35
66.45
$1.93
$2.61
77.74
+SAS
95.74
49.91
$4.29
72.48
52.95
Table 26: The numerical results on cross-dataset transfer and domain generalization.
34


Reviews:
[{"review_id": "T1Q672JjZV", "reviewer": "Reviewer_bxnj", "summary": "1. [Summary]\nThis paper focuses on Parameter-Efficient Fine-Tuning for vision language models. It discovers a group of black sheep, i.e., spurious attributes, on which VLMs inherently heavily rely, thereby leading to poor generalization and robustness. It introduces SPURIOUS ATTRIBUTE PROBING (SAP), aiming to identify and eliminate these problematic attributes, thereby improving the generalization of current attribute-based methods. Besides, it presents SPURIOUS ATTRIBUTE SHIELDING (SAS), a plug-and-play module seamlessly integrating into various PEFT methods to mitigate the influence of spurious attributes on predictions. Experiments show that the proposed method achieves good performances.\n2. [Strengths]\n- a. It proposes SPURIOUS ATTRIBUTE PROBING (SAP) and SPURIOUS ATTRIBUTE SHIELDING (SAS), which enhance the generalization of existing attribute-based methods is complementary to PEFT methods. \n- b. The experiments are extensive, and the performance of the proposed method is promising.\n- c. The method is efficient, intuitive and effective.\n3.1 [Reasons to Reject]\nAlthough this paper is well written with comprehensive evaluation and good results, there are still some issues.\nSeveral parts of this paper are not very clear and need further clarification. Please check the questions.\nIn addition, some key related VLM works are missed. Please check the questions.\n3. [Weaknesses] \n1. Fair comparison issue. It is not clear whether the performance gains come from involving additional training data as illustrated in Figure 2 (a) and (b) or the designed methods. What if the compared methods also involve the additional data in training? And how about the performance then? It seems that the performance gains are largely attributed to the additional training data. It would be better to add some additional ablation studies to address this concern.\n2. The core idea of this paper is to introduce attribute information (e.g., objects, parts and subparts) into vision-language model to enhance object recognition. The related works [A, B, C] are missed. Similarly, [A, B, C] enhance object recognition performance by introducing the additional information like multi-granularity object class information and hierarchical object class information. It would be better to discuss and highlight the similarity and difference between this paper and [A, B, C]. \n[A] Adversarial fine-grained composition learning for unseen attribute-object recognition\n[B] Open-Vocabulary Object Detection via Language Hierarchy\n[C] Multiple granularity descriptors for fine-grained categorization\n\n3. The paper method relies on constructing pseudo categories via generating or retrieving, which shares similar idea with the concept of retrieval-augmented generation (RAG). It would be better to discuss and clarify the similarity and differences with RAG, for example, RAG for object recognition as in [D].\n[D] Retrieval augmented classification for long-tail visual recognition.\n4. As the objective of this paper is to enhance vision-language models, it would be better to test/verify the proposed method on multiple vision-language models in addition to CLIP. Besides, the Section 2 Related Work missing an important paragraph of “Vision-language model.”. It would be better to add a new paragraph in the Section 2 Related Work to introduce background and related papers of VLM and cite the survey [E] which could be a good reference for new readers that are not very familiar with recent progresses in VLM.\n[E] Vision-language models for vision tasks: A survey\n\n\n4. Conclusion\nOverall, this work proposes to introduce attribute information into VLM to enhance object recognition performance with good performance gains. However, there are some details that need to be clarified, as listed in the questions. I would upgrade the score if the questions are well addressed.", "questions": "as shown in Summary", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "as shown in Summary", "weaknesses": "as shown in Summary", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H6HiyW0ld8", "reviewer": "Reviewer_9qjN", "summary": "This paper aims to address the issue of VLMs overly relying on a small subset of attributes in decision-making due to spuriously correlated attributes. The paper proposes SAP to identify and filter out spurious attributes to enhance the generalization of existing attribute-based methods. In addition, a plug-and-play module SAS that integrates into various Parameter-Efficient Fine-Tuning (PEFT) methods is designed to reduce the influence of spurious attributes on predictions. The experiments demonstrate that SAP and SAS significantly improve accuracy on distribution shifts across 11 datasets and 3 generalization tasks without compromising downstream performance.", "questions": "- Will the code be made publicly available for community use?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Novelty: The paper reveals the impact of spuriously correlated attributes on the predictions of Vision-Language Models and proposes innovative methods (SAP and SAS) to tackle the problem of spurious correlations in VLMs\n\n- Performance & Experiments: The proposed methods achieve state-of-the-art results across 11 datasets and 3 generalization tasks, enhancing the accuracy of VLMs on distribution shifts without sacrificing performance on downstream tasks. The figures of results effectively express the intended meaning and support the conclusions, as illustrated in Figure 3.\n\n- Presentation: The writing is fluent and understandable.", "weaknesses": "- Does the construction of pseudo categories using the synthetic method result in additional computational costs? And have any experiments been conducted to validate the computational efficiency of this method? Additionally, could you provide some analysis of the inference time based on GPT and diffusion-related methods?\n- The paper primarily focuses on vision-language recognition. Could the proposed method generalize to other modalities (such as video) or tasks (such as language reasoning) outside of this specific task in this paper?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DIiLEKHddE", "reviewer": "Reviewer_Yv7Q", "summary": "This paper investigates how spuriously correlated attributes can lead to poor generalization in Vision-Language Models (VLMs). To address this issue, the authors propose the SAP method for identifying and filtering out problematic attributes, thereby enhancing existing attribute-based methods. They also introduce a plug-and-play SAS module to mitigate these attributes' influence on prediction. Experiments are conducted on standard benchmark datasets.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Researching spuriously correlated attributes in VLMs is a less explored yet intriguing direction.\n2. The method of identifying spuriously correlated attributes is both reasonable and effective.\n3. The SAS module can be integrated with existing methods to validate its generalization capability.", "weaknesses": "1. Most existing attribute-based methods are zero-shot, whereas the attribute filtering in this paper appears to require a few labeled training images. This raises concerns about whether comparing it to zero-shot attribute-based methods is fair.\n2. Some of the latest methods in VLM adaptation have not been discussed. For example, [1,2], which explore VLM adaptation tasks from new perspectives, should be included in the discussion. It would be beneficial to verify the complementarity of the proposed method with these new approaches.\n[1] MMA: Multi-Modal Adapter for Vision-Language Models, CVPR2024\n[2] Dual memory networks: A versatile adaptation approach for vision-language models, CVPR2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uaA9gpPaE7", "reviewer": "Reviewer_rex7", "summary": "This paper proposes a new adaptation strategy for improving the generalization of attribute-based VLM recognition methods. The authors assess the detrimental impact of spurious attribute on adapting to novel domains, and propose spurious attribute probing and shielding strategies to mitigate this influence. By learning a subsidiary task to discriminate target categories and spurious attribute categories, the adaptation performance on out-of-distribution data is notably improved.", "questions": "1.It is advisable to provide more visualization examples in Figure 5 or in supplementary materials.\n\n2.Can you provide an ablation study of the performance when generating an equivalent number of positive samples for each category?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.The motivation of this paper appears interesting to me. The authors quantitively identify the influence of spurious attributes,  a key problem for out-of-distribution visual recognition.\n\n2.The idea of learning pseudo category to mitigate the impact of spurious attribute is novel. The proposed SAS and SAP bring consistent improvement across 10 baselines, demonstrating the method’s effectiveness.\n\n3.The paper is well-written, well-organized and easy to follow.", "weaknesses": "1.During fine-tuning, differentiating each category from the synthetic pseudo category is time-consuming and tedious. Although the authors apply selective optimization to maintain overall performance，they have not fundamentally addressed the prolonged optimization time for each category.\n\n2.Only three visualization figs are presented in Figure 5, and the results can not effectively demonstrate the removal of attention to spurious attribute features, e.g., Figure 5 (b).\n\n3.During training, an additional module generates negative samples to form a strong contrast from the target categories. There’s a possibility that the performance improvement is primarily attributed to data augmentation, as the authors also mentioned in the paper. However, the authors do not compare the performance when generating an equivalent number of positive samples for each target category.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)