A Sober Look at the Robustness of CLIPs to Spurious Features
Authors: Qizhou Wang
Yong Lin
Yongqiang Chen
Ludwig Schmidt
Bo Han
Tong Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: CLIP
Distribution Shift
Primary area: machine_vision

Abstract: Large vision language models, such as CLIP, demonstrate impressive robustness to spurious features than single-modal models trained on ImageNet. However, existing test datasets are typically curated based on ImageNet-trained models, which aim to capture the spurious features inherited in ImageNet. Benchmarking CLIP models based on the ImageNet-oriented spurious features may not be sufficient to reflect the extent to which CLIP models are robust to spurious correlations within CLIP training data, e.g., LAION. To this end, we craft a new challenging dataset named CounterAnimal designed to reveal the reliance of CLIP models on realistic spurious features. Specifically, we split animal photos into groups according to the backgrounds, and then identify a pair of groups for each class where a CLIP model shows high-performance drops across the two groups. Our evaluations show that the spurious features captured by CounterAnimal are generically learned by CLIP models with different backbones and pre-train data, yet have limited influence for ImageNet models. We provide theoretical insights that the CLIP objective cannot offer additional robustness. Furthermore, we also re-evaluate strategies such as scaling up parameters and high-quality pre-trained data. We find that they still help mitigate the spurious features, providing a promising path for future developments.

A Sober Look at the Robustness of CLIPs to Spurious Features
Qizhou Wang
Yong Lin
Yongqiang Chen Tong Zhang
Ludwig Schmidt
Bo Han't
TMLR Group, Department of Computer Science, Hong Kong Baptist University
"The Hong Kong University of Science and Technology
Thee Chinese University of Hong Kong UUniversity of Washington University of Illinois Jrbana-Champpla
https://coonteraiinallggthbbbbbu
Abstract
Large vision language models, such CLIP, demonstrate impressive robustness spurious features than single-modal models trained on ImageNet. However, existing test datasets typically curated based ImageNet-trained models, which aim to capture the spurious features inherited in ImageNet. Benchmarking CLIP models based on the ImageNet-oriented spurious features may not be sufficient reflect the extent to which CLIP models are robust to spurious correlations within CLIP training data, e.g., LAION. To this end, we craft a new challenging dataset named teetterAna AounterAniman designed to reveal the reliance of CLIP models on realistic spurious features. Specifically, we split animal photos into groups according to the backgrounds, and then identify pair of groups for each class where CLIP model shows gegh-performance drops across the two groups. Our evaluations show that the spurious features captured by CounterAn: models with different backbones and pre-train data, yet have limited influence for ImageNet models. We provide theoretical insights that the CLIP objective cannot offer additional robustness. Furthermore, we also re-evaluate strategics such as scaling up parameters and high-quality pre-trained data. We find that they still help mitigate the spurious features, providing a promising path for future developments.
are generically learned by CLIP
Introduction
Large vision language models (LVLMs) have demonstrated huge success across wide range of vision and multi-modal tasks, surpassing conventional ImageNet (-trained) models by a remarkably large margin [T] LVLMs are typically trained with or based on Contrastive Language Image Pre-training (CLIP) 2 on an unprecedented scale of real-world vision and language data such as LAION [3]. which are significantly larger than ImageNet. The huge success of CLIP has presented paradigm shift for modern vision and vision-language models to conduct the pre-training from ImageNet benchmarks to web-scale multi-modal datasets
A key signature of CLIP models is the impressive robustness against various ImageNet-oriented distribution shifts (2). which shown to be prohibitive ImageNet models (5). The performance
"Equal contributions. *Corespondence to Bo Han @hhnnt@@comphhhhuucduuu
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Photos of Ice bear in snow background (easy, acCu 97.62)
Photos of Ice bear in grass background (hard, accu 70.91)
We showcase CounterAnimal examples from the class of ice bear, separated into easy and hard groups with different backgrounds (i.e., snow and grass). The zero-shot performance of CLIP-LATOM4400N-ETTBBBB3 drops from 97.62% (easy) to 70.91% % (hard).
Figure I:
boosts over ImageNet models seem to suggest that CLIP resolves distribution shifts, thereby sparking rich discussion about its rationale 6 g
9 [O]. However, the elephant in the room is that adopted testsets (i.e., ImageNet variants) to evaluate the robustness of CLIPs are primarily designed for ImageNet-based models [5]1] These datasets may not correctly reflect the exact robustness of CLIP, given that CLIP models are trained on a large amount of data that may include, and possibly extend beyond those ImageNet variants during pre-training [10]. In this paper, we investigate the robustness of CLIP to distribution shifts caused by the presence of spurious features. These features are highly correlated with labels, but this correlation may break down under distributional shifts 12 13
14 [5] 16, 17 [8 19 [0].
We raise challenging research question in the following:
Is there a benchmark that reflects the exact reliance on spurious features of CLIP?
Sadly, most of the existing benchmarks [2] 23 are tailored primarily for ImageNet models, which are unsuitable for CLIP. To fill this gap, we introduce new testset, named CounterAnimal, specifically designed for assessing the robustness of CLIP models against real-world spurious features. Figure 1 presents several examples of CounterAnimal where data divided into two groups, a) the easy group: animals in commonly appeared backgrounds that the CLIP models make correct predictions, and b) the hard group: animals in less commonly yet still plausible backgrounds, where the CLIP models are likely to misclassify them. Intuitively, the easy part captures some real-world biases that the web-scale data may naturally inherit. Hence, by comparing the performances of the two groups, one can quantify to what extent the model relies on spurious features.
More specifically, the Counter Anmi al dataset is curated based on raw photos collected from iNaturalis
Table 1: vs. 1000 results of exemplary ani- mal classes within the CounterAnmial dataset for SLIP-LAION40OM-I Vi3-B/32. "bkg" denotes the background label, "accu" (%) denotes the zero- shot accuracy, and "drop" (%) denotes the drop in accuracy between easy and hard groups.
F The construction pipeline consists of steps. a) Data collection: querying iNatural- ist with each animal class, where select some of the animal names from the ImageNet- IK dataset (25). Data curation: manually cleans- ing low-quality photos that potentially contain ambiguity and corruption. c) Background la- beling: manually annotating photos with their respective backgrounds. selected from the label space of the candidate backgrounds. d) Spurious discovering: preserving classes and associated data based on the decrease in zero-shot perfor-
objoct label
easy
hard
drop
bkg
aceu
bkg
accu
ice bear
snow
97.62
graas
70.91
26.71
black ewan
93.63
68.87
24.76
flamingo
water
79.70
sky
55.45
24.25
vulture
sky
87.76
41.84
dung beetle
earth
56.92
hand
17.02
39.90
mance (i.e., evaluating based on pre-trained CLIP models without fine-tuning) when shifting the backgrounds. The resulting Counter inimal dataset covers total of 45 animal classes, and ends up with 7,174 easy photos and 5,926 hard photos, aligning with the standard size as an evaluation dataset, such as 126 27). Moreover, ELIP-LAION400M, -VTTTB//M----TB/33 is used as the proxy CLIP model in spurious discovering (ef., Appendix C.5
for the model naming rules).
We evaluate the CLIP models on our CounterAnmial with various backbones, e.g., ViT (28). along with different pre-train datasets. e.g., LAION [3) We also consider more advanced LVLMs like MiniGPT4 29) and LLLa VA [30]. We employ two evaluation setups crafted for different families of models (cf., Appendix C a) vs. 1000 setup: using the full ImageNet- K class names as the candidate label space and b) I vs. 20 setup: using the top-20 most confusing classes regarding -OL--LAINNOOMM--T--BBB-B
as the candidate label space. We provide some of results in Table highlighting the key observations in the following:
and Figure 2
https: www.:/aaaantttttttmmmmmmmm www.auurrals...rgbbbaeraaataaaaa

---

ImageNet
yex
Imagehet
ymx
600
AiniGPTeWicc Mice
40
1aW05 5-77
eaay
larger
largrr
larger dutaset
larger model
(a) 1000 (label space of ImageNet-IK)
(b) 1 vs. 20 (20 most confusing labels per class)
The easy vs. hard performance (%) for CLIP, ImageNet models, and more advanced LVLMs, ie., MiniGPT4 and LLaVA. The marker size indicates the backbone scale and the color shade indicates pre-train data scale. We highlight the CLIP models pre-trained on high-quality datasets, i.e., DataComp (CLIP-DC) and Data Filtering Networks (CLIP-DFN). We linearly fit the trends for CLIP (CLIP, CLIP-DC, and CLIP-DEN) and ImageNet models to show their effective robustness. We also depict the perfect trend, i.e., y I, where the models will not learn any bias.
Figure 2:
Counter Amall captures general spurious correlations within CLIP. As exemplified in Table I we observe a significant drop of CLIP-LATON40OM-VV VVT-OO/MM-TTB//-- in zero-shot accuracy from the easy to hard groups for each class. Furthermore, the observed biases in CLIP MAAAIINNOOOMVVT-B//AA generalize to other CLIP configurations, with non-trivial performance drop from the easy hard groups across various backbones and pre-train datasets as in Figure 2 It implies that CounterAnimal characterizes some general spuriousness common large-scale multi-modal datasets.
also
ImageNet models are more robust to spurious correlations captured by CounterAnimal, Fig- ure also illustrates the performance changes of ImageNet models (colored red). Compared with CLIP models (colored in blue), we find that ImageNet models exhibit stronger robustness to spurious correlations captured by CounterAnimal. Our findings contrast with previous studies that assess the ImageNet variants (2) highlighting that CLIP models do not always generalize better than ImageNet models. underscores the necessity of choosing appropriate benchmarks to comprehensively assess the robustness of different models and training schemes.
the sizes and the color shades of the markers to indicate the scales of backbones and the pre-train datasets, respectively. Overall, larger CL.IP backbone models (i.e., larger markers) can improve the effective robustness, implying that scaling up backbones may enhance model performance against spurious features. In contrast, increasing the scale of the pre-train dataset (i.e., darker markers) does not yield the same improvement, implying that collecting more data alone cannot rectify much bias, which provides some new understanding addition to the data-centric perspective 6
Larger CLIP models are more robust. Shown also in Figure 2
CLIP models trained on high-quality data are more robust. We categorize CLIP models into two distinct groups according to the pre-train data quality: a) CLIP-DC using DataComp [ and CLIP-DFN employing Data Filtering Networks [31, as well as b) those pre-trained on datasets that lack stringent curation, labeled simply as CLIP. The results indicate that CLIP models pre-trained high-quality datasets demonstrate enhanced robustness in general. It suggests that enhancing data quality remains a promising strategy for mitigating the spurious features.
The CLIP objective may not offer additional robustness. Complementary to our empirical observations, we also provide theoretical explanations for the reasons why CLIP learns spurious features. We further conduct confirmatory experiments that fine-tune CLIP models onto datasets with synthetic spurious features. The results align with observations Counter Animal that the CLIP objective can not offer additional robustness over standard single-modal supervised training.
Comparison with previous results. Our work presents new benchmark to effectively and system- atically evaluate the robustness CLIP models, which complements the literature in understanding the generalizability of CLIP models and LVLMs. More specifically, 32 reports that CLIP models
3

---

may wrongly align co-occurred objects with their texts. 13 reports similar failure modes for more sophisticated LVLMs such MiniGPT4 or LLaVA. B4 finds that CLIP misaligned samples will further cause the hallucination of LVLMs. Complementary to these works, our study explicitly characterizes the spurious features captured by CLIP and explains the existence of the reported failure cases. Our study provides interesting empirical and theoretical counterexample to the previous beliefs for the substantial improvements in robustness for CLIP models, especially for those results observed on ImageNet variants 17 & 9 35 Based on the newly collected CounterAnimal dataset, we suggest that distribution shifts remain an open problem for CLIP models. Also, we need to be cautious about the test setups when evaluating new models pre-trate rre-trained on datasets that differ significantly in scales and distributions from traditional ImageNet models.
Comparison with previous benchmarks. There are many other datasets to study distribution shifts, e.g., ImageNet variants [21 26
DomainBed [41], and Wilds [42]. However, these datasets have biases when assessing the OOD robustness of CLIP models, as they may fail to represent the true OOD scenarios during CLIP training. Moreover, numerous recently released 45 have also explored distribution shifts. However, these studies primarily focus on synthetic distribution shifts, which may not fully represent real-world cases. In fact, it has been shown that previous OOD benchmarks are contained in CLIP training [T), making it hard to ablate ID/OOD cases for data in these benchmarks. Consequently, CLIP models have shown to be more robust than ImageNet models these contaminated datasets [46].
36
37
38
39
40),
datasets, such 22 23 43
44
2 Dataset and Evaluation Setups
To begin with, we describe the basic experimental setups, including the pipelines in constructing CounterAnimal, its key characteristics, as well as the adopted evaluation settings.
2.1 Construction of CounterAnimal
We introduce the curation pipeline of our new dataset CounterAnimal, tailored for CLIP investi- gate spurious correlations. The pipeline consists of 4 steps as follows:
Data Collection. We query animal names listed in the ImageNet-Ii IK dataset and collect raw data via the search interface of iNaturalist, a global biodiversity data-sharing platform. We retrieve the latest 300-800 photos per animal class, organizing them based on the queried labels.
Data Curation. The collected raw samples are susceptible to noise and ambiguities. Therefore, we manually cleanse the low- low-quality data that fall into any one of the following 4 situations: label noise, feature noise, obscurity, and clarity. Label noise refers to cases where photos do not belong to the queried classes; feature noise refers to cases where some pixels are disrupted or missing; obscurity occurs when photos belong to more than one object class; clarity issues refer to cases where animal objects are largely occluded by the backgrounds or other irrelevant objects. It also includes the cases where animal objects do not occupy the majority of the space in photos.
Background labeling, We consider a typical form of spurious features where the backgrounds of photos can be biased [47). To identify such data for CLIP models, we manually label the backgrounds for the curated data. The considered class space of backgrounds is defined as fol- lows: ground, water, earth, sand, snow, grass, human, sky, road, rock, shrub, indoor, tree, and outdoor. Note that the class space of backgrounds above is not entirely orthogonal due to the inherent ambiguity: Some backgrounds may be ambiguous and some photos may contain more than one background. Nevertheless, we try our best to determine the assigned background labels for each animal class and exclude those photos challenging to be labeled.
Spurious Discovery. For each class, we quantify the impacts of spurious correlations to CLIP models by comparing the performances on the associated samples across different backgrounds. We take those classes as containing spurious features on which we observe a relatively obvious decrease in accuracy when changing backgrounds. In realization, we adopt the checkpoint of LAPPLLAANNNOMMMVVTT-BBB
for evaluation, where the prompt for its text encoder is "A photo of <object label>.", and the space of <object label> is the ImageNet- class names, i.e., we follow an 1 vs. 1000 setup. Then, we consider the classes where the zero-shot accuracy varies by more than 5% when changing backgrounds as the cases where CLIP model has learned the spurious features. The data with the preserved classes and backgrounds are used to create our final

---

115
100
1030
20
147
45
00220
77
79
23:00
200
2200
28
1112 13
20
24
25
27
29
31
3334435
37
39
40
cless
Figure 3: The data layout across various animal classes. The horizontal axis denotes the class IDs and the vertical axis denotes the number of photos for the easy and hard groups, respectively.
67.2
35
40.3
12
403
29.1
20
10.20
23 25
27
28
3031
34
36
40
63
Figure 4: The vs. 1000 performance drop (%) with CLIP MLLP-LATON4OON-VTT-BBB axis denotes the class IDs and the vertical axis denotes the percentage points of decline.
The horizontal
CounterAnimas dataset. Photos with the highest CLIP accuracy are assigned to the easy group, and those with the lowest CLIP accuracy are assigned to the hard group. We further refine the collected data remove any mistake that the labelers may made during data curation and background labeling.
Our objective in developing CounterAnimal is to reflect the spurious correlations learned by CLIP. Therefore, need to employ the CLIP models for dataset curation and thus ensure the construction effectively biased towards CLIP configurations [26]. In Appendix [ we further show that our data curation pipeline is general and reliable to characterize the spurious features within the considered models. Moreover, our experimental results later in Section 3 will corroborate that the spurious features captured by our CounterAnimal dataset are general across different CLIP setups and may not be so influential for ImageNet benchmarks, These findings will justify that our crafted testset satisfies our primary objective characterizing the spuriousness for CLIP specifically.
2.2 Characteristics of CounterAnimal
We depict the data layout in Figure 3 and visualize the zero-shot gaps for each animal class in Figure 4 where we use CLIP- MOOP--LAON40OM-VV-----
as our referred model. Please refer to the detailed object/background names concerning the easy and hard groups Appendix B Recalling that, when CLIP models resort to the shortcut of data, the model performance will heavily correlate with the backgrounds presented in the easy group yet compromised when coming the hard group. Accordingly, Figure implies a reliance for the CLIP models on the backgrounds.
2.3 Evaluation Setups
We evaluate a series of CLIP models on the CounterAnimal dataset for their zero-shot performance. For each class, we use the pre-defined prompt of "A photo of <object label>." as in our data collection procedure and the similarity between image and text embeddings in classification. By default, we use the label space of the ImageNet- K dataset and report the top-1 accuracy, i.e., the 1 vs. 1000 setup. Moreover, when involving more advanced LVLMs, we adopt the vs. 20 setup where we employ the top-20 most confusing classes regarding CLIP-LAION40OM- ViT-//33 as the candidate label space. For re-productivity, we adopt the pre-trained CLIP checkpoints from OpenCLIP (48) and ImageNet model checkpoints from the PyTorch repository. The model naming rules in Appendix C.5 and the evaluation details are discussed in Appendix
Experimental Analysis
Our experiments center on the evaluation and the analysis of our CounterAnimal dataset. In Section 3.1 we examine the generality of the captured spurious correlations. In Section 3.2 we explore the potential facets that affect the robustness of CLIP models. Section we extend the evaluation to broader family of models with different training paradigms.
5

---

3.1 Generality of the Spurious Correlations
we discover spurious correlations using CLIP-LATONAOON--VE -iT-- and collect associated data to build the CounterAnime dataset. A critical problem then arises: Is our dataset general benchmark to examine spurious correlations of CLIP with other pre- train datasets and backbones? Hence, we need to examine whether the biases in the Counter imal dataset can hinder the robust- ness of other CLIP models, where we con- sider two situations: fixing pre-train datasets while varying backbones and b) varying pre- train datasets while fixing backbones.
In Section 2.1
67.1
69.1
72.9
67.1
48.7
37.0
37.0
easy hard
VIr882
VI58/16
VITL/14
LAIONAGOM
OpenAi
LAION2B
(a) LAION40OM
(b) ViT-B/32
Figure 5: The 1 vs. 1000 results for varying CLIP setups beyond TLIP-LATON4OOM-VITT fixing the pre-train dataset be LAION40OM and b) fixing the backbone to be ViT-B/32.
a)
Varying Backbones. We fix the pre-train dataset to be LAION40OM and explore two other
backbones within the ViT family [28] i.e., F16-B/16 and ViT-L/14. Their zero-shot results are de- picted in Figure There remains a drop above 17 percentage points for both the cases of ViT-B/16 and ViT-L/14. suggests that the CounterAn dataset captures some general spurious shifts that are at least present in the pre-train dataset of LAION40OM.
Varying Pre-train Datasets. We fix the backbone be ViT-B/32 and consider other pre-train datasets. Here, we consider LAION2B and the closed-source dataset used by OpenAI. Their easy and hard results are Figure (tb). Here, the spurious features affect the zero-shot robustness of CLIP models trained on both LAION2B and by OpenAl, indicating that our CounterAnima dataset possesses some realistic shifts that are contained various CLIP setups. Therefore, we conclude that CounterAn Animal captures some general spurious features learned by CLIP models.
3.2 Scaling up May Relieve Spurious Correlations
We extend our evaluations to a wider range of CLIP models with different scales of pa- rameters and pre-train data. The results are summarized in Table and and further depicted in Figure a). Generally speaking, perfor- mance drops can be observed across all con- sidered CLIP configurations, indicating that CLIP models in various scales still learn spuri- features. More specifically,
Table 2: The vs. 1000 results for CLIP check-
points on the CounterAnimal dataset. The pre-train datasets with high-quality data are marked by
backbone
pro-train dataset
easy
hard
drop
RM-1001
OpenAl
64.27
45.15
19.12
RM-50x4
OpenAL
investigate
V1T-B/16
LATOK400M
73.11
52.17
20.94
VIT-B/16
OpenAT
the influence of a) parameter scales and b) pre-train data scales in CLIP models on the sensitivity of spurious features. We exclude the backbone of ViT-B/32 and the dataset of LAION40OM to avoid biases data collection.
V1T-B/16
DataComp18
80.36
64.24
16.12
V1T-B/16
LAION2B
20.00
ViT-B/16
DFF2B
85.03
70.61
14.42
ViT-B/32
LAIOK400%
67.13
36.95
30.18
V1T-B/32
OpenAI
69.
45.62
23.51
VIT-B/32
DataConp18'
75.96
53.74
22.22
VIT-B/32
LAION26
48.74
24.20
Scaling up Pre-train Data. To test the im- pacts enlarging scales of pre-train datasets, we consider two CLIP backbones, namely, ViT-B/16 and ViT-L/14, along with a se- ries of pre-train datasets of increasing sizes. The results are summarized in Figure 6 We observe that scaling up the data scale does not necessarily reduce the performance drop. suggesting that directly enlarging the scale of pre-traie pre-train data alone cannot enhance robustness. One possible explanation is that larger datasets do not imply fewer biases, whereas the CLIP models will still inherit the spurious correlations therein.
VIT-L/14
LATOK400M
80.90
63.31
17.59
VIT-L/14
OpenAl
ViT-L/14
DataComp18
89.29
79.90
9.39
ViT-L/14
LAION2B
VIT-L/14
DFN28
90.77
80.55
10.22
VIT-1/14-336
OpenAI
73.14
13.21
ViT-H/14
LAIDW2B
85.74
73.13
12.61
VIT-H/14
DFU5B
79.13
9.42
VIT-G/14
LAIOW2B
86.81
73.32
13.49
Vi7-big0/14
LAIOM26
87.57
76.96
10.61
Scaling up CLIP Model Sizes. We also explore the connection between model scales and spurious conrelations. In Figure 7 we consider two pre-train datasets, namely, LAION2B and the close-soured

---

25.7
86.4
64.2
70.3
66.3
82.
73.2
66.3
23.1
73.3
77.
64.74
731
56.6
53.2
56.
45.0
48.6
Openik
DitaCompls
LAIONZS
Opondi
DataConp.l 18
LAOHZE
wrenss
WEHILL
vibaigeis4
8N-101
Rik Soxs
viaste
viuna
WFLL4-306
(a) V1T-B/16
(b) ViT-L/14
(a) LAION2B
(b) OpenAl Checkpoints
Figure 6: vs. 1000 results for varying CLIP Figure vs. 1000 results for varying CLIP setups with different pre-train datasets.
setups with different backbones.
30.2
20.5
20.3
222
242
1.44
16.12
15.5
102
LAONAOOM
Openai
LAKCHD
DetaCan Brlle
LANOMAOOM OpnnAl
LAUGNOS
DetaCompll
OpanAi
DatuComp
imagethal
LaIoNaap Opanki
Datec DataComp LAINN2
imugeteet
(a) ViT-B/16
(b) ViT-L/14
(a) ViT-B/16
ViT-B/32
Figure 8: vs. 1000 drops for varying CLIP Figure 9: vs. 1000 drops for varying training
setups with filtered and unfiltered pre-train data. setups with CLIP and ImageNet supervision.
data from OpenAl, along with backbones of increasing scales. We observe a clear trend indicating that larger models exhibit better performance against spurious correlations. It may tell us that larger models possess stronger robustness, making them less prone to the shortcuts of spurious features.
Data Quality Matters. Moreover, we observe that the results obtained with DataComp- and DFN- trained CLIPs exhibit better performance and smaller drops across backbones, Figure 8 offers their comparisons. We notice that these datasets have been stringently filtered and thus possess high-quality data. It indicates that enhancing data quality is still promising way improve OOD generalization.
Our analysis focuses on absolute performance drop. In Appendix we strengthen our conclusions by incorporating the analysis based on effective robustness 5 where our findings still hold.
3.3 Evaluations for other Learning Paradigms
We extend our evaluations broader families of mod- els, including ImageNet- IK supervised models and more advanced LVLMs, such as MiniGPT4 and LLaVA.
Table 3: The 1 vs. 1000 performance for ImageNet models Counter Animal.
ImageNet Models. We first extend our evaluations to in-
clude ImageNet models. The main results are summarized 9 further illustrates the accu- racy drops of various CLIP models, comparison Ima- geNet models. Surprisingly, we find that ImageNet models are more robust to spurious features This finding indicates that our CounterAnimal specifi- cally characterizes the spurious features that are unique to CLIP configurations. Additionally, it indicates that spu- rious correlations in large-scale multi-modal data are dis- tinct from that of the ImageNet scenarios which are widely used conventional single-modal supervised learning. It further highlights the importance of proposed dataset, which is especially suitable to study the spurious correla- tions for Dision-language pre-training.
backbone
essy
drop
in Table 3 Moreover, Figure
AlexNet
59.56
39.24
20.31
73.37
56.12
17.25
VOG-13
75.33
58.43
16.90
CounterAniman
RI-188
74.36
56.07
18.29
RN-34
78.31
17,30
RM-50
81.44
66.07
15.37
RX-101
81.76
68.18
13.57
VIT-B/16
ViTTB/32
79.84
64.36
15.48
VIT-1/16
VIT-1/332
81.23
67.54
13.69
ConvNext-9
88.27
79.97
8.30
Convilxtt-B
88.60
80.53
8.07
ConvNext -1
89.12
81.47
7.65
Advanced LVLMs. We further evaluate for more ad-
vanced LVLMs, which align CLIP visual encoders with advanced large language models like Vi-
7

---

Table 4: The 1 vs. 20 results of CounterAnimal for advanced LVLMs and several CLIP models.
More results of CLIP models and ImageNet models can be found Appendix F.
LVLMS
essy
hard
drop
NiniGPT4- Kin1GFT4-Viccuma7
Viccuna78
47.99
39.73
8.26
LLeVA1. 5-78
40.06
30.09
9.97
80.90
63.31
17.59
CLIP-
CLIF-OpeuAT-VIT-L/EE
85.38
70.28
15.10
79.90
CLIP-LAICW2B-VTT-LLE
82.23
66.27
15,96
90.77
80.55
10.22
cuna [49]. To reduce inference costs, our evaluation follows the 1 vs. 20 setup. We summarize their results in Table 4 along with the I vs. 20 results for several CLIP models (cf., Appendix for more results). We further depict the full results in Figure bb). As we can see, these advanced LVLMs have lower performance yet smaller drops, but the spurious features in CointerAnimal still impact them.
4 Understanding Why CLIPs Rely on Spurious Features
To better understand the observed phenomena in Section 3
we present a theoretical analysis of why the CLIP models rely spurious features. We begin by establishing the setup for analyzing multi-modal contrastive learning following [9].
Definition 1 (Multi-modal Dataset). Consider n image-text pairs {(x), x both image æj
and text z are generated from the latent factor 2ú. where = [zinv: 2xpu, E IR2 is composed of an invariant feature Zinv (pinoy), Aie) and a spurious feature Zspu [[ a, ppp) with
Pr(a- y) Pspu otherwise -y- y the label uniformly drawn from 11 The training data D is drawn with 2 Papu and OOD data DD' is drawn with a Papu 2"
We employ two linear encoders: 91 R - R² for the image modality and gr RR - R for the text modality. implemented as (x1) W jx and gr(rr) = W WIEr with W E R and
WT E Raxxdr_ The encoders are trained through the linearized contrastive loss 9 50) that mimics the CLIP dynamics:
LCLP = 2n(n - 1) ΣΣ jot
(sis - -8s) + 2n(n 1) ΣΣ - sw) + W.W. (1) Ju
(s
where 8iy = ga) (2=f) is the similarity with respect to the i-th image and j-th text represen- tations. Once the CLIP (g), has been trained, the performance will be measured in zero-shot manner by matching the most similar caption with the corresponding object name filled in, such as "a photo of <object label>" " [2] Intuitively, once the model focuses more on invariant features, will have better zero-shot classification accuracy across different distributions. Nevertheless, in the following theorem, we justify that CLIP remains to learn use spurious features, aligning with our experimental observations on the CounterAnimal dataset.
Theorem 1. Given a multi-modal dataset (Def. [ with suitable variance in the features Tsne = (1)) > Sppa- and spurious features with large spurious correlation Pspa 1 o(1), overparameterized CLIP model where n. = w(1), dy = 2(n) and dr = =22()) if the spuri- ous features (e.g., backgrounds of the image) takes up relatively large amount of the image
( = 11 o(1), the CLIP model achieves large error in zerv-shol accuracy in the OOD test data where + y:
Hapu. 2 42n 2 Pine = L, then with high probability of at least 1 1-0( polytn)
Errigs, gr) 2 - -}(K)) -()),
and a small error in the OOD test data where = y::
Acc(gr. (T) 2 1 - (k22) - (1))
=1-((Kg)
where K1 1 Poy (apopaPave 1pa?
Kg
2M Paru 2
and T
V1+02 Pay th (2 Pope 1
denotes the CDF of standard normal distribution.
8

---

94.4
95.7
94.1
94.8
97,9
97.1
69.7
68.3
71..
21.0
alrplane
5.3
0.8
0.2
1.3
1.7
background
aero-shot
supervised ob)
abjbkg
zero-shot
supervised
abl
ob|bkg
(a) OpenAI -RN-A
(b)
phoro cyan bockground
phote tackgrund
Training Data
Data
Figure 11: CLIP performance on ColoredCDCO. "super- vised" refers to supervised trained models, while "obj" and "objbkg" refer to using different prompts to fine-tune CLIPs.
Figure 10: Illustration of ColoredCOCo.
We leave more theoretical details well the proof Appendix D due to space limit. Intuitively, Theorem T iimplies that once there exists a relatively strong correlation between the object captions and the parts of image backgrounds. CLIP will learn align the backgrounds, i.c., spurious features, with object captions. Although our theory discusses simplistic case of one invariant and one spurious feature, there could exist more features describing the objects and even more features describing the backgrounds, CLIP will fail to robustly align the visual features of objects to its captions, once there exists spurious correlation between any of the background features with the object caption. Our theory is the first to provably demonstrate the drawbacks of CLIPs in OOD generalization, providing the foundation for future developments tackling the issue
To verify our theory, we construct multi-modal datasets named ColoredCOCO following (51] It con- tains 9 classes and the spurious correlation in the training part is 80%. i.e., each class has correlation of 80% to specific biased color and 20% uniformly correlates to 10 different randomly chosen colors, cf., Figure IO] The OOD datasets are built with classes randomly correlating other 8 biased colors. We consider two prompts with different descriptiveness: a) obj: : "a photo of <object label>" and b) obj "a photo of <object label> <color label> background", with either objects or both objects and backgrounds.
We tune the treitrained CLIP models using the CLIP objective, which has been shown to be most robust to distribution shifts [52). In addition, we also incorporate the baseline of full fine-tuning with MLP onto the image encoder using the ERM objective. As shown in Figure [1] fine-tuning with CLIP objective based on neither of the prompts provides any non-trivial robustness against the vanilla full fine-tuning. The results further verify theory. Nevertheless, the degraded robustness of CLIP could also be caused by the weak language understanding capability of the BERT encoder in the CLIP. To this end, we also conduct additional experiments with a perfect language encoder setting. The results are given in Appendix D.4 Nevertheless, we find that CLIP still performs similarly to ERM and is prone to distribution shifts even with perfect captions.
Conclusion
In this paper, we highlight biases in previous evaluations for assessing the robustness of CLIP models, primarily relying ImageNet variants. Such improper benchmarking would cause illusions that CLIP models seem to resolve spurious correlations, particularly in comparison with ImageNet models. motivates us to craft the new testset, named CounterAniman which specifically designed to probe the natural spurious correlations between animal and their backgrounds. The spuriousness is general across different CLIP setups and exerts relatively small
captured by CounterAnime impacts on the ImageNet benchmarks, thereby specifically capturing the spurious correlations within CLIP setups. Our experiments on CounterAnima show that many conventional strategies, e.g.. increasing backbone scales and improving pre-train data quality, remain effective in enhancing the robustness of CLIP models. Moreover, we present theoretical analysis for the reasons of the CLIP objective to learn biases. Overall, we provide platform for future developments of more advanced and robust CLIP and vision-language models, and we hope our presented experiments can offer sober look at the robustness of CLIP models spurious correlations.

---

Acknowledgments
The authors would like to express their sincere gratitude the anonymous reviewers and the area chairs for their thorough review and constructive feedback. Their insightful comments and valuable suggestions have significantly enhanced the quality and clarity of this manuscript. We deeply appreciate their time and effort in helping us improve our work.
References
[1] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
[2] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In JCML, 2021.
[3] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. In NeurlPS, 2022.
[4] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang. Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. arXiv preprinl arXiv:2304. /40004//408, 2023.
[5] Zhouxing Shi, Nicholas Carlini, Ananth Balashankar, Ludwig Schmidt, Cho-Jui Hsich, Alex Beutel, and Yao Qin. Effective robustness against natural distribution shifts for models with different training data. In NeurIPS, 2023.
[6] Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (CLIP). In JCML, 2022.
[7] Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang, and Tatsunori Hashimoto. Is caption worth a thousand images? a study on representation learning. In ICLR, 2023.
[8] Imant Daunhawer, Alice Bizeul, Emanuele Palumbo, Alexander Marx, and Julia E. Vogt. Identifiability results for multimodal contrastive learning. In ICLR, 2023.
[9] Yihao Xue, Siddharth Joshi, Dang Nguyen, and Baharan Mirzasoleiman. Understanding robustness of multi-modal contrastive learning to distribution shift. arXiv preprint 2rXiv:231000497) 2023.
[10] Prasanna Mayilvahanan, Thaddäus Wiedemer, Evgenia Rusak, Matthias Bethge, and Wieland Brendel. Does CLIP's generalization performance mainly stem from high train-test similarity? In NewrlPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models, 2023.
[11] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roclofs, Raphael Gontijo Lopes, Hannanch Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In CVPR, 2022.
[12] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- 2019.
mization. arXiv preprint
[13] Xiao Zhou, Yong Lin, Weizhong Zhang, and Tong Zhang. Sparse invariant risk minimization. In International Conference on Machine Learning, pages 27222-27244. PMLR, 2022.
[14] Yong Lin, Hanze Dong, Hao Wang. and Tong Zhang. Bayesian invariant risk minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16021-16030, 2022.
[15] Xiao Zhou, Yong Lin, Renjie Pi, Weizhong Zhang, Renzhe Xu, Peng Cui, and Tong Zhang Model agnostic sample reweighting for out-of-distribution learning. In International Conference on Machine Learning, pages 27203-27221. PMIR, 2022.
[16) Yong Lin, Fan Zhou, Lu Tan, Lintao Ma, Jiameng Liu, Yansu He, Yuan Yuan, Yu Liu, James Zhang, Yujiu Yang, et Continuous invariance learning. arXiv preprint arXiv: 13.00534 2023.
10

---

[17] Yong Lin, Lu Tan, Yifan Hao, Honam Wong, Hanze Dong. Weizhong Zhang, Yujiu Yang, and Tong Zhang. Spurious feature diversification improves out-of-distribution generalization. arXiv preprint arXiv:2309. 17230, 2023.
[18] Xiaoyu Tan, Lin Yong, Shengyu Zhu, Chao Qu, Xihe Qiu, Xu Yinghui, Peng Cui, and Yuan Qi. Provably invariant learning without domain information. In International Conference Machine Learning, pages 33563-33580. PMLR, 2023.
[19] Yonggiang Chen, Kaiwen Zhou, Yatao Bian, Binghui Xie, Bingzhe Wu, Yonggang Zhang, MA KAILI, Han Yang, Peilin Zhao, Bo Han, and James Cheng. Pareto invariant risk minimiza- tion: Towards mitigating the optimization dilemma in out-of-distribution generalization. In The Eleventh International Conference on Learning Representations, 2023.
[20] Yonggiang Chen, Wei Huang, Kaiwen Zhou, Yatao Bian, Bo Han, and James Cheng. Under- standing and improving feature learning for out-of-distribution generalization. In Advances in Neural Information Processing Systems, 2023.
[21] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. NeurIPS, 2019.
[22] Joshua Vendrow, Saachi Jain, Logan Engstrom, and Aleksander Madry. Dataset interfaces: Diagnosing model failures using controllable counterfactual generation. arXiv preprint arXiv:2302.07866 2023.
[23] Viraj Prabhu, Sriram Yenamandra, Prithvijit Chattopadhyay, and Judy Hoffman. Lance: Stress- testing visual models by generating language-guided counterfactual images. NeurlPS, 2023.
[24] Xiaodan Li, Yuefeng Chen, Yao Zhu. Shuhui Wang, Rong Zhang, and Hui Xue. Imagenet-e: Benchmarking neural network robustness via attribute editing. In CVPR, 2023.
[25] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
[26] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural
adversarial examples. In CVPR, 2021.
[27] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
[28] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[29] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint ar0iv:2304. 10592, 2023.
[30] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
[31] Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309. 2tt::2239./72555 17425, 2023.
[32] Yu Yang, Besmira Nushi, Hamid Palangi, and Baharan Mirzasoleiman. Mitigating spurious correlations multi-modal models during fine-tuning. arXiv preprint arXiv:2304.00 03330403916,, 2023.
[33] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint, arXiv:2305. 10355, 2023.
[34] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint 20X1022401.06209 2024.
[35] Qi Zhang, Yifei Wang, and Yisen Wang. On the generalization of multi-modal contrastive learning. In ICML, 2023.
[36] Haohan Wang, Songwei Ge. Zachary Lipton, and Eric Xing. Learning robust global represen- tations by penalizing local predictive power. NeurIPS, 2019.
[37] Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. ArXiv preprint 200.:2000009994, 2020.
11

---

[38] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness natural distribution shifts in image classification. NeurIPS, 2020.
[39] Vaishaal Shankar, Achal Dave, Rebecca Roelofs, Deva Ramanan, Benjamin Recht, and Ludwig Schmidt. Do image classifiers generalize across time? In ICCV, 2021.
[40] Yong Lin, Shengyu Zhu, Lu Tan, and Peng Cui. Zin: When and how to learn invariance without environment partition? NeurIPS, 2022.
[41] Ishaan Gulrajani and David Lopez-Paz. In search lost domain generalization. arXiv preprint arXiv:2007.014324 2020.
[42] Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08733 2019.
[43] Gregory Plumb, Marco Tulio Ribeiro, and Ameet Talwalkar. Finding and Fixing Spurious Patterns with Explanations. arXiv preprint Sttiv:2106.02110 2021.
[44] X Li, Y Chen, Y Zhu, S Wang, R Zhang. and H ImageNet-E Xue. Benchmarking neural network robustness via attribute editing. In CVPR, 2023.
[45] Chenshuang Zhang, Fei Pan. Jummo Kim, In So Kweon, and Chengzhi Mao. Imagenet-d: Benchmarking neural network robustness on diffusion synthetic object. In CVPR, 2024.
[46] Yann Dubois, Yangjun Ruan, and Chris Maddison. Optimal representations for covariate shifts. In NeurlPS 2021 workshop distribution shifts: connecting methods and applications, 2021.
[47] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In ICLR, 2020.
[48] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR, 2023.
[49] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, al. Judging llm-as-a-judge with mt-bench and chatbot arena. arviv preprint
0..-2336005685 2023.
[50] Ryumei Nakada, Halil Ibrahim Gulluk, Zhun Deng, Wenlong Ji, James Zou, and Linjun Zhang. Understanding multimodal contrastive learning and incorporating unpaired data. In AISTAT, 2023.
[51] Faruk Ahmed, Yoshua Bengio, Harm van Seijen, and Aaron Courville. Systematic generalisation with group invariant predictions. In ICLR, 2021.
[52) Sachin Goyal, Ananya Kumar, Sankalp Garg. Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In CVPR, 2023.
[53] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model unified interface for vision-language multi-tast learning. arXiv preprint arXiv:2310.099 00.0478, 2023.
[54] Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterizzttoon exacerbates spurious correlations. In ICML, 2020.
12

---

A Broader Impacts and Limitations
The current community often overestimates the robustness of CLIP models, largely due to the potentially misleading reliance on ImageNet variants for testing. To address this issue, we propose new testset, named CounterAnimal specifically tailored for CLIP models. Our findings indicate that CLIP models may not be as robust to distribution shifts as previously believed. Our dataset serves as real-world benchmark, poised to be meaningful for the subsequent works to understand and enhance CLIP concerning their OOD robustness. For real-worle applications, the understanding of spurious correlations for CLIP is also critical. We raise practical concerns when deploying CLIP models, which pertain to fairness and potential biases that may arise from inherent spurious correlations. We also present general strategies and theoretical analysis understand the spurious correlations within CLIP models, which may motivate subsequent works to further enhance CLIP in real-world applications. However, although dataset reaches the bar standard evaluation dataset, its research potential can be further benefited from expanding the scale of our dataset, diversifying the raw data sources beyond iNaturalist, broadening the semantic scope beyond animal classes, and studying other testbeds beyond the ImageNet benchmarks. the future, we will extend our focus beyond animal subjects and include wider array of high-quality data that suitable for evaluating the robustness of CLIP and more advanced LVLMs.
B Dataset Composition
We release our dataset CounterAnimal structured as follows: CounterAnimal
ostrich
easy-ground
figurel. jpeg
figure2.jpg
hatd-water
figure1. jpeg
figure2. jpg
brambling
easy-green
figure1. jpeg
figure2.
hard-sky
f figurel. jpeg
figure2. jpg
Overall, the CounterAnimal dataset is organized by the object names. The data therein are further separated into two parts, i.e., the easy and hard groups, where the background name is also provided for each sub-directory. By evaluating accuracy with respect to the easy and hard groups, one can quantify the impacts of the spurious correlations captured by CounterAnimal. We further summarize the ImageNet animal objects as well as the group names for the easy and hard groups in Table
C Experimental Configurations
In this section, we provide more details about our experimental configurations.
C.1 Hardware Configurations
experiments are realized by Pytorch 1.81 with CUDA 11.1, using machines equipped with GeForce RTX 3090 GPUs and AMD Threadripper 3960X Processors.
13

---

Table 5: The object names and the background names in the CounterAnimal dataset. The full names of labels are presented following the fashion of the ImageNet- IK dataset.
ID obyex label
hard
1D
dbjoct bed
hard
ID
objon lshel
hand
catzich, otzutàto
camaler cor
ground
vater
brankling, Pringilla
grass
8ky
bolbol
eay
grand
vater ousel,
4iyger
Matez
gEcund
Vel Saze
aky
builtrag, rans
cataoheizzo
Halos
ground
tuctie, caeusta cacessa
Rigerthend, Tiggerbedd
Mater
gouud
box Tursle, tosteise
g0050
021
coDloo iguant, 1 Igiara
Aguaza Lgcans
0breb
shiysall. whiptail
LEcacas crocodile,
busan
ogaaa
crocadylix
13
bogtcas putz
sand
kingarake
doake
carth
harventzan, longinga, Pialangium
16
uater anake
17
whrob
18
scorpton
Indoor autdocr
1in
15
CATASSDIA
B7ARK
20
Sant pede
Tedo07
blace grouze
BTARK
Eren
pratesm csscomt, profzše grouse, prafete fool echidra, aptny
XD. paapp-tcsattteee cockatoo,
22
ptartigaa
XEOM
23
gra22
24
Zakatoe galerito,
tree
grwa
cacatiia golerita
CIRELE accabiaa
black muan,
vater
ground
unteatez, sassetez Filleen
graRz
tree
ciconia nigea
black stork
grazs
sky
Floningo
Vater
DAy
K10S0
policas
HETCE
3ky
Log dog.
byana cage busting
kyans, byaens
34 vulpes yulpes
200d
fos, slopex Lagopus
enck, Fells
Fanthera
37
beaste, panthers 1eo
Ho., king of
1200
38
cbestab, chotal,
acitonys Juhatcs
39
Lce beaz, polar besz, srauk tarthinee, theisretor exritimus
0309
dung bestin
warch
huevnn
41
escade, cieaie
tree
heeas
besver
Water
Digcorn, baghoen sheep,
clearron
83-200
2ock
aldk
g3000
Mater
45
octer
Hator
1200
C.2 Candidate Label Space
We consider two different label spaces of candidate labels: a) using the full ImageNet- I class names and b) using the top-20 most confusing classes for more conputing-ntenssiee intensive models like MiniGPT4. It leads to the following two evaluation setups, i.e., the vs. 1000 setup and the vs. 20 setup.
1 vs. 1000 Setup. As default option, is suitable given that the object labels for Counter immal all belong to that of the ImageNet- K dataset. Furthermore, this choice also reflects more realistic situation in the open world, where have a vast number of candidate labels and the failure cases of LVLMMs are common.
the full label space of the ImageNet- IK dataset, which
1 vs. 20 Setup. To suit more advanced LVLMs of which the inference costs are much higher than CLIP models, we constrain the sizes of candidate label space for each class. Specifically, based on WAA.IIINOOOO-VV-----
we select the top-20 most confusing labels, which is calculated by the average cosine similarity for both the easy and hard groups
C.3 Evaluation Metrics
Now, we discuss the evaluation metrics. Typically, they are applied the easy and hard groups separately when we evaluate the robustness of various models.
Class-wise Accuracy. We are interested in the effects of spurious features for each class. Therefore, we calculate the prediction accuracy specifically for photos within each class. It can be referred to as the class-wid wise accuracy, which is given by
ACC(label)
Σ TE T
1 label].
Irabel
where Iiabel is the indices of photos belonging to label and y; is the predicted label for the i-th image. The class-wise accuracy reflects the class-level model reliability against spurious correlations.
Average Accuracy. Upon the class-wise accuracy, we can calculate the average performance of models, namely,
ACC =
Σ
ACC(label).
4
IabelEC
Compared to the conventional average accuracy, i.e.,
[ EIEZ 1( = gt} with I the image indices and gt the true labels, our definition of the average accuracy further offsets the impact of class
14

---

Table 6: Adopted versions of CLIP checkpoints employed in our main experiments.
hackhone
pre-train
checkpoint
V
LAIO840OM
ES1 BSSK XL S13B B90K E31
V1T-B/16
ViT-B/18
DataConp18
VIT-B/32
LAI01400M
VrT-B/32
LAION2B
S34B B7%K
ViT-B/35
DataComp1B
XL. $1.3 B9OK
ViT-L/14
LAIONN00M
ES1
ViT-L/14
LAION2B
S32B B82K
VIT-L/14
DataComp18
VIT-H/14
LAIOW2B
S32B B79K
ViT-G/14
LAIOW2B
S34B B88K
V:T bag-b1g0/14
LAION2B
S34B B160K
ConvNaxt--B
LATON40OM
$138 BSIK
B82K
imbalance. We default using the average accuracy, and present the results without balancing in Tables 17118 for CLIP and ImageNet models.
Accuracy Drop. To quantify the spurious correlations that make CLIP models fail, measure the performance drop when moving from the easy and hard groups. At the class level, the accuracy drop is defined the class-wise accuracy of easy minuses that of hard. At the dataset level, it is the average value for the class-level accuracy drop.
C.4 Evaluation Details of MiniGPT4 and LLaVA
To evaluate LVLMs with backend of language models, we follow the common practice that constructs questions to prompt LVLMs 29 30 Specifically, we construct the question as:
What is the main object in the image?
and then calculate the language modeling loss with respect to the answer:
A <object name>
for each ImageNet class name. Meanwhile, we also try another question prompt that is widely used in training MiniGPT4 and LLaVA 330 53):
Describe this image in detail.
while the performance will generically decrease. In addition, when we switch to the object-centric evaluation protocol as [3
Is there <object name> in the image?
or
Is this image photo of <object name>?
and evaluate the answer with Yes for each class, we observe a severe performance decrease as LVLMs easily hallucinate the objects. If we strictly follow the evaluation metrics of (33 by simply fetching the answers instead of comparing the losses, there exist lots of hallucinated objects by LVLMs in our dataset.
C.5 CLIP Naming Rules
For the CLIP checkpoints, we adopt the naming rule of <dataset> is the name of pre-train datasets and <backbone> is the specific name of backbone models. For example, LLIP--LATN44OM- -VT-/0//-----/22 indicates the B1T-B/32 model CLIP-trained on LAION40OM. Different training setups are considered in OpenCLIP, and the versions of the adopted checkpoints are summarized in T Moreover, in Table beyond the adopted ones.
where
we consider the results of checkpoints
15

---

D Theoretical Understanding of CLIP's Robustness to Spurious Features
We provide more detailed setup and analysis in complementary to Section
D.1 Detailed Theoretical Setup
We begin by introducing more details about the data generation process following the literature 9 500 [54].
Definition (Multi-modal Dataset), Consider n image-text pairs {(x) x))))
) Ji=1 both image x'} E Rdd and Text wy E BRd are generated from the underlying latent factor 2; E IR. The samples are generated as follows:
2 1iimoy Eapul E IR2 is composed of a invariant feature Zine ~ NN of and spurious feature Zapa ~ ~N(pppa dpp)) with Pr(a 3) = Pspu otherwise yy, is the label uniformly frownfrrm {-1,1}, D is drawn with 1/2 S Pepu I 1 while the OOD test data D is drawn uniformly with Papu 1/2.
Given z. the x at modality M is generated via x = D (x) EEM, with DM E Rd xt and EM NNO,, / dd a..)- The matrix Das E Re x1 with dm is a matrix with orthonormal columns which can be considered dictionary matrix.
y' + m1 MapuPspu + 12.s
With the definition, we can write every z² =
where Th,s 72, are two Gaussian
variables in the definition.
CLIP Training. We employ two linear encoders gr : RRd, - R for the image modality and @T Rd - R for the text modality, implemented as gf (æ,) = W, æ f and (T() = WTæT with W, E Raxd: and Wr E IR Axxrr respectively. The encoders are trained through the linearized 50 that mimics CLIP training dynamics:
contrastive loss
LCUP
ΣΣ jtt
Sty ss))
2n(n 1)
Σ
Σ (s s.) ( W Wrll.
2n(n 1)
where st = gr (*t)) gr(+) is the similarity with respect to the i-th image and j-th text representa- tions, and W wrll the a regularization term with p > O.
Zero-shot Inference. Once the CLIP model (Q.gr) is trained. the performance will be measured in zero-shot manner by matching the most similar caption such as 'a photo of {object name] across different object name as class names. Meanwhile, one could also leverage several prompts and leverage the average text embeddings across the available prompts to facilitate the evaluation (2] The prompt with respect to y could be modeled as P,, Dr 21yy), where Dr is the prompt transformation matrix. Then, the zero-shot accuracy of CL.P could be formalized follows:
Acc(gı, 9r) = ( (arg max 91 (z gr (Pg),)),
while the error is (rr(gI.gT) invariant features, it will have a better zero-shot classification accuracy across different distributions.
Acc(gr, Intuitively, once the model extracts more of the
D.2 Proof for Theorem
Theorem 2 (Restatement of Theorem I]. Given multi-modal dataset (Def.) 2 with suitable variance
in the features Jsnv O(1) > aspu. and spurious features with a large spurious correlation
Pepu = 1 - o(1), an overparameterized CLIP model where w(1), d = Ω()) and dr = (m)), the spurious features (e.g., backgrounds of the image) takes up relatively large amount of the
image Mapn. > 22nt22 2 Piin = 1, then with high probability of at least 1 1 O( (ootinn)
o(1),
the CLIP model achieves large error in zero-shot accuracy in the OOD test data where + y:
Err(gr,gr gT) > 1 -4(K1) o()),
16

---

and a small envor in the OOD test data where y:
Acc(gi, (T) 2
21- 21-((K2)) - -1);;
where K1 (n PappppaPan 1) denotes the CDF of a standard normal distribution.
PappuPapi
K2
-2 P Panu 6aa
and T
v (n Po t Papapupaps 11) 1)
Proof. We will introduce some useful lemmas to help with our proof.
Lemma 1 (9). The minimizer of linearized CLIP loss W W satisfies the following with probability of at least 1 - O (poly(n)) such that,
T WT D1
1+ oino
2psppPopy 1] + o
Dille S O(VE))
PPPapuPppu
apu
p
where eo = =v logn
Intuitively, the lemma indicates the importance of the training distribution, that the minimizer of CLIP will converge to the data characteristics of the latent features of the training distribution.
Then, consider the case where the model is inferred onto test sample with y 1, -1 Then, with the aforementioned lemma, we have
x w
1 =T/Di D1 p
1 o
2psppuPppu + Japu
Di I
||
>,,
1 O(VEo) |و
ppspuPppu
p
Then, notice that
1 o Psp
2pspuPapu - 1 1 o
- =
Dr
DJ D (( PPppppppp 1).
-
(5)
When CLIP makes an incorrect prediction, we have
o W Wi #=1 < ot W W of 1
Then, we have
J D
1 1 o 2pspuPopu
2 UpspuPsp + Jap
8=1 D D
1 0 (VVeloogn) <
(6)
T D
1 1 0 2PappPopu
2ppp o² spm
0 O0vcologg
with Eq. 5 plugged in, denote 61 = =O log log we further have
-2 (1) )n + (-1 + 72)
(2HupaPPpp (2) 1) c1] 0.
Since Mn (1 (mn) 172 (2HopuPspa
is Gaussian variable follows the distribution of
m/ (1 + ) + 7/2 (2Pspu 1) ~ NO (1 ) Po!n + (2HaspuPopu - 1)² 2 spu
then, have
Pr(-2 [1 m1 - 1 (p) - - >
02 2u +
(1 + 2 knv
sn + (2HaspaPpp - 1)202 Spm 2o?
(8)
=1- Φ{
Finor
2 2psppPspp +
o iuw
2o? + (2/4 1) spu
17

---

Table 7: Comparison between CLIPs and standard supervised learning on ColoredCO0
backbone
pre-train
appmach
out-of-distriuttion
drop
RXi-50
OpenAI
zero shot
69.67
68.33
1.34
OpenAI
95.67
RM-50
OpenAI
objbkg
94,11
0.22
93.89
BN-50
OpenAI
supervised
94.44
5.33
89.11
VIT-B/16
OpenAI
zem shot
73,11
71.22
1.89
VIT-B/16
OpenAI
obj
97.89
ViT-B/1K
OpenAI
objhkg supervised
97.11
1.67
95.44
V1T-B/16
OpenAI
94.78
1.33
93.45
where is the CDF of the standard Gaussian distribution.
Then, suffices to know that the 2pppnPonet Erry E is lower bounded by o( ((L+o Po +(MsspuPape 1) ), which also applies to the case -1,a= 1.
Ginu +2- +2
Similarly, given the case y a, as the model fits the spurious feature, could derive the lower
bound for its Ace by leveraging the spurious features as O( V(I+o ( (2psspu - 1)"
-2ptapuPapu -i
D.3 More Details on ColoredCO0 Experiments
To further validate our theoretical results, we construct the ColoredCOO dataset following 51 More specifically, ColoredCO0 is constructed as follows:
The dataset contains classes of COCO objects. The spurious correlation in the trainset is 80% such that each class has correlation of 80% specific biased color and 20% uniformly correlates to 10 sufficiently different randomly chosen colors.
The OOD testsets are constructed with classes randomly correlating to 8 biased colors different from the one correlated in the training set.
Then, we further generate two prompts for each sample:
1. obj: a photo of <object label>;
2. objbkg: a photo of <object label> <color label> background
We tune the pre-trained CLIP models using the CLIP objective based on the OpenCLIP library. We consider the learning rate of 3, 1e 5), with weight decay {le- 1, 1e and a warmup of {0, 1000} steps. We select the model according to the best in in-distribution test performance. The detailed results are given in Table 7 we can see, the CLIPs finetuned using either the CLIP objective the standard supervised training both exhibit high sensitivity to the spurious features.
le 5}.
D.4
More Details Multi Experiments
One possible explanation for the failure of CLIP objective in ColoredCOCO is that, the language encoder of the CLIP models may understand the captions well. Therefore, further construct new setup called MultiCols weltiiooooreddM@SS where each image contains only the digit information from MNIST dataset and the color information. Therefore, can directly derive the one hot encoding for all of the useful factors in the dataset.
Data. We consider multi-class classification setting with number of classes less than 2. The objects are the
Training data: Fix two class (0/1) and color (r/g), they spurious correlated by correlation
Pepu- The invariant feature's 's correlation with labels is Pine-
Test data (Rand): All classes and the colors randomly correlated, given class, Pspa 1/k.
Test data (Rev): All classes and the colors are reversely correlated, Pspu is 10% 0/1 classes and 1/k for others.
18

---

Z
Figure 12: Examples of MultiCooooeddMIIT dataset.
In Figure 12 we give some examples for the MulticooorredMniss dataset.
Experimental setting. We compare the standard supervised training and CLIP. To avoid noises or information loss in encoding language modality, we consider the perfect language supervision for single model. Given batch of image and caption representations {(h" " h h for image-caption pair, the CLIP objective aims to
max M, M,h - - (M, M. V
where M, Rd and M - the learnable projection layers for image and caption represen- tations. Assuming the perfect language encoding as the one-hot encoding for all possible object and background appearance h [0, 1]10++18, and M, can simply be an identity matrix, then Eq. can be considered as classification task for objects and backgrounds respectively:
max CE(M M,h" h)),
(10)
where the labels are simply the one-hot encodiigs of the objects and the backgrounds, and the classifier is M M, x- For the selliioolrr ColooedNIIS task where there is only one object and background (i.e., color), implement Eq. TO we only need to construct an additional classification head for the background. Given the aforementioned setup, conduct experiments comparing CLIP-based contrastive learning to the standard supervised learning. The results are given Table As we can see, both contrastive learning and supervised learning perform similarly across different numbers of classes and bias degrees.
19

---

Table 8: Comparison of standard supervised Mul MmltiCollredMIIS dataset.
learning and contrastive
learning on
samples
Pinc
Pepu
(Rand)
(Rev.)
rest
2 10,610
0.9
0.75
Contrastive
87.42:10.79
81,87::1. .66
n/a
0.9
Supervised
86.44:10.90
80.22-4:1733
2
10,610
0.9
0.9
Contrastive
71.56.11.79
50.08 1.9.7
m/a
Supervised
71.62:4:155
50.134:3.24
2
10,610
0.75
0,75
Contrastive 65.06 15.06-22.21
43,18+33,78
n/a
10,610
0.75
0.75
Supervised 65.01:1:.68
43.76.13.44
n/a
10,610
0.75
0.9
Controstive 53.73=+1.08
1..4244:.74
2
10,610
0.75
0.9
Supervised $3.89:1:0966
17.14::1.88
n/a
15,578
0.9
0.75
Contrastive 85.03-+1.25
35.86:20.70
81:88:00:52
15,578
0.9
0,75
Supervised
19.20=11.91
1.103=11110
15,578
0.9
0.9
Contrastive 69.05-1-2.26
45.55::4.52
88.60: 1.20
15,578
0.9
Supervised
68.29=+1.37
44.7443..50
08.43=0.89
3
15,578
0.75
0.75
Contrastive 61.57:1:2.86
37.76
63.53
Supervised
59.51-12.22
15,578
0.75
0.9
Contrastive 42.47-12.48
7.08 1.10
71.07
15,578
0.75
0.9
Supervised 41.60=21.67
8.18
71.89 71,89-+110
25,538
0.9
0.75
Contrastive 86.06-10.56
82.41110.77
88. 0.339
5
25,538
0.9
0.75
Supervised 85.60:10.77
80.99::0.99
87. 07.76=0.57
0.9
0.9
Contrastive
18.15.10.42
25,538
0.9
0.9
Supervised 70.73=#1.41
43.47=24.01
0.,
25,538
0.75
0.75
Contrastive 61.15=11.10
33.97-43.70
71.88
25,538
0.75
0,75
Supervised
1..69:1..229
33.66:1:.118
68.75
25,538
0.75
0.9
Contrastive 35.37
4.60.10.45
72.47.10.55
25,538
0.75
0.9
Supervised 34.82:4 1.97
5.44:10.70
69. 19.38--0.55
30,044
0.9
0,75
Contrastive 85.76220744
8188111111
00.4400544
6
30,044
0.9
0.75
Supervised 85.84:1:0.81
81.81:11.77
06.29
30,044
0.9
0.9
Contrastive 70.99.12.39
40.07:110.53
86.57 0.449
6
30,044
0.9
0.9
Supervised 70.97.1.2.45
40.63.19.81
86.25
30,044
0.75
0.75
Contrastive 1.105=
32.70:1:4500
70.76=00.0
30,044
0.75
0.75
Supervised 19.49:t:1.26
33.94:13.69
67.91
30,044
Contrastive 38.96.12.55
30,044
0.75
0.9
Supervised 35.85-22.27
0.87:t0.71
68.36=0091
40, 170
0.9
0.75
Contrastive 84.81:10.86
80.54
8641110.40
0.9
0.75
Supervised 85.49=10.67
1..47
86.78
8 40,170
0.9
0.9
Contrastive 71.75:1:1.65
89.85-18.81
86.3410.36
Supervised
8 40,170
0.75
0.75
Contrastive 63.73-1
31.46-17.20
01.58
8 40,170
0.75
0.75
Supervised 62.22:42.00
13:1::66.54
70.58
8 40,170
0.75
0.9
Contrastive 43.91-122.36
5.11110..8
70.76100.0
8
40,170 170
0.75
0.9
Supervised 40.39:12.82
5.28:1:0.2
70.43 0.64
0.9
Contrastive
10 50,000
0.9
0.75
Supervised
85. 15.10::00.67
86. 0
10 50,000
0.9
0.9
Contrastive 73.79=211.43
48.02-1:5550
85.18
10
50,000
0.9
0.9
Supervised 74.97=11..99
52.09-15.72
0.224
10 50,000
0.75
0.75
Contrastive 65.31=#1.43
32.31166.73
69.67 00.53
Supervised 66.00:11..22
36.35::5.59
70.27=0.30
10 50,000
0.75
0.9
Contrastive 48.03
5.53.11.25
69. 19.13
Supervised 46.83-1..33
5.72-1
69.92.10.37
20

---

E Ablation Studies
In this section, we present ablation studies further validate the feasibility of our data curation process.
Biasing to ImageNet Setups. We follow the curation procedure while using ImageNet models (i.e., ResNet50-ImageNet) to construct easy and hard splits, where we name the corre- sponding dataset CounterAnimal-I.. We present the results of CLIP and ImageNet models on AounterAnimal-I in Tables 9 TO respectively. The effective robustness is further shown in Fig- ure 13 Contrary the observations within original CounterAnimal, CLIP models can demonstrate better robustness against spurious features within CounterAnimal-3 I. It aligns with our expectation since different training data (e.g.. LAION for CLIPs, and ImageNet for ImageNet models) follow different distributions and naturally contain different spurious features. It also demonstrates the generality of our data curation method reveal the spurious features for different kinds of the models. We also list the background names for easy and hard splits with respect to some of the selected classes in Table IT. As observed, using different models split data will capture very different spurious features. It highlights the necessity to curate an OOD testset for CLIP models, as CLIP models learn different spurious features than ImageNet models.
Correctness vs. Frequency. We further explain why easy and hard examples can characterize spurious features within CLIP setups. In general. spurious features can be caused by biases inherent in the data distribution concerning backgrounds. For example, for the animal class of ice bear, the background of ice more common than other backgrounds, such grass, thus causing spurious correlations learned by CLIP models. Therefore, we investigate in terms of the background frequency, employing the searching tool of Have Been Trained that can retrieve images from LAION5B closely matching given class name. We examine 10 animal classes as our case studies. For each class, we collect the top 100 most relevant images and tally the occurrences of the backgrounds of our consideration. It is important note that our counting process excludes cartoon images, irrelevant photos, corrupted photos, and those featuring multiple distinct animal subjects or ambiguous backgrounds. The results are summarized in Table T2 As we can see, in general, the spurious features captured align with our conjecture that hard examples contain uncommon backgrounds in the CLIP training data, e.g., LAIONSB, further justifying the feasibility of our CounterAnimal assessing the robustness of CLIP models in real-world situations.
httpes//aavellleetrriie...cc
21

---

Table 9: The vs. 1000 results for CLIP checkpoints on the CounterAnimal-1 dataset.
backbone
pre-train dataset
easy
hard
drop
RX-50
OpenAI
18.
R0-101
OpenAl
61.22
40.25
20.97
B22 50-50xx
OpenAl
64.40
47.85
16.55
RN-50
OpenAI
72.00
57.65
14.35
RN-50 64
OpenAl
81.41
68. 36
13.05
LAION400M
53.22
V1T-B/16
OpenAl
73.46
56.56
17.10
ViT-B/16
DataComp18
79.33
ViT-B/15
LAION2B
68.66
52.13
16.53
V17-B/16
DPH28
83.39
75
14.64
V1T-B/32
LATDN40OM
57.32
37.61
19.71
ViT-B/32
OpenAl
66.95
47.12
19.84
DataComp18
VaT-B/32
LAIOX28
67.37
47.64
19.73
VIT-B/S2-255
DataComp1B
78.18
17.39
ViT-L/14
LAION400M
77.96
60.85
17.11
ViT-L/14
OpenAl
81.67
67.55
14.12
ViT-L/14
DataConp18"
88.87
77.06
11.82
ViT-L/14
LAIOX26
78.89
63.14
13.75
VIT VIT-L/14
L/14-336
OpenAI
84.09
71.62
12.47
ViT-H/14
LAIOW2B
83,77
71.04
12.72
ViT-H/14
DPKEB
89.32
79.65
9.68
VIT-R/14-384
DPKS8
92.55
19
9.36
ViT-G/14
84.46
16.31
ViT-bigG/16
LAIOX26
86.39
74.03
12.36
Convliext-B
LAIOK400M
ConvNext--BB
LAIOX28
57.19
38.74
18.45
Table 10: The vs. 1000 performance for ImageNet models on the CounterAnimal-1 dataset.
backbone
hard
Alexliat
31.87
VGG-11
73.82
46.66
27.15
VGG-13
74.32
48.08
26.24
VGG-19
77.07
52.77
24.30
RN-16
73.08
49.19
23
13-34
R0
80.71
$2.97
27.74
VIT-B/16
85.06
66.64
18.42
VIT-B/32
78.27
55.42
22.85
VIT-1/16
83.65
64.03
VIT-L/32
80.28
59.17
21.11
S
ConvNext.
88.92
74.48
14.44
ConvXext.-1.
77.21
CLIP-DC
CLIP-DFN
ImageNet
y-x
80
70
60
50
70
80
90
common
larger dataset
larger model
Figure 13: The easy verus hard performance (%) for CLIP and ImageNet models CounterAnimal-I The 1 vs. 1000 setup is considered.
22

---

Table 11: Selected animal object and background names in CounterAnimal and CounterAnimal-1 We bold the background names differently between Anunter imiml and CounterAnimal-I.
object label
easy
hard
easy
hard
Ostrich
ground
water
ground
Brambling
sky
water
Bul
sky
tree
sky
grass
Vluture
Box tur tle
grass
earth
grass
water
Common iguana
Whaptail
earth
human
water
shurb
Agama
grass
Crocodile
carth
grass
earth
free
Table 12: The number of photos counted with respect easy and hard backgrounds, based on the searching tool of Have I Been Trained.
object label
easy
name
number
namee
nimber
ground grass
brambling
skey
17
sky
grass
water ouzel
water
31
ground
bullfrog
water
28
ground
19
rulture
sky
box turtle
grass
earth
3
loggerhead
grass
whiptail
earth
58
human
2
agama
tree
african crocodile
earth
15
hognose snake
earth
grass
14
king snake
earth
24
21
garter snake
36
earth
28
vater snake
ground
harvestnan
shrub
40
rock
27
scorpien
indoor
outdoor
tarantula
41
grass
centipade
indoar
grass
black grouse
41
ptarmigan
Show
13
grass
15
prairis
grass
sulphur-cress crested cockatoo
51
grass
14
black exan
water
ground
echidna
tree
0
black stork
grass
35
sky
20
flaningo
sky
bittern
grass
28
tree
9
pelican
sky
lion
sand
22
water
19
african hunting dog
grass
78
tree
3
byena
36
rond
red fox
grass
road
23
grass
26
jaguar
water
tree
3
lion
grass
tree
cheetah
26
tree
2
ice bear
snow
grass
bestle
earth
52
human
cicada
tree
13
human
grass
bighorn
grass
20
rock
3
mink
grass
water
23

---

VIT-8/16
VIT4/34
LAION20
OpenW
80
70
herd
hard
170
60
60
50
60
40
590
540
90
easy
(a) Backbones
(b) Datasets
Data Quality
Figure 14: Comparison for the effective robustness with respect to a) different backbones, b) different pre-train datasets, well high-quality (HQ) and low-quality (LQ) pre-train datasets.
More Results
In this section, we present more experimental results support our claims.
Effective Robustness. In Section 3] we mainly examine the absolute robustness to assess and compare the OOD performance across various CLIP setups, which well known to be sensitive to the original value scales. Therefore, in Figure [4] we apply the measures of effective robustness [5] conclusions. Overall,
further substantiate that the benefits derived from increasing model scales and enhancing data quality notably outweigh those obtained merely expanding dataset sizes.
previous conclusions are upheld, demonstrating
Top-5 Results for CLIP models. We present vs. 1000 results for more CLIP checkpoints on the CounterAnimal dataset in Table B which is extension of Table 2 Moreover, present more results for the evaluations on CounterAnimal supplementing our analysis of CLIP models under spurious correlations. To begin with, report the top-5 scores under the 1 1000 setup. where we check the target label is one of the top-5 model predictions. The results are summarized in Table [4] Comparing with the top-1 results in Table [3] find that there is still large performance gap between the easy and hard groups, indicating that the label confusion quite diverse and not limited the top two classes.
Other Versions of Pre-train Datasets. OpenCLIP provides other CLIP checkpoints beyond our adopted ones. Table 15 summarizes the results of CLIP models similar to Table 2 while using different versions of checkpoints. As we can see, the performance for both easy and hard is very stable across varying versions, except for DataComp1B. The reason is that their various checkpoints use subsets of DataComp where XL indicates the fully DataComp1B, L indicates a 140M subset, M indicates a 14M subset, and S indicates 1.4M subset.
further consider the prompt setups following OpenAl CLIP 2 us- ing average text embeddings over 80 predefined prompts as the final text embeddings. The results are summarized in Table T6 we can see, the average performance for both the easy and hard groups generally improves to percentage points over the results of our simpler prompt. However, main conclusion remains unchanged: the ImageNet models generally exhibit better performance and smaller drops. Another interesting finding is that when evaluating with CLIP-LAION4OOM-ViTT (the CLIP checkpoint employed in our data collection), the performance drop with OpenAl prompts is not as high as that of simple prompt used in Table 2 It indicates that our curation procedure mainly overfit the adopted prompt instead the particular CLIP checkpoint.
Results OpenAT Prompts.
Average Performance without Balancing. We by default adopt the balanced average accuracy to offset the impacts of class imbalance. In Tables 17 we further summarize the results without class balance, following [ Eiez 1 (ūx = gt). As we can see, the performance drop remains obvious, and similar conclusions be drawn as the balanced results: Backbone scales more important for spurious robustness than pre-train dataset scales, and b) ImageNet models are more reliable when facing spurious features in CounterAnimal.
1 vs. 20 Results for CLIP and ImageNet Models. We adopt the vs. setup for the evaluations
of more advanced LVLMs in Table For fair comparison, further summarize the I vs. 20

---

Table 13: The vs. 1000 results for CLIP checkpoints on CounterAnimal. The pre-train datasets with high-quality data are marked by
backbone
pre-train dataset
easy
hard
drop
OpenAl
RM--101
OpenAT
64.27
45.15
19.12
RM-500x
OpenAT
RN--00
OpenAl
76.43
59.13
17.30
RN-50
OpenAT
80.25
66.77
13.48
ViT-B/16
LAION400M
52.17
ViI-B/16
OpenAl
73.08
56.56
16.52
DataComp18
V1T-B/16
LAION2B
73.18
53.18
20.00
ViT-B/16
DFW28
85.03
ViT-B/32
LAION400M
67.13
36,95
30.18
ViT-B/32
OpenAl
69.13
45.6 62
23.51
ViI-B/32
DataComp18
ViT-B/38
LAIOX26
72.94
48.74
24.20
VIT-B/32-22
DataComp18
ViT-L/14
LAION40OM
80.90
63.31
17.59
ViT-L/14
OpenAl
85.38
70.28
15.10
ViT-L/14
DataCompi8
89.29
79.90
9.39
ViT-L/14
LAIOX26
82.23
06.27
15.96
VIT-L/14--0 L/14-344-36
OpenAl
86.36
73.14
13.21
LATOX28
ViT-R/14
DFKS8"
88.55
79.13
9.42
VIT-B/14-384
DFM5B
90.23
83.67
6.56
ViT-G/14
LAION2B
86.81
73.32
13.49
ViT-bigG/16
LAIOX28
87.57
76.96
10.61
ConvNext-B
LAION400M
ConvNoxt-
LA1OX28
61.03
39.91
21.12
results for CLIP models in Table and and for ImageNet models in Table 20 As sec, there does not exist a significant change in performance drop compared to vs. 1000 results, indicating that mistakes made by CLIP models are relatively concentrated. As in Figure
also depict the easy versus hard performance for various learning setups with their names, following the vs. 1000 setup 20 setups in Figure [6]
in Figure 15 and
Class-wise Results. In Tables 22 we summarize the detailed results of the class-wise accuracy for the main results in Figure 5 We further depict the drop in accuracy in Figure 17 Generally speaking, the spurious features found CLIP-LATDN4OON-ViT-B///
can also fail other CLIP setups, and the general trends of decline are preserved class-wise. However, there are some cases where the drop in accuracy between easy and hard is negative, c.g., for data in class ID 33 and 42. It means that for these cases, our collection pipeline may have large overfit to the adopted CLIP setup, i.c., CLIP-LAION4OOM VVT--B32TT
25

---

Table 14:
The 1 vs. 1000 results with top-5 performance scores for CLIP checkpoints on CounterAnimal The treitrain datasets with high-quality data are marked by
backhone
pre-train dataset
easy
hard
drop
RX-50
OpenAl
91.02
77.15
13.87
RMI-101
OpenAT
89.04
79. 98
9.06
OpenAI
83.65 65
7.55
RNI-50x16
OpenAl
92.72
87.65
7.55
OpenAl
ViT-B/16
LAION400M
92.54
84.03
8.51
Vi7-B/16
OpenAl
94.74
88.21
6.53
ViT-B/16
DataComp18
95.04
90.89
4.15
ViT-B/16
LATUN26
91.04
84. 64
6,40
ViI-B/16
95.45
ViI-B/32
LAION<O0M
87.54
71.48
16.U6
OpenAl
ViT-B/32
DataCompi8
92.60
85. 88
6.72
Vi7-B/32
LAION2B
90.73
81.47
9.25
VIT-B/32-255
DataComp18
94.26
5,93
ViT-L/14
LAION400M
94.33
88. 73
5.60
ViT-L/14
OpenAl
ViT-L/14
DataComp18
97.36
95.10
2.26
LAIOX28
ViT-L/14
DPN28
96.76
94.55
2.23
VIT-L/14-380
OpenAl
96.50
94.
2.30
ViT-H/14
LATUV26
95.26
91.72
3.55
VIT-H/14
DENSB
97.03
94.51
2.52
VIT-B/14-366
V1T-C/14
LAIOX2B
95.30
91.20
4.10
VST-big0/14
ConvNext-BB
LAION400M
81.67
69.90
11.77
ConvMext-
LATON28
82.64
73.27
9.37
Table 15: The vs. 1000 performance with other versions of CLIP checkpoints in OpenCLIP.
backhone
pre-train dataset
chockpoint
bard
drop
VIT-B/16
LAIGN4DON
E32
73.59
52.53
21.06
BTT-
DataCoup1B DataComp1B
XL $13B B90K
80.36
64.24
16.12
B/16
SIB B8K
65.80
44.14
21.66
VAT-B/32
LAION40ON
E31
67.13
36.95
30.18
BTT-
LAION2B
E16
71.32
47.21
24.11
ViT-B/32
LAION2B
S34B B79K
72.94
48.74
24.20
VIT-B/32
DataComp18
XI $13B B90K
75.96
53.74
22.22
VIT-B/32
DataConp18
M S128M B4K
25.91
11.65
14.26
DatsCoup
VIT-L/14
LAIGN4DON
ViT-L/14
LAION40ON
E32
81.11
63.87
17.24
VIT-G/14
LATON2B
$12B B42K
83.72
68.46
15:26
VIT-G/14
LAION2B
S34B B8SK
86.81
73.32
13.49
26

---

Table 16: The 1 1000 performance using prompts of OpenAI CLIP. The pre-train datasets with high-quality data are marked by
pre-train dataset
easy
bard
drop
RM-E0
OpenAl
64.55
44.20
20.35
OpenAT
BM- 4
OpenAl
69.62
53.68
15.93
RN-50
OpenAl
84.78
72.13
12.65
RN-50
OpenAl
84.33
72.02
12.31
ViI-B/16
LAION-O0M
76.20
58.17
18.18
OpenAl
V1T-B/16
DataComp18
82.85
69.77 79.74
13.11
ViT-B/16
LAIOX28
74.08
ViT-B/16
DFN2B
85.20
74.33
10.87
Vi7-B/32
LAION400%
66.68
43.22
23.46
ViT-B/32
OpenAI
67.23
47.11
20.12
ViI-B/32
DataComp12 1B"
76.00
59.23
16.77
BB33
LAIOX28
UNT-B/32-22
DataComp18
79.77
64.20
15.57
ViT-L/14
LAION400M
81.22
15.91
ViT-L/14
OpenAl
85.76
73.23
12.53
Vi7-L/14
DataComp18
89.56
81.21
8.35
ViT-L/14
LATON28
83.43
69.44
13.99
ViT-L/14
DPK28
90.45
82.28
8.17
VIT-L/14-336
OpenAI
ViT-H/14
LAIUX2B
86.11
75 30
10.81
ViT-H/14
DPKSB
91.33
85.20
VIT-B/14-384
DPM58
92.20
88.01
4.19
ViT-G/14
LATOX28
87.17
77.20
10.97
VST-bigt/14
LAIGN28
87.57
76.96
10.61
ConvNext-
LAIOK400M
60.20
44.15
16.05
Table 17: The 1000 performance on CounterAnimal for CLIP models, evaluating based on the accuracy without balancing. The pre-train datasets with high-quality data are marked by *
pre
easy
drup
RM-50
OpenAI
64.59
38.40
19
RM--111
OpenAl
64.18
43.99
19
RN50-X4 X4
OpenAI
70.76
46.91
23.85
RN50
OpenAT
77.26
RN50 ---00-x69 X64
OpenAl
82,88
62.84
20.04
ViT-B/16
LATON400%
75.58
48.46
27.12
ViT-B/16
OpenAI
73.94
53.93
20.01
ViT-B/16
DataComp18'
81.83
61.47
36
VIT-B/16
LAION26
74.97
$1.20
23.77
V1T-B/16
DFW28
86.10
67.55 95
18.
ViT-B/32
LAION400M
69.02
ViT-B/32
OpenAl
68.84
44..17
24.67
ViT- BB32
DataComp18
78.16
51.50
26.66
ViT-B/32
LATON28
74.23
36
27.87
VIT-B/32-256
DataConp18
82.38
58. 56
23.82
LAION400M
VIT-L/14
OpenAl
85.29
69. 25
16.04
ViT-L/14
DataComp15
90.79
77.28
13.51
VST-L/14
LAION2B
83,47
62.33
21.14
ViT-L/14
DPH28
91.81
13.71
VIT-1/11-3356
OpenAT
86.40
72.40
14,00
ViT-H/14
LAIOX26
87.10
69.84 69.84
17.26
VLT B-T-B/14-30
DPMS8
92.29
80.95
11.34
ViT-G/14
LAION2B
88.09
69.96
18.13
ViT-big0/14
LAION2B
83.47
73.45
15.02
ConvNaxt-BB
LATON40OM
60.16
34.27
25.89
-B
LAIUX2B
60.65
38.64
27

---

Table 18: The I vs. 1000 performance on CounterAnimal for ImageNet models, evaluating based on the accuracy without balancing.
backhone
hard
drop
Alezliet
37.20
VGG 11
75.92
53.35
22.57
VGG-13
77.23
VGG-19
79.40
58.93
20.47
IN-18
76.46
52.79
23.67
R3-34
80.38
57.80
22.58
RN-50
83.52
62.97
20.54
ViT-B/16
86,97
71.62
15.35
VIT-B/32
82,03
61.71
20.32
ViT-L/16
85,96
70.21
15.75
ViT-L/32
82.89
64.64
18.25
ConvKext-8
89.88
76.61
13.27
ConvKext-BB
90.27
77.51
12.76
ConvNext-L
90.67
78.34
Table 19: The versus 20 performance on CounterAnimal for CLIP models. The pre-train datasets with high-quality data are marked by
buckhome
pre train dalasel
easy
hard
drup
RM-50
OpenAl
67.41
43.63
23.78
RM--101
OpenAT
66.92
19.69
BM- 4
OpenAl
71.82
50.50
21.32
RM-50
OpenAl
78.60
60.63
17.97
OpenAl
82.33
13.28
ViT-B/15
LAIONOOОM
75.51
$4.59
20.92
OpenAl
V1T-B/16
DataConp18"
82.02
66.02 06.02
16.00
ViT-B/16
LAIOX2B
75.85
48
ViT-B/16
DFM2B
86.04
72.13
13.91
ViT-B/32
LAION400M
70.46
31.02
ViT-B/32
OpenAl
ViT-B/38
DataComp18 1B
78.58
56.32
22.26
LAIOX28
VIT-B/32-255
DataComp18'
83.05
63.99 98.98
19.07
ViT-L/14
LAION400M
82.27
64.89
17.38
ViT-L/14
OpenAl
86.38
72.12
14.26
ViT-L/14
DataComp18
90.13
80.46
9.67
ViT-L/14
LAION26
ViT-L/14
DPW28
91.29
81.23
10.05
VIT-L//14-336
OpenAT
ViT-H/14
LAIUX2B
86.75
74.29
12.46
ViT-H/14
DFMSB
89.13
79.79
9.35
ViT F2T-B/14-384
DPH5B
90.70
84.00
6.70
VIT-G/14
LATOX28
87.74
74.
13.63
ViT-big0/14
ConvWlext-B
LATON40OM
64.85
39.71
25.14
ConvNext-BU
LAIOX28
Table 20: The versus 20 performance on CounterAnimal for ImageNet models.
backbone
easy
bard
Alezliet:
67.71
46.43
21
VGG-11
77.25
60.19
17.06
VGG-13
79.07
62.02
17.04
VGG-19
18
78.11
59.47
18.64
R3-50
83,72
68.60
15.
RN-101
84.13
70.77
13.37
ViT-B/16
86.57
76.88
9.69
VST-B/32
82.56
68.30
14.26
VsT-L/32
83.86
71.00
12.86
ConvXext-8
81.61
ConvKext-B
89.58
82.32
7.26
ConvNext-L
89.84
82.67
7.17
28

---

imageNat-Conffente Conffeettl
Imagiieel-Covveeee
Da Daaa3mm1B MITILZZ
ImageNee-VTTBBLU6
LAIOniB
ImageNee-VITL.I6
Imaneket-aN -N100
Imagehiet RNSO
An SESSEE VITALLL
image/el-A1//00g
DuroComolB
AION4OOM
Imegetees/GE1
DatsComp18 256
Imagehet-rGG12 Opepdd RN50016
Imagelae-Vvacg SpchAIN//B/LLE Imagehoc RM
Opeaki RN Sww.AIAI T/8/12
ORenIAI RNSO
AMOMZ Noncheatttyy
LAGON AAI LAIO COOM Con 400M//17/8/22
50
60
80
Figure 15: The easy versus hard performance (%) for CLIP and ImageNet models, following the 1 vs. 1000 setup. We also present the model setups for each easy-hard result pair.
29

---

InlageNa ConvNestL mmagehketCComeeeli ConvNeett
(mageNe::Crrhiiite
Dar Comple VTTLL1
Imsgeket-WTTL.226
242B VIT/K-4
M AOBMUUBL
Dbentt RM50055
ImageN VOTTS
ImageNet-WaGL
maageNet---e -
4ON40OM VIT1144
ImageNes-vGGy
Imagelet VGG11
RN50X16
60
imagehet--
Cata Sem.el VIT///2
AITT2S/VITB/S22
N50wd
50
ImageNet Alexte A-
MinGPPA-MiccuaT
LLVA115-7B
Figure 16: The easy versus hard performance (%) for CLIP, ImageNet models, and more advanced LVLMs, following the 1 vs. 20 setup. We also present the model setups for each easy-hard result pair.
30

---

Table 21: Class-wise I vs. 1000 performance CounterAnimal for different backbones CLIP-
trained on LAION40OM.
class CLIP-LATORACOH-TT
CLIP-LAIDX-0OWN -IT-BB/2
hard
drop
hard
drop
hard
dmp
71.36
64.60
6.76
79.61
57.52
22.09
93.20
91.15
2,05
87.18
69.37
17.81
78.63
49.55
29.08
94,02
75.68
18.34
90.00
70.99
19.01
81.15
48.15
33.01
94.23
90.12
4,11
76.19
67.35
87.76
41.84
45.92
97.96
82.65
15.31
88.32
67.72
20.60
73.36
48.10
25.26
83.94
68.35
15.59
78.64
43.96
34.68
73.64
18.68
54,96
81.36
69.23
12.13
44.00
25 25.23
73.85
24.85
87.69
74.00
13.69
9 74.00
37.50
36.50
$4.00
30.83
23.17
$4.00
39.17
14,83
79,92
11
62.43
28.87
33.55
74.26
28.87
45.39
60.95
42.96
17.99
83.52
51.19
72.53
35.71
36.81
72.62
16.39
64.04
26.83
37.21
22.17
2.44
19.73
17.24
7.32
9.92
63.60
53.06
10.54
32.46
22.45
10.01
64.04
44.90
19. 14
16
82.12
13.50
68. 13.62
68. 87
1.23
67.65
88.08
$0.92
37.16
68.35
54.92
13:43
29.11
4.17
24.95
87.34
69.32
18.02
83.98
74.05
81.82
43.67
38.
91.34
70.89
20.46
67.21
59.62
7.60
55.74
20.19
35.55
75.41
70.
5.22
21
67.31
37.12
30.19
73.08
43.94
29.
71.15
56.82
14.34
23
85,33
50.57
34.75
$9.85 85.85
17.24
42.60
83.78
41.38
42.40
25
98.04
88.68
9.36
93.63
68.87
24.76
99.02
86.79
12.23
5.60
1.81
3.79
20.00
4.07
15.93
43.20
8.60
34,60
86.42
62.42
77.78
14.77
78.52
6.66
28
65.48
27.72
37.76
79.70
$5.45
24.25
91.37
82.18
9.19
30
96.98
82.83
14.15
86.21
71.72
14.49
99, 14
91.94
5,20
0.49
32
95.71
84.72
10.99
85.24 24
63.89
21.35
98.57
97.22
1.35
83.24
80,00
3.24
92.20
$2.00
10.20
86.42
80.00
6.42
23
10.18
71.43
5.49
35
76.42
36.13
40.29
67.48
26.05
41.43
88.62
61.34
27.27
37
79.47
62.61
16.86
74.90
45.95
28.96
93. 16
82.43
10.72
96.70
77.36
11
83.02
15.09
39
99,21
79.09
20.12
97.62
70.91
26.71
100.00
90.91
9.09
49.23
23.40
25.83
56.92
17.02
39,90
$8.46
14.89
43,57
$6.82
23.87
42
75.73
85.00
-9,27
84.47
67.00
17.47
90.29
93.00
-2.11
44
22.08
3.92
18.16
18.18
0.00
18.18
72.73
24.51
48.22
72.52
51.43
72.52
31,57
80.92
53.33
27.58
31

---

Table 22: Class-wise vs. 1000 performance CounterAnimal for ViT-B/32 CLIP-trained on
different datasets.
LATON2B
B/32
LATON40OO
SLIP-OPenAI....
class
hard
drop
hard
drop
hard
drop
86.41
79.65
6.76
79.61
57.52
22.09
81.55
66.37
15.18
2
86.32
72.97
13.35
78.63
49.55
29.08
85.47
58.56
26.91
91.54
74.69
16.85
81.15
48.15
33.01
77.69
38.27
39.42
75.51
55.10
20.41
87.76
41.84
43.92
61.22
38.78
22.45
83.58
64.56
19.02
73.36
48.10
25.26
77.74
65.82
11.91
72.27
29. 67
42.60
73.64
18.68
$4.96
88
67.03
21.60
92.31
77.50
14.81
73.85
49.00
87
72.50
9
44.00
25. 00
19.00
54.00
30.83
23.17
70.00
35.83
34.17
68.64
45.07
23.57
74.26
28.87
45.39
53 53.85
11.27
42.58
82.42
41.67
40.75
72.53
35.71
36.81
78.02
90
16.12
31.53
11.38
20.14
22.17
2.44
19.73
38
16.26
22.66
14
60.09
47.96
12.13
32.46
22.45
10.01
17 9.
18.37
-0.38
16
71.52
15.95
55.57
68.87
1.23
67.65
72.85
8.59
64.26
67.09
39.77
27.41
29.11
4.17
24.95
74.68
25.76
48.93
68.40
60.76
7.64
81.82
43.67
38.15
77
$1.27
26.22
73.77
54.81
18. 16.96
55.74
20.19
35.55
70.49
34.62
35.88
21
69 23
31.82
37.41
73.08
43.94
29.14
67.
27.27
40.03
23
64.86
37.93
26.93
59.85
17.24
42.60
60.62
32. 18
28.43
95.
25
91.67
57.55
34.12
93.63
68.87
24.76
96.57
83.96
12.61
26
13.60
0.45
13.15
20.00
4.07
15.93
15.20
0.45
14.75
66.67
48.
18.3 34
77.78
14.77
77.78
69.13
8.65
28
68.53
49.50
19.02
79.70
55.45
24.25
63.45
16.83
46.62
30
93.10 10
61.65 62
31 .49
86.21
71.72
14.49
82.76
30.30
$2.46
63.21
23
82.76
42.45
40.31
32
95.71
84.72
10.99
85.24
63 89
21.35
88 57
72.22
16.35
33
85.84
80.00
5.84
92.20
82.00
10.20
76.59
80.00
-3.41
36.
69.
59.05
69.
13.74
35
78.05
21.85
$6.20
67.48
26.05
41.43
73.17
48.74
24.43
37
83.27
51
31 .47
74.90
45.95
28 96
85.55
62.22
19.34
92.92
87
24.06
80.66
55
78.77
47.17
39
96.03
76.36
19.67
97.62
70.91
26.71
100.00
93.64
16.36
40
64.62
34.04
30.57
56.92
17.02
39.90
56.92
25.53
31,39
42
72,82
69.00
3.82
84.47
67.00
17.47
71.84
71.00
0.84
44
63.64
19.61
44.03
18.18
0.00
18.18
10.39
2.94
7.45
70.99
48.57
22.42
72.52
40.95
35.88
30.48
32

---

67.7
56.6
45.0
40.5
40.3
39.9
29.1
1411
24.93
19.7
28.54
29.9.
34.8
24.2
29
18.2
7.
16
1922
20
22
23
25
28
30
39
42
44
dasss
CLIP CLIP-LAION40OM WWT.---i-------
B//32
17.8
0.4
8.8
5.22
36
0..
39.3
97.6
30.20.
2.4g
24
37.8
24
2.23
16.89
25.5
(b) CLIP-LAION40OM-VV -VT--B11VI
56.5
43.6
V.24.1
15.35
13.34.
6.
182.5
14.5
39.6
B.8
34.5
6.79
27.3
o 45.
5
3.8
27.6
(c) CLIP-LAION40OM ViT-L/14
44.5
40.9
48.9
25
30.4 94.
15.1
31.81.5
31.5
30.81.7
13.3
16.
20.1
22.4
7.6
3.9
2
8
11
14
16
1920 21
22
24
25
26
27
28
30
33
34
36
39
41
44.45
(d) CLIP-LAION2B-VE B/32
64.3
52.9
45.6 52.5
26.9
394
34
22.7
26
105
$2.
32.5
31.4
15.4
5.2
2.10.8
7.55
0.2
5
8
10
12
14
16
18
1970
21 22
23
24
28
35
4445
class 1D
CLIP-OpenAI-VIT-BB/2
Figure 17: The performance drop (%) between easy hard on varying CLIP setups. The horizontal axis denotes the class ids and the vertical axis denotes the class-wise accuracy drop.
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claim is that CLIP may not rely less on spurious features compared ImageNet models. To support our claim, we curate new evaluation dataset named
CounterAnimal specifically tailored for a comparative evaluation for the robustness of
CLIP. Furthermore, we also conduct comprehensive experiments in Section and theoretical justification in Section E both of which fully support our main position.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitation of our work in Appendix especially focusing on the further improvements for our dataset.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
34

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: In Definition 1 we clearly outline the assumptions made for the subsequent theoretical analysis presented in Theorem TT, In Appendix [D] present further details re- garding our assumptions, proofs, and the associated experiments to validate their feasibility.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: In Section we elaborate on the exact process used construct the CounterAnimal dataset, including details about the raw data sources, the construction pipeline, and the dataset configurations. Moreover, for reproducibility, most our experi- ments utilize open-sourced model checkpoints from the OpenCIII and PyTorch repositories.
We also comprehensively document the evaluation setups in Appendix C our dataset and the evaluation codes in the future. Guidelines:
We will release
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
35

---

(d)
reproducible, they We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We establish anonymous repository for the access to our dataset, which can found at the link of
htt/s//fboo33311668880 provide clearer and easier access instructions once the paper accepted.
We will
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
mublic/guides httil///uuiide//oouuuuinssssssss woweeSuumissiooooooooo...o.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. //nips.cc/publll wwwp//bllccggguiiceee Siweeee///ooSSeeessseeeeee ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We utilize open-sourced model checkpoints from OpenCLIP and PyTorch. We
collect the Counter An dataset following the precise pipelines in Section 2.1 Moreover, detail the evaluation setups of our experiments in Appendix C
We utilize open-sourced model checkpoints from OpenCLIP and PyTorch; we have collected the GounterAniman dataset according to the procedures outlined in Section 2.1; and we provide detailed descriptions of our evaluation setups in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
36

---

Answer: [No]
Justification: We utilize open-sourced model checkpoints without additional training or fine- tuning, and employ the complete dataset of CounterAnimal conduct our experiments. There is no stochastic factor that may appear in our experiments, and thus we do not need to report the error bars for results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror hars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details our hardware configurations in Appendix C.1 However, we do not list memory and time requirements since our experiments are not computationally demanding.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Our research conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
37

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the broader impacts of our work in Appendix A focusing impacts for both the research community and the real-world applications.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: To avoid any safety risk, we control the label space for our Counter dataset and manually cleanse any unsafe image during our data collection procedure.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
38

---

Answer: [Yes]
Justification: For models, only open-sourced checkpoints are adopted, following the licenses and https: /ttt:///gttubbb... http://........oooo//o//rrrr/r/a http://oo/maan//EEEEES.E. The versions of the adopted OpenCLIP checkpoints be found in Table 6 For scraped data, only use those photos from iNaturalist following the CC BY-NC license, which are allowed for scientific usage. Please refer the following link for the details about CC BY-NC: ht https::/ccreativecommee org//icensessby-nc///
from htttps://githbb.. http::/wwww.o.oooooooaaooooooooa
Guidelines:
The answer NA means that the paper does not use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papprswit thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our CounterAnimal dataset follows the MIT license. Permission is hereby granted, free of charge, any person obtaining a copy of this software and associated documentation files, to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software
Guidelines:
The answer NA means that the paper does release new assets. Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: The CounterAnimal dataset is constructed with human labeling, yet without crowdsourcing. The labeling and filtering procedure adhere strictly to the pipeline outlined in Section 2 Our project does not involve any form of compensation or profit. All individuals involved in the manual data curation are from our team, and they all have been notified and agreed to grant consent to the use of the data in this work. this work.
39

---

The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
40


Reviews:
[{"review_id": "qBnkutpo76", "reviewer": "Reviewer_2MXa", "summary": "This work asks one interesting question: \"Do CLIP models always generalize better than ImageNet models?\" Driven by this question, this work proposes a new benchmark dataset named CounterAnimal. This dataset consists of a) the common group: comprising animals in common backgrounds, and b) the counter group: including animals in plausible yet unusual backgrounds. The main idea is that the performance drops from the common to counter groups quantify the reliance on spurious background features for animal predictions. The main observation is that CLIP models exhibit notable performance drops when tested on the counter group. In comparison, ImageNet models can be more robust than CLIP models.", "questions": "- ImageNet models are not so biased toward spurious correlations compared with CLIP models. Why? Is this because the ImageNet training set does not have too many examples that exhibit spurious correlations?\n\n- While I appreciate this work includes results on ColoredCOO and ColoredMINIST, some other spurious correlation benchmarks (e.g., WaterBirds) would be greater if they were also included.", "limitations": "The dataset is proposed, so a discussion on the potential bias/ privacy is needed. I appreciate this work highlights the future improvement of expanding semantic scope, data source, and ImageNet testbed.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It is always good to see a new and novel dataset proposed for evaluating CLIP and ImageNet-trained models. The proposed dataset CounterAnimal is complementary to existing datasets that cannot reflect the robustness of CLIP models to spurious correlations. \n\n- The dataset construction is well-presented. The statistics, curation, background labeling, and spurious discovery are well introduced in Section 2\n\n- The analysis around spurious correlation is good. This work tries to give insights from several aspects, such as pre-trained datasets, scaling up, and learning paradigms. The observations are sound to me.", "weaknesses": "-  I found the analysis of why CLIPs rely on spurious features interesting. However, I think the claim is somewhat \"obvious\": there exists a relatively strong correlation between the object captions and the parts of image backgrounds, CLIP will learn to align the backgrounds, i.e., spurious features. If the training dataset contains many examples of spurious correlations, then models will tend to be biased.\n\n- I am curious about why ImageNet models may not be so influenced by the spurious bias in CounterAnimal. Is this because the ImageNet training set does not have too many spurious correlation examples? Or ImageNet has a spurious bias but such bias is different from the one in CounterAnimal? Please provide a discussion or share some insights on this question. \n\n- This paper adopts absolute performance drop in Section 3.3. Such a metric may not be so robust. For example, model A drops from 40 to 39, and model B drops from 90 to 89. They drop the same but the I would say model B is better. Please comment on this, and discuss the metric of absolute performance drop.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and suggestions! Please find our responses below. \n \n> Q1. I think the claim is somewhat \"obvious\": there exists a relatively strong correlation between the object captions and the parts of image backgrounds, CLIP will learn to align the backgrounds, i.e., spurious features. If the training dataset contains many examples of spurious correlations, then models will tend to be biased.\n \nA1. We need to clarify that, our theoretical analysis has unique and significant values for its implications to both theory and practice:\n\n- From the theoretical perspective, it complements the literature of theoretical studies about CLIPs. Prior to this work, previous works such as [1,2] mainly focus on developing theories and experiments justifying the superior OOD generalization capabilities of CLIPs in ImageNet variant tests. To the best of our knowledge, **our theory is the first to provably demonstrate the drawbacks of CLIPs in OOD generalization**, providing the foundation for future developments tackling the issue.\n\n- As a direct implication, the theory shows that CLIP training objectives can not offer additional robustness against spurious correlations. Consequently, the biases in large-scale multimodal training data such as LAION-5B is the major source for the spurious correlations learned by CLIPs. **However, prior to our work, there is no proper benchmark capturing the spurious correlations for large-scale multimodal training data**.\n\n\n[1] Identifiability Results for Multimodal Contrastive Learning, ICLR’23.\n\n[2] Does CLIP’s generalization performance mainly stem from high train-test similarity? ICLR’24.\n\n\n \n> Q2. I am curious about why ImageNet models may not be so influenced by the spurious bias in CounterAnimal. Is this because the ImageNet training set does not have too many spurious correlation examples? Or ImageNet has a spurious bias but such bias is different from the one in CounterAnimal? Please provide a discussion or share some insights on this question.\n \nA2. Thank you for the insightful question. First, we need to clarify that, ImageNet models still learn the spurious correlations as evidenced by the performance gaps in CounterAnimal benchmark. One reason for the lower performance gaps of ImageNet models is that, **CounterAnimal is specifically designed to reflect the biases of the CLIP pretrained datasets**, similar to ImageNet variant test sets which are designed for ImageNets[3,4]. A key takeaway for the phenomenon is that, we need to carefully choose suitable benchmarks to evaluate the OOD robustness of different models. \n\n[3] Do ImageNet Classifiers Generalize to ImageNet? ICML’19.\n\n[4] Natural Adversarial Examples, CVPR’21.\n\n\n> Q3. This paper adopts absolute performance drop in Section 3.3. Such a metric may not be so robust. For example, model A drops from 40 to 39, and model B drops from 90 to 89. They drop the same but the I would say model B is better. Please comment on this and discuss the metric of absolute performance drop.\n\nA3. Thank you for the suggestion. We further depict the lines of effective robustness, where we found that the conclusions remain the same. However, the improvement from increasing model scales and improving data quality is much higher than simply scaling up the datasets. \n\n> Q4. While I appreciate this work includes results on ColoredCOO and ColoredMINIST, some other spurious correlation benchmarks (e.g., WaterBirds) would be greater if they were also included.\n\nA4. We would like to note that the WaterBirds benchmark is similar to ColoredCOO, that synthesizes unnatural images by composing birds and different image backgrounds. Essentially it can be considered as a binary classification variant of ColoredCOCO and ColoredMNIST, hence the results of ColoredCOO and ColoredMNIST also generalizes to WaterBirds. \nIn addition, we need to clarify that the main focus of this work is to identify natural spurious correlations learned by CLIPs instead of the synthetic ones. CounterAnimal essentially captures the desired real-world spurious features, differing from previous synthetic benchmarks like WaterBirds.\n\n> Q5. The dataset is proposed, so a discussion on the potential bias/ privacy is needed. \n \nA5. As discussed in question 12 of the checklist (see Appendix), we use the publicly available data from the internet, following the CC BY-NC license, which permits scientific use. We have revised the Broader Impacts in our manuscript to include a discussion regarding the issue.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sKA3ZjPX6H", "reviewer": "Reviewer_KCA4", "summary": "In this work, the authors create an evaluation dataset comprising two groups, one with animals in usual backgrounds (common group) and another with unusual backgrounds (counter group). They then evaluate a suite of models of different backbones, model sizes, and datasets. They find that CLIP models do poorly than ImageNet-trained models, and generally high quality data or bigger model size improves counter group accuracy.", "questions": "1. **Biased dataset:** The dataset is split into to common and counter group using a CLIP model. Therefore by construction, the CLIP models will perform poorly and it is no surprise that the ImageNet-trained models do a bit better. One could construct a split of this dataset where ImageNet models do better than the CLIP models. If the dataset was collected in a model agnostic way then the conclusions could potentially be more interesting. \n\n2. **On the premise:** As such the premise or the primary question seems a bit vacuous. Models arguably learn different features and there will exist some type of evaluation where one does better than the other. But are there useful tasks/evaluations where ImageNet models are preferred over CLIP models? That is an interesting open question. This work doesn't necessarily start from there and create a benchmark that is supposed to represent a task. The authors rather create a biased dataset that by design make CLIP models perform poorly. Therefore the primary premise of the work seems erroneous. There is some value in the other evaluations so maybe the paper could be rewritten by positioning things differently. \n\n3. **Lack of novelty:** Keeping the primary result aside, other conclusions like that better datasets or model sizes improve robustness are not new. Please see [1, 2, 3, 4]. \n\n[1] [Geirhos 2021] https://proceedings.neurips.cc/paper/2021/hash/c8877cff22082a16395a57e97232bb6f-Abstract.html\n\n[2] [Idrissi 2022] https://arxiv.org/abs/2211.01866\n\n[3] [Fang 2022] https://arxiv.org/abs/2205.01397\n\n[4] [Nguyen 2022]  https://arxiv.org/abs/2208.05516", "limitations": "The authors have addressed some limitations. For the rest, please see my questions block.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The CounterAnimal dataset is a nice contribution that can be of value to the community.\n2. The authors have evaluated a number of models on the dataset and that too could be of value to the community.", "weaknesses": "Please see questions for more information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and suggestions! Please find our responses below. \n\n> Q1. Biased dataset: The dataset is split into common and counter groups using a CLIP model. Therefore, by construction, the CLIP models will perform poorly, and it is no surprise that the ImageNet-trained models do a bit better. One could construct a split of this dataset where ImageNet models do better than the CLIP models. If the dataset was collected in a model agnostic way then the conclusions could potentially be more interesting. \n\nA1. We need to clarify that, **one of the major objectives of this work is to construct a benchmark specifically to reflect the spurious correlations learned by CLIPs**. To characterize the spurious features captured by CLIP, it is reasonable to use a CLIP model to curate the data. Our experiments in Section 3.1 justify that spurious features captured by CounterAnimal are general across different CLIP setups, and our experiments in Section 3.3 verify that the spurious features within CounterAnimal may not be so influential for ImageNet benchmarks. These results justify that our crafted CounterAnimal satisfies our original goal. \n\nFurthermore, we would like to note that, **previous comparisons between CLIPs and ImageNet models on ImageNet variant test sets are also biased**. However, biases are not entirely bad. The past few years witnessed a lot of developments built upon the ImageNet variant test sets. Therefore, **our benchmark share the same goal to provide a testbed for developing more advanced CLIP and vision-language models**. \n\n> Q2. On the premise: Are there useful tasks/evaluations where ImageNet models are preferred over CLIP models? That is an interesting open question. Therefore the primary premise of the work seems erroneous. There is some value in the other evaluations so maybe the paper could be rewritten by positioning things differently. \n\nA2. As explained in A1, our benchmark indeed captures spurious features learned by CLIP models. Future works can be developed upon our benchmark to mitigate the spurious correlations learned by CLIP models, similar to those built upon ImageNet variant testsets. \n\nMoreover, we need to clarify that the comparison between ImageNet and CLIP models is not to argue which model is the best universally. Rather, **we would like to highlight the biases existing in previous evaluations of CLIPs’ OOD robustness using ImageNet variant testsets**. Benchmarking models with improper test sets would cause illusions that CLIP models seem to resolve the spurious correlations, especially compared with ImageNet models. However, the experiments with CounterAnimals provide a sober look of the vulnerability of CLIP models to spurious correlations, and provide a platform for future developments of more advanced and robust CLIP and vision-language models. \n\nFinally, we understand that our original title may bring unnecessary misunderstandings about our work. We thus propose to revise it to `A Sober Look at the Robustness of CLIPs to Spurious Features` to more precisely reflect the contents of this work. Please kindly let us know if you have better suggestions to resolve the misunderstandings!\n \n\n> Q3. Lack of novelty: Keeping the primary result aside, other conclusions like that better datasets or model sizes improve robustness are not new. \n\nA3. As clarified in A1 and A2, CounterAnimal is more suitable than ImageNet variant test sets to benchmark the OOD robustness of CLIP models. As the main comparison results between CLIPs and ImageNet models already differs from the previous studies, extending the benchmarking of CLIP models to more variants is necessary to verify the conclusions of previous works. **We do not claim we are the first to discover those findings, rather, we are verifying those findings in order to provide insights of developing more robust CLIP and vision-language models**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZlFRMjXMk8", "reviewer": "Reviewer_iLSC", "summary": "This paper presents CounterAnimal, an evaluation dataset featuring two subsets: animals with common backgrounds and those with unusual backgrounds. The images were sourced from iNaturalist. Data with high CLIP accuracy are categorized as \"Common\",  while those with low CLIP accuracy are labeled as \"Counter\".  Results shows that CLIP models experience a greater accuracy drop compared to ImageNet models when tested on this dataset.", "questions": "Please refer to the weaknesses.", "limitations": "This paper has discussed the limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper analyzes multiple factors affecting CLIP accuracy, including model size and training data quality.\n- The paper combines both experimental results and theoretical analysis. The analysis in Section 5 is interesting and novel.\n- The paper is well-written and easy to follow.", "weaknesses": "- The proposed dataset is not sufficiently robust to analyze the influence of spurious bias, as this is not the only difference between the common and counter datasets.\n  - To analyze the accuracy drop caused by spurious features such as background, the background should be the only difference between common and counter image pairs. Prior work [4,5] has proposed such datasets focusing on background.\n  - In the proposed dataset, other factors may influence the model accuracy gap besides background. For instance, as shown in Figure 1, the more varied gestures of ice bears on the right compared to the left could be a contributing factor to the accuracy drop.\n\n- Current experiments cannot conclusively show that ImageNet models generalize better than CLIP.\n\n   - As the common and counter groups are selected according to the CLIP accuracy (see line 165 in the paper),  they indicate easy and hard samples for CLIP.    Since ImageNet models have different training characteristics, it is natural that hard cases for these models may differ from those for CLIP, resulting in a smaller performance drop for ImageNet models. This result cannot support that ImageNet models are more robust than CLIP models.\n   -  The accuracy drop from common to counter group can be greatly influenced by the model used to divide the common and counter dataset. Using the combined proposed common and counter dataset, a new Common' and Counter' dataset can be created based on the accuracy of ImageNet models. What is the impact of this dataset division on the accuracy drop for different models?\n\n\n- Prior studies[1,2,3,4,5,6] have proposed datasets specifically to analyze the influence of background, which are not discussed in this work.  These datasets can be used for CLIP evaluation as they do not overlap with the CLIP training set. Additionally, creating datasets based on model accuracy in this work is similar to the approach in [6].\n\n[1] Noise or Signal: The Role of Image Backgrounds in Object Recognition.\n\n[2] Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models, NeurIPS 2019.\n\n[3] Dataset Interfaces: Diagnosing Model Failures Using Controllable Counterfactual Generation.\n\n[4] ImageNet-E: Benchmarking Neural Network Robustness via Attribute Editing, CVPR 2023.\n\n[5] LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Images, NeurIPS 2023.\n\n[6] ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object, CVPR2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and suggestions! Please find our responses below. \n \n> Q1. The proposed dataset is not sufficiently robust to analyze the influence of spurious bias, as this is not the only difference between the common and counter datasets.\n \nA1. In our study, we primarily focus on the real-world spurious bias caused by backgrounds, while we also try our best to avoid the influence of other factors. As mentioned in Sec 2.1, during data curation, we took measures to control the influence of other covariates, e.g., ensuring that the main objects are in the major position. To further understand whether other factors such as gestures will introduce unintentional bias into the dataset, we count the proportion of different gestures within the class of Ice Bear. The results are listed as follows, which do not have an obvious difference in the distribution of gestures between common and counter scenarios. We will add the related discussion and explore other unintentional factors that may affect model predictions in our revision. \n\n\n| Ice Bear | Common | Counter |\n|----------|--------|---------|\n| Stand    | 89%    | 84%     |\n| Lying    | 13%    | 16%     |\n\n> Q2. Current experiments cannot conclusively show that ImageNet models generalize better than CLIP.\n\nA2. We apologize for any confusion in our description. In general, our conclusion that ImageNet models generalize better than CLIP models is limited to CounterAnimal. This is a preferred observation as it aligns with **our main goal of crafting a dataset that characterizes the spurious features that may widely exist in CLIP setups**. Prior to our work, there is no specialized benchmark curated for CLIPs to study the robustness of CLIPs to the real-world spurious correlations. Instead, previous works predominantly focus on the spurious features in ImageNets. Studying the OOD robustness of CLIPs on ImageNet specialized benchmarks is biased and may bring illusions about the robustness of CLIPs. \n\n\n> Q3. Prior studies [1,2,3,4,5,6] have proposed datasets specifically to analyze the influence of background, which are not discussed in this work. These datasets can be used for CLIP evaluation as they do not overlap with the CLIP training set. Additionally, creating datasets based on model accuracy in this work is similar to the approach in [6].\n\nA3. Many thanks for your suggested paper. Most of these suggested papers have shown that CLIP models are more robust to the associated spurious features than ImageNet models, indicating that they did not intend to capture spurious features within CLIP setups. The CounterAnimal remains the first dataset that can find some spurious features that are predominant for the CLIP setups while might not be so strong for the ImageNet benchmarks, which is the uniqueness of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "soygMlaZn2", "reviewer": "Reviewer_BaJ3", "summary": "The authors aim to investigate spurious correlations learned by CLIP models. For this, they curate a novel dataset where animals are organized into common and uncommon backgrounds, e.g. a polar bear is more likely encountered in snow than on grass. The authors then perform experiments where they benchmark various CLIP and ImageNet models on the curated dataset. They observe that CLIP models suffer from spurious correlations which stem from changing the background.", "questions": "I have written a very detailed review above. I expect clarifications with respect to the raised points, at the very least in the \"Major\" paragraph.", "limitations": "The limitations discuss the comparison in performance of the CLIP vs ImageNet models which I believe cannot be made due to the methodological issues in this work.", "rating": 6, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "I think the issue of spurious correlations is important and one needs to understand how and whether VLMs learn spurious features. The paper presents many experiments and shows that scale or backbone capacity do not improve the effective robustness on CounterAnimal which is interesting.", "weaknesses": "The paper has many issues, both in terms of writing and the methodology which need to be fixed.\n\n### Major:\n**The authors missed important previous works**: The paper “Recognition in Terra Incognita” is very related to this work and also proposes a “dataset designed to measure recognition generalization to novel environments” based on camera traps. The dataset is sorted according to difficult environments for different animals, which makes it very similar to CounterAnimal. I think the authors need to cite and discuss this paper. Currently, I do not understand the benefit of having a new dataset in addition to the already present one. The waterbirds dataset is also highly similar and should be discussed (https://arxiv.org/pdf/1911.08731). The authors cite that paper, but do not discuss it in the Related Work section, nor put it into context with CounterAnimal. The backgrounds challenge (https://github.com/MadryLab/backgrounds_challenge) is also highly related and should be discussed. In general, the related work section is very weak, given how extensively spurious correlations and worst-group-accuracy have been studied. Another important work to be discussed would be \"Finding and Fixing Spurious Patterns with Explanations\" (https://arxiv.org/abs/2106.02112).\n\n**The naming of the common vs counter groups is misleading**: \nLine 165: “Photos with the highest CLIP accuracy are assigned to the common group, and those with the lowest CLIP accuracy are assigned to the counter group.” I have a major understanding issue here. As far as I understood the paper before this line, the goal was to put images with common backgrounds into the common group and images with uncommon backgrounds into the counter group. This is also depicted in Fig. 1 or Table 1. The caption in Fig.1 says that “Most ice bears appear in a snow background (i.e., common), while it also is reasonable to find some ice bears in a grassy environment (i.e., counter)”. But here, the authors write that accuracy has actually been used to separate images into these groups? But then the frequency of the co-occurrence of certain backgrounds and classes has not been taken into account, or rather, it is a conjecture that those backgrounds where the CLIP model has higher accuracy on are more “common”? \n\n**The terms \"effective robustness\" and \"robustness\" are used interchangeably which is wrong and confusing**:\nI think the paper conflates the terms “robustness” and “effective robustness” which is confusing. When looking at effective robustness plots, such as in Fig. 2, we are interested in the residual difference between the measured value and the value predicted by the linear fit. As I can see, all plotted markers (CLIP and ImageNet) lie on their respective linear fits, and none of the interventions, such as CLIP-DC or CLIP-DFN offer any effective robustness benefits. It is though true that the **absolute** robustness numbers are overall higher for the CLIP-DFN models, for larger models or models trained on more data. I am however confused by the authors discussion of this observation. On the one hand, they write that larger CLIP models are more robust but increasing the dataset size does not yield improvements. First, I am confused whether they mean “effective robustness” or “robustness” here. Second, I do not see the effect the authors are describing: Both more data and larger backbones have higher absolute robustness but the same effective robustness as the other models. The statement “CLIP models trained on high-quality data are more robust” is also confusing, because it is not clear whether “robustness” or “effective robustness” is meant. \n\n**Due to methodology issues, results on CLIP models cannot be compared to results on ImageNet models (or other advanced LVLMs):**\nLine 60: “d) Spurious discovering: preserving classes and associated data based on the decrease in zero-shot performance (i.e., evaluating based on pre-trained CLIP models without fine-tuning) when shifting the backgrounds.” This step is really unclear. Do the authors curate the dataset based on the zero-shot accuracy of a CLIP model? From the introduction and the abstract, it sounds like the authors want to benchmark the robustness of CLIP vs ImageNet models on this custom dataset. But then, it is strange that CLIP models also seem to be used during the curation process. After reading the more detailed description in line 156, I think the statements made in line 85 are misleading. The authors write “ImageNet models are more robust to spurious correlations captured by CounterAnimal” and “Compared with CLIP models (colored in blue), surprisingly, we find that ImageNet models exhibit a stronger robustness to the spurious correlations in CounterAnimal.” Given that CounterAnimal has been curated based on the performance drop of a CLIP model, I find it very unsurprising that CLIP models perform worse on it compared to ImageNet models. I think that if CounterAnimal had been curated based on an ImageNet-trained ResNet50, the trend would have been reversed. I think all statements comparing CLIP and ImageNet trained models on CounterAnimal need to be relaxed and I think that this comparison is quite meaningless because of the described selection bias. I think that the whole Section 3.3. is misleading for this reason and statements such as the following cannot be made given the methodology issues: “Surprisingly, we find that ImageNet models are more robust to spurious features in the CounterAnimal dataset. This finding may contradict the common belief [Radford et al., 2021, Shi et al., 2023] that the CLIP models tend to be more robust to spurious correlations than single-modal supervised learning.” Similarly, the conjecture paragraph from line 265 onwards is wrong and cannot be made. \n\nFor the same reason, the comparison to advanced LVLMs in line 273 onwards cannot be made.\n\nFigure 1: Are these examples cherry-picked or are they representative of the data points present in CounterAnimal? I am asking this, because of the Winoground dataset [A]. This dataset tests the compositionality of VLMs by forcing a model to match two captions to two respective images. Winoground has later been criticized because the two images in the choice process are not equally hard [B]. For example, the model needs to match “the glass is on the grass” and “the grass is in the glass” to the corresponding images. However, there is much more grass in the image matching to the first caption, and the model likely picks that image for both captions just because there is more grass and it makes the decision in a bag-of-words-manner. To summarize, Winoground did not control for object size, orientation and other confounders. In Fig.1, it appears that the main objects (the polar bears) are equal in size, so size could be excluded as a possible confounder? Did the authors consider this possibility, i.e. that the drop in performance could be explained by other differences in the images from the respective domains?\n[A] https://arxiv.org/abs/2204.03162\n[B] https://arxiv.org/abs/2211.00768\n\n### General:\nLine 25: please cite CLIP\n\nLine 64: “The resulting dataset covers a total of 45 animal classes, ends up with 7,174 common photos and 5,926 counter photos, aligning with the standard size as an evaluation dataset [Recht et al., 2019, Hendrycks et al., 2021].” -> I do not understand this statement. Different test sets have different numbers of test images. ImageNet Val has 50k images for example. In what sense are the presented numbers standard?\n\n\nLine 94: “Overall, larger CLIP backbones (i.e., larger markers) can improve the effective robustness, implying that scaling up backbones may enhance robustness against spurious features.” -> I do not see this in Fig. 2. The larger markers appear to be on the fitted line, same as the smaller markers. Effective robustness measures the difference with respect to the linear fit, and there is none for the larger CLIP backbones. Please clarify this point.\n\n\nLine 146: “feature noise involves severe feature corruptions” -> Please be more specific here. What do you mean with feature noise? Do features refer to animals features such as missing ears or such? Or to the images themselves?\n\nLine 147: “clarity issues arise when animal objects are not in major positions” -> unclear formulation: what is a major position? Do the authors mean that the animals are too small or not in the center of the image?\n\nLine 153: “Note that the class space of backgrounds as above is not entirely orthogonal with each other due to the inherent ambiguity of the real-world situations. Nevertheless, we try our best to discern the assigned background labels within each animal class.” -> This is unclear. How many images would be ambiguous? I could imagine that many images would have two backgrounds, such as e.g. grass and sky or snow and water. For example, the last image in Fig. 1 on the left has both snow and water. It is not clear to me that only picking the snow background and ignoring the water is correct here. Further, at least for CLIP, the caption can contain several background keywords.\n\nFurther, I imagine animals occur in all kinds of environments, but there are only two backgrounds for each animal. Were the other images also discarded?\n\nLine 214: “Therefore, we conclude that our CounterAnimal dataset possesses some realistic shifts that are generally contained in large-scale pre-training data, regardless of backbones.” This conclusion cannot be drawn from this experiment since the backbone has not been varied here.\n\n### Section 4:\nThe proposed experiment is very similar to the well-known ShiftMNIST [D] or ColoredMNIST [E] datasets, which test the influence of spurious correlations. The findings here are not novel and should be brought into perspective with previous work. I do not understand how Fig. 11 relates to the text. What is “supervised”, “obj”, “objbkg”?\n[D] https://arxiv.org/pdf/1811.00401\n[E] https://arxiv.org/pdf/1907.02893\n\n### Typos, grammar:\nThe quality of the text is poor on some occasions which makes reading and understanding the paper difficult. The manuscript would benefit from a round of proof-reading. Some statements and formulations should be made more precise.\nLine 32: “The performance boosts over ImageNet models seem to suggest that CLIP resolves distribution shifts and thus spark a rich discussion about its rationale.” Strange formulation. How can “distribution shifts be resolved”? Please rephrase for clarity.\n\nLine 112: “More specifically, [Yang et al., 2023] report that CLIP models may misaligned frequently co-occured objects with the corresponding texts.”\n\nLine 115: “[Tong et al., 2024] find that CLIP misaligned samples will further cause the hallucination of LVLMs.” I do not understand this statement, grammar errors.\n\nLine 132: “Meanwhile, many existing datasets, e.g., DomainBed and Wilds, do not have overlapped label space with ImageNet, making the comparison between ImageNet and CLIP models hard.” There is a version of DomainBed [C] where the dataset has been filtered to only include classes compatible with ImageNet, such that an evaluation of ImageNet models is possible out-of-the-box.\n[C] https://openreview.net/pdf?id=LiC2vmzbpMO\n\nLine 171: “Recalling that, when CLIP models resort to the shortcut of data, the model performance will heavily correlate with the backgrounds presented in the common group yet is compromised when coming to the counter group.” Grammar errors, I do not understand this sentence. What is “the shortcut of data”?\n\nLine 208: “It suggests that the CounterAnimal dataset captures some general spurious shifts that at least commonly present in the pre-train dataset of LAION400M.” grammar\n\nLine 213: “Here, the spurious features degenerate the zero-shot robustness of CLIP models trained on both LAION2B and by OpenAI.” Typo? “degenerate”?\n\nLine 243: “In Figure 7, we consider two pre-train datasets, namely, LAION2B and the close-soured data from OpenAI” typo\n\nLine 297: “Nevertheless, in the following theorem, we justify that CLIP remains learning to use spurious features, aligned with our experimental observations on the CounterAnimal dataset.” grammar\n\nStrange space break between line 310 and 311.\n\n# Summary of the review:\nWe could fix the naming convention from \"common\" and \"counter\" to something like \"hard\" and \"easy\" since accuracy has been used rather than frequency of certain backgrounds to classify backgrounds into certain groups. Based on my arguments below, I believe we cannot compare CLIP models to ImageNet models on the proposed dataset in any sensible way due to the introduced selection bias. I believe the very title of the paper is misleading since the posed question cannot be answered based on the methodology issues. But if we remove the claims about comparing ImageNet models and CLIP models, then, the main point of the paper is that there exist backgrounds which are harder for CLIP models, given certain classes, and other backgrounds which are easier. I don't think that this observation is particularly interesting on its own. The authors did not relate the hardness of the backgrounds to their frequency in the pretraining dataset or anything else. The observation that backgrounds matter is also not novel but quite well-known and the authors do not offer a solution. Further, the writing is quite poor and confusing on many occasions; I provided many examples of incorrect and confusing sentences below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and suggestions! Please find our responses below. \n \n>Q1. The authors missed important previous works.\n\nA1. Many thanks for your suggestion. CounterAnimal utilizes high-quality data available on the internet, whereas Terralcognita relies on camera trapping data, which are not as widely spread. We believe CounterAnimal is preferred over Terralcognita as CounterAnimal much align with the training / deployment setups of CLIPs. Moreover, to compare with ImageNet models, we should ensure that we align the label space of ImageNet, satisfied by CounterAnimal but not by Terralcognita. WaterBird also has such a problem, and is limited to simple binary classification. For the Background Challenge, they study the robustness from another perspective, i.e., w/ or w/o the background. However, to find spurious features within CLIP datasets, we need to compare model performance across different backgrounds. At last, SPIRE masks a part of objects from the original image, which may introduce new factors to degenerate model performance.\n\n>Q2. The naming of the common vs counter groups is misleading. \n\nA2. Using accuracy instead of frequency makes the data collection procedure more effective in finding spurious features that make CLIP models fail. We will follow your suggestion and change the names of groups.\n\n>Q3. The terms \"effective robustness\" and \"robustness\" are used interchangeably which is wrong and confusing.\n\nA3. We agree that we need to clarify the term “robustness” in Section 3. We further depict the lines of effective robustness, where we found that the conclusions remain the same. However, the improvement from increasing model scales and improving data quality is much higher than simply scaling up the datasets. We will add the related discussion in our revision.\n\n>Q4. Due to methodology issues, results on CLIP models cannot be compared to results on ImageNet models (or other advanced LVLMs).\n\nA4. In general, we do not intend to claim that ImageNet models are generally more robust to spurious features. Instead, our goal is to emphasize that spurious features within CounterAnimal may not be as influential for ImageNet models, supporting that CounterAnimal captures spurious features within CLIPs, underscoring the significance of CounerAnimal.  For previous works, they assess CLIP robustness using OOD datasets primarily for ImageNet datasets. Such a tendency may not fully reflect the CLIP robustness, as spurious features within ImageNet datasets may not be learned by CLIP models. Our CounterAnimal fills this gap, offering a more comprehensive perspectives when studying CLIP robustness. \n\n>Q5. Did the authors consider this possibility, i.e. that the drop in performance could be explained by other differences in the images from the respective domains?\n\nA5. we have tried our best to do the quality control, and get rid of the influence of other factors. We also evaluate some potential confounders like gestures, as in the response to Reviewer iLSC, and find that it is relatively equally distributed across groups. Please kindly let us know if you feel there is another potential factor that may affect the performance.\n\n>Q6. Different test sets have different numbers of test images. ImageNet Val has 50k images for example. In what sense are the presented numbers standard?\n\nA6. We aim at clarifying that our CounterAnimal (about 13K) aligns with the standard size as an evaluation dataset, resembling the scales of other many popular OOD evaluation datasets such as [1-2]. We will make our discussion clearer in our revision. \n\n[1] The many faces of robustness: A critical analysis of out-of-distribution generalization. \n\n[2] Do ImageNet Classifier Generalize to ImageNet?\n\n>Q7. Unclear description in the dataset construction procedure in Section 2.1. \n\nA7. In data curation, feature noise refers to cases where some pixels are disrupted or missing. Clarity issues refer to cases where animal objects are largely occluded by backgrounds or other irrelevant objects. It also includes cases where animal objects do not occupy most of pictures. In background labeling, there are two cases where one image can be assigned more than one background label, i.e., some backgrounds can be confusing and other may possess more than one background. Due to space limit, we will further clarify our data construction procedure in our revision. \n\n>Q8. Further, I imagine animals occur in all kinds of environments, but there are only two backgrounds for each animal. Were the other images also discarded?\n\nA8. Yes, we only preserve two groups for each animal, aiming to better capture spurious features captured of CLIPs. In Sec 3.1, we further show that the identified spurious features are general across different CLIP setups, justifying that these spurious features will not be largely bias towards a particular CLIP checkpoint. \n\n>Q9.  “Therefore, we conclude that our CounterAnimal dataset possesses some realistic shifts that are generally contained in large-scale pre-training data, regardless of backbones.” This conclusion cannot be drawn from this experiment.\n\nA9. We have shown that using different backbones will always lead to poor performance on the counter group in Fig 7. We will make our discussion clearer in our revision. Thank you for your suggestion. \n\n>Q10. The proposed experiment is very similar to the well-known ShiftMNIST [D] or ColoredMNIST [E] datasets, which test the influence of spurious correlations. \n\nA10. The experiments are presented to echo our theoretical analysis that CLIP learns to align spurious features with object captions. For the legends in Fig. 11, “supervised” refers to the results of supervised trained models, while “obj” and “objbkg” refer to use different prompts to fine-tune CLIPs. We will add more discussion about the table legends in Fig. 11.\n\n>Q11. Typos and grammar issues. \n\nA11. Thanks for pointing out our typos. We will correct them in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all the reviewers for their careful reviews and constructive feedback. We are also grateful for reviewers’ recognitions of our efforts on dataset constructions, the empirical findings, as well as the theoretical analysis. In response, we would like to emphasize the contributions of this paper and clarify the potentially misleading content.

**Dataset Contribution**. We would like to highlight the biases existing in previous evaluations of CLIPs’ OOD robustness using ImageNet variant test sets. **Benchmarking models with improper test sets would cause illusions that CLIP models seem to resolve the spurious correlations, especially compared with ImageNet models**. It motivates us to craft the CounterAnimal dataset, specifically capturing the spurious correlations within CLIP setups. The experiments with CounterAnimals provide a sober look of the vulnerability of CLIP models to spurious correlations and provide a platform for future developments of more advanced and robust CLIP and vision-language models. 

**Comparative Experiments**. In general, **It is not our intention to claim that ImageNet models are inherently more robust to spurious features than CLIP models**. The comparative experiments between CLIP and ImageNet models in our study demonstrate that the features captured by CounterAnimal are generalizable across different CLIP setups. Furthermore, our experiments detailed in Section 3.3 suggest that the spurious features identified by CounterAnimal may have limited influence on ImageNet benchmarks.

Moreover, following the suggestions from the reviewers, we have applied measures of effective robustness to further support our conclusions in Section 3.3. These results are detailed in the appendix. Overall, our previous conclusions are upheld, with additional findings indicating that **improvements from increasing model scales and enhancing data quality significantly outweigh the benefits of merely scaling up datasets**.

Finally, we understand that our original title may bring unnecessary misunderstandings about our work. We thus propose to revise it to `A Sober Look at the Robustness of CLIPs to Spurious Features` to more precisely reflect the contents of this work. Please kindly let us know if you have better suggestions to resolve the misunderstandings!

Meta-review: **Summary**
CLIP is believed to be robust to distribution shifts due to its performance on ImageNet shifts. This paper presents a new evaluation dataset named CounterAnimal that contains a common group of animals in common backgrounds and a counter group including animals in plausible yet unusual backgrounds. When evaluated with this dataset, CLIP exhibits performance drops challenging common assumptions on the robustness of this model.

**Recommendation**
Accept. All the reviewers tend towards acceptance and the authors did a good job with the rebuttal. I also believe this work is interesting as it challenges a common assumption about CLIP in the research community.
Decision: Accept (poster)