A Sober Look at the Robustness of CLIPs to Spurious Features

Authors: qizhou  wang
yong  lin
yongqiang  chen
ludwig  schmidt
bo  han
tong  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 08:37:45

Abstract: Large vision language models, such as CLIP, demonstrate impressive robustness to spurious features than single-modal models trained on ImageNet. However, existing test datasets are typically curated based on ImageNet-trained models, which aim to capture the spurious features inherited in ImageNet. Benchmarking CLIP models based on the ImageNet-oriented spurious features may not be sufficient to reflect the extent to which CLIP models are robust to spurious correlations within CLIP training data, e.g., LAION. To this end, we craft a new challenging dataset named CounterAnimal designed to reveal the reliance of CLIP models on realistic spurious features. Specifically, we split animal photos into groups according to the backgrounds, and then identify a pair of groups for each class where a CLIP model shows high-performance drops across the two groups. Our evaluations show that the spurious features captured by CounterAnimal are generically learned by CLIP models with different backbones and pre-train data, yet have limited influence for ImageNet models. We provide theoretical insights that the CLIP objective cannot offer additional robustness. Furthermore, we also re-evaluate strategies such as scaling up parameters and high-quality pre-trained data. We find that they still help mitigate the spurious features, providing a promising path for future developments.

Main paper: 
 introduction: 
 Large vision language models (LVLMs) have demonstrated huge success across a wide range of vision and multi-modal tasks, surpassing conventional ImageNet (-trained) models by a remarkably large margin [1] . LVLMs are typically trained with or based on Contrastive Language Image Pre-training (CLIP) [2] on an unprecedented scale of real-world vision and language data such as LAION [3] , which are significantly larger than ImageNet. The huge success of CLIP has presented a paradigm shift for modern vision and vision-language models to conduct the pre-training from ImageNet benchmarks to web-scale multi-modal datasets [4] .A key signature of CLIP models is the impressive robustness against various ImageNet-oriented distribution shifts [2] , which is shown to be prohibitive to ImageNet models [5] . The performance boosts over ImageNet models seem to suggest that CLIP resolves distribution shifts, thereby sparking a rich discussion about its rationale [6, 7, 8, 9, 10] . However, the elephant in the room is that adopted testsets (i.e., ImageNet variants) to evaluate the robustness of CLIPs are primarily designed for ImageNet-based models [5, 11] . These datasets may not correctly reflect the exact robustness of CLIP, given that CLIP models are trained on a large amount of data that may include, and possibly extend beyond those ImageNet variants during pre-training [10] . In this paper, we investigate the robustness of CLIP to distribution shifts caused by the presence of spurious features. These features are highly correlated with labels, but this correlation may break down under distributional shifts [12, 13, 14, 15, 16, 17, 18, 19, 20] . We raise a challenging research question in the following:Is there a benchmark that reflects the exact reliance on spurious features of CLIP? Sadly, most of the existing benchmarks [21, 22, 23, 24] are tailored primarily for ImageNet models, which are unsuitable for CLIP. To fill this gap, we introduce a new testset, named CounterAnimal, specifically designed for assessing the robustness of CLIP models against real-world spurious features. Figure 1 presents several examples of CounterAnimal where data are divided into two groups, a) the easy group: animals in commonly appeared backgrounds that the CLIP models make correct predictions, and b) the hard group: animals in less commonly yet still plausible backgrounds, where the CLIP models are likely to misclassify them. Intuitively, the easy part captures some real-world biases that the web-scale data may naturally inherit. Hence, by comparing the performances of the two groups, one can quantify to what extent the model relies on spurious features. Table 1 : 1 vs. 1000 results of exemplary animal classes within the CounterAnmial dataset for CLIP-LAION400M-ViT-B/32. "bkg" denotes the background label, "accu" (%) denotes the zeroshot accuracy, and "drop" (%) denotes the drop in accuracy between easy and hard groups. More specifically, the CounterAnmial dataset is curated based on raw photos collected from iNaturalist 3 . The construction pipeline consists of 4 steps. a) Data collection: querying iNaturalist with each animal class, where we select some of the animal names from the ImageNet-1K dataset [25] . b) Data curation: manually cleansing low-quality photos that potentially contain ambiguity and corruption. c) Background labeling: manually annotating photos with their respective backgrounds, selected from the label space of the candidate backgrounds. d) Spurious discovering: preserving classes and associated data based on the decrease in zero-shot performance (i.e., evaluating based on pre-trained CLIP models without fine-tuning) when shifting the backgrounds. The resulting CounterAnimal dataset covers a total of 45 animal classes, and ends up with 7,174 easy photos and 5,926 hard photos, aligning with the standard size as an evaluation dataset, such as [26, 27] . Moreover, CLIP-LAION400M-ViT-B/32 is used as the proxy CLIP model in spurious discovering (cf., Appendix C.5 for the model naming rules).We evaluate the CLIP models on our CounterAnmial with various backbones, e.g., ViT [28] , along with different pre-train datasets, e.g., LAION [3] . We also consider more advanced LVLMs like MiniGPT4 [29] and LLaVA [30] . We employ two evaluation setups crafted for different families of models (cf., Appendix C): a) 1 vs. 1000 setup: using the full ImageNet-1K class names as the candidate label space and b) 1 vs. 20 setup: using the top-20 most confusing classes regarding CLIP-LAION400M-ViT-B/32 as the candidate label space. We provide some of results in Table 1 and Figure 2 , highlighting the key observations in the following: may wrongly align co-occurred objects with their texts. [33] reports similar failure modes for more sophisticated LVLMs such as MiniGPT4 or LLaVA. [34] finds that CLIP misaligned samples will further cause the hallucination of LVLMs. Complementary to these works, our study explicitly characterizes the spurious features captured by CLIP and explains the existence of the reported failure cases. Our study provides interesting empirical and theoretical counterexamples to the previous beliefs for the substantial improvements in robustness for CLIP models, especially for those results observed on ImageNet variants [7, 8, 9, 35] . Based on the newly collected CounterAnimal dataset, we suggest that distribution shifts remain an open problem for CLIP models. Also, we need to be cautious about the test setups when evaluating new models pre-trained on datasets that differ significantly in scales and distributions from traditional ImageNet models.Comparison with previous benchmarks. There are many other datasets to study distribution shifts, e.g., ImageNet variants [21, 26, 36, 37, 38, 39, 40] , DomainBed [41] , and Wilds [42] . However, these datasets have biases when assessing the OOD robustness of CLIP models, as they may fail to represent the true OOD scenarios during CLIP training. Moreover, numerous recently released datasets, such as [22, 23, 43, 44, 45] , have also explored distribution shifts. However, these studies primarily focus on synthetic distribution shifts, which may not fully represent real-world cases. In fact, it has been shown that previous OOD benchmarks are contained in CLIP training [10] , making it hard to ablate ID/OOD cases for data in these benchmarks. Consequently, CLIP models have shown to be more robust than ImageNet models on these contaminated datasets [46] . 
dataset and evaluation setups: 
 To begin with, we describe the basic experimental setups, including the pipelines in constructing CounterAnimal, its key characteristics, as well as the adopted evaluation settings.We introduce the curation pipeline of our new dataset CounterAnimal, tailored for CLIP to investigate spurious correlations. The pipeline consists of 4 steps as follows:Data Collection. We query animal names listed in the ImageNet-1K dataset and collect raw data via the search interface of iNaturalist, a global biodiversity data-sharing platform. We retrieve the latest 300-800 photos per animal class, organizing them based on the queried labels.Data Curation. The collected raw samples are susceptible to noise and ambiguities. Therefore, we manually cleanse the low-quality data that fall into any one of the following 4 situations: label noise, feature noise, obscurity, and clarity. Label noise refers to cases where photos do not belong to the queried classes; feature noise refers to cases where some pixels are disrupted or missing; obscurity occurs when photos belong to more than one object class; clarity issues refer to cases where animal objects are largely occluded by the backgrounds or other irrelevant objects. It also includes the cases where animal objects do not occupy the majority of the space in photos.Background labeling. We consider a typical form of spurious features where the backgrounds of photos can be biased [47] . To identify such data for CLIP models, we manually label the backgrounds for the curated data. The considered class space of backgrounds is defined as follows: ground, water, earth, sand, snow, grass, human, sky, road, rock, shrub, indoor, tree, and outdoor. Note that the class space of backgrounds as above is not entirely orthogonal due to the inherent ambiguity: Some backgrounds may be ambiguous and some photos may contain more than one background. Nevertheless, we try our best to determine the assigned background labels for each animal class and exclude those photos challenging to be labeled.For each class, we quantify the impacts of spurious correlations to CLIP models by comparing the performances on the associated samples across different backgrounds. We take those classes as containing spurious features on which we observe a relatively obvious decrease in accuracy when changing backgrounds. In realization, we adopt the checkpoint of CLIP-LAION400M-ViT-B/32 for evaluation, where the prompt for its text encoder is "A photo of <object label>.", and the space of <object label> is the ImageNet-1K class names, i.e., we follow an 1 vs. 1000 setup. Then, we consider the classes where the zero-shot accuracy varies by more than 5% when changing backgrounds as the cases where CLIP model has learned the spurious features. The data with the preserved classes and backgrounds are used to create our final CounterAnimal dataset. Photos with the highest CLIP accuracy are assigned to the easy group, and those with the lowest CLIP accuracy are assigned to the hard group. We further refine the collected data to remove any mistake that the labelers may made during data curation and background labeling.Our objective in developing CounterAnimal is to reflect the spurious correlations learned by CLIP. Therefore, we need to employ the CLIP models for dataset curation and thus ensure the construction is effectively biased towards CLIP configurations [26] . In Appendix E, we further show that our data curation pipeline is general and reliable to characterize the spurious features within the considered models. Moreover, our experimental results later in Section 3 will corroborate that the spurious features captured by our CounterAnimal dataset are general across different CLIP setups and may not be so influential for ImageNet benchmarks. These findings will justify that our crafted testset satisfies our primary objective in characterizing the spuriousness for CLIP specifically.We depict the data layout in Figure 3 and visualize the zero-shot gaps for each animal class in Figure 4 , where we use CLIP-LAION400M-ViT-B/32 as our referred model. Please refer to the detailed object/background names concerning the easy and hard groups in Appendix B. Recalling that, when CLIP models resort to the shortcut of data, the model performance will heavily correlate with the backgrounds presented in the easy group yet is compromised when coming to the hard group. Accordingly, Figure 4 implies a reliance for the CLIP models on the backgrounds.We evaluate a series of CLIP models on the CounterAnimal dataset for their zero-shot performance.For each class, we use the pre-defined prompt of "A photo of <object label>." as in our data collection procedure and the similarity between image and text embeddings in classification. By default, we use the label space of the ImageNet-1K dataset and report the top-1 accuracy, i.e., the 1 vs. 1000 setup. Moreover, when involving more advanced LVLMs, we adopt the 1 vs. 20 setup where we employ the top-20 most confusing classes regarding CLIP-LAION400M-ViT-B/32 as the candidate label space. For re-productivity, we adopt the pre-trained CLIP checkpoints from OpenCLIP [48] and ImageNet model checkpoints from the PyTorch repository. The model naming rules are in Appendix C.5 and the evaluation details are discussed in Appendix C.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitation of our work in Appendix A, especially focusing on the further improvements for our dataset. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
experimental analysis: 
 Our experiments center on the evaluation and the analysis of our CounterAnimal dataset. In Section 3.1, we examine the generality of the captured spurious correlations. In Section 3.2, we explore the potential facets that affect the robustness of CLIP models. In Section 3.3, we extend the evaluation to a broader family of models with different training paradigms. In Section 2.1, we discover spurious correlations using CLIP-LAION400M-ViT-B/32 and collect associated data to build the CounterAnimal dataset. A critical problem then arises: Is our dataset a general benchmark to examine spurious correlations of CLIP with other pretrain datasets and backbones? Hence, we need to examine whether the biases in the CounterAnimal dataset can hinder the robustness of other CLIP models, where we consider two situations: a) fixing pre-train datasets while varying backbones and b) varying pretrain datasets while fixing backbones.V iT -B /3 2 V iT -B /1 6 V iT -L /Varying Backbones. We fix the pre-train dataset to be LAION400M and explore two other backbones within the ViT family [28] , i.e., ViT-B/16 and ViT-L/14. Their zero-shot results are depicted in Figure 5 (a). There remains a drop above 17 percentage points for both the cases of ViT-B/16 and ViT-L/14. It suggests that the CounterAnimal dataset captures some general spurious shifts that are at least present in the pre-train dataset of LAION400M.Varying Pre-train Datasets. We fix the backbone to be ViT-B/32 and consider other pre-train datasets. Here, we consider LAION2B and the closed-source dataset used by OpenAI. Their easy and hard results are in Figure 5 (b). Here, the spurious features affect the zero-shot robustness of CLIP models trained on both LAION2B and by OpenAI, indicating that our CounterAnimal dataset possesses some realistic shifts that are contained in various CLIP setups. Therefore, we conclude that CounterAnimal captures some general spurious features learned by CLIP models. We extend our evaluations to a wider range of CLIP models with different scales of parameters and pre-train data. The results are summarized in Table 2 and further depicted in Figure 2 (a). Generally speaking, performance drops can be observed across all considered CLIP configurations, indicating that CLIP models in various scales still learn spurious features. More specifically, we investigate the influence of a) parameter scales and b) pre-train data scales in CLIP models on the sensitivity of spurious features. We exclude the backbone of ViT-B/32 and the dataset of LAION400M to avoid biases in data collection.Scaling up Pre-train Data. To test the impacts of enlarging scales of pre-train datasets, we consider two CLIP backbones, namely, ViT-B/16 and ViT-L/14, along with a series of pre-train datasets of increasing sizes.The results are summarized in Figure 6 . We observe that scaling up the data scale does not necessarily reduce the performance drop, suggesting that directly enlarging the scale of pre-train data alone cannot enhance robustness. One possible explanation is that larger datasets do not imply fewer biases, whereas the CLIP models will still inherit the spurious correlations therein.Scaling up CLIP Model Sizes. We also explore the connection between model scales and spurious correlations. In Figure 7 , we consider two pre-train datasets, namely, LAION2B and the close-soured data from OpenAI, along with backbones of increasing scales. We observe a clear trend indicating that larger models exhibit better performance against spurious correlations. It may tell us that larger models possess stronger robustness, making them less prone to the shortcuts of spurious features.(a) LAION2B R N -1 0 1 R N -5 0 × 4 V iT -B /1 6 V iT -L /1 4 V iT -L /Data Quality Matters. Moreover, we observe that the results obtained with DataComp-and DFNtrained CLIPs exhibit better performance and smaller drops across backbones, Figure 8 offers their comparisons. We notice that these datasets have been stringently filtered and thus possess high-quality data. It indicates that enhancing data quality is still a promising way to improve OOD generalization.Our analysis focuses on absolute performance drop. In Appendix F, we strengthen our conclusions by incorporating the analysis based on effective robustness [5] , where our findings still hold. We extend our evaluations to broader families of models, including ImageNet-1K supervised models and more advanced LVLMs, such as MiniGPT4 and LLaVA.ImageNet Models. We first extend our evaluations to include ImageNet models. The main results are summarized in Table 3 . Moreover, Figure 9 further illustrates the accuracy drops of various CLIP models, in comparison to Ima-geNet models. Surprisingly, we find that ImageNet models are more robust to spurious features in CounterAnimal. This finding indicates that our CounterAnimal specifically characterizes the spurious features that are unique to CLIP configurations. Additionally, it indicates that spurious correlations in large-scale multi-modal data are distinct from that of the ImageNet scenarios which are widely used in conventional single-modal supervised learning. It further highlights the importance of our proposed dataset, which is especially suitable to study the spurious correlations for vision-language pre-training.Advanced LVLMs. We further evaluate for more advanced LVLMs, which align CLIP visual encoders with advanced large language models like Vi- cuna [49] . To reduce inference costs, our evaluation follows the 1 vs. 20 setup. We summarize their results in Table 4 , along with the 1 vs. 20 results for several CLIP models (cf., Appendix F for more results). We further depict the full results in Figure 2 (b). As we can see, these advanced LVLMs have lower performance yet smaller drops, but the spurious features in CounterAnimal still impact them.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: In Definition 1, we clearly outline the assumptions made for the subsequent theoretical analysis presented in Theorem 1. In Appendix D, we present further details regarding our assumptions, proofs, and the associated experiments to validate their feasibility. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
understanding why clips rely on spurious features: 
 To better understand the observed phenomena in Section 3, we present a theoretical analysis of why the CLIP models rely on spurious features. We begin by establishing the setup for analyzing multi-modal contrastive learning following [9] .EQUATIONinv ) and a spurious feature z spu ∼ N (µ spu a, σ 2 spu ) with Pr(a = y) = p spu otherwise a = -y. y is the label uniformly drawn from {-1, 1}. The training data D tr is drawn with 1 2 ≤ p spu ≤ 1 and OOD data D * is drawn with a p spu = 1 2 .We employ two linear encoders: g I : R d I → R h for the image modality and g T : R d T → R h for the text modality, implemented as g I (x I ) = W I x I and g T (x T ) = W T x T with W I ∈ R h×d I and W T ∈ R h×d T . The encoders are trained through the linearized contrastive loss [9, 50] that mimics the CLIP dynamics:EQUATIONwheres ij = g I (x i I ) T g T (x j T )is the similarity with respect to the i-th image and j-th text representations. Once the CLIP (g I , g T ) has been trained, the performance will be measured in a zero-shot manner by matching the most similar caption with the corresponding object name filled in, such as "a photo of <object label>" [2] . Intuitively, once the model focuses more on invariant features, it will have a better zero-shot classification accuracy across different distributions. Nevertheless, in the following theorem, we justify that CLIP remains to learn to use spurious features, aligning with our experimental observations on the CounterAnimal dataset. Theorem 1. Given a multi-modal dataset (Def. 1) with suitable variance in the features σ inv = Θ(1) > σ spu , and spurious features with a large spurious correlation p spu = 1 -o(1), an overparameterized CLIP model where n = ω(1), d M = Ω(n) and d T = Ω(n), if the spurious features (e.g., backgrounds of the image) takes up a relatively large amount of the image 1), the CLIP model achieves a large error in zero-shot accuracy in the OOD test data where a ̸ = y:µ spu ≥ σ 2 inv +2 2 ≥ µ inv = 1, then with a high probability of at least 1 -O( 1 poly(n) ) = 1 -o(Err(g I , g T ) ≥ 1 -Φ(κ 1 ) -o(1),and a small error in the OOD test data where a = y: Figure 11 : CLIP performance on ColoredCOCO. "supervised" refers to supervised trained models, while "obj" and "objbkg" refer to using different prompts to fine-tune CLIPs.Acc(g I , g T ) ≥ 1 -Φ(κ 2 ) -o(1), where κ 1 = σ 2 inv +2-2µspupspu √ (1+σ 2 inv ) 2 σ 2 inv +(2µspupspu-1) 2 σ 2 spu , κ 2 = -2µspupspu-σ 2 inv √ (1+σ 2 inv ) 2 σ 2 inv +(2µspupspu-1) 2 σ 2We leave more theoretical details as well as the proof to Appendix D due to space limit. Intuitively, Theorem 1 implies that once there exists a relatively strong correlation between the object captions and the parts of image backgrounds, CLIP will learn to align the backgrounds, i.e., spurious features, with object captions. Although our theory discusses a simplistic case of one invariant and one spurious feature, there could exist more features describing the objects and even more features describing the backgrounds. CLIP will fail to robustly align the visual features of objects to its captions, once there exists a spurious correlation between any of the background features with the object caption. Our theory is the first to provably demonstrate the drawbacks of CLIPs in OOD generalization, providing the foundation for future developments tackling the issue.To verify our theory, we construct multi-modal datasets named ColoredCOCO following [51] . It contains 9 classes and the spurious correlation in the training part is 80%, i.e., each class has a correlation of 80% to a specific biased color and 20% uniformly correlates to 10 different randomly chosen colors, cf., Figure 10 . The OOD datasets are built with classes randomly correlating to other 8 biased colors.We consider two prompts with different descriptiveness: a) obj: "a photo of <object label>" and b) objbkg: "a photo of <object label> in <color label> background", with either objects or both objects and backgrounds.We tune the pre-trained CLIP models using the CLIP objective, which has been shown to be most robust to distribution shifts [52] . In addition, we also incorporate the baseline of full fine-tuning with a new MLP onto the image encoder using the ERM objective. As shown in Figure 11 , fine-tuning with CLIP objective based on neither of the prompts provides any non-trivial robustness against the vanilla full fine-tuning. The results further verify our theory. Nevertheless, the degraded robustness of CLIP could also be caused by the weak language understanding capability of the BERT encoder in the CLIP. To this end, we also conduct additional experiments with a perfect language encoder setting.The results are given in Appendix D.4. Nevertheless, we find that CLIP still performs similarly to ERM and is prone to distribution shifts even with perfect captions.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: In Section 2.1, we elaborate on the exact process used to construct the CounterAnimal dataset, including details about the raw data sources, the construction pipeline, and the dataset configurations. Moreover, for reproducibility, most of our experiments utilize open-sourced model checkpoints from the OpenCLIP and PyTorch repositories. We also comprehensively document the evaluation setups in Appendix C. We will release our dataset and the evaluation codes in the future. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the broader impacts of our work in Appendix A, focusing on our impacts for both the research community and the real-world applications. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion: 
 In this paper, we highlight biases in previous evaluations for assessing the robustness of CLIP models, primarily relying on ImageNet variants. Such improper benchmarking would cause illusions that CLIP models seem to resolve spurious correlations, particularly in comparison with ImageNet models. It motivates us to craft the new testset, named CounterAnimal, which is specifically designed to probe the natural spurious correlations between animal and their backgrounds. The spuriousness captured by CounterAnimal is general across different CLIP setups and exerts relatively small impacts on the ImageNet benchmarks, thereby specifically capturing the spurious correlations within CLIP setups. Our experiments on CounterAnimal show that many conventional strategies, e.g., increasing backbone scales and improving pre-train data quality, remain effective in enhancing the robustness of CLIP models. Moreover, we present a theoretical analysis for the reasons of the CLIP objective to learn biases. Overall, we provide a platform for future developments of more advanced and robust CLIP and vision-language models, and we hope our presented experiments can offer a sober look at the robustness of CLIP models to spurious correlations. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: To avoid any safety risk, we control the label space for our CounterAnimal dataset and manually cleanse any unsafe image during our data collection procedure. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The current community often overestimates the robustness of CLIP models, largely due to the potentially misleading reliance on ImageNet variants for testing. To address this issue, we propose a new testset, named CounterAnimal, specifically tailored for CLIP models. Our findings indicate that CLIP models may not be as robust to distribution shifts as previously believed. Our dataset serves as a real-world benchmark, poised to be meaningful for the subsequent works to understand and enhance CLIP concerning their OOD robustness. For real-world applications, the understanding of spurious correlations for CLIP is also critical. We raise practical concerns when deploying CLIP models, which pertain to fairness and potential biases that may arise from inherent spurious correlations. We also present general strategies and theoretical analysis to understand the spurious correlations within CLIP models, which may motivate subsequent works to further enhance CLIP in real-world applications. However, although our dataset reaches the bar as a standard evaluation dataset, its research potential can be further benefited from expanding the scale of our dataset, diversifying the raw data sources beyond iNaturalist, broadening the semantic scope beyond animal classes, and studying other testbeds beyond the ImageNet benchmarks. In the future, we will extend our focus beyond animal subjects and include a wider array of high-quality data that are suitable for evaluating the robustness of CLIP and more advanced LVLMs.We release our dataset CounterAnimal structured as follows:Overall, the CounterAnimal dataset is organized by the object names. The data therein are further separated into two parts, i.e., the easy and hard groups, where the background name is also provided for each sub-directory. By evaluating accuracy with respect to the easy and hard groups, one can quantify the impacts of the spurious correlations captured by CounterAnimal. We further summarize the ImageNet animal objects as well as the group names for the easy and hard groups in Table 5 .In this section, we provide more details about our experimental configurations.All experiments are realized by Pytorch 1.81 with CUDA 11.1, using machines equipped with GeForce RTX 3090 GPUs and AMD Threadripper 3960X Processors. We consider two different label spaces of candidate labels: a) using the full ImageNet-1K class names and b) using the top-20 most confusing classes for more computing-intensive models like MiniGPT4. It leads to the following two evaluation setups, i.e., the 1 vs. 1000 setup and the 1 vs. 20 setup.1 vs. 1000 Setup. As a default option, we use the full label space of the ImageNet-1K dataset, which is suitable given that the object labels for CounterAnimal all belong to that of the ImageNet-1K dataset. Furthermore, this choice also reflects a more realistic situation in the open world, where we have a vast number of candidate labels and the failure cases of LVLMs are common.1 vs. 20 Setup. To suit more advanced LVLMs of which the inference costs are much higher than CLIP models, we constrain the sizes of candidate label space for each class. Specifically, based on CLIP-LAION400M-ViT-B/32, we select the top-20 most confusing labels, which is calculated by the average cosine similarity for both the easy and hard groups.Now, we discuss the evaluation metrics. Typically, they are applied to the easy and hard groups separately when we evaluate the robustness of various models.Class-wise Accuracy. We are interested in the effects of spurious features for each class. Therefore, we calculate the prediction accuracy specifically for photos within each class. It can be referred to as the class-wise accuracy, which is given byACC(label) = 1 |I label | i∈Ilabel 1{ŷ i = label},where I label is the indices of photos belonging to label and ŷi is the predicted label for the i-th image. The class-wise accuracy reflects the class-level model reliability against spurious correlations.Average Accuracy. Upon the class-wise accuracy, we can calculate the average performance of models, namely,ACC = 1 |L| label∈L ACC(label).Compared to the conventional average accuracy, i.e., 1 |I| i∈I 1{ŷ i = gt} with I the image indices and gt the true labels, our definition of the average accuracy further offsets the impact of class imbalance. We default to using the average accuracy, and present the results without balancing in Tables 17-18 for CLIP and ImageNet models.Accuracy Drop. To quantify the spurious correlations that make CLIP models fail, we measure the performance drop when moving from the easy and hard groups. At the class level, the accuracy drop is defined as the class-wise accuracy of easy minuses that of hard. At the dataset level, it is the average value for the class-level accuracy drop.To evaluate LVLMs with a backend of language models, we follow the common practice that constructs questions to prompt LVLMs [29, 30] . Specifically, we construct the question as:What is the main object in the image?and then calculate the language modeling loss with respect to the answer:A <object name> for each ImageNet class name. Meanwhile, we also try another question prompt that is widely used in training MiniGPT4 and LLaVA [30, 53] :Describe this image in detail.while the performance will generically decrease. In addition, when we switch to the object-centric evaluation protocol as [33] :Is there a <object name> in the image?or Is this image a photo of <object name>?and evaluate the answer with Yes for each class, we observe a severe performance decrease as LVLMs easily hallucinate the objects. If we strictly follow the evaluation metrics of [33] by simply fetching the answers instead of comparing the losses, there exist lots of hallucinated objects by LVLMs in our dataset.For the CLIP checkpoints, we adopt the naming rule of "CLIP-<dataset>-<backbone>", where <dataset> is the name of pre-train datasets and <backbone> is the specific name of backbone models. For example, CLIP-LAION400M-ViT-B/32 indicates the ViT-B/32 model CLIP-trained on LAION400M. Different training setups are considered in OpenCLIP, and the versions of the adopted checkpoints are summarized in Table 6 . Moreover, in Table 15 , we consider the results of checkpoints beyond the adopted ones.We provide a more detailed setup and analysis in complementary to Section 4.We begin by introducing more details about the data generation process following the literature [9, 50, 54] . Definition 2 (Multi-modal Dataset). Consider n image-text pairs {(x i I , x i T )} n i=1 , both image x i I ∈ R d I and text x i T ∈ R d T are generated from the underlying latent factor z i ∈ R l . The samples are generated as follows:EQUATIONinv ) and a spurious feature z spu ∼ N (µ spu a, σ 2 spu ) with Pr(a = y) = p spu otherwise a = -y, y is the label uniformly drawn from {-1, 1}, D tr is drawn with 1/2 ≤ p spu ≤ 1 while the OOD test data D * is drawn uniformly with p spu = 1/2.• Given z, the x at modality M is generated viax M = D M µ M (z)+ξ M , with D M ∈ R d M ×land ξ M ∼ N (0, σ 2 ξ /d m I dm ). The matrix D M ∈ R dm×l with d m > l is a matrix with orthonormal columns which can be considered as a dictionary matrix.With the definition, we can write everyz i = y i + η 1,i µ spu p spu + η 2,iwhere η 1,i , η 2,i are two Gaussian variables in the definition.CLIP Training. We employ two linear encoders g I : R d I → R h for the image modality and g T : R d T → R h for the text modality, implemented as g I (x I ) = W I x I and g T (x T ) = W T x T with W I ∈ R h×d I and W T ∈ R h×d T , respectively. The encoders are trained through the linearized contrastive loss [9, 50] that mimics CLIP training dynamics:EQUATIONwhere s ij = g I (x i I ) T g T (x j T ) is the similarity with respect to the i-th image and j-th text representations, and||W T I W T || 2F is the a regularization term with ρ > 0. Zero-shot Inference. Once the CLIP model (g I , g T ) is trained, the performance will be measured in a zero-shot manner by matching the most similar caption such as 'a photo of {object name}' across different object name as class names. Meanwhile, one could also leverage several prompts and leverage the average text embeddings across the available prompts to facilitate the evaluation [2] . The prompt with respect to y could be modeled as p y = D T E[z t |y], where D T is the prompt transformation matrix. Then, the zero-shot accuracy of CLIP could be formalized as follows:EQUATIONwhile the error is Err(g I , g T ) = 1 -Acc(g I , g T ). Intuitively, once the model extracts more of the invariant features, it will have a better zero-shot classification accuracy across different distributions. ≥ µ inv = 1, then with a high probability of at least 1), the CLIP model achieves a large error in zero-shot accuracy in the OOD test data where a ̸ = y:1 -O( 1 poly(n) ) = 1 -o(Err(g I , g T ) ≥ 1 -Φ(κ 1 ) -o(1),and a small error in the OOD test data where a = y:Acc(g I , g T ) ≥ 1 -Φ(κ 2 ) -o(1),whereκ 1 = σ 2 inv +2-2µspupspu √ (1+σ 2 inv ) 2 σ 2 inv +(2µspupspu-1) 2 σ 2 spu , κ 2 = -2µspupspu-σ 2 inv √ (1+σ 2 inv ) 2 σ 2 inv +(2µspupspu-1) 2 σ 2 spu and Φdenotes the CDF of a standard normal distribution.Proof. We will introduce some useful lemmas to help with our proof.). The minimizer of linearized CLIP loss W * T I W * T satisfies the following with a probability of at least 1 -O( 1 poly(n) ) such that,||W * T I W * T - 1 ρ D I 1 + σ 2 inv 2µ spu p spu -1 2µ spu p spu -1 1 + σ 2 spu D T T || 2 ≤ 1 ρ O( √ ϵ 0 ),whereϵ 0 = O( log n n ).Intuitively, the lemma indicates the importance of the training distribution, that the minimizer of CLIP will converge to the data characteristics of the latent features of the training distribution.Then, consider the case where the model is inferred onto a test sample with y = 1, a = -1. Then, with the aforementioned lemma, we have|x T I W * I W * T x ŷ T - 1 ρ x T I D I 1 + σ 2 inv 2µ spu p spu -1 2µ spu p spu -1 1 + σ 2 spu D T T x ŷ T || 2 ≤ ||x I ||||x ŷ T || 1 ρ O( √ ϵ 0 ) ≤ 1 ρ O( √ ϵ 0 log n).(4) Then, notice thatEQUATION2 )(2µ spu p spu -1).(5) When CLIP makes an incorrect prediction, we havex T I W * I W * T x ŷ=1 T < x T I W * I W * T x ŷ=-1 T .Then, we haveEQUATIONwith Eq. 5 plugged in, denote ϵ 1 = O( √ ϵ 0 log n), we further haveEQUATIONSince where Φ is the CDF of the standard Gaussian distribution. Then, it suffices to know that the Err y=1,a=-1 is lower bounded by Φ(EQUATIONσ 2 inv +2-2µspupspu+ϵ1 √ (1+σ 2 inv ) 2 σ 2 inv +(2µspupspu-1) 2 σ 2 spu), which also applies to the case y = -1, a = 1.Similarly, given the case y = a, as the model fits the spurious feature, we could derive the lower bound for its Acc by leveraging the spurious features as Φ(-2µspupspu-σ 2 inv √ (1+σ 2 inv ) 2 σ 2 inv +(2µspupspu-1) 2 σ 2 spu).To further validate our theoretical results, we construct the ColoredCOO dataset following [51] . More specifically, ColoredCOO is constructed as follows:• The dataset contains 9 classes of COCO objects. The spurious correlation in the trainset is 80% such that each class has a correlation of 80% to a specific biased color and 20% uniformly correlates to 10 sufficiently different randomly chosen colors. • The OOD testsets are constructed with classes randomly correlating to 8 biased colors different from the one correlated in the training set.Then, we further generate two prompts for each sample:1. obj: a photo of <object label>; 2. objbkg: a photo of <object label> in <color label> backgroundWe tune the pre-trained CLIP models using the CLIP objective based on the OpenCLIP library. We consider the learning rate of {1e -3, 1e -4, 1e -5}, with a weight decay of {1e -1, 1e -3, 1e -5}, and a warmup of {0, 1000} steps. We select the model according to the best in-distribution test performance. The detailed results are given in Table 7 . As we can see, the CLIPs finetuned using either the CLIP objective or the standard supervised training both exhibit high sensitivity to the spurious features.One possible explanation for the failure of CLIP objective in ColoredCOCO is that, the language encoder of the CLIP models may not understand the captions well. Therefore, we further construct a new setup called MultiColoredMNIST, where each image contains only the digit information from MNIST dataset and the color information. Therefore, we can directly derive the one hot encoding for all of the useful factors in the dataset.Data. We consider a multi-class classification setting with a number of classes no less than 2. The objects are the In Figure 12 , we give some examples for the MultiColoredMnist dataset.Experimental setting. We compare the standard supervised training and CLIP. To avoid noises or information loss in encoding language modality, we consider the perfect language supervision for a single model. Given a batch of image and caption representations {(h xi , h ci )} B i , for a image-caption pair, the CLIP objective aims toEQUATIONwhere M x ∈ R d×hx and M d×hc c are the learnable projection layers for image and caption representations. Assuming the perfect language encoding as the one-hot encoding for all possible object and background appearance h ci ∈ [0, 1] |O|+|B| , and M c can simply be an identity matrix, then Eq. 9 can be considered as a classification task for objects and backgrounds respectively:EQUATIONwhere the labels are simply the one-hot encodings of the objects and the backgrounds, and the classifier is M T c M x . For the MultiColoredMNIST task where there is only one object and background (i.e., color), to implement Eq. 10, we only need to construct an additional classification head for the background. Given the aforementioned setup, we conduct experiments comparing CLIP-based contrastive learning to the standard supervised learning. The results are given in Table 8 . As we can see, both contrastive learning and supervised learning perform similarly across different numbers of classes and bias degrees. In this section, we present ablation studies to further validate the feasibility of our data curation process.Biasing to ImageNet Setups. We follow the same curation procedure while using ImageNet models (i.e., ResNet50-ImageNet) to construct easy and hard splits, where we name the corresponding dataset as CounterAnimal-I. We present the results of CLIP and ImageNet models on CounterAnimal-I in Tables 9 10 We also list the background names for easy and hard splits with respect to some of the selected classes in Table 11 . As observed, using different models to split data will capture very different spurious features. It highlights the necessity to curate an OOD testset for CLIP models, as CLIP models learn different spurious features than ImageNet models.Correctness vs. Frequency. We further explain why easy and hard examples can characterize spurious features within CLIP setups. In general, spurious features can be caused by biases inherent in the data distribution concerning backgrounds. For example, for the animal class of ice bear, the background of ice is more common than other backgrounds, such as grass, thus causing spurious correlations learned by CLIP models. Therefore, we investigate in terms of the background frequency, employing the searching tool of Have I Been Trained4 that can retrieve images from LAION5B closely matching a given class name. We examine 10 animal classes as our case studies. For each class, we collect the top 100 most relevant images and tally the occurrences of the backgrounds of our consideration. It is important to note that our counting process excludes cartoon images, irrelevant photos, corrupted photos, and those featuring multiple distinct animal subjects or ambiguous backgrounds. The results are summarized in Table 12 . As we can see, in general, the spurious features captured align with our conjecture that hard examples contain uncommon backgrounds in the CLIP training data, e.g., LAION5B, further justifying the feasibility of our CounterAnimal in assessing the robustness of CLIP models in real-world situations. In this section, we present more experimental results to support our claims.Effective Robustness. In Section 3, we mainly examine the absolute robustness to assess and compare the OOD performance across various CLIP setups, which are well known to be sensitive to the original value scales. Therefore, in Figure 14 , we apply the measures of effective robustness [5] to further substantiate our conclusions. Overall, our previous conclusions are upheld, demonstrating that the benefits derived from increasing model scales and enhancing data quality notably outweigh those obtained by merely expanding dataset sizes.Top-5 Results for CLIP models. We present 1 vs. 1000 results for more CLIP checkpoints on the CounterAnimal dataset in Table 13 , which is an extension of Table 2 . Moreover, we present more results for the evaluations on CounterAnimal, supplementing our analysis of CLIP models under spurious correlations. To begin with, we report the top-5 scores under the 1 vs. 1000 setup, where we check if the target label is one of the top-5 model predictions. The results are summarized in Table 14 . Comparing with the top-1 results in Table 13 , we find that there is still a large performance gap between the easy and hard groups, indicating that the label confusion is quite diverse and not limited to the top two classes.Other Versions of Pre-train Datasets. OpenCLIP provides other CLIP checkpoints beyond our adopted ones. Table 15 summarizes the results of CLIP models similar to Table 2 while using different versions of checkpoints. As we can see, the performance for both easy and hard is very stable across varying versions, except for DataComp1B. The reason is that their various checkpoints use subsets of DataComp1B, where XL indicates the fully DataComp1B, L indicates a 140M subset, M indicates a 14M subset, and S indicates a 1.4M subset.Results of OpenAI Prompts. We further consider the prompt setups following OpenAI CLIP [2] , using average text embeddings over 80 predefined prompts as the final text embeddings. The results are summarized in Table 16 . As we can see, the average performance for both the easy and hard groups generally improves 1 to 3 percentage points over the results of our simpler prompt. However, our main conclusion remains unchanged: the ImageNet models generally exhibit better performance and smaller drops. Another interesting finding is that when evaluating with CLIP-LAION400M-ViT-B/32 (the CLIP checkpoint employed in our data collection), the performance drop with OpenAI prompts is not as high as that of our simple prompt used in Table 2 . It indicates that our curation procedure mainly overfit the adopted prompt instead of the particular CLIP checkpoint.Average Performance without Balancing. We by default adopt the balanced average accuracy to offset the impacts of class imbalance. In Tables 17 18 , we further summarize the results without class balance, following 1 |I| i∈I 1{ŷ i = gt}. As we can see, the performance drop remains obvious, and similar conclusions can be drawn as the balanced results: a) Backbone scales are more important for spurious robustness than pre-train dataset scales, and b) ImageNet models are more reliable when facing spurious features in CounterAnimal.1 vs. 20 Results for CLIP and ImageNet Models. We adopt the 1 vs. 20 setup for the evaluations of more advanced LVLMs in Table 4 . For a fair comparison, we further summarize the 1 vs. 20 Figure 15: The easy versus hard performance (%) for CLIP and ImageNet models, following the 1 vs. 1000 setup. We also present the model setups for each easy-hard result pair.Figure 16: The easy versus hard performance (%) for CLIP, ImageNet models, and more advanced LVLMs, following the 1 vs. 20 setup. We also present the model setups for each easy-hard result pair. NeurIPS Paper Checklist https://www.inaturalist.org/observationshttps://haveibeentrained.com 


Decision: **Summary**
CLIP is believed to be robust to distribution shifts due to its performance on ImageNet shifts. This paper presents a new evaluation dataset named CounterAnimal that contains a common group of animals in common backgrounds and a counter group including animals in plausible yet unusual backgrounds. When evaluated with this dataset, CLIP exhibits performance drops challenging common assumptions on the robustness of this model.

**Recommendation**
Accept. All the reviewers tend towards acceptance and the authors did a good job with the rebuttal. I also believe this work is interesting as it challenges a common assumption about CLIP in the research community.

Reviews: review: paper_summary: paper_summary: This work asks one interesting question: "Do CLIP models always generalize better than ImageNet models?" Driven by this question, this work proposes a new benchmark dataset named CounterAnimal. This dataset consists of a) the common group: comprising animals in common backgrounds, and b) the counter group: including animals in plausible yet unusual backgrounds. The main idea is that the performance drops from the common to counter groups quantify the reliance on spurious background features for animal predictions. The main observation is that CLIP models exhibit notable performance drops when tested on the counter group. In comparison, ImageNet models can be more robust than CLIP models.
strength_weakness: strength_weakness: -  I found the analysis of why CLIPs rely on spurious features interesting. However, I think the claim is somewhat "obvious": there exists a relatively strong correlation between the object captions and the parts of image backgrounds, CLIP will learn to align the backgrounds, i.e., spurious features. If the training dataset contains many examples of spurious correlations, then models will tend to be biased.

- I am curious about why ImageNet models may not be so influenced by the spurious bias in CounterAnimal. Is this because the ImageNet training set does not have too many spurious correlation examples? Or ImageNet has a spurious bias but such bias is different from the one in CounterAnimal? Please provide a discussion or share some insights on this question. 

- This paper adopts absolute performance drop in Section 3.3. Such a metric may not be so robust. For example, model A drops from 40 to 39, and model B drops from 90 to 89. They drop the same but the I would say model B is better. Please comment on this, and discuss the metric of absolute performance drop.
questions: questions: - ImageNet models are not so biased toward spurious correlations compared with CLIP models. Why? Is this because the ImageNet training set does not have too many examples that exhibit spurious correlations?

- While I appreciate this work includes results on ColoredCOO and ColoredMINIST, some other spurious correlation benchmarks (e.g., WaterBirds) would be greater if they were also included.
limitations: limitations: The dataset is proposed, so a discussion on the potential bias/ privacy is needed. I appreciate this work highlights the future improvement of expanding semantic scope, data source, and ImageNet testbed.
paper_summary: paper_summary: In this work, the authors create an evaluation dataset comprising two groups, one with animals in usual backgrounds (common group) and another with unusual backgrounds (counter group). They then evaluate a suite of models of different backbones, model sizes, and datasets. They find that CLIP models do poorly than ImageNet-trained models, and generally high quality data or bigger model size improves counter group accuracy.
strength_weakness: strength_weakness: Please see questions for more information.
questions: questions: 1. **Biased dataset:** The dataset is split into to common and counter group using a CLIP model. Therefore by construction, the CLIP models will perform poorly and it is no surprise that the ImageNet-trained models do a bit better. One could construct a split of this dataset where ImageNet models do better than the CLIP models. If the dataset was collected in a model agnostic way then the conclusions could potentially be more interesting. 

2. **On the premise:** As such the premise or the primary question seems a bit vacuous. Models arguably learn different features and there will exist some type of evaluation where one does better than the other. But are there useful tasks/evaluations where ImageNet models are preferred over CLIP models? That is an interesting open question. This work doesn't necessarily start from there and create a benchmark that is supposed to represent a task. The authors rather create a biased dataset that by design make CLIP models perform poorly. Therefore the primary premise of the work seems erroneous. There is some value in the other evaluations so maybe the paper could be rewritten by positioning things differently. 

3. **Lack of novelty:** Keeping the primary result aside, other conclusions like that better datasets or model sizes improve robustness are not new. Please see [1, 2, 3, 4]. 

[1] [Geirhos 2021] https://proceedings.neurips.cc/paper/2021/hash/c8877cff22082a16395a57e97232bb6f-Abstract.html

[2] [Idrissi 2022] https://arxiv.org/abs/2211.01866

[3] [Fang 2022] https://arxiv.org/abs/2205.01397

[4] [Nguyen 2022]  https://arxiv.org/abs/2208.05516
limitations: limitations: The authors have addressed some limitations. For the rest, please see my questions block.
paper_summary: paper_summary: This paper presents CounterAnimal, an evaluation dataset featuring two subsets: animals with common backgrounds and those with unusual backgrounds. The images were sourced from iNaturalist. Data with high CLIP accuracy are categorized as "Common",  while those with low CLIP accuracy are labeled as "Counter".  Results shows that CLIP models experience a greater accuracy drop compared to ImageNet models when tested on this dataset.
strength_weakness: strength_weakness: - The proposed dataset is not sufficiently robust to analyze the influence of spurious bias, as this is not the only difference between the common and counter datasets.
  - To analyze the accuracy drop caused by spurious features such as background, the background should be the only difference between common and counter image pairs. Prior work [4,5] has proposed such datasets focusing on background.
  - In the proposed dataset, other factors may influence the model accuracy gap besides background. For instance, as shown in Figure 1, the more varied gestures of ice bears on the right compared to the left could be a contributing factor to the accuracy drop.

- Current experiments cannot conclusively show that ImageNet models generalize better than CLIP.

   - As the common and counter groups are selected according to the CLIP accuracy (see line 165 in the paper),  they indicate easy and hard samples for CLIP.    Since ImageNet models have different training characteristics, it is natural that hard cases for these models may differ from those for CLIP, resulting in a smaller performance drop for ImageNet models. This result cannot support that ImageNet models are more robust than CLIP models.
   -  The accuracy drop from common to counter group can be greatly influenced by the model used to divide the common and counter dataset. Using the combined proposed common and counter dataset, a new Common' and Counter' dataset can be created based on the accuracy of ImageNet models. What is the impact of this dataset division on the accuracy drop for different models?


- Prior studies[1,2,3,4,5,6] have proposed datasets specifically to analyze the influence of background, which are not discussed in this work.  These datasets can be used for CLIP evaluation as they do not overlap with the CLIP training set. Additionally, creating datasets based on model accuracy in this work is similar to the approach in [6].

[1] Noise or Signal: The Role of Image Backgrounds in Object Recognition.

[2] Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models, NeurIPS 2019.

[3] Dataset Interfaces: Diagnosing Model Failures Using Controllable Counterfactual Generation.

[4] ImageNet-E: Benchmarking Neural Network Robustness via Attribute Editing, CVPR 2023.

[5] LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Images, NeurIPS 2023.

[6] ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object, CVPR2024.
questions: questions: Please refer to the weaknesses.
limitations: limitations: This paper has discussed the limitations.
paper_summary: paper_summary: The authors aim to investigate spurious correlations learned by CLIP models. For this, they curate a novel dataset where animals are organized into common and uncommon backgrounds, e.g. a polar bear is more likely encountered in snow than on grass. The authors then perform experiments where they benchmark various CLIP and ImageNet models on the curated dataset. They observe that CLIP models suffer from spurious correlations which stem from changing the background.
strength_weakness: strength_weakness: The paper has many issues, both in terms of writing and the methodology which need to be fixed.

### Major:
**The authors missed important previous works**: The paper “Recognition in Terra Incognita” is very related to this work and also proposes a “dataset designed to measure recognition generalization to novel environments” based on camera traps. The dataset is sorted according to difficult environments for different animals, which makes it very similar to CounterAnimal. I think the authors need to cite and discuss this paper. Currently, I do not understand the benefit of having a new dataset in addition to the already present one. The waterbirds dataset is also highly similar and should be discussed (https://arxiv.org/pdf/1911.08731). The authors cite that paper, but do not discuss it in the Related Work section, nor put it into context with CounterAnimal. The backgrounds challenge (https://github.com/MadryLab/backgrounds_challenge) is also highly related and should be discussed. In general, the related work section is very weak, given how extensively spurious correlations and worst-group-accuracy have been studied. Another important work to be discussed would be "Finding and Fixing Spurious Patterns with Explanations" (https://arxiv.org/abs/2106.02112).

**The naming of the common vs counter groups is misleading**: 
Line 165: “Photos with the highest CLIP accuracy are assigned to the common group, and those with the lowest CLIP accuracy are assigned to the counter group.” I have a major understanding issue here. As far as I understood the paper before this line, the goal was to put images with common backgrounds into the common group and images with uncommon backgrounds into the counter group. This is also depicted in Fig. 1 or Table 1. The caption in Fig.1 says that “Most ice bears appear in a snow background (i.e., common), while it also is reasonable to find some ice bears in a grassy environment (i.e., counter)”. But here, the authors write that accuracy has actually been used to separate images into these groups? But then the frequency of the co-occurrence of certain backgrounds and classes has not been taken into account, or rather, it is a conjecture that those backgrounds where the CLIP model has higher accuracy on are more “common”? 

**The terms "effective robustness" and "robustness" are used interchangeably which is wrong and confusing**:
I think the paper conflates the terms “robustness” and “effective robustness” which is confusing. When looking at effective robustness plots, such as in Fig. 2, we are interested in the residual difference between the measured value and the value predicted by the linear fit. As I can see, all plotted markers (CLIP and ImageNet) lie on their respective linear fits, and none of the interventions, such as CLIP-DC or CLIP-DFN offer any effective robustness benefits. It is though true that the **absolute** robustness numbers are overall higher for the CLIP-DFN models, for larger models or models trained on more data. I am however confused by the authors discussion of this observation. On the one hand, they write that larger CLIP models are more robust but increasing the dataset size does not yield improvements. First, I am confused whether they mean “effective robustness” or “robustness” here. Second, I do not see the effect the authors are describing: Both more data and larger backbones have higher absolute robustness but the same effective robustness as the other models. The statement “CLIP models trained on high-quality data are more robust” is also confusing, because it is not clear whether “robustness” or “effective robustness” is meant. 

**Due to methodology issues, results on CLIP models cannot be compared to results on ImageNet models (or other advanced LVLMs):**
Line 60: “d) Spurious discovering: preserving classes and associated data based on the decrease in zero-shot performance (i.e., evaluating based on pre-trained CLIP models without fine-tuning) when shifting the backgrounds.” This step is really unclear. Do the authors curate the dataset based on the zero-shot accuracy of a CLIP model? From the introduction and the abstract, it sounds like the authors want to benchmark the robustness of CLIP vs ImageNet models on this custom dataset. But then, it is strange that CLIP models also seem to be used during the curation process. After reading the more detailed description in line 156, I think the statements made in line 85 are misleading. The authors write “ImageNet models are more robust to spurious correlations captured by CounterAnimal” and “Compared with CLIP models (colored in blue), surprisingly, we find that ImageNet models exhibit a stronger robustness to the spurious correlations in CounterAnimal.” Given that CounterAnimal has been curated based on the performance drop of a CLIP model, I find it very unsurprising that CLIP models perform worse on it compared to ImageNet models. I think that if CounterAnimal had been curated based on an ImageNet-trained ResNet50, the trend would have been reversed. I think all statements comparing CLIP and ImageNet trained models on CounterAnimal need to be relaxed and I think that this comparison is quite meaningless because of the described selection bias. I think that the whole Section 3.3. is misleading for this reason and statements such as the following cannot be made given the methodology issues: “Surprisingly, we find that ImageNet models are more robust to spurious features in the CounterAnimal dataset. This finding may contradict the common belief [Radford et al., 2021, Shi et al., 2023] that the CLIP models tend to be more robust to spurious correlations than single-modal supervised learning.” Similarly, the conjecture paragraph from line 265 onwards is wrong and cannot be made. 

For the same reason, the comparison to advanced LVLMs in line 273 onwards cannot be made.

Figure 1: Are these examples cherry-picked or are they representative of the data points present in CounterAnimal? I am asking this, because of the Winoground dataset [A]. This dataset tests the compositionality of VLMs by forcing a model to match two captions to two respective images. Winoground has later been criticized because the two images in the choice process are not equally hard [B]. For example, the model needs to match “the glass is on the grass” and “the grass is in the glass” to the corresponding images. However, there is much more grass in the image matching to the first caption, and the model likely picks that image for both captions just because there is more grass and it makes the decision in a bag-of-words-manner. To summarize, Winoground did not control for object size, orientation and other confounders. In Fig.1, it appears that the main objects (the polar bears) are equal in size, so size could be excluded as a possible confounder? Did the authors consider this possibility, i.e. that the drop in performance could be explained by other differences in the images from the respective domains?
[A] https://arxiv.org/abs/2204.03162
[B] https://arxiv.org/abs/2211.00768

### General:
Line 25: please cite CLIP

Line 64: “The resulting dataset covers a total of 45 animal classes, ends up with 7,174 common photos and 5,926 counter photos, aligning with the standard size as an evaluation dataset [Recht et al., 2019, Hendrycks et al., 2021].” -> I do not understand this statement. Different test sets have different numbers of test images. ImageNet Val has 50k images for example. In what sense are the presented numbers standard?


Line 94: “Overall, larger CLIP backbones (i.e., larger markers) can improve the effective robustness, implying that scaling up backbones may enhance robustness against spurious features.” -> I do not see this in Fig. 2. The larger markers appear to be on the fitted line, same as the smaller markers. Effective robustness measures the difference with respect to the linear fit, and there is none for the larger CLIP backbones. Please clarify this point.


Line 146: “feature noise involves severe feature corruptions” -> Please be more specific here. What do you mean with feature noise? Do features refer to animals features such as missing ears or such? Or to the images themselves?

Line 147: “clarity issues arise when animal objects are not in major positions” -> unclear formulation: what is a major position? Do the authors mean that the animals are too small or not in the center of the image?

Line 153: “Note that the class space of backgrounds as above is not entirely orthogonal with each other due to the inherent ambiguity of the real-world situations. Nevertheless, we try our best to discern the assigned background labels within each animal class.” -> This is unclear. How many images would be ambiguous? I could imagine that many images would have two backgrounds, such as e.g. grass and sky or snow and water. For example, the last image in Fig. 1 on the left has both snow and water. It is not clear to me that only picking the snow background and ignoring the water is correct here. Further, at least for CLIP, the caption can contain several background keywords.

Further, I imagine animals occur in all kinds of environments, but there are only two backgrounds for each animal. Were the other images also discarded?

Line 214: “Therefore, we conclude that our CounterAnimal dataset possesses some realistic shifts that are generally contained in large-scale pre-training data, regardless of backbones.” This conclusion cannot be drawn from this experiment since the backbone has not been varied here.

### Section 4:
The proposed experiment is very similar to the well-known ShiftMNIST [D] or ColoredMNIST [E] datasets, which test the influence of spurious correlations. The findings here are not novel and should be brought into perspective with previous work. I do not understand how Fig. 11 relates to the text. What is “supervised”, “obj”, “objbkg”?
[D] https://arxiv.org/pdf/1811.00401
[E] https://arxiv.org/pdf/1907.02893

### Typos, grammar:
The quality of the text is poor on some occasions which makes reading and understanding the paper difficult. The manuscript would benefit from a round of proof-reading. Some statements and formulations should be made more precise.
Line 32: “The performance boosts over ImageNet models seem to suggest that CLIP resolves distribution shifts and thus spark a rich discussion about its rationale.” Strange formulation. How can “distribution shifts be resolved”? Please rephrase for clarity.

Line 112: “More specifically, [Yang et al., 2023] report that CLIP models may misaligned frequently co-occured objects with the corresponding texts.”

Line 115: “[Tong et al., 2024] find that CLIP misaligned samples will further cause the hallucination of LVLMs.” I do not understand this statement, grammar errors.

Line 132: “Meanwhile, many existing datasets, e.g., DomainBed and Wilds, do not have overlapped label space with ImageNet, making the comparison between ImageNet and CLIP models hard.” There is a version of DomainBed [C] where the dataset has been filtered to only include classes compatible with ImageNet, such that an evaluation of ImageNet models is possible out-of-the-box.
[C] https://openreview.net/pdf?id=LiC2vmzbpMO

Line 171: “Recalling that, when CLIP models resort to the shortcut of data, the model performance will heavily correlate with the backgrounds presented in the common group yet is compromised when coming to the counter group.” Grammar errors, I do not understand this sentence. What is “the shortcut of data”?

Line 208: “It suggests that the CounterAnimal dataset captures some general spurious shifts that at least commonly present in the pre-train dataset of LAION400M.” grammar

Line 213: “Here, the spurious features degenerate the zero-shot robustness of CLIP models trained on both LAION2B and by OpenAI.” Typo? “degenerate”?

Line 243: “In Figure 7, we consider two pre-train datasets, namely, LAION2B and the close-soured data from OpenAI” typo

Line 297: “Nevertheless, in the following theorem, we justify that CLIP remains learning to use spurious features, aligned with our experimental observations on the CounterAnimal dataset.” grammar

Strange space break between line 310 and 311.

# Summary of the review:
We could fix the naming convention from "common" and "counter" to something like "hard" and "easy" since accuracy has been used rather than frequency of certain backgrounds to classify backgrounds into certain groups. Based on my arguments below, I believe we cannot compare CLIP models to ImageNet models on the proposed dataset in any sensible way due to the introduced selection bias. I believe the very title of the paper is misleading since the posed question cannot be answered based on the methodology issues. But if we remove the claims about comparing ImageNet models and CLIP models, then, the main point of the paper is that there exist backgrounds which are harder for CLIP models, given certain classes, and other backgrounds which are easier. I don't think that this observation is particularly interesting on its own. The authors did not relate the hardness of the backgrounds to their frequency in the pretraining dataset or anything else. The observation that backgrounds matter is also not novel but quite well-known and the authors do not offer a solution. Further, the writing is quite poor and confusing on many occasions; I provided many examples of incorrect and confusing sentences below.
questions: questions: I have written a very detailed review above. I expect clarifications with respect to the raised points, at the very least in the "Major" paragraph.
limitations: limitations: The limitations discuss the comparison in performance of the CLIP vs ImageNet models which I believe cannot be made due to the methodological issues in this work.

Comments: title: Thank you
Response to the recent results
The updated abstract
Background frequency results and further clarfications
Response to the rebuttal
We used **effective robustness** instead of test accuracies as the main supporting experimental results
Final rating update
The original maunscript does not need significant modifications
Preliminary results of suggested experiments are released
Preliminary results of suggested experiments are released
Detailed performances on CounterAnimal-i
Preliminary results of model performances on splits created according to the accuracy of ImageNet models
Thank you
Response to Ethics Reviewer sfcY
Follow-up experiments on splitting prior test set into two datasets based on CLIP accuracy
Further clarification on the remaining questions
Thank you and we are working on the required experiments
Response
Follow-Up on Author Responses to Your Review Comments Before Aug 13th
comment: Dear Reviewer BaJ3,

We would like to thank you for your acknowledgment of our new results and the revised abstract! We are truly grateful for your insightful comments, which have helped improve our manuscript significantly!! 

Please feel assured we will follow your suggestions to extend the analysis and to incorporate all the results including those on `CounterAnimal-i` in future revisions. Thank you again for your valuable time and insights!
Thank you for the additional results and the revised abstract!

**Additional results on the occurrence frequency**: I think these results are very nice. It would be great if you could extend this analysis for all backgrounds and groups for the final version. If the trend holds that the rarest animal-background combinations are the most challenging ones, I think it would also be ok to keep the naming of "common" and "uncommon" since it would have been tested then. And also, this maybe sounds a bit more intuitive. I also like the link between the training data and the test sets that we see. 

**On the revised abstract**: I think it reflects the message of the paper better than the previous one. I especially like that the backgrounds are not split according to their commonness, but according to the accuracy drop. I also like that the phrasing about the ImageNet models is a bit more cautious. Maybe you would want to also mention CounterAnimal-i and summarize the findings for this dataset, as well.

I am raising my score to 6 now, especially because of the new analysis experiment, showing that the hardness of the backgrounds can indeed be tied to the co-occurrence frequency. I trust that the authors will conduct a larger scale analysis for the final version.

Best,
Reviewer BaJ3
> Revision to the paper

Thank you for acknowledging our revisions. Here we provide our reworked abstract for your reference:

```
Large vision language models, such as CLIP, demonstrate impressive robustness to spurious features than single-modal models trained on ImageNet. However, existing test datasets are typically curated based on ImageNet-trained models, that aim to capture the spurious features inherited in ImageNet. Benchmarking CLIP models based on the ImageNet-oriented spurious features may not be sufficient to reflect the extent to which CLIP models are robust to spurious correlations inherited in CLIP training data, e.g., LAION. To this end, we craft a new challenging dataset named CounterAnimal that is designed to reveal the reliance of CLIP models on realistic spurious features.  Specifically, we split animal photos into groups according to the backgrounds, and then identify a pair of groups for each class where a CLIP model shows high-performance drops across the two groups. Our evaluations show that the spurious features captured by CounterAnimal are generically learned by CLIP models with different backbones and pretraining data, yet have limited influence for ImageNet models. We provide theoretical insights that the CLIP objective cannot offer additional robustness. Furthermore, we also re-evaluate strategies such as scaling up parameters and high-quality pre-trained data and find that they still help mitigate the spurious features, providing a promising path for future developments.
```
Dear Reviewer BaJ3,

Thank you for providing further feedback about our work and for raising the rating. We believe all your remaining concerns are addressable! **Although there is limited time before the end of the discussion, we have conducted an investigation of the background frequency to clarify your main concern about the observed spurious correlations.** Please find our responses below:

> Residual differences in the effective robustness

We kindly refer Reviewer BaJ3 to the uploaded figure, where we separately draw the linear fits in terms of each intervention:
- In the uploaded Figure 1, we compared the effective robustness of CLIP models with `ViT/B/16` and `ViT/L/14`, shown as in the blue line and red line, respectively. It can be found that in the rightmost part of the x-axis, two red dots locate beyond the linear fit of the blue line, indicating an improvement of effective robustness of CLIP models based on larger backbone `ViT/L/14`.
- In the uploaded Figure 2, we compared the effective robustness of CLIP models trained with `LAION2B` and `OpenAI` data, shown as in the blue line and red line, respectively. As both of them are considered web data with simple filtering, they have similar data quality that CLIP models trained on either `LAION2B` or `OpenAI` do not lead to improvements in terms of effective robustness.
- In the uploaded Figure 3, we compared the effective robustness of CLIP models trained with high quality data (`HQ`) and relatively low quality data (`LQ`), shown as in the red line and blue line, respectively. It can be found that in the rightmost part of the x-axis, multiple red points locate beyond the linear fit of the blue line, indicating an improvement of effective robustness of CLIP models trained on high quality data (`HQ`).

We have supplemented the aforementioned discussion in our revised manuscript.

> Group naming issue

We have revised our manuscript to avoid the confusion of the group naming.

> Backgrounds frequency

Thank you for your insightful question. We need to clarify that, **our theoretical analysis in Section 4 indeed explains  why CLIP models perform worse on the "hard" backgrounds, as we clarified [in the response to Reviewer 2MXa](https://openreview.net/forum?id=wWyumwEYV8&noteId=BNamSMaVj3)**.

To provide more support of our theoretical explanation, we conduct an investigation in terms of the background frequency following your suggestion! Specifically, we adopt the searching tool of `Have I Been Trained` based on `clip-retrieval` to retrieve images from `LAION5B` closely matching to a given class name. 
- For each animal class, we obtain 500 images and count the frequencies for our considered backgrounds. 
- We control the sampled images aligned to the distribution of natural animal photos. For example, we filter out images with multiple distinct animal subjects or multiple distinct backgrounds. Due to time limit, we currently present three classes as follows and are extending our studies for revisions:

| class name | easy bkg |     | hard bkg |    |
|--------------|----------|-----|----------|----|
| Ice Bear     | Ice      |  83 | Grass    |  7 |
| Black Swan   | Water    | 101 | Earth    | 38 |
| Flamingo     | Water    | 111 | Sky      | 16 |

In general, it aligns with our conjecture that hard examples do contain uncommon backgrounds in the CLIP training data, e.g., `LAION5B`.   

We hope the supplemented results above could clarify the concern of Reviewer BaJ3 about our work.
I apologize for the late response. I have read the rebuttal, the other reviews and the comments.

**Q3. The terms "effective robustness" and "robustness" are used interchangeably which is wrong and confusing."**
The authors write "*We further depict the lines of effective robustness, where we found that the conclusions remain the same. However, the improvement from increasing model scales and improving data quality is much higher than simply scaling up the datasets.*" When we look at effective robustness, we are interested in the residual difference between the measurement and the linear fit. I do not see any interventions in Fig.2 which go beyond the linear fit. Could you please clarify more directly what is meant here?

**On the common vs unusual background issue**: It is good that the authors agreed to rename the groups to "easy" and "hard". Reading the other reviews, all reviewers "misunderstood" the naming convention. I find it very important to emphasize in the updated manuscript that the groups were created based on accuracy and not on occurrence frequency.

**On analyzing the backgrounds frequency**: Since writing that the "hard" group is the "uncommon" one feels so natural, this should be analyzed. I would like to suggest to the authors to perform a caption based analysis to check whether "hard" examples actually do contain uncommon backgrounds in their captions. I understand that parsing the LAION dataset for the images themselves is unfeasible. I of course do not expect results on this until tomorrow; I merely think that performing this experiment would actually justify the authors in writing "common" and "uncommon" backgrounds and would also link the learned spurious correlations nicely to the training data. As it stands, the authors have not offered an explanation for why CLIP models perform worse on the "hard" backgrounds, which has been asked by Reviewer **2MXa**. Reviewer **2MXa** wrote: *I think the claim is somewhat "obvious": there exists a relatively strong correlation between the object captions and the parts of image backgrounds, CLIP will learn to align the backgrounds, i.e., spurious features. If the training dataset contains many examples of spurious correlations, then models will tend to be biased.* This is a natural question which can be analyzed.

**On reframing the paper and the new title**: I agree with the other reviewers that reframing the paper to follow the spirit of ImageNet-A is a good idea. I agree with the other reviewers that a major revision might be necessary because the new title and the new framing effectively makes it a different paper. I think it might be helpful if the authors could post their reworked **abstract** here as it could be a good discussion base for the next reviewer-AC discussion stage. 

I greatly appreciate the new experiments and the authors' willingness to update their submission in such a major way. I am raising my score to 4, but would like to stress that I am very borderline on this. I am looking forward to the discussion with the other reviewers and AC on this submission. I felt in agreement with most of the other reviewers' comments and hope we can reach a collective decision together. I currently vote for a "weak reject" because I think that the paper lacks an analysis for why certain backgrounds are easy and other backgrounds are hard. I believe that the naming convention of "common" and "uncommon" aimed to (subconsciously) fill this gap by providing an untested hypothesis for the observed spurious correlations. I believe that merely presenting a dataset where CLIP trained models underperform is a bit weak and could be linked to the training data, as suggested by reviewer **2MXa**.
Dear Reviewer iLSC,

Thank you for acknowledging our responses and updating the rating lean to an acceptance. 

We feel necessary to clarify that **we do not use the test accuracy as the supporting experiments in our original manuscript**.
- **We use effective robustness[1] in the main figures to support our claims about the robustness of CLIP models on `CounterAnimal`**. The test accuracy results are adopted to provide more details. 
- In response to Reviewer BaJ3, we also provide **more figures of effective robustness** (as attached in [the general response](https://openreview.net/forum?id=wWyumwEYV8&noteId=yKMULf9RNp)) to support our claims that scaling up the model parameters and increasing the quality of data help with improving robustness on `CounterAnimal`.

Since our focus is to study the robustness of CLIP models under spurious features instead of comparing whether ImageNet models are more robust than CLIP models, the results and analysis in our original paper sufficiently help with addressing our main research question `Is there a benchmark that reflects the exact reliance on spurious features of CLIP?` in line 41, including  
- the curation procedure of `CounterAnimal`;
- the experimental results of effective robustness of CLIP models on `CounterAnimal`;
- the influence of factors such as parameters of models, and the quality of data to the effective robustness of CLIP models on `CounterAnimal`;
- the theoretical analysis of why CLIP models still learn spurious features;

**Therefore, it does not need significant revisions to accommodate with our new title, as all the original contents sufficiently address our main research question**. The revisions we need to make (most of which has already been done) are to:
- Adjust the abstract and introduction following [2], to avoid potential misunderstanding and precisely present the motivation of our work: We do not focus on the comparison of CLIP and ImageNet models but on curating a test set specifically for CLIP models to study the robustness of CLIP models against spurious features;
- Supplement the ablation studies with `CounterAnimal-i` to strengthen our motivation;

Please kindly let us know if our aforementioned revisions could address your concerns especially about the focus and the revisions of our work. We again thank you for your time and constructive comments!

**References**

[1] Measuring Robustness to Natural Distribution Shifts in Image Classification, NeurIPS'20.

[2] Natural adversarial examples, ECCV'21.
Thanks a lot to the authors for the detailed response and new experiments.  I appreciate the changes to the title and the additional experiments, which have improved this work a lot in the rebuttal. I also appreciate the contribution of creating a real-world dataset about backgrounds.  Considering all the comments and discussions,  I am raising my rating to 5.

I share Reviewer 2MXa's concern that the paper needs significant revisions due to the shift in title and focus. I want to emphasize the modifications in the experimental results under the new shift. The original paper takes test accuracy as main experimental results to support the old title. However, test accuracy is not solid enough to be the main results for the new title as they are heavily influenced by the model used for dataset splitting. More analysis and results on spurious features are needed.

Thanks again to the authors for their efforts during the rebuttal and discussion.
Dear Reviewer 2MXa,

Thank you for engaging in the discussion and acknowledging our motivation. We need to clarify that **as our focus is to study the robustness of CLIP models, the additional experiments do not necessarily introduce significant modifications**. 

Meanwhile, to avoid any potential misunderstandings, we have changed the title to `A Sober Look at the Robustness of CLIPs to Spurious Features` to **more precisely reflect the research problem that our original abstract and introduction are addressing**. 

To accommodate the concern of adopting CLIP models to curate the splits, we have revised our abstract and the introduction to more align with the corresponding work for ImageNet [1], which curates an OOD test set `ImageNet-a` according to the accuracy of ImageNet models. **We have revised our manuscript to clarify that it is a common practice in the literature to curate the OOD test set to evaluate the robustness of neural networks. The revisions are also acknowledged and suggested by Reviewer KCA4.**

Please kindly let us know if you still feel any additional revisions are needed. We would sincerely appreciate it if you could jointly take our clarifications above when making the final evaluation of this work.
Dear Authors,

Thank you for providing the response. I share the concern about using a CLIP model to curate the data. The newly added CounterAnimal-i dataset is helpful, but it could introduce differences in some of the observations made in the submission. For example, it raises questions about whether the title, “Do CLIP Models Always Generalize Better than ImageNet Models?” can still be fully addressed.

The authors now emphasize that “the focus of this work is to construct a test set for studying the robustness of CLIP models against real-world spurious features.” While this is a valid focus, it necessitates significant revisions to reflect this shift. For instance, the abstract and introduction currently do not align with this focus.

Given these points, I am actually in the boderline case. I would like to point out that the motivation behind the work is always good—whether it is to answer the title’s question or to study the real-world spurious features of CLIP models.

However, I would like to ***draw the Area Chairs’ attention*** to the fact that this submission needs significant modification, and some observations should be carefully adjusted based on the newly curated dataset. The presentation also needs to be revised to align with the goal of studying real-world spurious features of CLIP models.

Best regards,

Reviewer 2MXa
Dear Reviewer iLSC,

Following your suggestion, we curate a new test set `CounterAnimal-i` according to the accuracy of ImageNet models. The detailed results and discussions about `CounterAnimal-i`  are present in [the latest general response](https://openreview.net/forum?id=wWyumwEYV8&noteId=4IR8MTyJIm), as the experiments are also suggested by Reviewer KCA4.

The results of the experiments on `CounterAnimal-i` show that **curating the OOD test data according to different models will reveal different spurious features** and one needs to be cautious when selecting the proper OOD test data to evaluate the robustness against spurious features. Since most of the previous OOD test sets are designed for ImageNet models, it again highlights the necessity and significance of a test benchmark like CounterAnimal specifically for CLIP models.

Please kindly let us know whether the aforementioned responses address your concerns. We would sincerely appreciate it if you could jointly consider our responses above when making the final evaluation of our work. Thank you again for your time and insightful suggestions about our work!
Dear Reviewer KCA4,

We would like to thank you again for your time and constructive comments about our work. Following your suggestion, we curate a new test set `CounterAnimal-i` according to the accuracy of ImageNet models. The detailed results and discussions about `CounterAnimal-i`  are present in [the latest general response](https://openreview.net/forum?id=wWyumwEYV8&noteId=4IR8MTyJIm), as the experiments are also suggested by Reviewer iLSC. 

The results of the experiments on `CounterAnimal-i` show that **curating the OOD test data according to different models will reveal different spurious features** and one needs to be cautious when selecting the proper OOD test data to evaluate the robustness against spurious features. Since most of the previous OOD test sets are designed for ImageNet models, it again highlights the necessity and significance of a test benchmark like CounterAnimal specifically for CLIP models.

Please kindly let us know if our additional experiments could address your concerns. Thank you again for your valuable suggestions about our work!
Following the last response, we present the detailed results of CLIP and ImageNet models on `CounterAnimal-i` in this comment due to the character limits.

|   **CLIP**   |           | common | counter |  drop | **ImageNet** | common | counter |  drop |
|:------------:|-----------|:------:|:-------:|:-----:|:------------:|:------:|:-------:|:-----:|
|     RN50     |   OPENAI  |  60.9  |  42.56  | 18.34 |    AlexNet   |  59.27 |  31.87  |  27.4 |
|     RN101    |   OPENAI  |  61.22 |  40.25  | 20.97 |     VGG11    |  73.82 |  46.66  | 27.15 |
|    RN50-4    |   OPENAI  |  64.4  |  47.85  | 16.55 |     VGG13    |  74.32 |  48.08  | 26.24 |
|    RN50-16   |   OPENAI  |   72   |  57.65  | 14.35 |     VGG19    |  77.07 |  52.77  |  24.3 |
|    RN50-64   |   OPENAI  |  81.41 |  68.36  | 13.05 |   ResNet18   |  73.08 |  49.19  | 23.89 |
|   ViT/B/16   | LAION400M |  73.71 |  53.22  | 20.49 |   ResNet34   |  77.52 |  52.74  | 24.78 |
|   ViT/B/16   |   OPENAI  |  73.46 |  56.36  |  17.1 |   ResNet50   |  80.71 |  52.97  | 27.74 |
|   ViT/B/16   |  DATACOMP |  79.33 |   63.1  | 16.23 |   ResNet101  |  82.35 |  59.46  |  22.9 |
|   ViT/B/16   |  LAION2B  |  68.66 |  52.13  | 16.53 |   ViT/B/16   |  85.06 |  66.64  | 18.42 |
|   ViT/B/16   |   DFN2B   |  83.39 |  68.75  | 14.64 |   ViT/B/32   |  78.27 |  55.42  | 22.85 |
|   ViT/B/32   | LAION400M |  57.32 |  37.61  | 19.71 |   ViT/L/16   |  83.65 |  64.03  | 19.63 |
|   ViT/B/32   |   OPENAI  |  66.95 |  47.12  | 19.84 |   ViT/L/32   |  80.28 |  59.17  | 21.11 |
|   ViT/B/32   |  DATACOMP |  73.59 |  53.99  |  19.6 |  CONVNEXT-S  |  88.87 |  73.86  | 15.01 |
|   ViT/B/32   |  LAION2B  |  67.37 |  47.64  | 19.73 |  CONVNEXT-B  |  88.92 |  74.48  | 14.44 |
| ViT/B/32-256 |  DATACOMP |  78.18 |   60.8  | 17.39 |  CONVNEXT-L  |  89.88 |  77.21  | 12.67 |
|   ViT/L/14   | LAION400M |  77.96 |  60.85  | 17.11 |              |        |         |       |
|   ViT/L/14   |   OPENAI  |  81.67 |  67.55  | 14.12 |              |        |         |       |
|   ViT/L/14   |  DATACOMP |  88.87 |  77.06  | 11.82 |              |        |         |       |
|   ViT/L/14   |  LAION2B  |  78.89 |  63.14  | 15.75 |              |        |         |       |
|   ViT/L/14   |   DFN2B   |  88.72 |  77.51  | 11.21 |              |        |         |       |
| ViT/L/14-336 |   OPENAI  |  84.09 |  71.62  | 12.47 |              |        |         |       |
|   ViT/H/14   |  LAION2B  |  83.77 |  71.04  | 12.72 |              |        |         |       |
|   ViT/H/14   |   DFN5B   |  89.32 |  79.65  |  9.68 |              |        |         |       |
| ViT/H/14-384 |   DFN5B   |  92.55 |  83.19  |  9.36 |              |        |         |       |
|   ViT/G/14   |  LAION2B  |  84.46 |  68.16  | 16.31 |              |        |         |       |
|  ViT-bigG/14 |  LAION2b  |  86.39 |  74.03  | 12.36 |              |        |         |       |
|  CONVNEXT-B  | LAION400M |  52.06 |  38.85  | 14.22 |              |        |         |       |
|  CONVNEXT-BW |  LAION2B  |  57.19 |  38.74  | 18.45 |              |        |         |       |
Dear Area Chair BKNm and Reviewers,

We would like to thank you again for your valuable time and constructive comments about our work. We follow the suggestions of Reviewer KCA4 and iLSC to further curate a new `CounterAnimal-i` dataset, for which we follow the same curation procedure while using ImageNet models (i.e., `ResNet50-ImageNet`) to construct Common and Counter splits.

Here, again we would like to clarify that the objective of this work is not for comparing which model is universally more robust than the other. Rather, **the focus of this work is to construct a test set for studying the robustness of CLIP models against real-world spurious features**:
- The construction of CounterAnimal follows the common practice in the literature[1]. 
- Prior to CounterAnimal, most of the test sets are constructed for ImageNet models. Comparing CLIP models with ImageNet models on OOD test sets specifically designed for ImageNet models is insufficient to reflect the robustness of CLIPs against real-world spurious features.

Now we present the results of CLIP and ImageNet models on `CounterAnimal-i`. We first present the summary of key observations below and then give the detailed performances in the next comment due to the character limit.

- CLIPs demonstrate better robustness against the spurious features discovered with ImageNet models. **It aligns with our expectation since different training data (e.g., LAION for CLIPs, and ImageNet for ImageNet models) follow different distributions and naturally contain different spurious features**. It also demonstrates the generality of our data curation method to reveal the spurious features of different models.
- More concretely, we list the common and counter split names for some of the classes, where we bold the background names differently across `CounterAnimal` and` CounterAnimal-i`. From the results, we can find that, 
    - Using different models to split data captures the different spurious features. 
    - Both CLIP and ImageNet models are good at generalizing under certain spurious features while performing worse than the other under some other spurious features. **It again highlights the necessity to curate an OOD test set for CLIP models, as CLIP models learn different spurious features than ImageNet models**.


|               | CounterAnimal |           | CounterAnimal-i |           |
|---------------|---------------|-----------|-----------------|-----------|
|               | common        | counter   | common          | counter   |
| Ostrich       | ground        | **water** | ground          | **rock**  |
| Brambling     | grass         | **sky**   | grass           | **water** |
| Bulbul        | sky           | **tree**  | sky             | **grass** |
| Vluture       | sky           | tree      | sky             | tree      |
| Box turtle    | grass         | **earth** | grass           | **water** |
| Ccommon iguana | earth         | shrub     | earth           | shrub     |
| Whiptail      | **earth**     | **human** | **water**       | **shurb** |
| Agama         | rock          | **tree**  | rock            | **grass** |
| Crocodile     | earth         | **grass** | earth           | **tree**  |



- Furthermore, we give the averaged performance drops of CLIP and ImageNet models under `CounterAnimal` and `CounterAnimal-i`, respectively. From the table below, two interesting observations can be found that:
    - CLIP models show similar averaged performance drops across `CounterAnimal` and `CounterAnimal-i`, which demonstrates the robustness of our curation method in revealing the spurious features of CLIP models;
    - When considering the averaged performance drops, it can be found that both models demonstrate similar performance drops, which again shows the necessity of using proper benchmarks to evaluate the robustness of different models.

|                 | CounterAnimal | CounterAnimal-i | AVG.  |
|-----------------|---------------|-----------------|-------|
| CLIP models     | 16.96         | 15.71           | 16.34 |
| ImageNet models | 13.95         | 21.9            | 17.93 |

**References**

[1] Natural adversarial examples, ECCV'21.
Dear Ethics Reviewer RdC5,

Thank you for your time and efforts conducting ethics reivew of our work. We strictly follow the Code of Conduct and Code of Ethics of NeurIPS and carefully avoid any potential ethics considerations.
Dear Ethics Reviewer sfcY,

Thank you for your time and suggestions about the potential broader impacts of this work. We have revised our manuscript to supplement the discussions about the General Ethical Conduct.

> Q1.1 Individuals have been notified..

We would like to clarify that individuals involved in the manual data curation are mainly from our team, and they all have been notified and agreed to grant consent to the use of the data in this work.


> Q1.2 Consent from iNaturalist data

We ensure all the data curated from the iNaturalist is allowed to share for scientific use under the CC BY-NC license. We strictly follow the licence permits to use the data in this work.

> Q2 Potential Negative Social Impact

We have revised our manuscript to clarify that the objective of this work is not to compare the performances of the CLIP and ImageNet models. Rather, we strive to construct a test set for CLIP models to evaluate the robustness against the spurious correlations. 

The construction pipeline follows the common practice in the literature to construct similar test sets for ImageNet models[1].

To avoid any potential misunderstanding, we have also revised our title to `A Sober Look at the Robustness of CLIPs to Spurious Features` to more precisely reflect the contents of this work. The revisions have been kindly acknowledged by Reviewer KCA4 after reading our rebuttal.

Please kindly let us know if our clarifications above resolve the potential ethics concerns of this work. Thank you!

[1] Natural adversarial examples, ECCV'21.
Dear Reviewers,

This is a friendly reminder that the authors have submitted a rebuttal and the end of the discussion period is August 13th AoE. Due to the tight deadline please take a look at it as soon as possible and check if your questions/comments have been properly addressed. At the very least, please acknowledge that you have read the authors' response to your review.

Thank you for your time and effort!

AC
Dear Reviewer iLSC,

Following your suggestion of the last question, we conduct additional ablation study that uses ImageNet-a[f] to create common and counter splits based on the `CLIP-ViT/B/32-LAION400M`. Meanwhile, as the original ImageNet-a is curated based on the `ResNet-50` trained on ImageNet, therefore the accuracy of `ResNet-50-ImageNet` is 0.

In the table below, we present the accuracies of CLIP models and ImageNet models across the common and counter groups. It can be found that
- CLIP models demonstrate a larger performance drop from common to counter groups, which is as expected;
- However, **none of the ImageNet models have a better generalization performances than CLIP models, despite a smaller performance drop**. In contrast, in CounterAnimal, ImageNet models can achieve compeitive performances in the Common groups while suffer less performance drops, or better effective robustness[a].
- The results demonstrate the new splits are more challenging to generalize for both CLIP and ImageNet models. It is hard to dissect the influence of spurious features from the other factors such as label noises or model misspecification[g].


|     **CLIP**     |           | common | counter |  drop |  **ImageNet**  | common | counter | drop   |
|:------------:|-----------|:------:|:-------:|:-----:|:----------:|:------:|:-------:|--------|
|     RN50     |   OPENAI  |  0.502 |  0.144  | 0.358 |   AlexNet  |  0.04  |  0.012  |  0.028 |
|     RN101    |   OPENAI  |  0.606 |  0.199  | 0.407 |    VGG11   |  0.032 |  0.011  |  0.021 |
|    RN50-4    |   OPENAI  |  0.672 |  0.299  | 0.373 |    VGG13   |  0.048 |  0.012  |  0.036 |
|    RN50-16   |   OPENAI  |  0.748 |  0.449  | 0.299 |    VGG19   |  0.046 |  0.015  |  0.031 |
|    RN50-64   |   OPENAI  |  0.807 |  0.584  | 0.223 |  ResNet18  |  0.022 |  0.008  |  0.014 |
|   ViT/B/16   | LAION400M |  0.749 |  0.216  | 0.533 |  ResNet34  |  0.039 |  0.013  |  0.026 |
|   ViT/B/16   |   OPENAI  |  0.778 |  0.384  | 0.394 |  ResNet50  |    0   |    0    |    0   |
|   ViT/B/16   |  DATACOMP |  0.812 |   0.38  | 0.432 |  ResNet101 |  0.09  |  0.037  |  0.053 |
|   ViT/B/16   |  LAION2B  |  0.734 |  0.256  | 0.478 |  ViT/B/16  | 0.3767 |  0.1699 | 0.2068 |
|   ViT/B/16   |   DFN2B   |  0.832 |  0.386  | 0.446 |  ViT/B/32  | 0.1937 |   0.07  | 0.1237 |
|   ViT/B/32   | LAION400M |    1   |    0    |   1   |  ViT/L/16  |  0.286 |  0.138  |  0.148 |
|   ViT/B/32   |   OPENAI  |  0.665 |  0.205  |  0.46 |  ViT/L/32  |  0.232 |  0.093  |  0.139 |
|   ViT/B/32   |  DATACOMP |  0.689 |  0.192  | 0.497 | CONVNEXT-S |  0.473 |  0.264  |  0.209 |
|   ViT/B/32   |  LAION2B  |  0.704 |   0.15  | 0.554 | CONVNEXT-B |  0.527 |  0.297  |  0.23  |
|   ViT/L/14   | LAION400M |  0.858 |  0.354  | 0.504 | CONVNEXT-L |  0.538 |  0.343  |  0.195 |
|   ViT/L/14   |   OPENAI  |  0.872 |  0.608  | 0.264 |            |        |         |        |
|   ViT/L/14   |  DATACOMP |  0.902 |   0.62  | 0.282 |            |        |         |        |
|   ViT/L/14   |  LAION2B  |  0.856 |  0.441  | 0.415 |            |        |         |        |
|   ViT/L/14   |   DFN2B   |  0.914 |  0.593  | 0.321 |            |        |         |        |
| ViT/L/14-336 |   OPENAI  |  0.893 |  0.691  | 0.202 |            |        |         |        |
|   ViT/H/14   |  LAION2B  |  0.869 |  0.477  | 0.392 |            |        |         |        |
|   ViT/H/14   |   DFN5B   |  0.926 |  0.626  |  0.3  |            |        |         |        |
| ViT/H/14-384 |   DFN5B   |  0.961 |  0.745  | 0.216 |            |        |         |        |
|   ViT/G/14   |  LAION2B  |  0.877 |  0.494  | 0.383 |            |        |         |        |
|  ViT-bigG/14 |  LAION2b  |  0.908 |  0.597  | 0.311 |            |        |         |        |
|  CONVNEXT-B  | LAION400M |  0.509 |   0.13  | 0.379 |            |        |         |        |
|  CONVNEXT-BW |  LAION2B  |  0.546 |  0.189  | 0.357 |            |        |         |        |
Dear Reviewer iLSC,

Thank you so much for engaging in the discussion! Now we provide more clarification on the remaining questions:

> A new Common' and Counter' dataset can be created based on the accuracy of ImageNet models

We apologize for not clearly responding to this question in the rebuttal. As also requested by Reviewer KCA4, we are currently collecting the new dataset based on the accuracy of Imagenet models and will update you once we have the preliminary results very soon!

Nevertheless, we would like to clarify that, since previous ImageNet variant test sets are curated based on the accuracy of ImageNet models, the focus of this work is to provide a corresponding one for the CLIP models.

> Examples of how prior work has focused on the spurious features in ImageNets.

We briefly introduce some examples here and refer Reviewer iLSC to [a] for a detailed discussion of the examples:

- ImageNetV2[b]: a reproduction of the ImageNet test set, where the authors introduce the distribution shifts by rerunning the Imagenet curation pipeline to collect a new test set;
- ObjectNet[c]: a test set of objects in a variety of scenes with 113 classes that overlap with ImageNet, where the authors introduce the distribution shifts by curating images with varying object poses, locations, etc.;
- ImageNet-Sketch[d]: the authors curate images from Google Image with queries "sketch of __", where __ is the standard class name;
- ImageNet-R[e]:  the authors curate images containing various renditions (e.g., paintings, embroidery, etc.) of ImageNet object classes;
- ImageNet-A[f]: the authors curate a hard version of the ImageNet test set with adversarial filtration based on the correctness of a ImageNet-trained model;

All the aforementioned works introduce distribution shifts against the original ImageNet test sets in a way that they find that ImageNet models perform badly on the newly curated test sets. Based on those ImageNet variant test sets, [a] conducted extensive experiments and found that the performance gains of ImageNet models on the original ImageNet test sets can hardly be generalized to the ImageNet variant test sets. Therefore, it is suspected that the ImageNet models learn some spurious features from the original ImageNet train set, that do not hold on the ImageNet variant test sets.

> The differences between this paper and prior studies 

We apologize for any confusion. The uniqueness of this work compared to those referred by Reviewer iLSC is that:
- [1,2] are mainly designed for ImageNet models. Especially, in ObjectNet[2], CLIP models have been shown to be more robust than ImageNet models;
- [3,4,5,6] mainly consider **synthetic distribution shifts instead of natural distribution shifts**. We acknowledge the value of the synthetic distribution shifts in debugging neural networks, and have already revised our manuscript to discuss them. Natural distribution shifts may better reflect the robustness against real-world spurious features [e]. 

>  How this paper differs from splitting prior test set into two datasets—Counter and Common—based on CLIP accuracy

Thank you for this insightful question! 
- First, merely splitting test data according to the accuracy without considering the group information (e.g., backgrounds), will also involve additional factors, since the low-accuracy group will contain samples that are also affected by label noise or misspecification together [f]. Consequently, **the accuracy difference between the two groups may not uniquely reflect the influence of spurious features**. Note that our goal is to study the influence of spurious features. Therefore, we first labeled the background and then measured the differences between background groups, in order to measure the influence of spurious features in the backgrounds;
- As for the previous datasets specially designed for Imagenet models such as [1,2] referred by Reviewer iLSC, splitting them according to CLIP accuracy can not get rid of influence by other factors such as label noises, and may not reflect the real-world spurious features **uniquely in the CLIP training data**, as ImageNet models and CLIP models will both perform badly on the "counter/hard" splits. 
- As for the synthetic datasets [3,4,5,6], they may not reflect the **real-world spurious features** in the CLIP training data.


**References**

[a] Measuring Robustness to Natural Distribution Shifts in Image Classification, NeurIPS'20.

[b] Do ImageNet Classifiers Generalize to ImageNet? ICML'19.

[c] ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models, NeurIPS'19.

[d] Learning Robust Global Representations by Penalizing Local Predictive Power, NeurIPS'19.

[e] The many faces of robustness: A critical analysis of out-of-distribution generalization, ICCV'21.

[f] Natural adversarial examples, ECCV'21.

[g] ZIN: When and How to Learn Invariance Without Environment Partition? NeurIPS'22.
Dear Reviewer KCA4,

Thank you for acknowledging our revisions to the title and the paper an improvement that warrants acceptance! 

In addition to the existing revisions, we promise that we will revise our manuscript more aligned with [1] following your suggestion. 

Meanwhile, we are also working on creating a similar common/counter split based on the ImageNet-trained models. We will share more details during the discussion period very soon once we have some preliminary results!


**References**

[1] Natural adversarial examples, ECCV'21.
Thanks to the authors for providing the rebuttals, which partially addressed my concerns.

* Regarding Q2,  I appreciate the adjustment to emphasize the spurious features found in CLIP setups. However, I did not find any discussion related to my original second point in Q2, which is quoted below:

  > The accuracy drop from common to counter group can be greatly influenced by the model used to divide the common and counter dataset. Using the combined proposed common and counter dataset, a new Common' and Counter' dataset can be created based on the accuracy of ImageNet models. What is the impact of this dataset division on the accuracy drop for different models?

* Regarding Q2, could you please clarify the following point? For example,  I would appreciate some examples of how prior work has focused on the spurious features in ImageNets.

  >  Prior to our work, there is no specialized benchmark curated for CLIPs to study the robustness of CLIPs to the real-world spurious correlations. Instead, previous works predominantly focus on the spurious features in ImageNets. 

* Regarding Q3, the difference between this paper and prior studies remains unclear, which also affects the clarity of the paper's contribution. As also noted by reviewers KCA4 and BaJ3, the dataset creation makes it unsurprising that ImageNet models show a smaller gap between the two subsets. Therefore, it is confusing to claim that:

  > The CounterAnimal remains the first dataset that can identify some spurious features predominant in CLIP setups **while these features might not be as strong for the ImageNet benchmarks**, which is the uniqueness of our paper.

* Regarding Q3.  Prior studies have pointed out the spurious bias of the CLIP model about backgrounds, Therefore, a direct way to find CLIP-specific spurious bias is by splitting prior test set into two datasets—Counter and Common—based on CLIP accuracy. Could you further clarify how the proposed dataset differs from splitting prior test sets?
Thank you for your response and the new title—it’s definitely an improvement. I believe this paper warrants acceptance, but additional revisions are necessary. The paper would benefit from aligning more closely with the direction of Hendrycks et al. (2019) in terms of the motivation. Specifically, the ImageNet/ResNet comparison needs to be reframed, as its current presentation is potentially misleading. 

One potential experiment that could add value is identifying a common/counter split of the same ~13K dataset for ImageNet-trained models. Since you have the classes and backgrounds marked, it could be interesting to compare the differences between CLIP-based splits and ResNet-based splits. This comparison might reveals insights into the nature of spurious correlations in these two models. Incorporating these experiments and suggested changes could significantly enhance the manuscript. Hoping that the authors could make these changes, I am increasing my score.

Hendrycks et al. (2019) Natural Adversarial Examples
Dear Reviewers,

Thank you for your time and effort in reviewing the manuscript. Please kindly review and acknowledge the authors' rebuttal, and feel free to request any additional clarification if needed.

Your AC