In Search of Forgotten Domain Generalization

## Abstract

Abstract Out-of-Domain (OOD) generalization is the ability of a model trained on one or more domains to generalize to unseen domains. In the ImageNet era of computer vision, evaluation sets for measuring a model’s OOD performance were designed to be strictly OOD with respect to style. However, the emergence of foundation models and expansive web-scale datasets has obfuscated this evaluation process, as datasets cover a broad range of domains and risk test domain contamination. In search of the forgotten domain generalization, we create large-scale datasets subsampled from LAION—LAION-Natural and LAION-Rendition—that are strictly OOD to corresponding ImageNet and DomainNet test sets in terms of style. Training CLIP models on these datasets reveals that a significant portion of their performance is explained by in-domain examples. This indicates that the OOD generalization challenges from the ImageNet era still prevail and that training on web-scale data merely creates the illusion of OOD generalization. Furthermore, through a systematic exploration of combining natural and rendition datasets in varying proportions, we identify optimal mixing ratios for model generalization across these domains. Our datasets and results re-enable meaningful assessment of OOD robustness at scale—a crucial prerequisite for improving model robustness.

## 1 Introduction

Foundation models have revolutionized our world, demonstrating remarkable capabilities in solving grade school math problems, writing creative essays, generating stunning images, and comprehending visual content openai2023gpt4; ChatGPT; ramesh2022hierarchical. One notable example is CLIP (radford2021learning), a vision-language model pretrained on a vast dataset of image-text pairs, which forms the backbone of numerous other foundation models (ramesh2022hierarchical; liu2023visualinstructiontuning). CLIP has achieved unprecedented performance across a wide range of benchmarks spanning many domains—a sharp contrast to models from the ImageNet era, which struggled to generalize from a training domain mostly consisting of natural photographs to stylistically different domains such as ImageNet-Sketch (wang2019learning), ImageNet-R (hendrycks2020many), and DomainNet (peng2019moment).

Domains, while not always rigorously defined, emerge from collecting data from specific sources and conditions. Some domains, like *natural images* or *renditions*, are better delineated, allowing the creation of datasets like the ones mentioned above. Out-of-domain (OOD) generalization refers to a model’s ability to perform well on data from domains other than its training domain(s) (wang2022generalizingunseendomainssurvey). In this work, we collectively refer to the domain represented by ImageNet-Sketch, ImageNet-R, DomainNet-Painting, DomainNet-Clipart, DomainNet-Sketch, and DomainNet-Quickdraw as the rendition domain, since it contains images that are renditions of natural objects and scenes. Generalization to the rendition domain (especially OOD) is crucial for aligning models with human perception, as humans can interpret abstract visual renditions, while machines tend to rely heavily on textural cues (hendrycks2020many; geirhos2018imagenet).

CLIP’s strong performance in several domains, including renditions, is attributed to its vast training distribution, rather than its contrastive learning objective, language supervision, or dataset size (fang2022data). However, fang2022data do not specify what characteristics of the training distribution drive this performance.
CLIP could be learning more robust representations due to the diversity of natural images in its training set—or it may simply have been exposed to many datapoints from the (assumed to be OOD) test domains during training.
Indeed, mayilvahanan2024 revealed that CLIP’s training data contains exact or near duplicates of samples of many OOD datasets. Yet, they showed that CLIP still generalizes well when this *sample contamination* is corrected.
However, their analysis failed to account for *domain contamination*.

In contrast to sample contamination, domain contamination does not care about duplicates of specific datapoints but instead checks whether crucial aspects of a test domain are included in the training domain, e.g., by including images with different content but similar style to test samples. For example, after the correction by mayilvahanan2024, although not duplicates, many other rendition images remained in CLIP’s training set (refer to Tab. LABEL:tab:domain_composition).s from the test domain. This leads us to our central question:

*To what extent does domain contamination explain CLIP’s performance on renditions?*

We address the central question with the following contributions:

- •
Constructing Clean Single-Domain Datasets: To rigorously test whether CLIP’s success in the rendition domain stems from their exposure during training, we first train a domain classifier to distinguish natural images from renditions (Sec. 3.2). By applying the domain classifier to a deduplicated version of LAION-400M, we create and release two datasets: LAION-Natural contains $57\text{\,}\mathrm{M}$ natural images; LAION-Rendition consists of $16\text{\,}\mathrm{M}$ renditions of scenes and objects. Additionally, we refine existing rendition OOD benchmarks (ImageNet-R, ImageNet-Sketch, etc.) by removing samples that do not belong to the corresponding domain (LABEL:sec:subsampling_datasets).
- •
Refining the Evaluation of CLIP’s OOD Performance: Using LAION-Natural, we demonstrate that CLIP trained only on natural images significantly underperforms on rendition domain shifts (LABEL:sec:laion_nat_v_random). This suggests that its original success stems from domain contamination, not from an intrinsic OOD generalization ability (see Fig. 1 for a summary).
- •
Investigating Domain Mixing and Scaling Effects:
Our single-domain datasets enable analyzing the effects of training on controlled mixtures of natural and rendition images across scales (LABEL:sec:laion_mix).
We identify the optimal mixing ratio for the best overall performance and show the degree to which training on one domain enables some generalization to the other.

Through this work, we aim to shed light on the limitations of foundation models like CLIP in handling OOD generalization and provide valuable datasets and tools to the community for further exploration. Fig. 1 illustrates our core methodology.

## 2 Related Work

#### Measuring the OOD Generalization of CLIP Models

We aim to understand the OOD generalization capabilities of CLIP from a data-centric viewpoint.
While multi-modal training with rich language captions does seem to contribute to robustness against distribution shifts (xue2024understanding), fang2022data demonstrated that the nature of CLIP’s training distribution (as opposed to its mere size, its specific training objective, or natural language supervision) causes strong performance on various distribution shifts.
However, it is unclear what aspects of the data distribution drive the robustness gains.
mayilvahanan2024 remove images highly similar to the test sets to show that data contamination and high perceptual similarity between training and test data do not explain generalization performance. While their data pruning technique removes some samples from LAION-400M that lie outside the natural image domain, they do not address domain generalization: They only account for the part of a domain covered by existing test sets and give no guarantee that all images of a given domain were removed.
In another line of work, nguyen2022quality discover that a model’s effective robustness (fang2022data; taori2020measuring) on a test set interpolates when training data is compiled from various sources.
However, they only consider mixing datasets that each cover multiple domains.
In this work, we take their analysis further and show how mixing two data sources from distinct domains interpolates the effective robustness on those domains.
Our study’s title is inspired by gulrajani2020search, who studied generalization from multiple distinct source domains. In contrast, we focus on generalization from single or mixed source domains to unseen domains.

#### Domain Classification

The primary goal of our work necessitates creating web-scale datasets of different domains. This entails building a robust domain classifier that can reliably distinguish natural images from renditions. This task can be regarded as classifying the style of an image, which Gatys2015ANA proposed to measure using Gram matrices and which has been widely explored since then (sandoval2019; menis-mastromichalakis2020; sandoval2018; joshi2020; garcia2018; chu2018; bai2021).
More recently, cohen2024ask use a fine-tuned CLIP model from OpenCLIP (ilharco_gabriel_2021_5143773) to distinguish between ImageNet and test sets with a domain shift, such as ImageNet-Sketch, ImageNet-R, and ImageNet-V2 (recht2019imagenet).
wang2023evaluating and somepalli2024measuring develop a dataset classifier using a backbone trained by self-supervised learning and classification through retrieval via a database.
liu2024decade report high performance when training image classifiers to distinguish between different large-scale and diverse datasets.

## 3 Constructing Clean Single-Domain Datasets

To answer our central question—how much of CLIP’s performance on renditions can be explained by domain contamination—we must filter out datapoints from specific domains within web-scale datasets. Similar to how ImageNet is compared to ImageNet-Sketch and ImageNet-R, or how DomainNet-Real is compared to DomainNet-Sketch (Quickdraw, Infograph, Clipart, and Painting), we aim to create clean *natural* and *rendition* datasets from LAION by building a domain classifier to distinguish between these domains.

To build a robust domain classifier, we first create a labeled dataset where each class represents a distinct domain.
The labeling process is outlined in Sec. 3.1, and we explore different ways to build a domain classifier in Sec. 3.2. Further, in LABEL:sec:testing_domain_classifier, we employ the best-performing classifiers to analyze the composition of different training and test sets and finally use it to subsample LAION-Natural and LAION-Rendition in LABEL:sec:subsampling_datasets.

#### LAION-200M

For the remainder of this work, we substitute LAION-400M with LAION-200M, which we obtain by de-duplicating LAION-400M based on perceptual similarity as introduced by abbas2023semdedup. Both abbas2023semdedup and mayilvahanan2024 demonstrate that CLIP trained on LAION-200M obtains comparable downstream performance while greatly reducing the computational burden of training models from scratch and analyzing the dataset.

### 3.1 Labeling

LAION-200M contains diverse images from a multitude of sources. The images vary from naturally occurring to synthetically generated. We encourage the reader to glance at LABEL:fig:laion_random to get a sense of the dataset and the difficulty of determining the domain of each image.
We aim to classify these images mainly as natural or renditions. We also add an extra ambiguous class for images with elements of both domains, images with elements of neither, and edge cases.

We manually label images based on a labeling handbook derived from analyzing the existing OOD test sets, which we outline in LABEL:sec:appx_labeling. In general, we adopt a texture-centric approach to distinguish renditions of a scene or object from their natural depictions. That is, depictions where fine-grained texture information is preserved are generally considered natural, while depictions with simplified or flat textures are considered renditions. Fig. 2 illustrates this demarcation on samples from LAION-200M, ImageNet test sets, and DomainNet test sets.

Figure: Figure 2: Labeled natural, ambiguous, and rendition samples from different datasets. Natural images are photos or high-quality renders with minor filters that preserve fine-grained textures, while renditions are typically sketches, paintings, or graphics with flat or simplified textures. Images with elements of both, such as collages or natural images with large stylized elements, and images mainly containing text are labeled as ambiguous.
Refer to caption: x2.png

To further ease the labeling procedure, we first build a rough binary classifier by fine-tuning CLIP ViT-L/14 with a linear readout to differentiate between some of the natural ImageNet and DomainNet test sets (namely, ImageNet-Val, ObjectNet (barbu2019objectnet), ImageNet-V2, ImageNet-A (hendrycks2021natural), and DomainNet-Real) and rendition test sets (namely, ImageNet-Sketch, ImageNet-R, DomainNet-Painting, DomainNet-Sketch, and DomainNet-Clipart). We use this classifier to roughly pre-label samples before they are annotated by a human. The annotator verifies and potentially updates the labels for $25$ images at a time (see LABEL:fig:labeling_setup).

Overall, we label $19\,000$ random images from LAION-200M and $1000$ images from each of the ImageNet and DomainNet test sets ($12\,000$ in total).
Notably, almost all ImageNet and DomainNet test sets usually assumed to contain only images of a single domain exhibit some domain contamination.
We discuss this in detail in LABEL:sec:testing_domain_classifier.
LABEL:tab:labeling_nos contains a detailed breakdown of labels for each dataset.
We show more samples grouped by domain for each dataset in LABEL:fig:imagenet_a and LABEL:fig:dn_sketch_1k_visualization.

### 3.2 Training and Choosing the Domain Classifier

With the domain-labeled dataset, we can train a domain classifier to partition all of LAION-200M into natural images, renditions, or ambiguous images.
Since we aim to obtain datasets containing only images from a single domain, we need a domain classifier that is as precise as possible.
To this end, we train classifiers on $13\,000$ labeled LAION-200M images, retaining $3000$ samples each for a validation and test set.
From the domain classification literature discussed in Sec. 2, we evaluate four methods with publicly available code. All methods build on CLIP ViT-L/14 pretrained on LAION-2B, which we choose for its balance between accuracy and inference speed. For brevity, we present the two methods we finally employ here, and refer the reader to  LABEL:sec:appx_dc_full for a detailed description, results, and comparisons with other approaches.

#### Density Ratios

cohenwang2024ask aim to estimate the probability that a given sample is drawn from a reference distribution $p_{\text{ref}}$. Since high dimensional density estimation is challenging, they build a classifier to distinguish between a reference and a shifted distribution and compute the density ratio $\frac{p_{\text{ref}}}{p_{\text{shifted}}}$ which they threshold at $0.2$ to classify a given sample. We deploy their method unchanged to our task.
We obtain two binary classifiers, DR-N and DR-R, that distinguish natural from non-natural samples and renditions from non-renditions, respectively.

#### Fine-Tuning

We fine-tune pretrained CLIP ViT-L/14’s image encoder with a randomly initialized linear readout on the training dataset to obtain a ternary classifier, dubbed FT.

We use the validation set to determine the two best domain classifiers, one for natural images and one for renditions. Since the domain classifier should maximize precision above all else, we set the confidence threshold for each model such that it achieves $98\text{\,}\%$ per-class precision.
We then pick the classifier with the highest per-class recall to minimize the number of datapoints that are discarded when subsampling LAION-200M to build LAION-Natural and LAION-Rendition.
We choose FT, the fine-tuned ternary classifier, and DR-R, the binary classifier using density ratios, to detect natural and rendition images, respectively. We use these classifiers for all subsequent experiments.
Sec. 3.2 reports these models’ precision and recall on the natural and rendition class across ImageNet and DomainNet test sets. For comparison to other methods see  LABEL:sec:appx_dc_full.
For raw accuracy numbers of all models, which in general are high for most, refer to LABEL:tab:appx_dc_natural and LABEL:tab:appx_dc_stylistic in LABEL:sec:appx_raw_dc_performance.
We also assess the quality of the labels and the domain classifier’s predictions in LABEL:sec:label_quality, finding them to be robust even in the presence of label noise during training.

Table: Table 1: We chose the best natural classfier and the best rendition classifier between a binary classifier based on (DR) (cohenwang2024ask) and a ternary classifier using a linear readout based on fine-tuned CLIP model (FT). All models use CLIP ViT-L/14 pretrained on LAION-2B. We report precision and recall for the natural class (top) and rendition class (bottom) on ImageNet (IN) and DomainNet (DN) test sets and average performance across all test sets. For each class, we select the classifier with the highest validation-recall.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. Is there a way of automating the dataset slip so as to train the model on a large scale dataset?

2. Would it make sense to generate "un-natural" images i.e. data that does not relate to real world categories to explore this further. For instance, maybe texture images generated from explicitly disjoint data distributions?
soundness: 4
strengths: Here’s a refined take on the strengths for your review:

1. **Extremely important problem**: This paper addresses a critical issue in machine learning—distinguishing true OOD generalization from data contamination—by revisiting fundamental challenges of OOD generalization in the context of large-scale vision-language models. Their work highlights the limitations of current evaluation practices, underscoring the importance of ensuring genuine robustness and generalization capabilities.

2. **Impactful finding**: The finding that models like CLIP may not truly generalize when faced with rigorously OOD data is significant, as it challenges current assumptions about the robustness of these models. This insight has substantial implications for future work, suggesting that existing performance metrics may overstate the true generalization abilities of these models.

3. **Useful datasets**: By creating the LAION-Natural and LAION-Rendition datasets—both strictly OOD to ImageNet and DomainNet—this work offers a valuable resource for researchers focused on OOD robustness. These carefully constructed subsets provide a way to assess and enhance model generalization on web-scale datasets without contamination.

4. **Useful methodology**: The methodology for creating disjoint, OOD subsets of LAION is a wel established approach in the field, and a solid blueprint for other research seeking to evaluate OOD performance rigorously. 

5. **Results/Figures/Tables are very high quality**: The results, figures, and tables presented in the paper are meticulously crafted, enhancing the clarity and interpretability of the findings. They effectively illustrate the study's key points and provide a solid foundation for future discussions on OOD generalization and robustness evaluation.
summary: This paper investigates if the recent purported gains in generalization reported for Large scale vision language models come from true generalization beyond the data distribution, or from data contamination. To this end, they present a methodology for constructing disjoint subsets of the LAION dataset, which serve as OOD counterparts to ImageNet and DomainNet. This ensures no data contamination, and allows investigation of true OOD generalization capabilities of these models. Their main finding is like CLIP like models to not generalize well when such disjoint sets are constructed carefully. This suggests that while these models appear to generalize, they are merely performing well because they have already seen data close to the test set.
weaknesses: 1. **Missing Literature:** The field of OOD (out of distribution) generalization is very rich, and there are several related papers that are not cited. It would be nice to have this work related to existing works in the field (See below)

### References

a. Liu, J., Shen, Z., He, Y., Zhang, X., Xu, R., Yu, H. and Cui, P., 2021. *Towards out-of-distribution generalization: A survey*. arXiv preprint arXiv:2108.13624.

b. Koh, P.W., Sagawa, S., Marklund, H., Xie, S.M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R.L., Gao, I. and Lee, T., 2021, July. *Wilds: A benchmark of in-the-wild distribution shifts*. In International conference on machine learning (pp. 5637-5664). PMLR.

c. Madan, S., Henry, T., Dozier, J., Ho, H., Bhandari, N., Sasaki, T., Durand, F., Pfister, H. and Boix, X., 2022. *When and how convolutional neural networks generalize to out-of-distribution category–viewpoint combinations*. Nature Machine Intelligence, 4(2), pp.146-153.

d. Gulrajani, I. and Lopez-Paz, D., 2020. *In search of lost domain generalization*. arXiv preprint arXiv:2007.01434.

e. Madan, S., You, L., Zhang, M., Pfister, H. and Kreiman, G., 2022. *What makes domain generalization hard?* arXiv preprint arXiv:2206.07802.

f. Arjovsky, M., Bottou, L., Gulrajani, I. and Lopez-Paz, D., 2019. *Invariant risk minimization*. arXiv preprint arXiv:1907.02893.

g. Arjovsky, M., 2020. *Out of distribution generalization in machine learning* (Doctoral dissertation, New York University).

2. **Smaller training dataset size:** One of the key claims in papers presenting vision language models is the idea of Scaling Laws---behavior emerges from large scale datasets which cannot be seen in smaller models. Here, CLIP was trained on 19,000 images which is too small and thus it remains unclear if such generalization errors will continue in extreme dataset sizes. If nothing else, it is worth discussing this point in the paper.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - Q1: Even after reading the manuscript a few times, the main goal of this work is unclear to me. Previous work on the domain generalization setting has shown that neural networks do struggle to generalize across domains [1], even with powerful regularization and data augmentation approaches to mitigate domain shift. Given that, why exactly would CLIP be expected to generalize well OOD with limited training data? Given that, it seems to me the result/motivation presented in Figure 1-D is exactly what is expected to be given the IID assumption. Can the author kindly elaborate on their motivation? 

- Q2: Authors mentioned in line 62 that "Domains, while not always rigorously defined, emerge from collecting data from specific sources
and conditions." However, there is a lot of effort from the machine learning to rigorously define what a domain is. For example, the authors can refer to the work by Ben David et al. [2]

- Q3: Details about the annotation task are missing, please add the following information to the manuscript: how many annotators were considered, inter-rater agreement, how annotators were chosen and compensated, description of the task set up.

- Q4: The authors mentioned in Line 257: "Since human annotators can make mistakes, and LAION-200M’s domain
composition is inherently unknown, we use our domain classifiers to understand it.". However, I am confused by this statement. The domain classifiers the authors refer to were trained on human-labelled examples, which means they can also inherit the mistakes made by annotators, this approach also seems to be potentially affected by human mistakes (as well as the classifier's mistakes).  

- **Comment that has not affected my score:** The choice of paper title seems odd to me. The title is quite similar to the reference "In search of lost domain generalization", which can be confusing on its own, but the both papers are also considerably different in the questions asked and methodology, making it even more odd that they have such similar names. Moreover, the title "In search of forgotten domain generalization" does not reflect well the content of this submission as it gives the impression that the contributions are broader than they actually are, since this work only focused on the CLIP model and a specific type of domain shift.     


[1] Gulrajani, Ishaan, and David Lopez-Paz. "In search of lost domain generalization." arXiv preprint arXiv:2007.01434 (2020).

[2] Ben-David, Shai, et al. "A theory of learning from different domains." Machine learning 79 (2010): 151-175.
soundness: 3
strengths: - S1: This work raises awareness of the importance of curating and performing in-depth analysis of large-scale datasets.

- S2: One of the main contributions of this work is to release two curated partitions of the LAION dataset with non-overlapping domains. This will promote and facilitate future research on single-source domain generalization for foundation models such as CLIP.
summary: This work focuses on studying the out-of-distribution generalization capabilities of the CLIP model. To do so, authors consider the scope of *rendition* images and propose to use a domain classifier to detect images from this particular domain from the LAION dataset used to train the CLIP model. By training CLIP with this non-contaminated version of the training data, authors showed that the performance of the model on rendition images decayed, indicating that the original model performance on these images could be explained by the presence of such examples in the training set.
weaknesses: - W1: The motivation for the main question this work aims at answering ("shed light on the limitations of foundation models like CLIP in handling OOD generalization") is not quite clear from the manuscript. In a few words, it seems this work is showing that when CLIP is trained and tested on the same distribution, it performs well, and performance is harmed once a particular domain is removed from the training set and the model is tested on it. What is the exact new insight from this result? Please clarify this in the manuscript.

   - W1.1: In a similar vein, previous work [1] has pointed out that CLIP has limited OOD generalization capabilities and also proposed fine-tuning approaches to mitigate such issues.

- W2: The approach used for data curation relies on collecting human-labeled to train a domain classifier, making it specific to the setting studied in the paper, i.e. rendition vs natural images. 

- W3: In Table 2, the authors report the number of natural/ambiguous/rendition images resulting from the filtering with the domain classifier. However, it is possible to see that in most of the cases a considerably high number of images (> 60% in some cases) is deemed as ambiguous. This seems to indicate that either such domain classifier is not a good model to distinguish natural vs rendition images, or that it could be the case that it does not make sense to consider both domains as two distinct sources of data.

- W4: Lack of overall clarity. Some statements such as "We conclusively demonstrate that domain contamination in the training set is what drives CLIP’s robustness", however, if rendition data is present in the training data, then rendition data is no longer OOD with respect to the training distribution. 


[1] Shu et al, CLIPood: Generalizing CLIP to Out-of-Distributions, ICML, 2023.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: I have a few more questions that would love to discuss with the author. However, these questions should not count towards the evaluation of the paper.
1. Do you think the improved generalization comes from merely more data points or a more sophisticated loss or learning paradigm (i.e. contrastive learning and aligning with language)
2. Despite this paper is mainly on CLIP and contrastive vision language model, the main-stream of foundation models are generative models that are instruction-tuned and aligned. Do you think this work and its conclusion will generalized to LLMs? Specifically, one interesting new thing that was not used in CLIP training is the instruction tuning that enables the model to generalize across tasks. In that sense, models will learn to generalize across tasks instead of data points. For example, MLLMs can be trained to generalize to the task of discriminating between any two domains given by the user. Given such ability, will the MLLMs have better OOD generalization?
soundness: 3
strengths: 1. This paper studies a valuable problem that has seldom been studied. Indeed, the OOD problem, which seems to be less of a problem in the era of LLM, might be an illusion due to the scaling of the massive data. This problem should be taken seriously, as there are still areas where collecting data is extremely difficult and scaling will likely fail.
2. Sound analysis that supports the claim.
summary: This paper reintroduces the OOD problem in the era of the foundation model. They propose the argument that the phenomenon that foundational models are no longer  susceptible to OOD samples is merely an illusion due to the *domain contamination*, which they define as "whether crucial aspects of a test domain are included in the training domain, e.g., by including images with different content but similar style to test samples"

They then introduce cleaned single-domain datasets to study this problem, discovering that CLIP trained only on natural images significantly underperforms on rendition domain shifts. They further investigate the domain mixing and scaling that affects OOD performance.
weaknesses: 1. Figure 6 A is a bit confusing, as the result of the best rendition-to-natural ratio 1:3 and 1:1 can not be read in this figure. I suggest adding ratio labels to the color scale or individual data points in the figure.
2. Can you provide some discussion on the "true effectiveness" of the domain classifier trained and evaluated on the curated domain datasets but is used in a different and much larger dataset, i.e. Laion-200M?
    * Will there be OOD shift between the curated domain datasets and Laion-200M?
    * If so, how to make sure the domain classifiers are good?
    * Maybe you can sample a few classified samples in Laion-200M and verify the result by humans.
    * Will adding more domain classifiers and performing an ensemble help?
    * I suggest that the authors include a dedicated subsection in their methodology or results discussing the domain classifier's performance on LAION-200M, including human verification of a sample of classifications and exploration of ensemble methods.

3. Some typos, e.g data points instead of datapoints on lines 76, 82, etc.
4. Overall, I believe this is a technically sound paper with rigorous analysis that supports its main argument.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: No
soundness: 3
strengths: 1. This paper gives a very detailed analysis that clarifies that the success of CLIP mainly comes from the training data rather than its own training paradigm, which can give some inspiration to many researchers.

2. The authors give the optimal mixing ratio of rendition and natural image, which allows for better generalization performance of models trained on this dataset.

3. The authors conduct extensive experiments and analyses to support the claim that "the success of the base model in OOD settings does not originate from its own characteristics", which enhances its credibility.
summary: The paper focuses on exploring the phenomenon of test domain contamination in the era of foundation models. The paper constructs two different style datasets, LAION-Natural and LAION-Rendition, which are strictly OOD relative to the corresponding ImageNet and DomainNet test sets in terms of style. It examines whether the good OOD performance of CLIP results from test set contamination or from its training paradigm. The findings indicate that CLIP's success stems from domain contamination rather than an intrinsic OOD generalization ability. Additionally, the authors suggest how the mixing ratio of Rendition and Natural samples in the training set can optimize model performance on the test set.
weaknesses: 1. The main issue with this paper is that it appears to draw a conclusion "he success of CLIP mainly comes from the training data rather than its own training paradigm" that lacks sufficient insight for subsequent researchers. While the question of the optimal ratio of natural and rendition images in the training set is indeed illuminating, the authors do not seem to have conducted tests with models of different architectures. Additionally, I believe that this experiment is important because it can ensure that the optimal ratio is not sensitive to the model's architecture.

2. Although I have experience in OOD research, I am not up to date with the latest techniques and conclusions in this area. Therefore, I will be observing the perspectives of other reviewers as well as the authors' responses. Overall, the paper was thorough and solid, but it seemed to lack novelty.for me.

Meta-review: The paper addresses a critical and timely issue of out-of-distribution (OOD) generalization in the era of foundation models.
It challenges the common assumption that models like CLIP inherently generalize OOD, providing evidence that their strong performance often stems from domain contamination in training datasets.

The paper introduces two rigorously curated datasets, LAION-Natural and LAION-Rendition, which enable meaningful evaluation of OOD generalization. These datasets will have substantial utility for future research in improving model robustness and generalization.

Overall, this paper is well-written, addresses a significant problem, provides high-quality datasets and experiments, and has received strong support from reviewers. Given its contributions to OOD evaluation and potential to inspire further research in robust model development, it is suggested that this paper be accepted.

Decision: Accept (Spotlight)