Does CLIP’s generalization performance mainly stem from high train-test similarity?
Authors: Prasanna Mayilvahanan
Thaddäus Wiedemer
Evgenia Rusak
Matthias Bethge
Wieland Brendel
Venue: ICLR 2024 poster (2024)
Keywords: robustness
foundation models
CLIP
LAION
ImageNet
generalization
OOD robustness
distribution shift
vision language models
self-supervised learning
contrastive learning
ObjectNet
ImageNet-R
ImageNet-Sketch
ImageNet-A
ImageNet-V2
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: CLIP's ability to generalize to standard OOD benchmarks does not mainly stem from highly similar images in its training dataset.

Abstract: Foundation models like CLIP are trained on hundreds of millions of samples and effortlessly generalize to new tasks and inputs. Out of the box, CLIP shows stellar zero-shot and few-shot capabilities on a wide range of out-of-distribution (OOD) benchmarks, which prior works attribute mainly to today's large and comprehensive training dataset (like LAION). However, it is questionable how meaningful terms like out-of-distribution generalization are for CLIP as it seems likely that web-scale datasets like LAION simply contain many samples that are similar to common OOD benchmarks originally designed for ImageNet. To test this hypothesis, we retrain CLIP on pruned LAION splits that replicate ImageNet’s train-test similarity with respect to common OOD benchmarks. While we observe a performance drop on some benchmarks, surprisingly, CLIP’s overall performance remains high. This shows that high train-test similarity is insufficient to explain CLIP’s OOD performance, and other properties of the training data must drive CLIP to learn more generalizable representations. Additionally, by pruning data points that are dissimilar to the OOD benchmarks, we uncover a 100M split of LAION (¼ of its original size) on which CLIP can be trained to match its original OOD performance.

Published as a conference paper at ICLR 2024
DOES CLIP'S GENERALIZATION PERFORMANCE MAINLY STEM FROM HIGH TRAIN-TEST SIMILARITY?
Prasanna Mayilvahanan 122,3-
Thaddäus Wiedemer 1,23+
Evgenia Rusak 1,2,3
Matthias Bethge'.2 1.2
Wieland Brendel
Uniiversity Tübingen Tübingen AI Center
"Max-Planck-Intinuu for Intelligent Systems, Tübingen
*ELLIS Institute Tübingen
prasanna Trasanna.mayili wwwillaaaaaannnnnnammmaa u ttueiiittteueeteee de, thaddaeus, wi eeeeeeeeerree gmai com
ABSTRACT
Foundation models like CLIP are trained on hundreds of millions of samples and effortlessly generalize to new tasks and inputs. Out of the box, CLIP shows stellar zero-shot and few-shot capabilities on a wide range of out-of-distribution (OOD) benchmarks, which prior works attribute mainly to today's large and comprehen- sive training dataset (like LAION). However, it is questionable how meaningful CLIP's high zero-shot performance is as it seems likely that web-scale datasets like LAION simply contain many samples that are similar to common OOD bench- marks originally designed for ImageNet. To test this hypothesis, retrain CLIP on pruned LAION splits that replicate ImageNet's 's train-test similarity with respect common OOD benchmarks. While we observe performance drop on some benchmarks, surprisingly. CLIP's overall performance remains high. This shows that high train-test similarity is insufficient to explain CLIP's performance, and other properties of the training data must drive CLIIP to learn good representations. Additionally, by pruning data points that are dissimilar to the OOD benchmarks, we uncover 100M split of LAION (14 of its original size) on which CLIP can be trained to match its original performance.
INTRODUCTION
Large models like GPT-4 (OpenAI, 2023; Schulman al., 2022), CLIP (Radford et al., 2021) I), or LLaMa (Touvron et al., 2023) are changing the technological and academic landscape with their unprecedented performance and breadth of viable applications. core characteristic of these Foundation Models (Bommasani et al., 2021) is that they are trained on hundreds of millions or even billions of data points scraped from the internet. For example, OpenCLIP (Schuhmann et al., 2022), the open-source version of CLIP (Radford et al., 2021), is trained on LAION-400M, web-scale dataset with a wide variety of image-text pairs (Schuhmann et al., 2021). CLIP forms the backbone of generative models like DALL-E2 (Ramesh et al., 2022) and is known for its remarkable pero-shot and few-shot performance on wide range of tasks, specifically on out-of-distribution (OOD) benchmarks like SeageNet-Sketch (Wang et al., 2019), ImageNet-R (Hendrycks et al., 2020), etc.
Prior work has shown that CLIP's stellar performance stems mainly from its data distribution (Fang et al., 2022; Radford et al., 2021). Nevertheless, it remains unclear which specific properties of the training distribution, such as its scale, diversity, density, or relation to the test set, drive performance. OOD benchmarks like SSekehht and ImageNet-R were initially designed in reference to ImageNet- 1 (Deng et al., 2009), which had served as the primary dataset driving progress in machine vision for several years before the emergence of web-scale datasets. ImageNet-Sketer ImageNet-R, and others are considered OOD because they share the same content (i.e., classes) as ImageNet- lk but are dissimilar in terms of style, pose, scale, background, or viewpoint. There no guarantee that
"Equal contribution. Code available ht https: http::///gttttt wttp//wwwt//tetttt. http:///eeneeeegooooeeeei------.

---

Published as a conference paper at ICLR 2024
these datasets are also dissimilar LAION-400M. We provide evidence in Fig. I where choose samples from ImageNet- and ImageNet-R and examine their nearest perceptual neighbors in LAION-400M, and ImageNet-Traiie Train. We find highly similar neighbors and even exact duplicates in LAION-400M while neighbors in ImageNet- Train are relatively dissimilar. In other words, models trained on LAION-400M may perform well on conventional OOD benchmarks simply due to being trained on semantically and stylistically similar data points. Naturally, the question arises:
Does CLIP's accuracy on OOD benchmarks mainly stem from highly similar images in its train set?
By highly similar images, we mean images that are stylistically and semantically more similar to the test sets than any image in ImageNet-Ik is. To answer this question, we make the following contributions:
Sec. 4.1, we begin by introducing percepiual similarity (Ilharco et al., 2021), which has previously been shown to capture stylistic and semantic similarity between images (Fu et al., 2023; Gadre et al., 2023; Zhang et al., 20 We show in Sec. 4.2 that the similarity of nearest neighbors under this metric generally impacts CLIP's performance. Specifically, we (i) observe high correlation between zero-shot accuracy and nearest-neighhor similarity of test samples and (ii) demonstrate that similarity-based pruning of the training set greatly affects CLIP's performance.
Based on these insights, we compare the distribution of nearest-neighhor similarities of different training sets in Sec. 4.3 and find that they differ substantially. We hypothesize that CLIP's high performance might be largely explained by the training samples that cause this difference, which we term highly similar images.
Sec. 4.4 formalizes the notion of highly similar images based on the similarity gap of two training distributions. Under this formalization, highly similar images of LAION-400M lie within the similarity gap of ImageNet- any image in ImageNet- Train is. We go on to show how pruning can align the similarity gap of both distributions, such that test sets are as dissimilar to pruned --10--000--splitt as they are to ImageNet- Train.
to given test set, i.e., are more similar to test samples than
As our central result in Sec. 5, we surprisingly find that training CLIP on the curated subsets only marginally decreases performance on the corresponding OOD benchmarks (Tab. 1). We conclude that high train-test similarity cannot fully explain CLIP's remarkable performance, and other properties of LAION-400M must play a role.
To facilitate future research into the impact of training on the performance of vision-language foundation models, we curate 100M subset of LAION-400M (1/4 of its original size) on which CLIP maintains its full OOD benchmark performance (Sec. 4.2 & B.4).
RELATED WORK
Mcasuring OOD generalization To assess expected model performance the wild, researchers use different test sets that are considered OOD with respect to the training distribution. The terms OOD generalization, (distributional) robustness, or just generalization are used interchangeably by the community. This work mainly focuses on standard datasets that share classes with ImageNet. They include: image renditions (ImageNet-RR Hendrycks et al., 2020). unusual camera views and object positions (ObjectNet; Barbu et al., 2019), images selected to be difficult for ImageNet-trained ResNet-50s (ImageNet-A; Hendrycks et al., 2021) and sketches of ImageNet classes (ImageNet- Sketch; Wang et al., 2019). We also consider two datasets commonly considered in-distribution, namely ImageNet-Val (Deng et al., 2009), and ImageNet-V2 (Recht et al., 2019).
ID vs. OOD generalization While researchers treat the test sets listed above as OOD with respect to the training distribution when they study robustness, this core assumption is rarely scrutinized. Large-scale language-image models such as CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021), 1), or BASIC (Pham et al., 2021) claim exceptional OOD generalization and zero-shot capabilities. Fang et al. (2022) probe which aspects of the models- like language supervision, cost function, or training distribution are related to model's effective OOD robustness and find that differences in the distribution play key role. Further, Nguyen et al. (2022) find that combining data from multiple sources for training interpolates the model's effective robustness on an OOD test set between the
2

---

Published as a conference paper at ICLR 2024
performance of the model trained either data source. Here, we aim extend the findings of Fang et al. (2022) and Nguyen et al. (2022) by evaluating whether high similarity between training and test set is the main driver of CLIP's claimed performance, or whether CLIP is truly better at generalizing across larger distribution shifts.
ImageNet- - Sketch
Decreasing similarity
LAION-400M
ImageNet- Train
LAION-40OM
ImageNet-
ImageNet- R
LAION-400M
ImageNet-Train
LAION-400M
ImageNet-Tee Train
ImageNet Val
LAION-400M
ImageNet- Train
LAION-400M
ImageNet- Train
Figure 1: Similarity of common benchmarks to LAION-400M and ImageNet-Trrai.. We show nearest neighbors of ImageNet-Sketchl ImageNet-R and ImageNet- Val samples LAION-400M and ImageNet-Trree ordered by decreasing perceptual similarity. We omit duplicates within these nearest neighbors. Perceptual similarity is cosine similarity computed CLIP's image embedding space (see Sec, 4) and can be thought of as measuring the perceptual closeness images in terms of content and style. LAION-400M clearly contains more similar images to samples from ImageNet-Sketch and ImageNet-R, in contrast ImageNet-Traain -er-r is more similar to ImageNet-Van Val. More details in App. G.

---

Published as a conference paper at ICLR 2024
Query image
Nearest neighbor
Similarity
0.75
0.25
Figure 2: Relation between perceptual similarity and visual closeness of nearest neighbors. Query images are sampled from ImageNet-Skent Sketch (top row) and are connected to their nearest neighbor in LAION-400M (bottom row). As in Fig. 1, perceptual similarity is simply the cosine similarity measured in CLIP ViT-B/16+'s image embedding space.
EXPERIMENTAL DETAILS
This section contains technical specifics of image-to-image similarity computation, training details, deduplication, and LAION-200M. Readers can skip this section and return to it when they seek details on the aforementioned. For computing umage-to-image similarity, measuring duplicates, and pruning data points, we use CLIP ViT-B/16+'s B/16+'s image embedding space. For all our pruning experiments, we train CLIP ViT-B/32 (Dosovitskiy et al., 2020) for 32 epochs with batch size of 33,600 on one node with eight A100 GPUs (training takes several days, depending on the dataset size). We use the implementation provided by Ilharco et al. (2021) and stick to their settings for learning rate, weight decay, etc. Our downloaded version of LAION-400M contains only 377M images overall due to missing or broken links, compared to the original 400M used in OpenCLIP (Ilharco et al., 2021).
LAION-200M Abbas et al. (2023) show that pruning exact duplicates, near duplicates, and se- mantically very similar samples within LAION-400M (not yet taking any test sets into account) can reduce dataset size by up to % without performance degradation. We re-implement their method to generate our baseline LAION split containing 199M samples, which we refer to as LATON-200M. This step is important to make training multiple instances CLIP feasible, and we observe that the incurred drop in performance is negligible (compare Tab. 1).
THE SIMILARITY HYPOTHESIS
This section first illustrates how perceptual similarity can be quantified (Sec. 4... Based on this metric, we demonstrate that CLIP's performance on a test set is strongly related to the nearest-neighbon similarity between LAION-400M and test set (Sec. 4.2). Further, we show that nearest-neighbor similarities differ between LATON-400M and ImageNet-Trail which leads to the hypothesis that this difference explains CLIP's high classification accuracy on ImageNet-basee test sets (Sec. 4.3). Finally, phrase this hypothesis in terms of highly similar images, which leaves us with an interventional method to test this hypothesis (Sec. 4.4).
4.1 QUANTIFYING PERCEPTUAL SIMILARITY
Abbas et al. (2023) demonstrated that nearest neighbors in the image embedding space of CLIP share semantic and stylistic characteristics. We illustrate this in Fig. 2, where plot samples from ImageNet-Sketch and their nearest neighbors in LAION-400M for different similarity values. Visually, the similarity scores correlate well with the closeness of the image pairs. This is corroborated by other works that demonstrate high perceptual alignment between CLIP's embedding similarity and human perception (Fu et al., 2023), or using it to sample ImageNet-like images from large dataset (Gadre et al., 2023), or building a similarity- -baed classifier (Zhang et al., 2021).
We follow these works and quantify perceptual similarity as the cosine similarity in CLIP VIT-B/16+'s image embedding space. App. ablates the choice of the model used to compute this metric. We denote the similarity of two samples I;, Jj
12"h
s s(11, xtz) R
R
1.
(1)
4

---

Published as a conference paper at ICLR 2024
1.0
Smaeehet-Sketch
JmageNet-Sketch
ImageNet-Val
ImageNet-R
0.65
ImageNet-val
0.8
ps 0.91
accuraty
0.45
Ps 0.92
accuracy
0.6
95
0.25
baseline
rand-pruned
far-pruned
0.4
nearpruned
0.05
0.4
0.7 0.8 percoptual similarity
150M number of pruned datapoints (in millions) number of pruned datapoints (in millions)
50M
5DM 100M
150M
Figure 3: Nearest-neighhor similarity is predictive of performance. Left: AAION4400M-trainee CLIP's top-1 classification accuracy on test samples is highly correlated their nearest-neighbor similarity Sisati Results are averaged over 0.05 similarity intervals. Center and right: Similarity- based pruning greatly impacts CLIP's top-1 classification accuracy. We train a baseline model on LAION-200M (see Sec. 3) and additional models on L.IOON-200-spliss created by random pruning, near-pruning (in order decreasing similarity), and far-pruning (in order of increasing similarity). Compared to training on 'rand-pruned splits (solid blue curve), training red curve) drastically decreases classification accuracy. Training on 'far-pruu 'ar-prunedd' splits (dashed blue curve) impacts accuracy comparatively little.
splits (solid
We now consider the relation between a training dataset DD and test set T. Using the similarity metric s, we can find the nearest neighbor in the test set for each training sample. This allows us to assign each training sample I; E 'D the nearest-neighhor similarity
S (D, = max stt, TET
(2)
In the same way, we can assign each sample T the nearest-neighhor similarity (D, (D, max 2(),
Stest,
=ED
4.2 NEAREST-NEIGHBOR SIMILARITY DRIVES PERFORMANCE
We can now examine the relationship between nearest-neighbor similarity and CLIP's zero-shot
classification performance.
Fig. 3 (left) illustrates that the nearest- nearest-neighbor similarity Siest, of test samples in ImageNet-Sketdde ImageNet-R, and ImageNet-Val to LAION-200M is good predictor of CLIP's top-1 accuracy on these samples. We observe a clear correlation between nearest- neighbo similarity and accuracy across datasets. For SaageeNe-SSktch.. for example, sketches without similar counterparts in LAION- 400M (similarity 0.38) are classified with 35 % accuracy, while sketches duplicated in LAION-400M (similarity close to reach up 69 % accuracy. We show additional correlation plots for ImageNet- based test sets in App. B and for other test sets in App. D.
We can observe the impact nearest-neighbor similarity on classification performance more directly by pruning samples from LAION-200M based on their nearest- nearest-neighhor similarity Stain, to a given test set, retraining CLIP, and evaluating its zero-shot classification performance on that test set. We compare three different pruning strategies: 'near- pruning' prunes in decreasing order of similarity (pruning samples with high neighbt similarity first), 'far-pruning' prunes in increasing order of similarity, and "rand-pruning' prunes randomly irrespective of similarity. All strategies produce 200 with 50M, 100M, and 150M pruned samples.
CLIP's zero-shot classification performance when trained on these splits is illustrated in Fig. 3 for ImageNet-Sketch and ImageNet-Vall The 'near-pruned" accuracy curve drops much quicker with decreasing dataset size than the 'rand-pruned" curve. This reiterates that CLIP's classification perfor- mance is directly related to the similarity of its training set to the test set. Additional visualizations for other datasets (both ImageNet-basee and otherwise) well as a comparison with IaageNet-trained models can be found in Apps. B and D. Note that since we prune large fractions of the training set here, the pruned images are not yet very specific to the test set used to compute Suain As
5

---

Published as a conference paper at ICLR 2024
Imagefeet Train
LAION-400M
300
200
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.4
0.5
0.6
0.7
0.8
0.9
1.0
p perceptual similarity to ImageNet -SStchh
perceptual similarity to ImageNeet-R
perceptual dimilarity to ImageNet-Aol
Figure 4: Nearest-neighhor similarity distribution differs between LAION-40MM and ImageNet- Train. The histograms display the similarity Stest, of samples in ImageNet-Sketee Sketch (left), ImageNet-R (center), and ImageNet- Val (right) to their nearest neighbors in LAION-400M (red) and ImageNet- Train (blue). ImageNet-Skette and ImageNet-R R are overall more similar to LAION-400M, while ImageNet- Tergeeee-Taain is more similar to ImageNet-Val Val.
result, pruning based on one ImageNeet-based dataset generally decreases performance across many ImageNet-based datasets, although not on not on other tasks (see App. B).
The observation so far is not surprising: Performance on the test set decreases in tandem with the training distribution's 's similarity to the test set. However, our results validate using similarity-basee pruning as an effective intervention that allows us to study how training samples impact performance on given test set. In the next sections, we will explore how to hone this method to arrive at more precise conclusion about the role of highly similar images.
Core set As aside, we notice that CLIP's performance when trained 'far-pruned' LAION- 200M-splits remains stable up until a dataset size of 100M (see Fig. 3). The performance even slightly surpasses the baseline, further indicating that dissimilar samples do not contribute CLIP's performance and instead act more like noise in the training data. Motivated by this performance, we extract a LAION-400M core set with only 100M images by 'far-pruning' based on not one but six common ImageNet-base models trained a de-duplicated dataset of the same size (Ilharco et al., 2021) and roughly matches the performance of LAION--000, -o--rrnndd model (see Appx. B.4). We release this core set to ease further exploration of the relationship between training distribution and CLIP's zero-shot performance.
benchmarks simultancously. CLIP trained on this core set outperforms
4.3
COMPARING YEAREST-NEIGHERS SIMILARITIES BETWEEN TRAINING SETS
Given the impact of nearest-neighbor similarity CLIP's zero-shot performance, it is natural to ask how LAION-400M's nearest-neige neighbo similarity compares to that of other datasets. Specifically, for ImageNet-bawse benchmarks like ImageNet-S -Skechh and ImageNet-R, nearest-neighbor similarities Si to LAION-400M and ImageNet-Trrin., We have already seen in Fig. 1 that compared to ImageNet-Train. LAION-400M seemed stylistically and semantically much more similar to ImageNet-Sketee and ImageNet-R. while the effect was reversed for ImageNet- Val. Using the notion of perceptual neighbt similarity, we can now fully capture the difference in similarity in principled manner. This is illustrated in Fig. 4, where we can now clearly observe that compared to ImageNet- Train, LAION 400M indeed overall more similar to ImageNet- -ketch and ImageNet-R. We show additional histograms for other test sets in Apps. B and D.
compare the distribution of
Moreover, Appx. A.2, we detail how many training samples are near duplicates (duplicates up to small shifts or crops) of the test sets. While we found 3. % of ImageNet-SSetch images to have duplicates in LAION-400MM there are only 0.04 % ImageNet-Skent duplicates in ImageNet-Train. On the other hand, ImageNet-Train contains duplicates of 2.67 % ImageNet- Val images as opposed to just 0. % ImageNet- Val images in LAION-400M
LAION-400M and ImageNet- Train
LATON-400M-trained CLIP has been reported to outperform ImageNet-trainee methods on ImageNet- Sketch and ImageNet-R, while underperforming on ImageNet- Val (see Tab. 1). In light of the above observation, this could well be explained not by LAION-400M's general scale and diversity but specifically by its fraction of training samples whose nearest-neighte similarity to the test set surpasses that of any sample in ImageNet-Trrin. We term those samples highly similar images. The

---

Published as a conference paper at ICLR 2024
small, sparse dataset
large, dense dataset
corrected dataset
train data points
test data points
pruned data points
similarity gap
Figure 5: Aligning the similarity gap of two datasets. larger, denser, more diverse dataset likely contains samples more similar to given test points than smaller, sparser one. To control for this, we compute the nearest-neighbor similarity of each test point to the smaller dataset (left) and prune points from the larger dataset that lie within this hull (center). We end up with a corrected large dataset replicating the similarity gap of the small one (right).
following section formalizes this concept and explains how we can refine the similarity based pruning from Sec. 4.2 to quantify their impact on CLIP's zero-shot classification performance.
4.4 SIMILARITY GAP AND HIGHLY SIMILAR IMAGES
Secs. 4.2 and 4.3 provide direct and indirect evidence that CLIP's performance on common ImageNet- based benchmarks might mainly stem from images in its training set that are highly similar to the test sets. We now formalize this notion and describe how to systematically test our hypothesis. To this end, we note that even for ImageNet- Train, the nearest-neighbor similarity Sust, differs across test samples. Our goal is to prune LAION-400 4-000 so that the pruned dataset replicates the nearest-neighhor similarities S of ImageNet- Train.
Let us consider that we now have two training datasets, denoted as Ds (small, like ImageNet- and Dr, (large, like LAION-400M), and still use test dataset T (like ImagehettSSetcch. For the sake of simplicity, we assume that Dss is a subset of 'D... We choose a similarity measure as in Sec. 4.2. We collect all neighbt similarities
Stest, (recall Eq. 3) in the set
S(D, T) = Ssess (D, j [1.17]]}
(4)
which we term similarity gup. We can think of this set full characterization of the training set's similarity to any point in the test set; compare Fig. 5.
Based on the assumption that the large dataset contains all samples from the small dataset, it follows that (Ds) I (DL).. In other words, the nearest-neighbo similarity to samples in the small training set is always smaller than equal the similarity to samples in the large training set. Consequently, on a per-sample basis, S(D,, T) is strictly larger than S(Ds. T,, i.e., the large dataset is generally more similar the test than the small dataset. We aim to identify maximally large subset D, C DD, of the large training set, such that its similarity gap S(Di. T)) is equal to the similarity gap (Ds, T) of the small dataset (on per-sample basis, meaning (Dss) (Ds)) for all samples). To achieve this, we examine each test sample & and remove any sample x E Dz for which the similarity s(t,, x) (Ds). We illustrate this procedure in Fig.
This method allows us to surgically remove highly similar images with respect to a given test set and reference training set. Compared to the unconstrained pruning in Sec. 4.2, this will remove far less samples from LAION-400M, and thus allows us to isolate the impact of highly similar images.
CORRECTING FOR HIGHLY SIMILAR IMAGES
We now apply the framework from Sec. 4.4 to remove highly similar images from LAION-200M. To ensure that TaageNet-Train and LAION-200M have the same similarity gap to the test sets, include all ImageNet- Train images in LAION-2000 with the caption "a photo of a (object class): )" We refer the reader to Appx. Sec. C for discussion the choice of ImageNet for experiments.
7

---

Published as a conference paper at ICLR 2024
Table 1: Corrected zero-shot performance of CLIP VIT-B/32. 'X-pruned' represents a pruned dataset from LAION-200M + ImageNet such that the similarity gap to 'X' is the same as the similarity gap of ImageNet to 'X'. The sizes of these subsets are subtracted from the LAION-200M + splits
ImageNet's size. Here, 'X' is one of the six standard ImageNet test sets. 'combined-prume ensure a similarity gap of LAION-200M and ImageNet-Train to all 6 test sets. CLIP's corrected zero- shot performance drops the most on Imagehee-SSketch and ImageNet-R with relative performance drop of 10.8 % and 4.8% % respectively. Red color indicates a drop in performance on the respective test set, and blue represents risc. Overall, high performance indicates that highly similar images do not play key role in explaining CLIP's generalization ability.
Top-1 Accuracy
Dataset
Size
Val
Sketch
A
R
V2
ON
OpenAI (Radford et al., 2021)
400
63.38
42.32
31.44
69.24
55.96
44.14
L-400M (Schuhmann et al., 2021)
403 000
62.94
49.39
21.64
73.48
55.14
43,94
L-200M
199 824 274
62.12
48.61
21.68
72.63
54.16
44.80
L-200M + IN-Train
200 589
68.66
50.21
23.33
72.9
59.7
43.99
val-pruned
-377
68.62
49.58
23.47
72.74
59.47
45.08
sketch-pruned
8342
68.34
44.78
22.7
69.35
59.52
44.12
a-pruned
-138852
68.85
50.25
22.99
72.44
60.05
44.43
r-pruned
5735749
68.71
46.92
23.44
69.48
59.6
45.08
v2-pruned
325
68.79
50.45
23.19
72.58
59.84
45.33
objectmet-pruned
025
68.75
50.14
22.70
72.82
59.37
43.73
combined-prune -pruned
12 352
759
68.05
44.12
22.15
67.88
58.61
44.39
As described in Sec. 4.4, we first compute the similarity gaps of the smaller dataset, i.e., ImageNet- Train, the samples in each of the six test sets. Pruning LAION-200M to these similarity gaps leaves us with six different base splits as shown in Tab. 1. We also generate a 'combined-prunce split that ensures ImageNet-Traan-likk similarity gap all test sets simultaneously. We can now train CLIP from scratch on these splits to obtain a corrected zero-shot performance and compare it to the accuracy of CLIP trained by OpenAl and OpenClip (Ilharco al., 2021; Radford et al., 2021).
The first important point to note in Tab. 1 is that for 'sketch-pruned and 'r-pruned' datasets, we prune 8.3M and 5.7M samples, respectively. For all other datasets, we prune only around 250K- 380K samples. We saw indications of this already in Sec. 4 when we looked at the distribution of nearest-neighhor similarities, see also Tab. 7. The number of pruned samples is also highly correlated with the respective accuracies. For CLIP trained the 'r-pruned' dataset and CLIP trained on the 'sketch-pruned" dataset, we observe 4.8 % relative performance decrease on ImageNet-R and 10.8 % relative performance decrease ImageNet-Skette compared to the baseline. There is also considerable performance change on ImageNet-R for Skketch-prunee and on ImageNet- Sketch for 'r-pruned'. This is reasonable there is some style overlap in Seaeehet-Sketch and ImageNet-R. For the other four base splits, we see less than % relative performance change on all six evaluation sets. The performance of the CLIP model trained on the 'combined prunddddpp''' split is lower than the baseline on all six eval sets, with sizeable drops in ImageNet-R and ImageNet-Sketch. We also observe similar trends when we do not add ImageNet-Trail the pruned datasets (refer to Tab, 4 in the Appx.).
6 DISCUSSION
We now return to our original question: Does CLIP's 's accuracy on OOD benchmarks mainly stem from highly similar images in its train set? To give definitive answer, we take a closer look at the CLIP model trained on sretch-pruneed This model's training set is as dissimilar to ImageNet-Skete as is ImageNet-Trail Train. features an accuracy of 68.34 % on ImageNet- Val. According to ImageNet- Train's 's effective robusiness line (Fang et al., 2022), at this performance level, we would expect accuracy of roughly 14 % on ImageNet- Sketch. Instead, we find an accuracy of 44. %. In other words, training on much larger dataset while keeping the similarity gap constant drastically increases generalization performance for CLIP (in this case, by a staggering 30 percentage points). This effect is even higher for other datasets. This indicates that CLIP's impressive performance is not much the result of
8

---

Published as a conference paper at ICLR 2024
high train-test similarity but that CLIP leverages its dataset scale and diversity learn more generalizable features.
What drives generalization? Generalization of vision-language models is complex subject where several factors like architectural choices, caption quality, training procedures, and data distribution play role. We focus the training distribution since prior works have studied the effect of the aforementioned factors on CLIP's generalization performance (e.g., Santurkar et al., 2022; Mintun al., 2021) and identified it as a prominent factor (Fang al., 2022). Many distribution properties could contribute to generalization performance, but based on raw visualizations of the involved datasets, highly similar images are clearly a factor. Our results only show that it is not the most salient factor and large chunk of performance remains to be explained. We leave the scrutiny of other likely factors like data diversity and density for future work. Our work should be interpreted as a step towards finding specific data properties that dictate generalization.
Measuring the true OOD performance Our analysis excluded training images from LAION with smaller similarity gap to test images compared to ImageNet Train. Another interesting analysis would be to prune LAION images to measure its true OOD performance. To remove all images of a certain domain, we need to be able to label each image as 'ID' or 'OOD', This essentially means that we need access to a domain classifier (which would also need near-perfect accuracy so that no images are overlooked). Even for the 'sketch' domain, where a classifier could conceivably be trained, it is unclear exactly how the classifier should demarcate this domain: Should the domain contain all sketches, even sketches with characteristics not present in Sketch?t What about tattoos or small sketches on objects in natural images? For other benchmarks, such as ImageNet-A, it is even less clear how the test images constitute well-separable domain of images. This vagueness in defining a domain based on a given test set prevents us from building a fair OOD setting, which is why we do not analyze or claim analyze this.
Similarity metric We defer the reader to Sec. E for a discussion and ablation on the choice of CLIP ViT-B/16+ as the similarity metric.
Highly similar images We want to clarify further the notion of highly similar images. In Secs. 4. 4.2, and 4.3, when we use the notion of similar images to given image sample, we refer to images with high perceptual similarity values with no precise constraint. In contrast, in Secs. 4.4 and 5 we impose a constraint that defines highly similar images to a sample as images that are closer to LAION-200M than ImageNet-Train based our perceptual similarity metric.
Does compositionality drive performance? In this work, we found that high train-test similarity is insufficient to explain CLIP's high generalization performance only excluded images that were highly similar to the training set to maintain the same similarity gap with respect to ImageNet Train, e.g. sketches of dogs if the test image was sketch of a dog. However, sketches of other animals and objects still remained CLIP's training set. An open question remains whether compositionality (Wiedemer et al., 2023) can close the gap between the object and its domain, ie. whether CLIP can generalize from sketches of cats and natural images of dogs to understanding sketches of dogs.
OOD test sets. In our analysis, we
7 CONCLUSION
CLIP has demonstrated unprecedented performance on common OOD benchmarks designed origi- nally for ImageNet. Given that the training dataset of CLIP so large and diverse, it is natural to wonder whether its performance stems from the sheer similarity of many training samples to the benchmarks. To the best of our knowledge, we are the first to systematically test if high train-test similarity dictates CLIP's generalization performance. In our work, we address this by pruning away samples from the training set that are more similar to the test sets than ImageNet samples. Models trained on the pruned dataset do not significantly lose performance and still exhibit stellar generalization capabilities far beyond performance serformance-matchedd ImageNet- trainned models. This indicates that high similarity to the test sets alone can not explain CLIP's generalization ability. We hope this result will prompt the community to investigate other factors that allow models to learn more generalizable features from web-scale datasets.
9

---

For all the basic details training. pruning, similarity computation, and other analysis, we defer the reader to Sec. Details of computing the similarities and its correlation accuracy is given in the caption of Figs. 2, 3, and Sec. 4.1. .. To perform the experiment that observes the effect of "near--prunna mnear-prunng'' and 'far-pruning'. we defer the reader Sec, 4.2 and the caption of Fig. 3. The core Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
methodology of our paper is clearly elucidated in Section 4.4. Furthermore, the details of generating
the datasets and training the models are given in the first and second paragraph of Sec. 5, and in the caption of Tab. 1.
AUTHOR CONTRIBUTIONS
The project was led and coordinated by PM. The method was jointly developed by PM, TW, with insights from ER, WB, MB. PM conducted all the experiments based on code jointly implemented by PM and TW. PM, TW, ER, and WB jointly wrote the manuscript with additional insights from MB. ER created all figures and visualizations with TW's help using data provided by PM and with comments from WB.
ACKNOWLEDGMENTS
We would like thank (in alphabetical order): Thomas Klein, George Pachitariu, Matthias Tange- mann, Vishaal Udandarao, Max Wolff, and Roland Zimmermann for helpful discussions, feedback, and support with setting up the experiments. This work supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS 18039A WB acknowl- edges financial support via an Emmy Noether Grant funded by the German Research Foundation (DFG) under grant no. BR 6382/1-1 and via the Open Philantropy Foundation funded by the Good Ventures Foundation. WB is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. This research utilized compute resources the Tübingen Machine Learning Cloud, DFG FKZ INST 37/1057-1 FUGG. We thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting PM, TW, and ER.
REFERENCES
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data- efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09544
2023.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Sara Beery, Arushi Agarwal, Elijah Cole, and Vighnesh Birodkar. The iwildcam 2021 competition dataset, 2021.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.07258. 2021.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map the world, 2018.
Deng, W. Dong, Socher, L.-. Li. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6): 2012.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An
10

---

Published as a conference paper at ICLR 2024
image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11929, 2020.
Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness contrastive language image pre- training (clip). In International Conference Machine Learning, pp. 6216-62666211 6234. PMLR, 2022.
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. arXiv preprint arXiv:2306. 090444, 2023.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis. Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannanch Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304. 2023.
Sujan Sai Gannamaneni, Arwin Sadaghiani, Rohil Prakash Rao, Michael Mock, and Maram Akila. Investigating clip performance for meta-data generation in ad datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3839-3849, 2023.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. CoRR, abs/2006. 16244, 2020.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15262-15271, 2021.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidr. Openclip, July 2021. URL https:tt//// http/oio.rg//////////////o.//o.t 02201///////////////// 5281/ zenodo... 5143773, If you use this software, please cite as below.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Tsung- Yi Lin, Michael Maire, Serge Belongie. Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Eric Mintun, Alexander Kirillov, and Saining Xie. On interaction between augmentations and corruptions in natural corruption robustness. Advances in Neural Information Processing Systems, 34:3571-3583, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems (NeurIPS) Workshops, 2011. /ttttp::/storraae https///sstorrggggggeeepppo... www.goools...coooo com/ pub-tools www.ttoossssscccccco pwwuppubiii-- wwtpooosspppicccppppaaaaeeeeeeee pw-ppuuiiaiiinnnn data/ 376488 paf.
Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. In Advances Neural Information Processing Systems, 2022.
OpenAl. Gpt-4 technical report, 2023.

---

Published as a conference paper at ICLR 2024
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scaling for zero-shot transfer learning. arXiv preprint arXiv:2111.10055 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. -748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25, 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? International conference on machine learning, pp. 5389-5400. PMLR, 2019.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprini arXiv: 1.08731, 2019.
Shibani Santurkar, Yann Dubois, Rohan Taori, Perey Liang, and Tatsunori Hashimoto. cap- tion worth a thousand images? a controlled study for representation learning. arXiv preprint arXiv:2207.07635, 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski. Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty- Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https::///// wwwp///pneeeeeeeeeeeeeeeeeoeeeea .et. forum 11dM3117---1.1-- 4.maMYY.
John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Fe- lipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, Rapha Gontijo Lopes. Shengjia Zhao, Arun Vijayvergiya, Eric Sigler, Adam Perelman, Chelsea Voss, Mike Heaton, Joel Parish, Dave Cummings, Rajeev Nayak, Valerie Balcom, David Schnurr, Tomer Kaftan, Chris Hallacy, Nicholas Turley, Noah Deutsch, Vik Goel, Jonathan Ward, Aris Konstantinidis, Wojciech Zaremba, Long Ouyang, Leonard Bogdonoff, Joshua Gross, David Medina. Sarah Yoo, Teddy Lee, Ryan Lowe, Dan Mossing, Joost Huizinga, Roger Jiang, Carroll Wainwright, Diogo Almeida. Steph Lin, Marvin Zhang, Kai Xiao, Katarina Slama, Steven Bills, Alex Gray, Jan Leike, Jakub Pachocki, Phil Tillet, Shantanu Jain, Greg Brockman, Nick Ryder, Alex Paino, Qiming Yuan, Clemens Winter, Ben Wang, Mo Bavarian, Igor Babuschkin, Szymon Sidor, Ingmar Kanitscheider, Mikhail Pavlov, Matthias Plappert, Nik Tezak, Heewoo Jun, William Zhuk, Vitchyr Pong. Lukasz Kaiser, Jerry Tworek, Andrew Carr, Lilian Weng, Sandhini Agarwal, Karl Cobbe, Vineet Kosaraju, Alethea Power, Stanislas Polu, Jesse Han, Raul Puri, Shawn Jain, Benjamin Chess, Christian Gibson, Oleg Boiko, Emy Parparita, Amin Tootoonchian, Kyle Kosic, and Christopher Hesse. Chatgpt. https: //tp:///pppaa/a/ com / b og/ / chatgpt, 2022. Accessed: 2023-05-13.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint 23X. 2023.
Haohan Wang. songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pp. 10506-10518, 2019.
Thaddäus Wiedemer, Prasanna Mayilvahanan, Matthias Bethge, and Wieland Brendel. Compositional generalization from first principles, 2023.
12

---

Published as a conference paper at ICLR 2024
Yonglong Tian, Yonglong Tian, Yonglong Tian, Olivier J. Hénaff, and Aaron van den Oord. Divide and Contrast: Self-supervised Learning from Uncurated Data. arXiv: Computer Vision and Pattern Recognition, 2021. DOI: MAG ID: 3161438416.
Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training- freee clip-adapter for better vision vision-language modeling. arXiv preprint orXiv2211..0992 2021.
13

---

Published as a conference paper at ICLR 2024
A DISTRIBUTIONAL DISSIMILARITIE, OF LAION-200M AND IMAGENET
A.1 NEAREST-NEIGHIBR SIMILARITY BETWEEN LAION IMAGENET-TRAII AND OTHER OOD DATASETS
As an extension of our analysis in Sec. 4, we plot the nearest-neighbor similarity between ImageNet- Train/LAION-400M and other OOD test sets, namely ImageNet-A (Hendrycks et al., 2021), Object- Net (Barbu et al., 2019) and ImageNet- V2 (Recht al., 2019), and display results in Figure 6. There are no significant differences in nearest- neighor similarity for these test sets. Similar to our results in Figure 3 (left), we find strong correlation between perceptual similarity to LATON-400M and the top-1 accuracy of our LAION-trained model.
60
ImageNet-Try
LAION 400M
100
40
COUnT
20
0
1.
ImsageNet-A
percepiual similarity ObjectNet
1.0
Imagetlet-A
Imageflet
ObjectNe:
60
0.8-
0.
0.93
40-
0.6-
Dy 9.9
0.2
0.3
0.5 0.6
0.7
0.8
0.9
1.0
0.5
0.6
0,7
0.8
0.9
1.0
perceptual simiarity ImageNett222
perceptual similarity
Figure 6: Similarity of nearest neighbors to test sets varies between LAION-400M and ImageNet- Train and correlated with performance. Histograms over the ncarest-neights similarity of test sets ImageNet-A (top left), ObjectNet (top right), and ImageNet- V2 (bottom left) to training sets LAION-400M (red) and ImageNet-Traie Train (blue). There are no significant differences in nearest- neighbor similarity for these test sets. Nearest-neighbor similarity of test points to LAION-400M samples and top-1 classification accuracy is strongly correlated (bottom right). Data points in the correlation plot are averaged over bins (interval 0.05) of the red histograms.
A.2 DUPLICATES
We do a duplicate analysis in Tab. 2. To estimate the number of test points with near duplicates, we project the test set and LA1ON-400M to CLIP's image embedding space and check if any point in LAION-400M lies in the vicinity (c = 0.05) of each of the query test points.
B ADDITIONAL. EXPERIMENTAL RESULTS
B. IMPACT OF NEAR/FAR PRUNNING ON ALL DATASETS
In Sec. 4.2, by using each of the test datasets ImageNet- Val and ImageNet- and near/far- paru/inngg- LAION-200M, we trained models and reported the performance the test datasets, respectively. We now plot the performance of these models on all six datasets in Figure 'Near Mlear-pruning' ("far-pruning') with ImageNet-Sketch results in lower (higher) performance than "near-pruning'
14

---

Published as a conference paper at ICLR 2024
Table Number of test points of OOD datasets for which we find near duplicates in ImageNet- Train and LAION-40MM. A data point is considered near duplicate (semantic duplicate) if the distance in the CLIP embedding space is less than 0.05 (Abbas al., 2023).
Duplicates
Dataset
Size
ImageNet- Train LAION-40M
ImageNet-Ve Val
50000
1336
70
ImageNet-Skette -kktch
50889
18
1553
ImageNet-R
30000
104
297
ImageNet-A
7500
10
5
ImageNet-V2 V2
10000
10
24
ObjectNet
18574
0
0
"ffrrpruniig"" with ImageNet- Val ImageNet-R and MaaggeNet-Sketel Likewise, 'near- mear-pruning' ('far (far-pruning)) with ImageNet- Val results in lower (higher) performance than 'near-pruning' ('far- pruning') with ImageNet-Sketee on ImageNet- Val, ImageNet- V2, and ObjectNet. This is expected because ImageNet-Sketer is characteristically closer to ImageNet-R, and ImageNet- Val closer to ImageNet- V2, ObjectNet, and ImageNet-A.
ImageNet-A
ObjectNet
ImageNet-R
baseline
0.65
rand- pruned
fa-ppruned-sketch
near-prueed-sktchh
far-pruned-val
accuracy
0.45
near-pruned-val
0.25
0.05
ImageNet-Skette Sketch
ImageNet-V2
ImageNet Val
0.65
0.45
0.25
0.05
50
100
150
50
100
150
50
100
150
number pruned datapoints (in millions)
number pruned datapoints (in millions)
number pruned datapoints millions)
Figure The effect of 'near-pruning' and 'far-pruning' with ImageNet-Sketch or ImageNet- Val as the query dataset the performance of all six test sets. CLIP's zero-shot accuracy function of the number of pruned points from LAION-2000 The baseline model is trained on de-duplicated LAION-400M, which we call LAION-200M To generate the 'near-pruned" datasets, remove the images the decreasing order of similarity (based on CLIP image-embedding similarity) to each of the sets ImageNet-Sketor and ImageNet- respectively. In contrast, the 'far-pruned" datasets are generated by dropping images in the increasing order of similarity values to the respective test sets. For the 'rand-pruned" datasets, we prune random points.
15

---

Published as a conference paper at ICLR 2024
MNIST
SVHN
0.65
baseline
rand-pruned
far par-pruned-sketre
0.45
near-pruned-sketch
far- prunedrunddall
accuracy
near-pruned-var
0.25
0.05
50
100
150
100
150
number pruned datapoints (in millions)
number of pruned datapoints (in millions)
Figure 8: The effect of 'near-pruning' and 'far-pruning' with ImageNet-Sketch ImageNet- Val as the query dataset on the performance on MNIST SVHN CLIP's zero-shot accuracy as a function of the number of pruned points from LATON-200M, The baseline model is trained de- duplicated LAION-400M, which we call LAION-200M. To generate the 'near-pruned" datasets, we remove the images in the decreasing order of similarity (based on CLIP image-embedding similarity) to each of the test sets ImageNet-Sketee and ImageNet-Vall respectively. In contrast, the 'far- tur-pruned' datasets are generated by dropping images the increasing order of similarity values to the respective test sets. For the rran-prrune datasets, we prune random points.
B.2 IMPACT OF NEAR/FAR PRUNING ON NON-IMAGENET--IKG DATASETS
In Fig. 7, we observe consistent trend across all datasets that near-pruning with respect to either ImageNet-Sketer or ImageNet- Val decreases performance while performance is stable (at times even increases) when doing far-pruning. These findings can be explained by two hypotheses:
1. The pruned images in the near-pruning setting are reasonably similar ImageNet-Sketel or ImageNet-Ve Val, thus we see a drop in performance when we train CLIP on the pruned datasets.
2. Near- sear-pruning with respect ImageNet- that are of the highest quality samples from LAION which will perform well on any downstream task, when trained upon.
ImageNet- Val results in pruming datapoints
To decide between these hypotheses, repeat the analysis dissimilar from ImageNet: SVHN (Netzer et al., 2011) and MNIST (Deng, 2012). We show the results in Fig. 8 and indeed observe reversed trend compared Fig. 7: Now, for several cases. near-pruning increases performance and far-pruning decreases it, respectively. Thus, the near-pruned datapoints are not comprised high quality samples which improve performance all downstream tasks, and pruning away images similar to either ImageNet-Sketod or ImageNet- Val only decreases performance on ImageNet-lin like datasets.
sets which very
B.3 NEAR/FAR PRUNING EXPERIMENTS ON IMAGENET-TREN TRAIN
We generate new datasets by near/far pruning datapoints on ImageNet-Train with test datasets ImageNet-Sketch and ImageNet-Vall Val. The similarities are computed in the pre-trained CLIP ViT- B/16+ embedding space. We then train ResNet18s until convergence using standard Py PyTorch hyperparameter settings. We report the absolute and relative performance (to the respective baseline) in Figure and 10. We observe that near/far pruning affects CLIP performance more than ResNet18,
16

---

Published as a conference paper at ICLR 2024
Smagehet-Sketch
ImageNet-Val
0.65
0.45
accuracy
0.25
CLIP
baseline
ResNet18
rand-prunce
far-pruned
0.05
near-pruned
100
50
25
100
75 final dataset size (in %)
25
final dataset size (in %)
Figure 9: Effect of pruning similar and dissimilar points to a given test set on both CLIP and ResNet performance. The baseline model of CLIP trained on de-duplicated LAION-400M, which we call LAION-200M. The baseline model of ResNet18 is trained on ImageNet- Train. To generate the 'near-pruned" datasets, we remove images decreasing order of similarity to ImageNet-Sketue or ImageNet-We Val (based on CLIP image-embedding similarity). In contrast, the 'far-pruned' datasets are generated by pruning images in the increasing order of similarity values to the respective test sets. For the 'rand-pruned' datasets, prune random points. Pruning similar images adversely affects performance compared to pruning dissimilar or random images. Generally, near/far pruning affects CLIP more than ResNet18.
B.4 CORE SET OF 100M
In Sec. 4.2, we identify a 100M core set of LAION-400M, which, when trained on, leads to a CLIP model that nearly matches the performance of a LLAION-00MM trained CLIP model on the six test datasets. Motivated by the performance increase of the 'far- 'far-pruning' technique in the previous results, we now build several core sets of 100M, which, when trained on, roughly match the performance of CLIP trained on LAION-400M. Instead of pruning from the farthest point to samples in just single test set in CLIP ViT-B/16H 16+'s embedding space, we now prune from the farthest point to samples from collection of test sets (all six ImageNet-1K OOD test sets). We do far-pruning with all of the test sets on both LAION- 200M and LAION-40 -40M to obtain datasets that we call 'all-far 'all-far-punedd pruned'. For comparison, also add the performance of CLIP trained far far-pruned datasets with query datasets as ImageNet-Sketch and ImageNet- Val, which we call 'sketch-fae sreeth--farppee tar-pruned' and 'val-far far-pruned', respectively.
We report the results in Tab. and observe that models trained on all of the splits are within % average accuracy range of CLIP trained LAION-400M The model with the highest average accuracy is trained on 'all-far-pruned (L-200M)', which is a dataset generated by pruning far or dissimilar images in LAION-200M with all test datasets as query datasets. This model also performs better than model trained on a dataset of the same size generated by the pruning technique SemDeDup (Abbas et al., 2023). SemDeDup aims to prune semantically similar data with minor loss in test performance. We do not suggest this coreset as alternative to other pruning or deduplication methods that are largely agnostic to the downstream test datasets. Instead, we here created coreset that is specifically designed to perform well on six OOD test sets to facilitate further research into what aspects drive generalization.
B.5 MAIN EXPERIMENTS WITHOUT ADDING IMAGENET-TRAIN
We repeat the experiments Sec. 5 without adding ImageNet-Treee Train to LAION-200M and report results in Tab. 4. We observe the same trends in Tab. 1.
17

---

Published as a conference paper at ICLR 2024
ImageNet-Sketor
ImageNet-Voe
1.00
0.80
relative accuracy
0.60
0.40
CLIP
baseline
0.20
ResNet18
rand-pruned
far-pruned
near-pruned
0.00
100
75
50
100
25
final dataset size (in %)
final dataset size (in %)
Figure 10: Effect pruning similar and dissimilar points to given test set on both CLIP and ResNet's performance relative to the baseline. The baseline model of CLIP is trained on de-duplicated LAION-400M, which we call LAION-200M. The baseline model of ResNet18 is trained ImageNee-Trainn To generate the "near-pruned" datasets, remove images in decreasing order of similarity to ImageNet-Sketch or ImageNet-Val (based on CLIP image-embedding similarity). In contrast, the 'far-pruned' datasets are generated by pruning images in the increasing order of similarity values to the respective test sets. For the 'rand-pruned' datasets, we prune random points. Pruning similar images adversely affects performance compared pruning dissimilar or random images. The relative performance drop curves indicate that near/far pruning affects CLIP more than ResNet18.
Table 3: Performance of 'far-pruned CLIP (ViT-B/32) on the six test sets. We do 'far-pruning' on LAION-400M with all test sets as query sets and obtain the dataset 'all-far far--rurrdddd (L-400M). Similarly, do 'far-pruning' on LAION-400 -400M with all with all 6 test sets query sets, ImageNet- Sketch, and ImageNet- Val to get the datasets far---eerr (L-200M)', akktch-far-prunee (L- 200M)', and "val-far-pruned (L-200M)) respectively. These models compared off the shelf CLIP model (Ilharco et al., 2021), 1), model trained on LAION-200M and model trained on SemDeDup (Abbas al., 2023) dataset of size 100M.
Top-1 Accuracy
Dataset
Size
Val
Sketch
A
R
V2
ON
Avg.
L-400M
400M
62.94
49.39
21.64
73.48
55. 14
43.94
51.09
L-200M
199.8M
62.12
48.61
21.68
72.63
54.16
44.80
50.67
all-far- far-pruned (L-400M)
100M
61.90
48.11
19.43
70.14
53.11
39.30
48.67
all-far-prunee (L-200M)
100M
62.80
49.23
21.6
72.3
54.72
43.64
50.71
val-far-pruned (L-200MM)
100M
62.79
47.53
21.65
70.40
54.35
43.70
50.07
Mketch-far-pruned (I-2200M)
100M
62.27
50.21
20.77
72.67
53.77
42.95
50.44
SemDeDup
100M
52.19
41.70
16.71
67.05
44.96 39.59
43.7
C ON THE CHOICE OF IMAGENET
We choose zero-shot classification on ImageNet and its distribution shifts as the main object study four work. Our analysis is agnostic these choices and one could potentially use other datasets like iWILDCam 2021 (Beery et al., 2021), FMoW (Christic et al., 2018), MS-COCO (Lin et al., 2015) and Flickr30k (Yonglong Tian al., 2021) and tasks like image retrieval. One reason why
18

---

Published as a conference paper at ICLR 2024
Table 4: Corrected zero-shot performance of CLIP VIT-B/32. 'X-pruned' represents pruned dataset from LAION-200M such that the similarity gap to 'X' is roughly the same as the similarity gap of ImageNet to 'X'. The sizes of these subsets are subtracted from the LAION-200M' size. Here, 'X' is one of the six standard ImageNet test sets. 'combined-pruned splits ensure similarity gap of LA1ON-200M and ImageNet-Train to all test sets. CLIP's corrected zero-shot performance drops the most on Imagehet-Sketch and ImageNet-R with relative performance drop of 11.08 % and 5.99 % respectively. Red color indicates a drop in performance on the respective test set. Overall, high performance indicates that highly similar images do not play key role in explaining CLIP's generalization ability.
Top-1 Accuracy
Model
Dataset
Size
Val
Sketch
R
V2
ObjectNet
ViT-B/32
OpenAl
400.000 000
63.38
42.32
31.44
69.24
55.96
44.14
ViT-B/32
L-400M
413 000 000
62.94
49.39
21.64
73.48
55.14
43.94
VIT-B/32
L-200M
199 824 274
62.12
48.61
21.68
72.63
54.16
44.80
ViT-B/32
val-pruned
-377 340
62.12
48.38
21.45
72.2
54.76
42.79
VIT-B/32
sketch-pruned
-8342 783
61.55
43.22
22.28
69.6
53.53
42.77
ViT-B/32
a-pruned
-13
62.49
48.49
21.63
72.15
54.38
43.25
VIT-B/32
r-pruned
-5735749
61.73
45.66
21.67
68.28
54.1
42,90
ViT-B/32
v2-pruned
-274325
62.48
48.62
22.13
72.3
53.83
43.38
ViT-B/32
objectnet-pruned
025
62.30
49.03
22.64
72.90
54.21
42,80
ViT-B/32
combined-prumed
- -12 759
61.5
41.97
21.72
67.25
53.65
42,23
ResNet-101
ImageNet-Ik
200
77.21
27.58
4.47
39.81
65.56
36.63
do not investigate this in our paper for the reason that several ImageNet distribution shifts are more human aligned than for the aforementioned datasets. For instance, there's 's perceptual demarcation between a sketch of dog (ImageNet-Skee Skeech)) and a natural image of a dog (ImageNet- Train). In contrast, it is unclear what the distribution shift of Flickr-30k is from MS-COCO. Additionally, retrieval tasks are more complex and highly sensitive to captions, demanding an analysis that factors in both images and texts. Another reason to choose zero-shot classification and ImageNet is that CLIP demonstrated unprecedented performance on ImageNet- 2021). Moreover, find iWILDCam and FMoW datasets problematic since CLIP's (ViT-B/32) zero-shot performance on them is rather low (7.45% and 12.96%) (Ilharco et al., 2021). We therefore chose ImageNet and its distribution shifts for our study and leave analysis on other datasets for future work.
distribution shifts (Radford et al.,
D SIMILARITY ANALYSIS ON CELEBA AND WATERBIRDS
Secs. 4 and only considered test sets with clear distribution shift with respect ImageNet- Train. However, the general method outlined in Sec. 4.4 is dataset-agnostic. To illustrate this point, we here consider test sets that exhibit distribution shifts with respect to other datasets. Specifically, consider CelebA (Liu et al., 2015) and Waterbirds Sagawa et al. (2019).
CelebA This dataset contains 202 599 celebrity images with 40 annotated attributes (Liu et al., 2015). Gannamaneni et al. (2023) showed that CLIP could zero-shot predict many attributes with high accuracy (between 53 and 97 % top-1 accuracy). We can split the data along each attribute to obtain training and test sets with specific distribution shift. In Fig. 11, repeat the similarity analysis from Sec. 4.3 for CelebA splits along the 'eyeglasses' and 'hat' attributes. The distribution of neirest-neighhor similarities to the test set CelebA-w/eeyeglasees (CelebA-w/-B) We again observe a strong correlation
differs between
LAION-400M and DelebA-w//oeeegglasee between the similarity of test sample to LAION-400M and CLIP's zero-shot accuracy predicting the 'gender' attribute.
Waterbirds We generate two sets of splits of this dataset. We first split by background (land or water) and obtain distribution shift from Waterbirds-lan llnd with 7051 images (6220 landbirds, 831
19

---

Published as a conference paper at ICLR 2024
120
CelebA w/o eyeglasses
1.0
LAION-400M
80
accuracy
0.9
count
Ps 1.0
0.8
0.7
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.75 perceptual similarity
1.0
perceptual similarity to CelebA w/ eyeglasses
CelebA w/o hat LAION-400M
0.8
100
75
count
accuracy
0.6
Ps 0.78
50
25
0.5
0.6
0.7
0.8
0.9
1.0
0.4
0.8 perceptual similarity
0.6
1.0
perceptual similarity to CelebA w/ hat
Figure 11: nearest-neighbor similarity distribution and correlation to zero-shot accuracy for CelebA. Left: The histogram shows the similarity of samples in CelebA-w/-eyeglasse to their nearest neighbors in LAION-400M (red) and CelebA-w/o-gyellase (bluc). Right: The strong correlation between perceptual similarity of test points to nearest neighbors in LAION-400M samples and CLIP's top-1 classification accuracy on Male/Female classification indicates that differences in similarity can expected impact the performance of LAION -triindd models. Data points in the correlation plot are averaged over bins (interval = 0.05) of the red histograms in the left plot.
waterbirds) to Waterbirds-watee with 4737 images (2905 landbirds, 1832 waterbirds). We then split into the core group and the worst group. The group consists of 8052 images of landbirds land or waterbirds on water. The worst group consists of 3736 images of landbirds on water or waterbirds land. In Fig. 12, repeat the similarity analysis from Sec. 4.3 for Waterbirds splits along the land/water and core/worst-group distribution shifts. The distribution of nearest-neighbor similarities the test Waterbirds-wai water (Waterbirds- worst group) differs between LAION-4000 -40M and Waterbirds-land (Waterbirds-comm corbbrrds--oreeeeeo core group), and we again observe strong correlation between the similarity test sample to LAION-400M and CLIP's zero-shot accuracy in predicting the landbird/waterbi class.
20

---

Published as a conference paper at ICLR 2024
50
Waterbirds Coregroup LAION-400M
0.8
0.6
Ps = 0.82
count
accuracy
25
0.4
0.2
0.0
0.5
0.6
0.8
0.9
0.4
0.6
0.8
1.0
perceptual similarity to Waterbirds Worst Group
perceptual similarity
50
Waterbird Land LAION-40OM
1.0
0.8
count
accuracy
Ps 0.98
25
0.6
0.4
0
0.2
0.4
0.8 perceptual similarity to Waterbird Water
0.6
1.0
0.4
0.8 perceptual similarity
0.6
1.0
Figure 12: nearest-neighbor similarity distribution and correlation to zero-shot accuracy for Waterbirds. Left: The histogram shows the similarity of samples in Waterbirds- to their nearest neighbors in LATON-400M (red) and Waterbirds-land (blue). Right: The strong correlation between perceptual similarity of test points to nearest neighbors in LAION-4002 -400 samples and CLIP's top- classification accuracy on landbird/waterbird classification indicates that differences in similarity can be expected impact the performance of LAION- models. Data points in the correlation plot are averaged over bins (interval = 0.05) of the red histograms in the left plot.
E COMPARING EMBEDDING METRICS
To our knowledge, perceptual similarity as measured in CLIP VIT-B/16+'s image embedding space is leading metric capture the semantic and stylistic similarity between images. While we found this metric to align well with our intuitive notion of similarity and believe it to have captured the vast majority highly similar images (see also Appx. G.2 where we visualize the pruned datasets), we cannot guarantee that all highly similar images were removed. While we believe, based on prior work (Fu al., 2023; Abbas et al., 2023; Gadre al., 2023; Zhang et al., 2021) and our analysis, that CLIP's 's embedding space sufficiently captures relevant features, in this section, we ablate the influence of the embeddings used to compute the perceptual similarity. Specifically, we compare CLIP ViT-B/16+ embeddings used throughout the main paper to the embeddings of ViT-L-14 trained on LAION-400M, much larger model.
We compute the nearest-neighhor similarities of ImageNet-Trran to the test sets using either embed- dings and compute their correlation. We summarize the results in table and find that the correlation is strong across test sets. Figure 13 also shows histograms of nearest-neighhor similarities using either embedding, revealing that the similarity distributions are very similar. Given these two comparisons, and considering that CLIP ViT-B/16+ embeddings are faster and cheaper to compute and have been shown to capture perceptual image similarities reasonably well by previous work (Abbas et al., 2023), use them throughout work.
21

---

Published as a conference paper at ICLR 2024
Table 5: Choice of embeddings has little impact on nearest-neighbor similarities. We compute nearest-neighbor similarities between ImageNet- Train the six tests on embeddings produced by CLIP ViT-B/16+ and ViT-L-11 and find a strong correlation across the bench.
Dataset
Ps
ImageNet- Val
0.93
Imagehet-SSktch
0.87
ImageNet-R
0.90
ImageNet-A
0.86
ImageNet-V2
0.93
ObjectNet
0.80
oval, dotaset
sketch
eval, dataset imagenet vol
eval. dataset imagenet
3000
1500
2000
Count
3500
1000
1000
500
250
embedding
eval dataset imagenet-a
eval daw.sstttett..... imagenet-22
eval_ datasatttet objecthet-subsamph
VTB plus 240, Jalonit0m w32 1axmn400m 452
500
400
1000
200
003
400
200
200
0.6
0.5
3.0
0.4
0.6
0.5
1.0
0.4
0.5
1.0
sims
simra
sims
Figure 13: Choice of embeddings has little impact on nearest-neighhor similarity distribution. We compute nearest-neighbor similarities between ImageNet- -rain the six tests on embeddings produced by CLIP ViT-B/16+ and ViT-L-14 and find their distributions visually very similar across the bench.
F DISTRIBUTION OF SIMILARITIES OF LAION- AND IMAGENET-TRAIN AFTER PRUNING
We analyzed nearest-neighbor similarity distribution of the test sets to LAION-200M and ImageNet- Train (see Figs. and 6). But what about the nearest-neighhor similarity distributions of LAION- 200M and ImageNet-Trane Train to the test sets, especially after pruning? We now answer this question to understand better where the training points are situated with respect to the pruning boundary.
For each data point in the pruned dataset, we compute the maximum similarity to the respective test set and divide it by the test point's similarity gap (i.e., neiiest similarity of the test point to LAION-200M before pruning). We call this quantity normalized similarity. Note that the normalized similarity values for samples in the pruned datasets are strictly smaller than 1.0 because samples with values greater than 1.0 are the ones that lie in the similarity gap and are pruned away.
Plotting the density of normalized similarities in Fig. 14 reveals that LAION -rrundd has a much wider distribution with smaller mode. Since normalized similarity closer to 1.0 indicates that the point lies closer to the similarity gap, a larger proportion of ImageNet- Train samples are close to the similarity gap compared to the proportion observed in LAION-200M.
22

---

Published as a conference paper at ICLR 2024
We also compare the total number of points that are close to the similarity gap (normalized similarity > 0.9) in Tab. 6. Due to LAION's scale, each pruned LAION split contains 5-20 times more data points close to the similarity gap than ImageNet- Train. We expect that this large and diverse set of samples close to the boundary greatly dictates CLIP's performance.
Normal lzed similarity distributtion
sllatch
Imagenat-voll
4.-
2.5
2.0
100
03-
0.2
04
0.6
10
0.6
0/2
1.0
02
0.1
0.5
10
Imagenet-r
Imaganet-22
objectnat
3.0
4.0
1aler200m
imagenetttrain
2.0
Deraay
Deralty
2.3-
1.5
1.0
03
0.2
0.5
0.8
1.0
0.2
0.4
3.6
0.8
1.0
0.0
02
3.1
0.6
0.8
1.0
Figure Density normalized similarity for LATON-pruned and ImageNet-Trrin. We observe wider density function for LAION- LAION-pruned prunedd with smaller mode. This indicates that ImageNet-Traii samples are generally more concentrated around the similarity gap.
Table 6: Total number of samples in LAION-pruned splits and ImageNet that lie near the boundary of the similarity gap for each test set. Closeness defined by a normalized similarity 0.9. While LAION- still has 5 to 20 times more samples close to the boundary than ImageNet-
samples are less concentrated around the gap (see Fig. 14), LAION-prune
Dataset
LATON-200M
ImageNet-Train
ImageNet-Sketto
855 133
131 087
ImageNet-Val Val
2344 086
531 982
ImageNet-A
1 111 150
138 975
ImageNet-R
73766362
121 160
ImageNet-Ve V2
1919
326517
ObjectNet
1558 301
52277
G NEAREST NEIGHBOR VISUALIZATIONSS
To generate nearest neighbors, we compute the nearest images of LAION in CLIP's (CLIP ViT-B/16+) image embedding space for each test image. After removing duplicates and near-duplicates within LAION, we visualize the top six images.
G.1 LAION-400M VS IMAGENET-TRAIN
Just like in Fig. 1, we plot the nearest neighbors in LAION-400M and ImageNet-Trase Train of random query images for each of the six datasets in Figs. 15, 16, 17, 18, 19, and 20.
23

---

Published as a conference paper at ICLR 2024
Decreasing similarity
ImageNet- Sketch
LAION-400M
ImageNet- Train
LEOR TEOC SEODE LOGD B
LAION-400M
00:000 CO0D 10000
ImageNet- Train
LAION-400M
ImageNet- Train
LAION-400M
ImageNet- Train
LAION-400M
ImageNet- Train
Figure 15: Nearest neighbors of randomly sampled Imagehet-Sketch queries in LAION-400M and ImageNet-Train ordered by decreasing perceptual similarity. We omit duplicates within the nearest neighbors. Perceptual similarity computed in CLIP's image embedding space and can be considered to measure the "perceptual closeness" of images in terms of content and style.
G.2 AFTER PRUNING
Tab. 7 reports the percentage of images in each of the six datasets that have higher similarity to AAJON-200MLLAON--HO
than ImageNet- Train. For each of the six test sets, we randomly sample query images that are more similar to LAION-200M than TeageNet-Train and plot the nearest neighbors in ImageNet-Trail Train, LAION-200M, and LAION-200M after pruning by the respective test in Figures 21, 22, 23, 24, 25, and 26.
24

---

Published as a conference paper at ICLR 2024
Decreasing similarity
ImageNet- - Val
LAION-400M
ImageNet-Train
LAION-400M
KRESSE
Yees
ImageNet- Train
TARNAE
LAION-400M
ImageNet-Train
LAION-400M
Kodak Saries 144
ImageNet- Trainn
LAION-400M
ImageNet- Train
Figure 16: Nearest neighbors of randomly sampled ImageNet- Val queries in LAION-400M and ImageNet-Train ordered by decreasing perceptual similarity. We omit duplicates within the nearest neighbors. Perceptual similarity is computed in CLIP's image embedding space and can be considered measure the "perceptual closeness" of images in terms of content and style.
Table 7: Percentage ( (%) of points in the test datasets for which the nearest neighbor is in LAION440MILATON-2OOM rather than ImageNet-Trrin.
Dataset
Size
LAION-400M
LAION-200M
ImageNet- Val
50000
16.80
14.88
ImageNet-Sketchh Sketch
50
97.94
97.45
ImageNet-R
30000
87.88
86.74
ImageNet-A
7500
47.39
45.53
ImageNet-V2
10000
38.95
35.48
ObjectNet
18574
63.24
61.62
25

---

Published as a conference paper at ICLR 2024
Decreasing similarity
ImageNet- -
LAION-400M
ImageNet-Trai Train
LAION-400M
ImageNet- Train
LAION-400M
ImageNet- Train
LAION-400M
ImageNet- Train
LAION-400M
ImageNet- Train
Figure 17: Nearest neighbors of randomly sampled ImageNet-A queries in LAION-400M and ImageNet-Train ordered by decreasing perceptual similarity. We omit duplicates within the nearest neighbors. Perceptual similarity is computed in CLIP's image embedding space and can be considered measure the "perceptual closeness" of images terms of content and style.
26

---

Published as a conference paper at ICLR 2024
Decreasing similarity
ImageNet
LAION-400M
ImageNet-Train
LAION-400M
ImageNet-Train
LAION-400M
ImageNet-Traii
LAION-40OM
ImageNet-Train
LAION-400M
ImageNet-Train
Figure 18: Nearest neighbors of randomly sampled ImageNet-R queries in LAION-40MM and ImageNet-Train ordered by decreasing perceptual similarity. We omit duplicates within the nearest neighbors. Perceptual similarity is computed CLIP's image embedding space and can be considered measure the "perceptual closeness" of images in terms of content and style.
27

---

Published as a conference paper at ICLR 2024
Decreasing similarity
ObjectNet
LAION-400M
ImageNet-Train
LAION-400M
ImageNet-
LAION-400M
ImageNet-Trail
LAION-400M
ImageNet- -rain
LAION-400M
ImageNet- Train
Figure 19: Nearest neighbors of randomly sampled ObjectNet queries in LAION-400M and ImageNet- Train ordered by decreasing perceptual similarity. We omit duplicates within the nearest neighbors. Perceptual similarity is computed in CLIP's image embedding space and can be considered to measure the "perceptual closeness" of images in terms of content and style.
28

---

Published as a conference paper at ICLR 2024
Decreasing similarity
ImageNet- V2
LAION-400M
ImageNet-Train
LAION-400M
ImageNet-Train
LAION-400M
ImageNet-
LAION-40OM
ImageNet-
LAION-400M
ImageNet-
Figure 20: Nearest neighbors of randomly sampled ImageNet-V2 queries in LAION-400M and ImageNet-Train ordered by decreasing perceptual similarity. We omit duplicates within the nearest neighbors. Perceptual similarity is computed in CLIP's image embedding space and can be considered measure the 'perceptual closeness" of images terms of content and style.
29

---

Published as a conference paper at ICLR 2024
ImageNet- Sketch
Decreasing similarity
LAION-200M
ImageNet-Train
LAION-200M pruned
LAION-200M
ImageNet-Trainn
LAION-200M pruned
LAION-200M
ImageNet-Train
LAION-200M pruned
LAION-200M
ImageNet-Trail Trainn
LAION-200M pruned
Figure 21: Nearest neighbors of Smagehet-Sketch images in LAION-200M, MateeNet-Train and "sketch-prumed" (LAION-2000 200M pruned) ordered by decreasing perceptual similarity. The query (base) images randomly sampled from the set of images that more similar LAION-200M than ImageNet-Train to see the effect of pruning (see Tab. 7). We omit duplicates within the nearest neighbors. Perceptual similarity is computed in CLIP's image embedding space and can be considered to measure the "perceptual closeness" of images in terms of content and style. LAION-200M. clearly contains more similar images to samples in the test set compared to ImageNet-Train or 'sketch- pruned".
30

---

Published as a conference paper at ICLR 2024
ImageNet- Val
Decreasing similarity
LAION-200M
ImageNet- Train
LAION-200M pruned
EAYT SHOP
SHOP
LAION-200M
BRITE LITE BASBER SHOP
BARBER
LASTEE
BARBER
ImageNet-Train
SHOP
SHOP
DARBET
BARRE
LAION-200M pruned
LAION-200M
MINLING
HIIII SNIDAII
ImageNet- Train
LAION-200M pruned
LAION-200M
Fallout
$2:54
ImageNet- Train
LAION-200M pruned
Figure 22: Nearest neighbors of ImageNet- Val images in LAION-200M, ImageNet- and 'val-pruned" (LAION-200M pruned) ordered by decreasing perceptual similarity. The query (base) images are randomly sampled from the set of images that are more similar to LAION-200M than ImageNet-Trainn see the effect of pruning (see Tab. 7). We omit duplicates within the nearest neighbors. Perceptual similarity is computed in CLIP's image embedding space and can be considered measure the "perceptual closeness" of images in terms of content and style. LAION-200M clearly contains more similar images to samples in the test set compared to 'val-pruned ImageNet-Trran Train images are in-distribution ImageNet- Val and, therefore, contain similar samples.
31

---

Published as a conference paper at ICLR 2024
ImageNet- A
Decreasing similarity
LAION-200M
ImageNet- Train
LAION-200M pruned
LAION-200M
FOO
ImageNet- Train
LAION-200M pruned
LAION-200M
ImageNet-
LAION-200M pruned
LAION-200M
ImageNet- Train
LAION-200M pruned
Figure 23: Nearest neighbors of ImageNet-A images in LAION-200M, ImageNet- and 'a- pruned' (LAION-200M pruned) ordered by decreasing perceptual similarity. The query (base) images are randomly sampled from the set of images that are more similar to LAION-200M than ImageNet-Train to see the effect of pruning (see Tab. 7). We omit duplicates within the nearest neighbors. Perceptual similarity is computed in CLIP's image embedding space and can be considered to measure the 'perceptual closeness" of images in terms of content and style. LAION-200M clearly contains more similar images to samples in the test set compared to 'val-pruned'; ImageNet- -raign images are in-distribution to ImageNet- Val and, therefore, contain similar samples.
32

---

Published as a conference paper at ICLR 2024
ImageNet- -
Decreasing similarity
LAION-200M
ImageNet- Train
LAION-200M pruned
LAION-200M
ImageNet-Trail
LAION-200M pruned
LAION-200M
ImageNet- Train
Inst
LAION-200M pruned
LAION-200M
ImageNet-Tras Train
LAION-200M pruned
Figure 24: Nearest neighbors of ImageNet-R images in LAION-200M, ImageNet-Traine and 'r- pruned' (LAION-20M pruned) ordered by decreasing perceptual similarity. The query (base) images are randomly sampled from the set of images that are more similar to LAION-200M than ImageNet-Train -rain to see the effect of pruning (see Tab. 7). We omit duplicates within the nearest neighbors. Perceptual similarity is computed in CLIP's image embedding space and can be considered to measure the "perceptual closeness" of images in terms of content and style. LAION-200M clearly contains more similar images to samples in the test set compared to 'val-pruned ImageNet-Traiin images are in-distribution to ImageNet- Val and, therefore, contain similar samples.
33

---

Published as a conference paper at ICLR 2024
ObjectNet
Decreasing similarity
LAION-200M
ImageNet- Train
LAION-200M pruned
LAION-200M
ImageNet- Train
LAION-200M pruned
LAION-200M
ImageNet- Train
LAION-200M pruned
LAION-200M
ImageNet-Tren Train
LAION-200M pruned
Figure 25: Nearest neighbors of ObjectNet images in LAION-200M, ImageNet-Train, and 'v2-pruned' (LAION-200M pruned) ordered by decreasing perceptual similarity. The query (base) images randomly sampled from the set of images that are more similar to LAION-200M than ImageNet- Traiin see the effect of pruning (see Tab. 7). We omit duplicates within the nearest neighbors. Perceptual similarity is computed "perceptual closeness" of images in terms of content and style. LAION-200M clearly contains more similar images to samples in the test set compared to 'val-pruned'; ImageNet- Train images are in-distribution to ImageNet- Val and, therefore, contain similar samples.
CLIP's image embedding space and can be considered to measure the
34

---

Published as a conference paper at ICLR 2024
ImageNet- - V2
Decreasing similarity
90
LAION-200M
ImageNet- Train
LAION-200M pruned
LAION-200M
ImageNet- Train
LAION-200M pruned
LAION-200M
ImageNet-
LAION-200M pruned
LAION-200M
ImageNet-Train
LAION-200M pruned
Figure 26: Nearest neighbors of ImageNet-Vi V2 images in LAION-200M, ImageNet-Trrain and 'objectnet-pruned"" (LAION-200M pruned) ordered by decreasing perceptual similarity. The query (base) images are randomly sampled from the set of images that are more similar to LAION-200M than ImageNet-Train to see the effect pruning (see Tab. 7). We omit duplicates within the nearest neighbors. Perceptual similarity is computed in CLIP's image embedding space and can be considered to measure the 'perceptual closeness" of images in terms of content and style. LAION-200M clearly contains more similar images to samples in the test set compared to 'val-pruned'; ImageNet- Traiin images are in-distribution to ImageNet- Val and, therefore, contain similar samples.
35


Reviews:
[{"review_id": "2LiP2hlqgQ", "reviewer": "Reviewer_fbpP", "summary": "This paper tries to uncover the reason for the good generalization of foundation models like CLIP. Specifically, we see that CLIP has a good zero-shot accuracy on OOD datasets, which is much higher compared to the models that are first trained on the labeled dataset (ID), such as ImageNet, and then tested on OOD datasets, such as ImageNet-Sketch.\n\nAs the pre-training data is large, it is highly likely to contain similar images to the OOD dataset. If that is the case, we might not call the test OOD dataset to be truly out-of-distribution, which might be the reason for the good performance of CLIP. \n\nAuthors try to find an answer to the above problem through various experiments. \n\nThey define the perceptual similarity of two images using the CLIP ViT-B/16 embedding of a pre-trained CLIP model. They find that the general OOD datasets like ImageNet Sketch have more common images with the LAION dataset than the ImageNet dataset.\n\nIn the first experiment, the authors removed images similar to the OOD dataset from the LAION dataset iteratively and measured the performance of models trained on a pruned dataset. As expected, they found a general trend of performance decrease, indicating that similar images in the training dataset had more importance.\n\nThe second experiment compared the models trained on small datasets such as ImageNet vs those trained on LAION. The authors define a metric called generalization gap, which is the set of minimum distance of each image in the test set with every image in the training set. They then remove the data points from the larger dataset (here, LAION) to have the same generalization gap as that of the ImageNet with the OOD dataset. They then trained a CLIP model on the pruned dataset. \nThey discovered some performance drops, but the performance was still high compared to the case where the model was just trained on the ImageNet training set. This led to them concluding that the high performance of the CLIP is mostly due to more training data rather than test train similarity.", "questions": "1. Are there experiments on other datasets unrelated to ImageNet? Having these experiments would be better to make a strong case.\n2. Why do we use only CLIP ViT-B/16 embeddings to filter? Won't larger CLIP models have a better embedding to filter out the data points?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is very well written. The experiments are clearly defined, and the motivation for each experiment is mentioned.", "weaknesses": "Even though authors perform many experiments, the experiments are performed on a single type of dataset, which are the variations of the ImageNet.\nIt would have been to include experiments with other OOD datasets such as iWILDCam or FMoW. Without these experiments it is not sure if these analysis is limited to one type of dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vb9UjzV7Iz", "reviewer": "Reviewer_MLz9", "summary": "The goal of this work is to understand whether the superior OOD performance of foundation models like CLIP is a result of the training dataset containing images that are very similar to the OOD test set. Towards this, the authors systematically create several splits of the base LAION dataset that was used for training OpenCLIP. Firstly, they find that pruning samples that are very similar to the OOD test sets results in a considerable drop in the OOD performance. However, by matching the train-test similarity with that of ImageNet for a fair comparison, the authors find that CLIP still shows significant gains when compared to an ImageNet pretrained model. Thus, although LAION contains images that are very similar to ImageNet-OOD test sets, this is not the key reason for better OOD generalization of CLIP. Understanding the reasons for better generalization of CLIP still remains an open question.", "questions": "Several additional experiments are required to be done, in order to emulate a true **OOD** setting, as discussed in the weaknesses section.\nFor example, one should remove all \"sketch\" style images as well, from L-200M+IN-Train (sketch-pruned) in order to test on the ImageNet-sketch test set for a true OOD evaluation. Otherwise, other sketch images that are present in the dataset can help bridge the domain gap between sketches and natural images, leading to an unfair OOD evaluation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The key finding that despite reducing the similarity between the training data and the test sets, there is an improvement in test set performance, is helpful. \n- Several insightful experimental results are presented, which is helpful for the community, especially given that these experiments are very computationally intensive. \n- The pruned datasets whose code is released, can help with further investigation on why CLIP models have better OOD performance.", "weaknesses": "- Although the results are interesting, my main concern is that the analysis is not sufficient to enable fair **OOD** testing. If the paper was about **ID** performance alone, removing train set samples that are similar to each test sample would have been sufficient, as reported in the paper. However, **OOD** implies that the **distribution** of images is unknown. So, to actually conclude that OOD evaluation is fair, all images from the test **domain** should have been removed, not only the images that are similar to every test set image. Therefore, as mentioned in the abstract, the term \"out-of-distribution generalization\" is still not meaningful even by training on the pruned datasets considered in the paper.\n- To elaborate, if there are 5 sketches of the class \"airplane\" in the dataset, all images that are close to these sketches are removed. But there may be other airplane sketches, which are farther away than the closest image in ImageNet-test set, which are not removed. Thus, it is not guaranteed that all images of the given classes and test **domain** are removed from the train set.\n- Further, there may be other objects, such as \"space shuttle\"  which are not included in ImageNet, thus sketches of space shuttles can be present in the training data, in addition to natural images of the same. Thus the domain \"sketch\" is not unseen by CLIP, and hence the evaluation is not truly OOD. \n\nMinor feedback: \n- Clarity of the abstract and contributions list needs improvement. It would be better to make a shorter summary of the key contributions.\n- It would be better to give a different name to \"generalization gap\" as this term is used in a different context.\n\nTypo:\n\n\"We provide anecdotal evidence in Fig. 1 where we choose samples from ImageNet-Sketch and ImageNet-R and examine their nearest perceptual neighbors in LAION-400M and **ImageNet**\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3ndp2xk1gY", "reviewer": "Reviewer_fHnd", "summary": "This paper studies the generalization behaviour of CLIP from the perspective of the training set. CLIP, as a pioneering foundation model, is well known for its exceptional generalization capability. However, it remains unclear whether such a good generalization capability stems from the web-scale training set, as it may well enclose samples similar to those on the test sets. This paper tackles this question by filtering out similar samples according to CLIP-embedding-based nearest neighbors. By creating a LAION subset that comes with as large a generalization gap as the ImageNet-1K training set, this work concludes that it is other factors, rather than the model has already seen samples with similar distribution, that leads to the outstanding ood generalization of CLIP models.", "questions": "What is the potential application of the finding in this paper? I know this is a pure analytical work, and believe its merit is above the aceeptance threshold of ICLR. But I am still intrigued to learn what is the broader impact of this work, like how this would enlighten future research or engineer endeavours.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper first studies the ood generalization behaviour of CLIP from the interesting perspective of train-test distribution similarity, and has drawn some intriguing conclusions, like a subset of 100M of LAION-400M is able to train a good-performing model on ood benchmarks.\n2. The experiments are well motivated and designed, resulting in compelling results that even without nearest neighbor samples, CLIP is still to maintain good performance on ood benchmarks.", "weaknesses": "1. This paper only works on common classification ood benchmarks, but neglecting a whole bunch of other CLIP application domains. In the very least, retrieval is the most foundamental test ground to probe how good a CLIP model is. The authors could use MSCOCO as the in-domain dataset for sample filtering, and use Flickr-30k as the out-of-domain dataset. Also, experiments on more diverse benchmarks, like VTAB (check datacomp paper for current best practices), are encouraged.\n2. Some crucial experimental details seem to be missing. For instance, how the nearest neighbor sets in section 4.1 are constructed are not mentioned at all (or mentioned in the appendix). How many samples are chosen in this set, and what is the threshold of CLIP score used here?\n3. The authors should compare the similarity of nearest neighbors to test sets between the whole LAION datset (after filtering in section 5/6) and ImageNet-Train. Otherwise, one could argue that, even though the closest sample in LAION and ImageNet training set is about the same far away to the test set, LAION has a lot more samples close to the closest sample than ImageNet (but not as close to the test set as the closest sample), and thus still has a unfair advantage on those ood benchmarks, compromising the validity of the conclusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PQVwf2mVBj", "reviewer": "Reviewer_X2Xy", "summary": "This paper studies the hypothesis “Does CLIP’s accuracy on test sets mainly stem from highly similar images in its train set?”. Their approach is to take a CLIP training dataset, LAION, and remove samples similar to OOD benchmarks, then retrain and evaluate the performance drop. In Section 4.1, they show that for some OOD datasets e.g., ImageNet-Sketch and ImageNet-R, there are more similar images to the OOD dataset in LAION than there are in the ImageNet-train set. These images are both semantically and stylistically more similar. They also observe a positive correlation between accuracy and having similar neighbors in the training set for each OOD benchmark. In the remainder of the paper, they prune the dataset by removing similar samples to OOD datasets and evaluate the performance of models trained on pruned datasets.", "questions": "- Figure 4: What if we perform a similar process for training on the ImageNet dataset? Do we observe a similar accuracy drop?\n- Figure 7 shows that removing samples similar to ImageNet-Sketch and ImageNet-Val result in lower performance in all other ImageNet OOD datasets as well. Could this mean that these samples are important for a general understanding of the ImageNet distribution?\n- Section 5: Would this method and any model trained on this dataset be considered as transductive learning? Because the test datasets would have been seen directly or indirectly by the model. Would this be against the license of datasets such as ObjectNet that say “ObjectNet may never be used to tune the parameters of any model.”?\n\nTypos:\n- Page 7: Let us for consider -> Let us consider", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Figure 1 clearly shows that the LAION-400M dataset contains semantically and stylistically similar images to OOD benchmarks while the ImageNet training set does not.\n- Figure 3 quantitatively shows the nearest neighbors of OOD datasets in LAION-400M are on average more similar than the nearest neighbors from ImageNet-train set except for the Imagenet-Val set itself. Fig 3.right also clearly shows that CLIP model performs better on samples that it has seen similar images of it in the training set.\n- Figure 4 shows that pruning LAION for similar samples to ImageNet-Sketch and ImageNet-Val results in substantial accuracy drop compared with pruning random samples. Hence showing that these similar samples are crucial for the effective robustness of CLIP models.", "weaknesses": "- Even though the results in Figure 4 show the near-pruned samples are crucial for OOD generalization of CLIP, they are not conclusive. In particular, the following two experiments would be useful:\ni) Does a model *only* lose OOD generalization on one benchmark or are these near-pruned samples crucial for all sorts of image-classification performance on any benchmark? This requires plotting the accuracy on two datasets, e.g., ImageNet-Val and ImageNet-Sketch, and showing that after pruning samples similar to ImageNet-Sketch, we only lose performance on ImageNet-Sketch.\nii) Do CLIP models lose OOD generalization more quickly than classification models trained on ImageNet? For this, one would prune ImageNet with a similar procedure for X% of samples and compare whether the accuracy drop in ImageNet models is slower than CLIP models. If not, then we would know that any OOD generalization we have been seeing could be due to seeing similar samples in the training set.\n\n- This paper is a good place to have a broader discussion on “What is zero-shot?” and “What is out-of-distribution robustness?” and I think the paper should expand on that. In particular, after showing that pruning loses OOD generalization, it is not clear what the community should do with these datasets and evaluation benchmarks. Should we say CLIP models have cheated and they are not zero-shot? Related to that, the objective of Sections 5 and 6 that “Correct for highly similar images” is not clear. Why would we want to prune our training datasets to get worse on some test benchmarks if those exact test samples do not appear in the training set? Why would we want models in Table 1 trained on pruned datasets that perform worse?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)