What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights
Authors: Xin Wen
Bingchen Zhao
Yilun Chen
Jiangmiao Pang
XIAOJUAN QI
Venue: NeurIPS 2024 poster (2024)
Keywords: CLIP
vision-language pre-training
robustness
long-tail
data imbalance
uncurated data
Primary area: machine_vision
TL;DR: We show that CLIP is more robust to long-tailed pre-training data than supervisied learning, and reveal the reasons behind.

Abstract: Severe data imbalance naturally exists among web-scale vision-language datasets. Despite this, we find CLIP pre-trained thereupon exhibits notable robustness to the data imbalance compared to supervised learning, and demonstrates significant effectiveness in learning generalizable representations. With an aim to investigate the reasons behind this finding, we conduct controlled experiments to study various underlying factors, and reveal that CLIP's pretext task forms a dynamic classification problem wherein only a subset of classes is present in training. This isolates the bias from dominant classes and implicitly balances the learning signal. Furthermore, the robustness and discriminability of CLIP improve with more descriptive language supervision, larger data scale, and broader open-world concepts, which are inaccessible to supervised learning. Our study not only uncovers the mechanisms behind CLIP's generalizability beyond data imbalance but also provides transferable insights for the research community. The findings are validated in both supervised and self-supervised learning, enabling models trained on imbalanced data to achieve CLIP-level performance on diverse recognition tasks. Code and data are available at: https://github.com/CVMI-Lab/clip-beyond-tail.

What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights
Xin Wen' Bingchen Zhao? Yilun Chen'
Jiangmiao Pang"i Xiaojuan Qi
The University of Hong Kong {wenxin, xjqi] hku.ht hk pangjiangmiao@ppla org
University of Edinburgh SShanghai AI Laboratory
Abstract
Severe data imbalance naturally exists among web-scale vision-language datasets. Despite this, we find CLIP pre-trained thereupon exhibits notable robustness to the data imbalance compared supervised learning, and demonstrates significant effectiveness in learning generalizable representations. With an aim to investigate the reasons behind this finding, conduct controlled experiments to study various underlying factors, and reveal that CLIP's pretext task forms a dynamic classifica- tion problem wherein only subset of classes is present in training. This isolates the bias from dominant classes and implicitly balances the learning signal. Further- more, the robustness and discriminability language supervision, larger data scale, and broader open-world concepts, which are inaccessible to supervised learning. Our study not only uncovers the mech- anisms behind CLIP's generalizability beyond data imbalance but also provides transferable insights for the research community. The findings are validated in both supervised and sufervupeee learning, enabling models trained on imbalanced data achieve CLIP-level performance on diverse recognition tasks. Code and data are available at: https: / // g thub. com cww/CVMI htwpa///llppboeoorreeeee
CLIP improve with more descriptive
1 Introduction
The development of contrastive language-image pre-training (CLIP) [36, 44, 57, 68, 93] has demon- strated unprecedented success in learning generalizable representations, empowering zero-shot vision tasks and robustness to natural distributional shifts. This success can be primarily attributed to the effective use of large-scale uncurated image captioning datasets collected from the web. recent trend involves delving into the distribution of these datasets and explicitly introducing interventions the curation process to create better data for training [29, 91]. However, limited research has been conducted analyzing the distribution of concepts/classes in these datasets and the behavior of CLIP under varying distributions. This work thus starts by presenting a concept-centric analysis of existing web-scale image-text datasets and models pre-trained accordingly (Fig. 1).
Motivation. Our motivation this study arises from an intriguing observation CLIP's zero-shot performance ImageNet: CLIP is notably more robust to pre-trained data imbalance than supervised learning. We examine various vision-language datasets at different scales, and analyze their distribu- tion with respect to ImageNet classes. We find that image-text datasets share extremely imbalanced class distribution (Fig. la). Interestingly, we find that the zero-shot classification performance of trained CLIP models more robust to this imbalance, especially compared to models obtained by supervised learning. This evidenced by a weaker correlation between a class's performance and its
CCoresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

107
MetaCLIP-2.58
MetaCLIP
Model CUP
LAION-2B
10a
LAION-28
10a
Correlation
Per-class accuracy
MetaCLIP-400M
MetaCLIP-40OM
Penclass #prediction
LAJON-400M
10
LAION-100M
10a
YFCC-15M
YFCC-15M
101
CC-12M
CC-12M
102
10a
LAIONet-3M
200
1000
0.0
0.4
0.6
0.8
ImageNet classes ranked by the class frequency 400M
Spearman corr. to per-class frequency
(a) Class frequencies (log scale) ranked by LAION-40MM
(b) Correlation between class-wise statistics.
Figure 1: Per-class statistics of image-text datasets and models trained on top. (a) highly imbalanced class distribution is shared across datasets. *bb) Compared to supervised learning (x SL), CLIP's performance (measured by accuracy) less biased by data frequency, and the classifier is notably uncorrelated (measured by model's 's number of prediction per class). Besides, the correlation narrows as data scales up. Both aspects indicate implicit beballncing mechanisms exist in CLIP.
frequency (Fig. 1b). This trend is consistent across CLIP models and pre-training datasets and even holds true for smaller-scal scale datasets like CC-12M [12]. This phenomenon inspires us study the underlying causes for CLIP's relative robustness toward data imbalance and what we can learn from.
Our study and findings. To answer the question above, we conduct controlled experiments to analyze factors including supervision signal and pretext task (Fig. 3), data distribution (Fig. 4), scale (Fig. 5), and open-world concepts (Fig. 6). Our extensive studies have led us the following findings:
Language supervision, particularly the texts with increased descriptiveness (informativeness), enhances both the robustness and discriminability CLIP, and preserves more feature variation.
CLIP's pretext task forms dynamic classification problems, wherein only subset of classes is present during training, effectively isolates biases to dominant classes, and balances learning signal.
Severe data imbalance in web datasets increases the risk of bias in models. However, distribution shift and higher data diversity in them can enhance robustness, albeit a trade-off in data efficiency.
CLIP's robustness and discriminability improve together with data scaling, benefitting from its ability utilize open-world data, privilege not accessible supervised learning.
Applications. Inspired by the findings of our study, we found that this robustness to data imbalance can be transferred to supervised and self-supervised learning models with simple techniques by making the classification task dynamic during training. Under extremely imbalanced data scenarios, we show that a vanilla classification model can also generalize well to tail (or even open-world, classes well as CL.IP via 1) fixing the classifier with class prototypes from pre-trained CI.IP text encoder, and 2) training with randomly subsampled vocabulary (results in Fig. 8, and analysis in also show improved transferability DINO [11] pre-trained on uncurated web data by simply randomly subsampling the prototypes training (Fig. 10).
Fig. 9). Beyond classification,
Summary. Our study is one of the pioneering efforts to explore CLIP's robustness in the context of imbalanced data distributions. Our exploration provides a comprehensive analysis that uncovers the mechanisms contributing to CLIP's robustness against data imbalance. As will demonstrate in this paper, the insights gained from our research are transferable to other domains, including supervised and self- self-supervised learning frameworks.
'MetCCLIP [91] is relatively more balanced than other datasets due concept re- eebbalnncing in curation.

---

Image from ImageNet
ImageNet-
Class frequency (linear scale)
Class: payphone
URL: flickr.com/...
Title: Phone Call at Night
Retrieve metadata
Desc.: might have thing with telephones...
with Flickr API
449K Images
Tags: phone, telephone, black and white,
999 classes
Image LAION-400M
LAIONet
Class frequency (linear scale)
Caption:
Crane Crane,loadingg container ship, by harbor,.. ImageNet
ImageNet # ImageNet
Def.: "crane, which is...
"crane (bird) is..." "s..."codi "container ship is...."
3.26M images 943 classes
Filter caption-to-defiilion
Class: crane, container ship
similarity of CLIP text encoder
Figure Curation process and distribution of datasets used in our controlled study. Top: IN-Caps [27] augments train images of ImageNet with texts by querying Flickr with image URLs. The texts include title, description, and tags. Bottom: LAIONet [77] is filtered subset LAION-400M [73], obtained by matching ImageNet classes with captions and filtering by CLIP text encoder for disambiguation.
2 Related work
CLIP''s distributional robustness. The debut of CLIP not only set the state-of-the-arr performance on conventional image classification benchmarks but also demonstrated unprecedented robustness to challenging distribution shifts. Studies have shown that this robustness stems from the diverse training distributions CLIP has seen during training time [27, 69]. Also, it is shown that the data quality plays an important role in enhancing the distributional robustness of CLIP [58]. It may seem that CLIP obtains the improvement distributional robustness due to the similarity of pretraining data to the distribution shifted data, but [55] shows that is not the case where after pruning similar data, CLIP still obtains strong robustness, indicating generalizable representations are learned.
Learning from uncurated data. Apart from robustness distribution shifts, previous works have also delved into the nature of uncurated large-scale datasets [35, 49, 77, 91]. Studies have shown that self- supervssdd learning can produce more robust models than supervised learning uncurated data [35, 49]. Moreover, focusing on learning of subsets the entire dataset [9, 82] has shown to further enhance self supervupe learning from uncurated data. On the learning on uncurated data, the language information has shown to help learn good representations [71]. Balancing the concept distribution of uncurated data has shown be a scalable way of learning good models [91]. However, the uncurated data is not all harmful for performance, the lower intra-class similarity of the data is shown to help preserve information/variitiion in representations [77], but at low data efficiency [85].
Generalization of vision models. One of the main themes of computer vision research in the era of deep learning is the search for more generalizable models. Works have focused on self-supervised pretraining with only images, among which contrastive learning [13] and self-distillation [11, 61] are shown be effective. With the introduction of large-scale image-text datasets [73, 74], there is huge interest in learning more generalizable vision representations from additional language supervision. While techniques for incorporating language supervision have been proposed [19, 36, 68, 72. 94], further exploration of how semantic grounding help improves the generalization is needed [21]. To fully utilize language supervision, using synthetic data from large language models improve language supervision is newly emerged research area [65.
3 What makes CLIP more robust to long-tailed pre-training data?
In the following, we conduct series of controlled experiments to systematically evaluate the role of various factors on the robustness of CLIP to data imbalance. These factors include supervision signal (Sec. 3.2), pretext task (Sec. 3.3), data distribution (Sec. 3.4), data scale (Sec. 3.5), and open-world concepts (Sec. 3.6). Moreover, we also provide some insights on CLIP's feature space in Sec. 3.7.
3

---

3.1 Setting
Datasets. Experiments in this study are conducted on variants two image-text datasets: ImageNet- Captions [27] and LAIONet [77] to allow better data-centric control. An overview is shown in Fig. 2. Both datasets provide images with their paired captions, and class labels on ImageNet. The captions of ImageNet-Captions are in the format of title, description, and tags (some can be missing for specific image), which allows control captions' descriptiveness. Images of LAIONet are drawn from LAION, which has higher intra-class variability and is extremely imbalanced across classes. This makes it more challenging to train on and allows isolating the effect data distribution.
Models. We consider both CLIP and supervised learning (SL) with ResNet-50 the backbone. Given that CNNs are generally considered less robust than ViTs [4], this choice also enables us infer the robustness of other models. For SI., align most details with CLIP [68] to rule out the effect of irrelevant factors. E.g. we use the same weak data augmentation as CLIP, adopt a prototypical classification head (i.e., fz-normalizinn both features and classifier weights), and apply a learnable temperature to logits. The training schedules of CLIP and SL follow [15] and [27]. respectively. Models are fully trained from scratch by default. More details are provided in Appx.
Metrics. We compute Spearman correlation coefficients [78] between class frequency and models' statistics (class-wise top-1 accuracy and number of samples predicted as each class). Besides, we also consider metrics from neural collapse literature [32, 63] for analyzing feature distribution. Formally, defining the global feature mean Nc A hi,e, class-level means H.c Avg, hi,c, within- class covariance Ew Avgr,o(hs. (e Me) and between-class covariance ΣB
A (G) -MG) the metrics are defined as:
Me Pe
1
NCI
= =CI=
NC2 Avge, | |ellle C-1
where it denotes the Moore-Penrose pseudoinverse, hi.e is the feature of the i-th example in class and C is the total number of classes. Intuitively, NC1 and NC2 measure the compactness and separation of clusters, respectively. NCI approaches zero when the within-class variation of features becomes negligible, and NC2 converges zero when classifiers reach maximal and equal margins (i.e., ETF structure) [63]. Note that these two metrics are originally defined as an average across classes, and it is simple to obtain per-elass NCI and NC2 metrics, measuring the variability of specific class or its average margin to all other classes.
3.2 (Descriptive) language as supervision signal
Setting. We start by examining the impact of language supervision, the primary distinction between CLIP and other contrastive learning approaches. This is done by creating texts with roughly monotonic
Prediction Bias Perspective
Overall Accuracy Perspective
Model
0.6
brttter
60
CLIP
x
SL
Correlation (Ireq, #pred.)
0.5
smaller socabulary
55
Better
X
Class Templates
Text Type
Synset Templates
0.4
Overall acc. (top-1,
50
Only
Thtle Only
45
Tags Only
0.3
40
Description
Tags Description
0.2
higher text descriptiv eness
35
Title Tags
Rull Text
30
Vocabulary Size
0.1
25
50
X 500
100
X 1000
0.0
20
200
0.25
0.30
0.35
0.40
0.45
0.50
0.55
0.25
0.30
0.35
0.40
0.45
0.50
0.55
Correlation (freq, vs. acc.)
Correlation (freq, vs. acc.)
Figure 3: Results on IN-Caps about text descriptiveness and vocabulary size. 1) Increasing text descriptiveness improves both robustness (a) and discriminability (b) of CL.P. but the tendency varies using less descriptive (template-based supervision. 2) The gap between SL and CLIP (a) implies CLIP re-bebb balances predictions, which is replicable by x subsampling the vocabulary SL trains with.

---

increasing descriptiveness given metadata of ImageNett-aptions. For the low-diversity texts, we create synthetic class-centric texts using classification templates from CLIP [68] given class names or synset [56]. The natural language-based texts are created by concatenating different types of captions (see Fig. 2), and the descriptiveness of language supervision is controlled by the number of text types used. More details are available in Appx. C.2.
Results. Fig. provide comprehensive comparison between model variants from different perspec- tives. Restricting our view to CLIP models in the first two subfigures, higher text descriptiveness results in improvements in both robustness and discriminability of CLIP, as shown by lower corre- lation (Fig. 3a) and higher overall accuracy (Fig. 3b, y-axis). On the other hand, relatively less descriptive texts show weaker results that are close to results of templated bansee based CLIP (Fig. 3a, r-axis). We see this as less descriptive texts could collapse to class-centric supervision without much additional variance. Despite this, predictions of template-bar based CLIP are still notably less biased by pre-training data than x SL (Fig. 3b), indicating other factors may re-balance CLIP's predictions.
3.3 Dynamic classification (using subsampled vocabulary) as pretext task
Setting. We note that the pretext of template-based CLIP still differs from x SL. Although both formed as discrimination tasks, the vocabulary (classes in a mini-batch) of CLIP is much smaller than SL (all classes). Take using a batch size of 1024 for instance, after deduplication, the vocabulary only contains around 600 classes (for ImageNet-Captions, If negative samples are not shared across devices, the vocabulary received by each GPU can be even smaller. In contrast, the vocabulary of SL is consistent: 1000 classes for ImageNet. Considering CLIP sees far more than 1000 classes from web-crawled dataset, the portion that CLIP's training vocabulary takes is even smaller. To isolate the influence of training vocabulary, we experiment by forming dynamic classifiers during SL training. This is done by randomly subsampling the vocabulary (candidate classes) to a smaller size during training. thus forming dynamic classification tasks similar to CLIP (see details in Appx. C.3).
Results. As shown in Fig. 3a, sampling a x smaller vocabulary notably reduces SL's prediction bias, and obtains robustness similar to template-basee based CLIP. Regarding the favorable vocabulary size, smaller ones are more effective in reducing prediction bias (Fig. 3a), and intermediate ones also improve accuracy (Fig. 3b). The preferred vocabulary size for TrrageNet-Caption is around 100.
Discussion. Our intuition of the phenomena above is that dynamic classification in some way achieves llass-level re-balancing. When the ground truth (GT) corresponds to a tail class, a small vocabulary isolates the negative impact of most head classes, avoiding bias towards them and enabling the model to focus on classifying the tail class itself. Besides, is worth noting that as demonstrated in [32, 50], optimization continues after the model's predictions reach zero error, and seeks minimum intra-class variability and maximum inter-class margin (especially larger margin around head classes). Thus when the GT is head class, this approach limits the number of negative classes and could prevent the model from excessively distorting the representations of them through over-optimization.
3.4 Data distribution (level of imbalance, web distribution shift, and intra-class diversity)
Motivation. Motivated by the findings of [27] regarding the impact of image distribution on CLIP's robustness to natural distribution shifts, our study also examines its influence on robustness to data imbalance. As shown in [77]. a higher filter threshold leads to more condensed image distribution, a result that is confirmed in Fig. 4a. We thus create LAIONet variants of different intra-class variations by adjusting this threshold. All variants in this section keep the data scale the same as ImageNet-Caption (0.45M). In addition, due to the disparity in class distribution between LAIONet and ImageeettCaattion Captions ('=freq") ) while preserving web image distribution. This variant is sampled from the full version (3. M)2 that uses threshold of 0.7. More details about datasets are provided in Appx. C.5.
also create variant that aligns with the class frequencies of ImageNet-
Results. A comparison between models trained on the aforementioned datasets is present in Fig. 4b. We find that web data is not naturally friendly for de- de-biasing, but could have made models more biased due to extreme data imbalance (comparing '=freq' with other columns). The distribution shift of web data could improve robustness if pre-trained text head is available (circles squares, last column). If not, scaling may help. Moreover, results with smaller thresholds also turn out to be more robust, indicating that higher intra-class data diversity (smaller threshold) improves robustness.
5

---

LAUORe (12r=0.7)
LA (ir- 2.5)
Rand Init head
Frazen CLIP head
Caps results
LAMONet (match freq)
LAUONet (thr00.75)
LAVoNet Cihe 0.521
Clazs Clazs frequency ( (legscale)
10
probabiityy density funetion
3.0
40
Convlaatio (freq, vs. are)
0.6
102
2.5
2.0
ImageNie scc. (top-1
35
renove dittri ghitt
intra-class diversity
102
3.5
0.4
2.0
imbolancee
10f
0
250
Ranieed classes
500
750
1000
Intraaclasss image-image CLIP sim.
0.2
0.4
0.6
0.8
1.0
0.7
0.75 0.0 0.82 =frey
0.7
0.75 0.0 0.82 =freq
LAIONet text sim. thresh.
LAIONet det sim. thresh.
(a) Distrib. of LAIONet variants (same scale as IN-Caps).
(b) Results of CLIP trained on LAIONet variants.
Figure 4: Results on LAIONet about data distribution (level of data imbalance, distribution shift, and data diversity). 1) Extreme data imbalance makes models more prone to bias (last column ws. others). 2) Distribution shift (oe vs. mm, last column) harms discriminability but could improve robustness if pre-trained text head is used. 3) Higher data diversity (smaller threshold) also improves robustness.
3.5 Data scaling (also achievable via language pre-training)
Motivation. We note that the correlations of CLIP in Fig. 3a (z-axis) are still higher than that of open-source models in Fig. lb. One key remaining factor is the scale of pre-training data (see Fig. 1b for large-scale results). Given that TongeNet-Captionss is small-scaled (see Fig. 2). experiments following are conducted on LAIONet. See Appxs. C.4 and C.5 for more details about the setting.
Rand init head
CLIP init head
Frozen CLIP head
Frozen RoBERTa head
SL w/ frozen CLPP head
b
Imagelett accuracy (top-1. %0
Corelation (freq, vs. acc.)
NC1 (intra variation
NC2 (inter class margin&)
0.
40-
0.6
20-
compacter clusters
larger margins
x
30
0.2
0.5
35
0.20
20.
0.4
20-
0.3
0.38
0.14
Train data scale (million)
Train data scale
on)
Train data scale (million)
Train data scale (million)
Figure 5: Results on LAIONet subsets about data scale and text encoder. 1) CLIP's discriminability
(a) and robustness (b) co-improve as data scales up, and can be boosted by pre-trained heads. 2) A frozen head helps CLIP preserve intra-class variation (c) while not harming margins (d), which can be lost if fine-tuned. It is also unattainable by SL even using the same head. 3) Language praitrain using CLIP is more favorable for image-text tasks than pure language modeling (e.g., RoBERTa [51]).
Fig. 5 presents the results obtained from uniformly subsampled subsets of LAIONet. These findings extend the scaling law: as data scales, ImageNet zero-shot accuracy (Fig. 5a) and models' robustness to data imbalance (Fig. 5b) improve simultaneously. We also provide a comparison between text encoders: training from scratch, initializing with pre-trained CLIP (frozen) or frozen RoBERTa [51], or fine-tuning the text encoder together. Frozen CLIP language head enables the vision model to leverage a well-established feature space as supervision, achieving better data efficiency (Fig. 5a) and robustness to data imbalance (Fig. 5b). Fine-tuning CLIP text head results in over-fitting (similar results with training from scratch), and RoBERTa does not suit the contrastive task and adversarially affects performance. Further investigation through NC- basdd metrics shows frozen heads effectively preserves intra-class variation (Fig. 5c), which is at risk of being lost when fine-tuned. Both frozen and fine-tuned heads contribute to inter-class margins (Fig. 5d), and if randomly initialized, sealing training data still can achieve improved margins. Compared to x SL, CLIP can better utilize web-crawled data and pre-trained text encoder (Fig. 5a). But note that when evaluating close-set accuracy, the data efficiency of CLIP is still much lower than SL trained on classification datasets (e.g., ImageNet).
Results.

---

3.6 Utilization of open-world concepts
Motivation. One overlooked factor in Sec. 3.5 (on IK ImageNet classes) is the existence of massive open-world concepts in web-crawled datasets. CLIP only requires weak image-text supervision and thus not bound by a pre-defined vocabulary. The open- world concepts may share useful information with close-set ones and generalization could happen when data scales up. This section presents experiments on ImageNet-Captions and YFCC-15M subsets that re- veal scaling effects of the number of concepts/classes. Results are shown in Fig. 6 and details of datasets can be found in Appx. C.S.
(freq, vs acc.)
Corr. (freq, vs #pred.)
Correlation on IN-100
IN-Caps-100
IN-Caps (10%)
IN- NN-Caps
#concept1
#daaa
b
Correlation on IN-1K
0.8
SL not utilize open-world data
Results. We present results CnaapiottCaatiion
0.6
subsets (evaluate on 100 classes) and YFCC- subsets (evaluate 1K classes) in Fig. 6 to validate this. IN-Caps-100 100 stands for a 100-class subset of ImageNet-Captionn
0.4
0.2
and IN-Caps (10%) denote 1K- class subset at the same scale as IN-Caps-100, In Fig. 6a, both SL and CLIP attain additional robust- ness from the scaling of concept and data. However, expanding the vocabulary for SL is label-expensive practice. Thus concepts other than ImageNet classes YFCC-15M do not benefit SL in Fig. 6b.
YFCC15M-CIs
#datat YFCC-154 15M
#conceptt
Figure 6: CLIP can benefit from open-world concepts. (a) Train on IN-Caps variants, and evaluate on 100 classes.
(b) Train on YFCC- 15M variants, and evaluate IK classes.
3.7 Understanding the feature distribution of CLIP pre-trained at scale
Setting. The results above have shown that the discriminability and robustness to data imbalance improve simultancously as pre-training data scales up (Sec. 3.5). Then if pre-trained on sufficient data, when does CLIP fail (Fig. 7a. what does data imbalance affect (Fig. 7a.2), and how are they reflected in the feature space (Fig. 7b)? To answer these questions, we consider 3 vision feature- related metrics (o NCI, NC2 NC2 and 2 text feature- relattd metrics (o NC2w. NC2 ) NC2 ar uses vision feature centers, and NC2w takes CLIP's text classifier as feature centers. Margins are computed as average over all other classes for NC2, and that to the nearest neighbor for "C2
NC1
NC2,
NC2g
NC2T
NC23)
MotsCLIP-2.58
MotaCLIP-2.58
Head
LAION-20
LAION-29
MetaCLIP-40OM
MetaCUP-400M
LAION-400M-
LAION-400M
YFCC-15M
VFCC-15M
CC-12M
CC-12M
LMM
LAIONet-3M
-0.2 Carr. to per class accuracy
0.2
0.2
Corr. to periclass data frequency
(b) SNE of CLIP text centers (pre-train on LAION-400 400M).
(a) Correlation between model (left), data (right), and feature statistics.
Figure 7: Inspecting CLIP's failures and effects of data imbalance from NC-based metries. 1) Fail classes of smaller-scale models (12/15M) are hardly discriminative to most classes, while larger-scale models (2 400M) only fail on some nearest-neighbor classes. 2) Data imbalance is weakly correlated with most feature statistics except NC2w, denoting denser head and coarser tail classes in text space.
Results. Cluster compactness ( NCI) does not show a strong correlation with CLIP's failures (Fig. 1a. and the frequent classes of LAION models tend to preserve more intra-class variation (Fig. 7a.2). Besides, there are some implications from the margin between class centers (o NC2).
7

---

For example, Fig. 7a.1 shows that the fail classes of smaller-scale models (12/15M) are hardly discriminative to most classes (e ar) while larger-scale models (2 400M) only fail on some nearest-neighhor classes (- NC This indicates that the failing classes already have good separation from most other classes, and the confusion primarily comes from very few hard classes. Regarding the effects of data imbalance on CLIP (Fig. 2a.2), we find strong connection to NC2w. denoting denser head and coarser tail classes in text space. I-SNE [86] of the class centers is provided in Fig. 7b for reference, and more visualizations of vision features can be found in Fig. 20.
Discussions. Though weakly correlated to the class frequency, CLIP's performance is still highly biased [87, 99]. If data imbalance is not the main cause, then what are other suspect of CLIP's failures? We hypothesize that ImageNet is intrinsically biased. The classes are not of equal difficulty [17] and some even ambiguous [6. 39, 75]. e.g., "sunglass" vs. "sunglasses". In this case, it is possible for a model trained on the balanced ImageNet to be biased [17]. and some errors are unsolvable no matter how much training data is added. Besides, CLIP leverages open- concepts in training, which are not counted for frequency but still could affect close-set performance. Moreover, such biases might be connected with CLIP's hallucination [31, 53, 92]. We believe these valuable questions to be explored. In supplement to this discussion, we also discuss CLIP's bias measured on broader sets of concepts in Appx. A.2 and the effects of data imbalance on CLIP in Appx. A.5.
4 Acquiring CLIP-level generalization
This section shows findings from CLIP's underlying mechanisms can be applied to both supervised learning (Sec. 4.1) and sulf-supe learning (Sec. 4.2) under severe data imbalance.
4.1 Data-imbalanced learning: an extreme case
In quest of the limit of CLIP's robustness to pre-training data imbalance, we create an extreme case based on ImageNet-Captions. trimming the tail classes to only one shot, or even completely zero shot (i.e,, an open-world setting). We then train models on this trimmed dataset, and evaluate performance on ImageNet regarding tail/other classes. As shown in Fig. 8, the scale of ImageNet-Captions (~0.45MM, CLIP trained from scratch also fails on tail classes when trained under severe data imbalance. Despite this, by adopting a pre-trained text encoder following Sec. 3.5, CLIP acquires open-world knowledge and demonstrates superior generalization on tail (and open- open-world) classes. Then how much can an SL. model acquire such generalization? Surprisingly, we find training it with x frozen class prototypes produced by CLIP text head is not effective. Instead, also x subsampling the vocabulary during training is necessary to achieve a similar level of generalization as CLIP.
CLIP
SL
Random Init head
Frozen CLIP head
Frozen CUP head + sub voc. (SL)
Train w/ 1-shot tail
b
Train w/ 1-shat tail
Train w/ O-shat tail
d
Train w/ O-shat tail
500
500
500
500
Number tail classes
200
200
200
200
100
100
50
50
50
50
20
20
20
20-
40
50
60
5
10
1.5
20
40
50
60
5
10 15
20
Avg head
(top-1,
Avg tail acc (top- 1, %6
Avg head & mid (top-1, Avg tail acc (top-1, %)
Figure 8: An extreme case: we train SL models to only one shot (a & b) or even zero shot (c & d), and evaluate the accuracy on the tail and other classes. CLIP with a frozen pre-trained text encoder shows superior generalization, which can be acquired by a x SL. model with x fixed class prototypes from CLIP and x vocabulary subsampling.
IN-Caps variants that have tail classes trimmed
To understand the underlying mechanisms, we present a case study on the affinity matrix between classifiers, and tail class accuracies under the zero-shot tail (50 classes) setting in Fig. 9. The affinity matrices of the classification head (see Fig. 9a, we subsample 100 classes for visualization) demonstrate that the learned tail prototypes collapse to singularity, while the class prototypes from
8

---

Classifier int)
Classifer er sim. (frozen CLIP)
Tail per-class acc, (top-1. %)
Tail per-class #predictions
1.0
0.8
0.8
voc.
0.8
0.6
60
0.4
0.4
022
0.2
20
0.0
0.0
(a) Affinity matrices the classification head.
(b) Distributions of models' per-class statistics.
Figure 9: A case study of SL. under the zero-shot tail setting. (a) SL models seek maximal margins between classifiers, and tail prototypes collapse together. Instead, CL.IP has healthier structure. (b) Using CLIP head solely is less effective, and voc. subsampling is needed for CLIP-like generalization.
CLIP maintain a healthier structure. Replacing the learned head with frozen CLIP prototypes alleviates classifier bias. However, per-class accuracies (see Fig. 9b) show that using this head alone merely effective, only small improvements are observed in very few classes, indicating that the representations are still biased. Additionally, applying vocabulary subsampling overcomes the hidden bias supervision, allows the representations to the manifold encoded by CLIP text embeddings, and generalizes to open classes that CLIP has seen in pre-training. We note that this setting shares similarities with gpen-vocabylary recognition. Surprisingly, we indeed find a similar technique (termed federated loss) used in open- vocabulary object detection (OVOD) [98], but few explorations exist the relevant literature. Our study provides a thorough analysis of this technique from another perspective, and we hope it can motivate future applications in this field.
4.2 Empowering self-supervised learning in-the-wild at scale
To show the universality of the aforementioned techniques, we also explore the application in improving self- supervisdd learning when pre-trained on imbalanced data. As discussed in [3, 61], DINO's performance is sensitive to the imbalance in web-crawled pre-training data, and thus data deduplication crucial process DINOv2 [61]. As discussed by recent study [30], the learnable prototypes of DINO (akin to the classifier of SL) may be biased to imbalanced data and many collapses (like Fig. 9a). We hypothesize that applying subsampling the prototypes may alleviate this phenomenon. Our intuition is that the operation resembles dropout and could encourage better utilization of the learned prototypes uncurated web data. Based on vanilla DINO [11], we randomly subsample prototypes (instead of using them all) during the calculation of the self-distillation loss (see details in Appx. D). All models are pre-trained for 100 epochs on LAIONet, and evaluated on the transfer learning benchmark of [40].
DINO, thus improving representations learned from
DINO variants on LAIONet vs. vanilla DINO on ImageNet
Pascalvoc2007
5.0
OxfordPets
Caltech101
CIfAR100
Flowers 102
Linear Probe A Score (%))
Birdsnap
Food101
CIFAR10
SUN397
2.5
0.0
FGVCAircraft
DTD
StanfordCars
-2.5
-5.0
Method
Vocabulary size
10.00
Vanilla DINO
1024
16384
Sub
4096
65536
-12.5
Figure 10: Transfer learning results of DINO variants pre-trained on LAIONet vs. vanilla DINO trained on ImageNet. Extreme data imbalance makes LAIONet much harder for DINO to learn transferrable representations. The vocabulary subsampling strategy effectively helps DINO alleviate such defects and generally match ImageNet-pretraing performance.

---

Results in Fig. 10 and Tab. 2 show that compared to pre pre-training on ImageNet, vanilla DINO's performance drops notably among 11 datasets out of 12. Instead, vocabulary- the gap by large margin, highlighting this technique's 's effectiveness on large-scale data in the wild. To rule out the influence of total vocabulary size (number of prototypes), we also train vanilla DINO with reduced vocabulary (16384). This model is notably weaker than that trained with subsampling (16384 for each training iter, 65536 in total), and supports the improvement's 's effectiveness.
narrows
5 Limitations, future work, and broader impacts
Limitations. Our study has focused on the robustness of CLIP-type models in relation to the data imbalance naturally raised from web data sources. We have demonstrated that our findings transferrable to the supervised and self-supervised learning setting for classification tasks. However, we acknowledge that our estimation of image-text datasets' concept frequency is based on simple rule-based pipeline, which could be prone to caption noise, multi-label, and ambiguity. Besides, CLIP models are not only employed for classification tasks, the study of leveraging CLIP for open-world detection or segmentation is the area our study does not cover. Additionally, given the nature of the web-based data sources used in our study, we acknowledge that the data may contain implicit bias or harmful information. We provide more discussions in Appx. A.
Future work. Our findings cover insights in language supervision, pretext task, data scaling, and concept scaling, but only small portion are validated in application. One direction for future work is to explore the use of language supervision and open-world data in recognition models. Besides, recent work [43] finds Adam optimizer to outperform (stochastic) gradient descent on heavy-tailed data, which can be another factor in CLIP's robustness and is worth further exploration. On the other hand, we are interested in extending our discovery to the open-world detection and segmentation tasks to see if our findings still hold under these more challenging scenarios.
Furthermore, as we have analyzed in our study, language supervision plays an important role in achieving such robustness to the data imbalance, thus are also interested in studying whether or not similar traces of generalization exist in (multi-modal) large language models (e.g.. Llama [83], BLIP-2 [45], LLaVA [48], efc.). However, despite being trained large-scale data with language supervision, recent works show that LLM/MLLMS still suffer from long-tailed training data [37, 46), and their performance is highly correlated with the frequency that corresponding knowledge appeared in training [1. 95]. This indicates that generative models might be intrinsically more prone to long-tailed data than contrastive models like CLIP, and injecting rebalancing mechanisms into the generative process could be valuable for future explorations.
Broader impacts. We provide an in-depth analysis of CLIP's robustness to data imbalance, which helps understand the effectiveness of CLIP. The techniques here are also shown to be effective for other domains (supervised learning and self-supervised learning) overcome biases in tail under- represented classes. Thus, we expect our work not to pose potential negative societal consequences but rather to improve society's 's overall equality and inclusiveness.
6 Concluding remarks
Our work starts with the observation that although web-crawled datasets share extremely im- balanced data distribution, CLIP is relatively more robust to it. Extensive studies on 1) language supervision, 2) pretext task, 3) web data distribution, 4) data scaling, and 5) open-world concepts reveal significant findings about the underlying mechanisms of this robustness. We have also demon- strated that these findings can be transferred classification and self-supervised learning methods, yielding improved generalization under pre- pre-training data imbalance. Our study uncovers key factors of CLIP's robustness to pre-training data imbalance, and provides new perspectives understand its generalizability. The insights learned are validated on tasks from extremely long-tailed supervised learning to self-supervised learning on web-crawled data. While CLIP has been a game changer in these research fields, it has long been utilized as is. Our study, instead, delved into the mechanisms behind CLIP, providing an opportunity to improve downstream tasks by leveraging the underlying mechanisms rather than relying solely on the model itself, with greater flexibility and adaptability.
10

---

Acknowledgments
This work has been supported by Hong Kong Research Grant Council Early Career Scheme (Grant No. 27209621), General Research Fund Scheme (Grant No. 17202422), and RGC Research Matching Grant Scheme (RMGS). Part of the described research work is conducted in the JC STEM Lab of Robotics for Soft Materials funded by The Hong Kong Jockey Club Charities Trust.
References
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. arXiv preprint (Xii:2240.004055. 2024.
[2] Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultancous clustering and representation learning. In The Eighth International Conference on Learning Representations, Virtual, 26 Apr-1 May 2020.
[3] Mido Assran, Randall Balestriero, Quentin Duval, Florian Bordes, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, and Nicolas Ballas. The hidden uniform cluster prior in self-supervixed learning. In The Eleventh International Conference Learning Reprexentations, Kigali, Rwanda. 1-5 May 2023.
[4] Yutong Bai, Jieru Mei, Alan L. Yuille, and Cihang Xie. Are Transformers more robust than CNNs? In Ranzato, A. Beygelzimer, Dauphin, Liang, and W. Vaughan, editors, Advances in Neural Infor- mation Processing Systems, volume 34, pages 26831 26843, Virtual, 6 14 Dec 2021. Curran Associates, Inc.
[5] Thomas Berg, Jiongxin Liu, Scung Woo Lec, Michelle L. Alexander, David W. Jacobs, and Peter N. Belhumeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 201 1-2018, Columbus, OH, USA, 23-28 Jun 2014. IEEE.
[6] Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with ImageNet? xrXiv22006.0/559 Jun 2020.
[7] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 101 - mining discriminative components with random forests. In D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, editors, Computer Vision - ECCV 2014, volume 8694 of LNCS, pages 446-461, Zurich, Switzerland, 6-12 Sep 2014. Springer.
[8] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Mathhij Douze. Deep clustering for unsupervised learning of visual features. In V. Ferrari. M. Hebert, C. Sminchisescu, and Y. Weiss, editors, Computer Vision ECCV 2018, volume 11118 of LNCS, pages 139-156, Munich, Germany, 8-14 Sep 2018. Springer.
[9] Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-training of image features on non-curated data. Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 2959-2968, Scoul, Korea, 27 Oct-2 Nov 2019. IEEE/CVE.
[10] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, volume 33, pages 9912 9924, Virtual, 6- Dec 2020. Curran Associates, Inc.
[11] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEF/CVE Inter- national Conference on Computer Vision (ICCV), pages 9650-9660, Virtual, 11-17 Oct 2021. IEER/CVE
[12] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3558 3388-556, Virtual, 19-25 Jun
2021. IEEE/CVE.
[13] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In H. D. and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of PMLR, pages 1597-1607, Virtual, 13-18 Jul 2020. PMLR.
[14] Xinlei Chen, Hao Fang. Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. arXiv:1544.003 Apr 2015.

---

[15] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vixion and Pattern Recognition (CVPR), pages 2818 2829,, Vancouver, Canada, 18- 22 Jun 2023. IEEE/CVE.
[16] Mircea Cimpoi, Subhransu Maji. Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3606-36 3613, Columbus, OH, USA, 23 28 Jun 2014, IEEE
[17] Jiequan Cui, Beier Zhu, Xin Wen, Xiaojuan Qi, Bei Yu, and Hanwang Zhang. Classes are not equal: An empirical study on image recognition faimess. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23283 23292, Seattle, WA, USA, 17-21 Jun 2024. IEEE/CVE.
[18] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hicrarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, Miami, FL, USA, 20-24 25 Jun 2009, IEEE.
[19] Karan Desai and Justin Johnson. VirTex: Learning visual representations from textual annotations. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11162-11173, Virtual, 19-25 Jun 2021. IEEE/CVE.
[20] Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. RedCaps: Web-curated image-text data created by the people, for the people. Information Procexsing Systems Truck on Datasets and Benchmarks, volume 1, Virtual, 6-14 Dec 2021.
Vanschoren and Young, editors, Proceedings of the Neural
[21] Benjamin Devillers, Bhavin Choksi, Romain Biclawski, and Rufin VanRullen. Does language help generalization in vision modcls? In A. Bisazza and O. Abend, editors, Proceedings of the 25th Conference on Computational Natural Language Learning, pages 171-182, Online, 10-11 Nov 2021. ACL
[22] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transforners for image recognition at scale. In The Ninth International Conference on Learning Representations, Virtual, 3-7 May 2021.
[23] Linus Ericsson, Henry Gouk, and Timothy M. Hospedales. How well do supe-supe models transfer? In Proceedings of the IELE/CVT Conference on Computer Vision and Pattern Recognition (CVPR), pages 5414-5423, Virtual, 19-25 Jun 2021. IEBE/CVF.
[24] Mark Everingham, Lue Van Gool, Christopher K.I. Williams, John Winn, and Andrew Zisserman. The Pascal visual object classes (VOC) challenge. International Journal of Computer Vision. 88:303-338, 2010.
[25] Lijie Fan, Dilip Krishnan, Phillip Isola. Dina Katabi, and Yonglong Tian. Improving CLIP training with language rewrites. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 15544-35575, New Orleans, LA. USA, 10-16 Dec 2023. Curran Associates, Inc.
[26] Lijie Fan, Kaifeng Chen. Dilip Krishnan, Dina Katabi, Phillip Isola. and Yonglong Tian. Scaling laws of synthetic images for model training... for now, In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 7382-7392, Seattle, WA, USA, 17-21 Jun 2024. IEEE/CVE.
[27] Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (CLIP). In K. Chaudhuri, S. Jegelka, L. Song. C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of PMLR, pages 6216 6234, Baltimore, MD, USA, 17-23 Jul 2022. PMLR.
[28] Li Fei-Fei, Rob Fergus, and Pictro Perona. Onc-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4):594 611, 2006.
[29] Samir Y. Gadre, Gabriel Ilharco, Alex Fang. Jonathan Hayase, Georgios Smymis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jicyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander J. Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. DataComp: search of the next generation multimodal datasets. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 27092 27 12, New Orleans, LA, USA, 10-16 Dec 2023, Curran Associates, Inc.
12

---

[30] Hariprasath Govindarajan, Per Sidén, Jacob Roll, and Fredrik Lindstee. On partial prototype collapse in clustering- based self-supervised learning. Submission The Twelfth International Conference on Learning Representations, 2024.
[31] Melissa Hall, Laurens van der Maaten, Laura Gustafson, Maxwell Jones, and Aaron Adcock. A systematic study of bias amplification. arXiv:2201.11706 Jan 2022.
[32] X.Y. Y. Han, Vardan Papyan, and David L. Donoho. Neural collapse under MSE loss: Proximity to and dynamics on the central path. The Tenth International Conference on Learning Representations, Virtual,
25 29 Apr 2022.
[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vixion and Pattern Recognition (CVPR), pages 770-778, Las Vegas, NV, USA, 26 Jun-1 Jul 2016. IEEE/CVE.
[34] Patrick Helber, Benjamin Bischke, Andreas Dengel. and Damian Borth. Introducing EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification. In IEEE International Geoscience and Remote Sensing Symposium, pages 204-207, Valencia. Spain, 22-27 Jul 2018. IEEE
[35] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Bue, E. Fox, and Garnett, editors, Advances in Neural Information Processing Systems, volume 32, pages 15584-1555 15595, Vancouver, BC, Canada, 14 Dec 2019. Curran Associates, Inc.
[36] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham. Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Ducrig. Scaling up visual and rision-language representation learning with noisy text supervision. In M. Mcila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of PMLR, pages 1904-4916, Virtual, 18 18-24 Jul 2021. PMLR.
[37] Nikhil Kandpal. Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of PMLR, pages 15696-15707, Honolulu, HI, USA, 23-29 Jul 2023, PMLR.
[38] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. The Fighth International Conference on Learning Representations, Virtual, 26 Apr-1 May 2020.
[39] Polina Kirichenko, Mark Ibrahim, Randall Balestricro, Diane Bouchacourt, Ramakrishna Vedantam, Hamed Firooz, and Andrew Gordon Wilson Understanding the detrimental class-level effects of data augmentation. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 17498-17526, New Orleans, LA, USA, 10-16 Dec 2023. Curran Associates, Inc.
[40] Simon Kornblith, Jonathon Shiens, and Quoc V. Le. Do better ImageNet models transfer better? In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 2661-2671, Long Beach, CA, USA, 16-20 Jun 2019. IEEE/CVE.
[41] Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3D object representations for fine-grained categorization. In JEEE International Conference on Computer Vision Workshops, pages 554-561, Sydney, NSW. Australia, 2-8 Dec 2013. IEEE.
[42] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[43] Frederik Kunstner, Robin Yadav, Alan Milligan, Mark Schmidt, and Alberto Bietti. Heavy-tailed class imbalance and why Adam outperforms gradient descent on language models. 24Xx:::400..9444 Feb 2024.
[44] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In K. Chaudhuri, S. Jegelka, I. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162 PMLR, pages
Baltimore, MD, USA, 17-23 Jul 2022. PMLR.
[45] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In A. Krause, E. Brunskill, K. Cho. B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning. volume 202 of PMLR, pages 19730-19743 Honolulu, HI, USA, 23 23-29 Jul 2023. PMLR.
13

---

[46] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao. and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In H. Bouamor, Pino, and K. Bali, editors. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292-305, Singapore, 6-10 Dec 2023, ACL.
[47] Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Under- standing the modality gap in multi-modal contrastive representation learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A, Oh, editors, Advances in Neural Information Processing Systems, New Orleans, LA, USA, 28 Nov-9 Dec 2022. Curran Associates, Inc.
volume 35, pages
[48] Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 34892-34916, New Orleans, LA, USA, 10-16 Dee 2023, Curran Associates, Inc.
[49] Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In The Tenth International Conference on Learning Representations, Virtual, 25-29 29 Apr 2022.
[50] Xuantong Liu, Jianfeng Zhang, Tianyang Hu, Hc Cao, Yuan Yao, and Lujia Pan. Inducing neural collapse in deep long-tailed learning. In F. Ruiz, J. Dy, and J.-W. van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of PMLR, pages 11534-11544, Valencia, Spain, 25-27 Apr 2023. PMLR.
[51] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. 1.Xiv:9907.11622 Jul 2019.
[52] Zhuang Liu, Hanzi Mao, Cuao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrlll, and Saining Xie. convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pateern Recognition (CVPR), pages 1 11976-11986, New Orleans, LA, USA, 19-24 Jun 2022. IEEF/CVE.
[53] Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. CREPE: Can vision-language foundation models reason compositionally? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 28 2818-2829 Vancouver, Canada, 18-22 Jun
2023. IEEE/CVE.
[54] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. fine-grained visual classification of aircraft. arXiv:1306.5157 Jun 2013.
[55] Prasanna Mayilvahanan, Thaddäus Wiedemer, Evgenia Rusak, Matthias Bethge, and Wieland Brendel. Does CLIP's generalization performance mainly stem from high train-test similarity? In The Twelfth International Conference on Learning Representations, Vienna, Austria, 7-11 May 2024.
[56] George A. Miller. WordNet: lexical database for English. Communications the ACM, 38(11)399441
Nov 1995.
[57] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. SLIP: Self-supervision meets language- image pre-training. In S. Avidan, Brostow, M. Cissé, G. M. Farinella, and T. Hassner, editors, Computer Vision Saion-ECCE ECCV 2022, volume 13686 of LNCS, pages 229,544, Tel Aviv, Israel, 23-27 Oct 2022. Springer.
[58] Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of CLIP. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 21455 21469, New Orleans, LA, USA, 28 Nov-9 Dec 2022. Curran Associates, Inc
[59] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Sixth Indian Conference Computer Vision, Graphics and Image Processing, pages 722-729. Bhubaneswar, India, 16 19 Dee 2008 IEEE
[60] OpenAL. GPT-4 technical report. arXiv:2303.0877 087744 Mar 2023.
[61] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang, Shang- Wen Li. Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.
14

---

[62] Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2Text: Describing images using 1 million captioned photographs. In J. Shawe-Taylor. R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, editors. Advances in Neural Information Processing Systems, volume 24, pages 1143-1151, Granada, Spain, 12-25 Dec 2011. Curran Associates, Inc.
[63] Vardan Papyan, X. Y. Han, and David L. Donoho, Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40)24452-24460 2020.
[64] Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong. Yanan Li, Deva Ramanan, James Caverlee, and Shu Kong. The neglected tails of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12988 12997, Seattle, WA, USA, 17-21 Jun 2024. LEEE/CVF.
[65] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3498 3505, Providence, RI, USA, 2012. JEEE.
[66] Karl Pearson and Francis Galton. Note regression and inheritance in the case of two parents. Proceedings of the Royal Society of London, 18:240-242, 1895.
[67] Hieu Pham, Zihang Dai. Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi- Vi-Ting Chen,
Luong, Yonghui Wu, Mingxing Tan, and Quoe V. Le. Combined scaling for zero-shor shot transfer learning. Newrocompuring, 555:126658,, 2023.
[68] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretehen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In M. Mcila and T. Zhang, editors, Proceed- ings of the 38th International Conference on Machine Learning, volume 139 of PMLR, pages 8748 88633, Virtual, 18-24 Jul 2021. PMLR.
[69] Vivek Ramanujan, Thao Nguyen, Sewoong Oh, Ali Farhadi, and Ludwig Schmidt. On the connection between pre-training data diversity and finc-tuning robustness. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 66426-66437, New Orleans, LA, USA, 10-16 Dec 2023. Curran Associates, Inc.
[70] Benjamin Recht, Rehecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of PMLR, pages 5389-5400, Long Beach, CA. USA. 9-15 Jun 2019. PMLR.
[71] Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang, and Tatsunori Hashimoto. Is a caption worth thousand images? A study on representation learning. In The Eleventh International Conference on Learning Representations, Kigali, Rwanda, 1-5 May 2023.
[72] Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus. Learning visual representations with caption annotations. In A. Vedaldi, H. Bischof, Brox, and J.-M. Frahm, editors, Computer Vision - ECCV 2020, volume 12353 of LNCS, pages 153-170, Online, 23-28 Aug 2020. Springer.
[73] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open dataset of CLIP- 400 million image-text pairs. wrXiv:2111.0211. Nov 2021.
[74] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski. Srivatsa Kun- durthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 25278-25294, New Orleans, LA, USA, 28 Nov-9 Dec 2022. Curran Associates, Inc.
[75] Jic-Jing Shao, Jiang-Xin Xin Shi, Xiao Wen Yang, Lan-Zhe Guo, and Yu-Feng Li. Investigating the limitation of CLIP models: The worst ports-performig categories. arXiv:2310.03324 Oct 2023.
[76] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual Captions: A cleaned, hyperymee, image alt-text datasct for automatic image captioning. In I. Gurevych and Y. Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2556-2565, Melbourne, Australia, 15-20 Jul 2018. ACL.
[77] Ali Shirali and Moritz Hardt. What makes ImageNet look unlike LAION, arXiv:2306.11 15769, Jun 2023.
15

---

[78] Charles E. Spearman. The proof and measurement of association between two things. The American Journal of Psychology, (5(1):22-00,, 1904.
[79] Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. WIT: Wikipedia- based image text dataset for multimodal multiiinuua machine learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 21, pages 2443-2449, Virtual, 2021. ACM.
[80] Bart Thomee, David Shamma, Gerald Friedland, Benjamin Flizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: the new data multimedia research. Communications of the ACM, 59 (2):64733, Jan 2016.
[81] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, editors, Computer Vision ECCV 2020, volume 12353 of LNCS, pages 776-794, Online, 23 28 Aug 2020. Springer.
[82] Yonglong Tian, Olivier J. Hénaff, and Aäron van den Oord. Divide and contrast: Self-supervised learning from uncurated data. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10063-10074, Virtual, 11-17 Oct 2021. IEFF/CVE.
[83] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothéc Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. 230. Feb 2023.
[84] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv:2307.09280 Jul 2023.
[85] Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip H. S. Torr, Adel Bibi, Samuel Albanie, and Matthias Bethge. No "zero-shot" without exponential data: Pretraining concept frequency determines multimodal model performance. arXiv:2404.04155 Apr 2024.
[86] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SEE Journal of Machine Learning Research, 986):2557922055 2008.
[87] Xudong Wang, Zhirong Wu, Long Lian, and Stella X. Yu. Debiased learning from naturally imbalanced pseudo-labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14647-14657, New Orleans, LA, USA, 19-24 Jun 2022. IEEE/CVE.
[88] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongic, and Pictro Perona. Caltech-UCSD birds 200. Technical Report CNS-TR 201, Caltech, 2010.
[89] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Q. Liu and D. Schlangen, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, 16-20 Nov 2020. ACL.
[90] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In IEEE Conference on Computer Vision and Patern Recognition, pages
San Francisco, CA, USA, 2010. IEEE
[91] Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. In The Twelfth International Conference on Learning Representations, Vienna, Austria, 7-11 May 2024.
[92] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Y. Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, Kigali, Rwanda, 1-5 May 2023.
[93] Xiaohua Zhai, Xiao Wang. Basil Mustafa, Andreas Steiner, Daniel Keysers. Alexander Kolesnikov, and Lucas Beyer. LiT: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18123- 111333,,3, New Orleans, LA. USA, 19-24 Jun 2022. IEEE/CVE.
16

---

[94] Yuhao Zhang, Hang Jiang. Yasuhide Miura, Christopher D. Manning, and Curtis P. Langlotz. Contrastive learning of medical visual representations from paired images and text. In Z. Lipton, R. Ranganath, Sendak, Sjoding, and Yeung, editors. Proceedings of the 7th Machine Learning for Healthcare Conference, volume 182 of PMLR, pages 2-25, Durham, NC, USA, 5-6 Aug 2022. PMLR.
[95] Yuhui Zhang, Alyssa Unell, Xiaohan Wang. Dhruba Ghosh, Yuchang Su, Ludwig Schmidt, and Serena Yeung-Levy. Why are visually-grouus goluunedg language models bad at image classification? arXiv preprint arXiv:2405. 15, 2024.
[96] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6):1452 1464, 2018.
[97] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. BBN: Bilateral- network with cumulative learning for long- tailet visual recognition. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 9719-9728 Virtual, 14-19 Jun 2020. IEEE/CVE.
[98] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühh, and Ishan Misra. Detecting twenty- thousand classes using image-level supervision. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, editors, Computer Vision ECCV 2022, volume 13669 of LNCS, pages 350-368, Tel Aviv, Israel, 23-27 Oct 2022. Springer.
[99] Beier Zhu, Kaihua Tang, Qianru Sun, and Hanwang Zhang. Generalized logit adjustment: Calibrating fine- tuned models by removing label bias in foundation models. In A. Oh, T. Neumann, A. Gloherson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 64663-646 64680, New Orleans, LA, USA, 10-16 Dec 2023. Curran Associates, Inc.
17

---

What Makes CLIP More Robust to Long-tailed Pre-training Data? A Controlled Study for Transferable Insights (Supplementary Material)
Xin Wen' Bingchen Zhao?
Yilun Chen
Jiangmiao Pang"t Xiaojuan Qi
'The University of Hong Kong
University of Edinburgh SShanghai AI Laboratory {wenxin, xjqi>Beee hku. hk sangjlangmiao@@jjlat org
Contents
A Extended discussions
19
A.I
What makes a good correlation indicator for per-class statistics?
19
A.2
Correlation statistics on broader sets of concepts
19
A.3
Distributional convergence of large-scale image-text datasets
20
A.4
Concept frequency estimation compared concurrent work
20
A.5 Is data imbalance not a concern for CLIP?
21
A.6 Motivation behind the choice of factors to study
21
A.7 Can CLIP achieve robust generalization to extremely rare concepts?
22
Details about class frequency estimation
22
B.1
Preliminaries
22
B.2 Obtaining class frequency statistics
23
B.3
Open-source CLIP models
23
C Details about the controlled study
23
C.I Training details
................................
23
C.2 Details about text formation in IaaeeNetCaptionn
................................
23
C.3 Details about vocabulary subsampling in SL.
24
C.4 Details about models' heads
C.5 Details about image-text dataset variants
24
C.6 Evaluation setting
25
C.7 Computing resources
25
Details about DINO experiments
25
D.I Preliminaries
25
................................
D.2
Training details
................................
D.3
Transfer learning details
26
E
Extended results
26
E.1
Examples of class distribution and CLIP periomma........................
26
E,2
Extension of Fig. 1b with per-model results
26
E.3
Extension of Fig. with language pre-training
................................
27
E.4
Extended visualizations of CLIP's multi-modal feature space ................................
27
E.5
Original numeric data of DINO transfer learning results
28
E.6
Zooming in at the class distributions (linear scale)
28
18

---

St. Acc (IN-Caps]
CLIP Acc (IN-Caps)
WPredd (IN-Capst
CLIP #Pred (N-Capp)
102
106
323
Imagehee accoreity 114--1
93
imggeees a OF 1100
330
dtradiddioo par
strodiction
Class Frequeccy (lnwar scale)
Class frequeccy (fheer seale)
Class
Class frequeery Slneer sealel
(a) Correlation statistics of models pre-trained on ImageNet-Caption.
SL ACC ILAKONeD
2
CUP Act ILAIDNeti
3
SL o Pred (LAIONeD)
CUP MPred ILAIDNeb)
or 0.50
250
Insepette accuracy (topp-. 70
Pousee E3
Peatsarts (lgg miu00
Pasmort logind nasurmar Bagesiie
Feacar's riaggncul
Spsumuch aageedad
imsgetee atruaary (taps
30-
Spea ranrs Digind
4Proolc class
60-
40-
28
5
300
16
201
WP
15"
25"
23'
101
10
10"
20"
15'
36"
DF
10"
Class frequency (ngg seate]
Class frequenry (lgg sesle)
Class frequency tlgg scale)
Class frequeryy (log scale)
(b) Correlation statistics models pre-trained on LAIONet.
Ace (YFOC-15M
2
CLIP Acc (YFCC-15M)
SL #Preed (YFOC 15MI
CUP #Piee (5FCC135M)
393
1.0-
Imuugetet accuracy 11:0p- 760
Feasen E.g
Peansal (lgg)
Featur's riggeccc
24
Imagette accurscy Itsp1,
83-
16 10
23
UP
16
22
10
207
10"
31' 131
10
30
101 165
10
10
Class Frequency Ong scate)
Class Trequenry (bng scale)
Class
frequency (lgg scale)
Clase frequenry (log soale)
(c) Correlation statistics of models pre-trained on YFCC-ISM
Figure 11: Which is a better indicator for per-class statistics? (a) For less imbalanced IN-Caps, both Pearson's 'sson' r [66] and Spearman's is p [78] can model the correlation between statistics well. (b & c) For extremely imbalanced datasets (e.g.. LAIONet, YFCC-15M, and other web datasets), Peason's 's may fail even if class frequencies are processed to log scale. In contrast, Spearman's p remains robust.
A Extended discussions
A.1 What makes a good correlation indicator for per-class statisties?
Per-class statistics, especially class frequency data, can be of different levels of imbalance. A good correlation indicator should remain robust to the changes in imbalance levels and faithfully reflect the correlation between statistics. The commonly used Pearson correlation coefficient [66] (r) does not LAIONet, and
fit this criterion. We consider three datasets in this discussion: ImageNet-Caption YFCC-15M, which have increasing levels of data imbalance. As shown in Fig. 11, Pearson's 's can model moderate imbalance like ImageNet-Captions frequencies to log scale, but can fail if an extreme imbalance is met Fig. 1c..2). In contrast, the Spearman correlation coefficient [78] (p, defined as Pearson's 's r applied to data ranks) remains robust across scenarios. We thus take Spearman's p as the default correlation indicator used in this paper.
high imbalance like LAIONet if processing the
A.2 Correlation statisties on broader sets of concepts
Results in the main paper only consider the distribution of concepts/classes in ImageNet- IK. this discussion, we also consider the concept sets of broader datasets, including CUB [88], Food-101 [7].
19

---

CC-12M
YFCC-15M
LAION-40OM
MetaCLIP-40OM
LAION-28
MetaCLIP-.58
a
b
CUB
CUB
Food101
Food101
OxfordPets
OxfordPets
Flowers102
Flowers10Z 02
Places365
Places365
EuroSAT
EuroSAT
DTD
DTD
-0.4
0.2
0.0 0.2
0.4
-0.4
-0.2
0.2
0.4
Correlation to per-class accuracy
Correlation to per-class #prediction
Figure 12: Correlation statisties of CLIP evaluated broader sets of concepts. Models pre-trained at scale (2 400M) remain robust on most datasets except fine-trained (e.g., CUB and Flowers) and dopain-specific ones (e.g., EuroSAT). These data might be relatively rare on the web have significant gaps with other data, thus hard to benefit from scaling or generalization from existing data.
Oxford-III Pets [65], Flowers- 102 [59], Places365 [96], EuroSAT [34], and Describable Textures (DTD) [16]. Pre-trained CLIP models' correlation statisties these concept sets shown in Fig. 12. Models pre-trained at scale (2 400M) remain robust on most datasets. However, some fine-trained (e.g., CUB and Flowers-102) and domain-specift (e.g., EuroSAT) datasets tend to be harder to learn and easier to bias. These data might be relatively rare on the web and can have significant gaps with other data formats (satellite images are relatively uncommon), thus hard to benefit from scaling or generalization from existing data.
A.3
Distributional convergence of large-scale image-text datasets
Fig. the main paper has illustrated qualitatively that the class distributions of large-scale image-text datasets are roughly shared (correlated). Here, we also provide quantitative results about the correlation coefficients between the class distribution of different image-text datasets Fig. 13. Under most concept sets, the correlation is high and supports claim: there exists distributional convergence across large-scale image-text datasets. Results MetaCLIP [91] variants are relatively less correlated, which might be due to the re-balancing operation in the curation process.
A.4 Concept frequency estimation compared to concurrent work
Our estimation of concept frequency is based on simple rule-based pipeline (see details in Appx. B. B.2). which could be prone caption noise, multi-label, and ambiguity. A concurrent work by Parashar et al. [64] finds concept synonyms using ChatGPT [60], and estimates the class frequencies of each caption using Llama [84]. These advanced techniques may produce more accurate class frequencies. In Fig. 14, we provide the correlation coefficients between our estimations and the results of [64]. The high correlation across most datasets implies an agreement and verifies validity of our estimations. There is an exception for DTD [16], in which class names are about descriptive textures. This is more abstract than natural concepts and can more semantically ambiguous [64], and require more sophisticated designs in frequency estimation.
20

---

ImageNet
b
CUB
Food101
OxfordPets
CC-12M
1
1
YFCC-15M
0.92
1
0.7
-0.87
0.64
1
4DDM
0.91 0.75
1
00.74
0.85
-.88
0.85
0.75
0.51
1
MetaCLP-400M
0.76
0.63
0.76
0.5
0.49
0.52
0.68
0.63
0.67
0.63 0.68
0.77
1
LAION-28
0.53
0.77
0.99
0.76
0.74
0:85
0.97
0.52
0.89
0.86
0.93
0.66
0.82 0.55
0.94
0.8
MetaCLIP-2.55
0.72
0:88
0.77
0.75
0.5
0.49
0.52
0.52
0.68
0.63
0.57
0.66
0.65
0.55
0.76
0.98 0.8
Flowers102
Places365
g
EuroSAT
DTD
CC-12M
YFCC-15M
0.91
0.92
4DOM
MetaCLPP40OM
LAION-28
0.9 0.35 0.99
0.97
0.91
0.92
0.85
0.91
MetaCLIP-2.58
0.73 0.82 0.83
0.82
0.77
0.89
0.9
0.89
CC-12M
VFCC-15M
LAION-400M
MetacLiP-4N 400M
-00M
LAION-2B
MetaCLIP-2.5 58
CC-12M
YFCC-15M
LAION 400M
MetaCLip 400M
LAION-20
MetacLiP-2.88 58
CC-12M
YFCC-15M
LAION-400M
MetaCLIP 400M
LAKON-28
MetacLiP-2.N
585
CC-12M
VFCC-15M
LAION-100M
MetaCLi LP 400M
LAIDN-28
MetacLIP-255B
Figure 13: Correlation between class frequency statistics of different pre-training datasets under
different concept sets. There is convergence of data distribution over large-scale image-text datasets.
ImageNet
CUB
Food101
Flowers102
EuroSAT
DTD
Parashar et al.
LAION-400M
0.83
0.83
0.93
0.9
0.9
0.9
0.8
0.79
0.86
0.86
0.48
0.48
Farashar
Parashar
Parashar
Parashar
Parashaa
LAION-2B
0.83
0.84
0.91
0.91
0.92
0.79
0.88
0.5
LAION 400M
LAION-28
LAION 400M
LAION-28
LAION-400M
LAION-28
LAION-400M
LAION-28
LAION-400M
LAJON-28
LAION-400M
LAJON-28
LAION
Ours
Ours
Ours
Ours
Ours
Ours
Figure 14: Correlation between class frequency statistics of our estimations and concurrent results of Parashar et al. [64]. There is an agreement on most concept sets except DTD [16], which is about descriptive textures and can be more semantically ambiguous [64].
A.5 Is data imbalance not concern for CLIP?
As illustrated in Figs. 1b and 5, the discriminability and robustness to pre- pre-training data imbalance improve simultancously as data scales up. But neither does mean CLIP is unbiased (see discussions in Sec. 3.7), nor does it indicate CLIP is absolutely robust data imbalance. In Fig. 15, plot binned results of CLIP following Parashar et al. [64]. Looking at the average trend, the tail classes still of inferior performance. However, note that the standard deviation is high, indicating there are still many good-performing tail classes. Moreover, the figure also verifies CLIP is more robust than SL (Fig. 15a), and the harm of data imbalance alleviates as data scales up (Fig. 15b).
A.6 Motivation behind the choice of factors to study
The design of our study is largely influenced by [27]. which is among the first to study data's effect on CLIP's robustness. After ruling out the effects of language supervision and data distribution, we found there is still a notable gap between CLIP and SL in Fig. 3. We then exhausted every factor we could to align details between models, and finally found the pretext task of dynamic classification to be a key factor, which could implicitly di- classifiers, and reproducible by applying vocabulary subsampling. Besides that, we also considered other factors like the architecture of vision backbone and text backbone, vision pre-training, stronger data augmentation, larger batch size, and test-time prompts, and did not find noticeable effects. Additionally, we looked into the properties of the dataset instead of models and found that web data had mixed effects. Further, we extend the scaling law of CLIP and find open-world data to be an effective factor.
21

---

(YPCC-15M)
(YFCC-15M)
MPree (YFCC-15M)
(YFOC-15M)
122
231
200
Bet anrradctionn class
Imagehet
40
trequerry
trequeccy
frequency
frequency log
(a) Binned statistics models pre-trained on YFCC-15M (avg = std).
O.I Acc (LAOON-DDDM)
CLP Acc. (LAION-28)
CLIP alted (LAICN-100M)
CLIP alred (LAICN-28)
222
Irsipptte accumity 0ap-
03-
30
2
Glaas frequenicy (binned, log scal)
Class frequency (unreed, log scale)
Class frequency Ibimed, log scale)
Class frequency (baned, log scale)
(b) Binned statistics of models pre-trained on LAION 400M and LAION-2B (avg std).
Figure 15: CLIP can still be biased by pre-training data. It is relatively more robust than SL (a), and the bias reduces to extent data scales (b a), but the tail classes still underperform.
A.7 Can CLIP achieve robust generalization to extremely rare concepts?"
We indeed observe many. For example, among IK ImageNet classes, 29 classes appear in YFCC-15M less than 10 times, 20 classes appear less than 5 times, 6 classes appear 1 time, and 2 classes do not appear. Within these classes, CLIP trained accordingly from scratch has 2 50% accuracy on 12 classes. We provide detailed statistics in Tab.
Table 1: Results of the tail classes on YFCC-15M.
Freq. Acc.
5
3 2
44 98
42 62
88 94
16
82 44 72 22 48
The names of last-5 classes "potter's wheel", "Sussex Spaniel", "Curly-coated Retriever". "Kuvasz", and "Dandie Dinmont Terrier". We note that although our frequency calculation tries to maximize recall (e.g., matches class names to captions bag-of-words, and synsets), there may still be cases missed by us. Nevertheless, the results verify CLIP as good few-shot learner.
Besides YFCC-15M, we also provide examples of LA1ON-400M and MetaCLIP-400M in Fig. 17.
B Details about class frequency estimation
B.1
Preliminaries
Contrastive Language-Image Pre-training (CLIP). Taking paired image-caption data as input, the pretext task formulated as cross-modal contrastive learning task that discriminates the paired text from large batch of negative samples, and vice versa. After early explorations [19, 72, 94], emergent performance in representation learning, zero-shot evaluation, and distributional robustness was achieved by CLIP [68] and ALIGN [36] through training on datasets at unprecedented scale. Follow-up works include BASIC [67], LiT [93], BLIP [44], SLIP [57], etc. Without loss generality, this study takes special interest in CLIP.
Image-text datasets. Web-crawled image captioning data are typically formatted as image-text pairs, which can be crawled from the web. Existing works provide wide range of options across
22

---

scales, including MS-COCO [14]. CC-3M [76] and 12M [12], 1FCC-100M [80] and 15M [68]. WIT [79], SBU [62], RedCaps [20], LAION-400M [73] and 2B/5B [74], and MetaCLIP [91], etc.
This work considers those with both metadata and pre-trained CLIP publicly available, ie., CC-12M,
YFCC-15M, LAION-400//25 and MetaCL.IP-400//2.55.
B.2 Obtaining class frequency statisties
This study specifically examines the classes of ImageNet [18], which encompasses 1K common object categories. To obtain the class distribution on image-text datasets, we follow the common practice [27, 77, 91] query captions with class names and their WordNet [56] synset. In implementation, we also loosen the sub-string matching condition to set-level matching (overlooking the order of words) for a higher recall, and manually introduced negative words (e.g., 'vehicle', 'truck' for class 'ram', 'bird', and 'wing" for class 'crane') " to reduce false positives. Besides, we normalize letters to lowercase, remove non-letter and non-number symbols, and lemmatize words nouns. For MetaCLIP, which provides readily available distribution of 500K concepts, we simply summed up the statistics of target concepts (classes). And for other datasets, we ran the search over all captioning data.
B.3
Open-source CLIP models
The models are collected from the models of OpenCLIP [15]. We select models that have captions metadata of the pre- tre-training dataset publically available, and restrict the backbones to ResNet [33], ConwNeXt [52]. and VIT [22]. The remaining set comprises 41 models covering different model architectures (6 ResNets, 8 ConvNeXts, and 27 ViTs), model scales (ResNet-50/101, CenyNeXt- B/L/XI,, and ViT-B/L/H/G), data scales (from 12M to 2.5B), training schedules, and optimization techniques. An overview of the results of these models is provided in Fig. 18.
C Details about the controlled study
C.1 Training details
Our training settings follow the common practice in [27], CLIP experiments utilize cross-entropy losses and the AdamW optimizer. The initial learning rate is set to 0.001, and cosine-annealing learning rate schedule with 500 warmup steps is employed. The hyper-parameters for AdamW are 31 0.9, B2 0.999, and e 10- The batch size is set to 1024. Model training lasts for 32 epochs. We also tried training 90 epochs to match that of SL but found 32 epochs is enough for convergence and longer training has no notable benefit.
SL. models are trained using SGD with Nesterov momentum for 90 epochs. The weight decay is set to 0.0001, momentum to 0.9, and batch size 256. The initial learning rate is 0.1 and is decayed by 0.1 at epochs 30, 50, and 70.
To maximally align details with CLIP, both methods adopt the slightly modified ResNet structure as in [68]. The augmentation pipeline is also kept consistent: random resized crop to size 224 with a scale range of (0.9, 1.0). followed by normalization with a mean of (0.48145466, 0.4578275, 0. 0.40821073). and a standard deviation of (0.26862954, 0.26130258, 0.27577711). Note that this data augmentation pipeline is notably weaker than those commonly used by SL.
C.2 Details about text formation in ImageNet-Captions
For template-based captions, the caption of an image is generated using a randomly sampled template from 80 class templates provided in [68], e.g., "a photo of a [class]". If synsets are used, the class name [class] is also randomly sampled from its synsets. For natural-language captions, we refer Fig. (upper) for an example image and corresponding text metadata (including title, description, and tags). More examples can be found in Fig. 3 of [27]. The way captions are created simply by concatenating metadata together with spacing. E.g., if the [title] is "A phone call and night". and the [description] is "I might have thing with telephones... ", then the resulting caption [title description]- "A phone call and night might have thing with telephones... This follows the practice of [27], and is also the way CLIP [68] curates caption data from YFCC-15M.
23

---

C.3 Details about vocabulary subsampling in SL
The training vocabulary refers to the label set that a model classifies at a specific training iteration. Given mini-batch of samples, a minimal label set is formed as the union of all GTs in this mini-batch, If the expected vocabulary size not met, we additionally sample classes from the remaining, and the probability class is selected is determined by the frequency of the corresponding class in the pre-training data, Note that the sampling performed at the class level, which differs from the sampling strategies in long-tail learning that are done at the sample level. We also tried uniform sampling, i.., treating each class with equal probability, which yielded slightly weaker results.
Discussions. For SL, vocabulary subsampling refers to randomly reducing the size of candidate classes (akin to dropout on the classification head) when classifying an image during training. 1) Regarding how it works, Fig. 3a (ı-axis) shows it effectively reduces the model's predictions' correlation to class frequencies, a key indicator of classifier's bias. 2) Regarding why this technique can de-bias classifiers, our intuition is that this plays similar role to dropout: the classifier is regularized to put equal importance on all classes. Biases still cxist in the subsampled classes, but the gradients cancel out each other during training. 3) Regarding why frequency-basse sampling works better than dropping all classes with equal probability, we hypothesize that the dropping operation can de-bias the classifier regardless of how classes are selected, and sampling by frequency is more helpful for representation learning. The intuition comes from the finding in long-tail learning that resampling data by inverse frequency helps dias classifier, but harms representation learning [38, 97].
C.4 Details about models' heads
For CLIP experiments, the text encoder is trained from scratch by default. If the text encoder uses frozen CLIP, this means the text encoder is initialized by the pre-trained CLIP weights from [68]. During training, the parameters of the text encoder remain unchanged. In the CLIP init setting, after initialization, the text encoder is also fine-tuned in the training process. Further, for RoBERTa experiments, we follow the implementation of [15] and replace the text encoder with pre-trained RoBERTa [51] available on HuggingFace [89]. This is kept frozen during training, as we found fine-tuning it results NaN loss.
For SL experiments, we replace the commonly used linear classifier with a prototypical classifier to better follow CLIP's structure. This means the bias term in this linear layer is omitted, and both the feature from the backbone and the classificr's 's weight are ég fo-normalized, thus weights in the linear layer can be viewed as set of prototypes (feature vectors). To facilitate optimization, a learnable scaler with maximum scale of 100 is added as CLIP [68] to upscale logits. For the setting using fixed prototypes obtained from CLIP, we format each class to sentence using the template "a [c] [class]". feed them to the text encoder of a pre-trained CLIP, and keep the output class- wise text features as SL model's 's classification head/prototypes.
C.5 Details about image-text dataset variants
ImageNet-Captionn subsets. Starting from the original ImageNet-Captions [27], we take only image-text pairs that correspond to the 100 classes of Tian al. [81], thus obtaining 100-class subset called SaageNet-CCationss-O00 construct a subset that is of the same scale as Ceageeeet-Catiinns 100 but with the same number of classes as ImageNet-Capiee same scale of
Besides, we randomly sample from ImageNet-Caption and
This subset is called ImageNet-Caption (10%). Note that it is of the 100, and not necessarily 10% of IaageNet-Captions.
LAIONet variants. LAIONet [77] is a subset of LAION-400M [73] created by matching between ImageNet class synsets and captions. Items with low CLIP text similarity between the caption and class definition are filtered out to reduce label noise.
10a
subsample Rabi
100%
12.5%
20a
50%
6.25%
25%
3.125%
Our reproduction sets 0.7 as the default threshold, and 3.26M images are successfully crawled. Experi- ments in Sec. 3.4 consider LAIONet variants filtered with different text-definition similarity thresholds: 0.7, 0.75, 0.8, 0.82, and the sizes of corresponding LAION- 400M subsets are originally 3.26M, 1.93M,
102
207
200
400
600
800
1000
Classes ranked frequency of LAIDNet (threshold==.77
Figure 16: Distribution of LAIONet subsets.
24

---

0.88M, and 0.58M. We then randomly subsample them to be the same scale as ImageNet-Captionn (0.45M). Besides, the variant that matches the class distribution of ImageNet- Captionss sampled from the 3.26M version, and the scale is also kept the same as ToageNet-Captionn experiments in Sec. 3.5 use LAIONet subsets randomly sampled from the 3. 3.26M version (threshold set to 0.7), at the portion of 1/1, 3/2, 3/4, 1/s, 1/16, and 1/32, respectively. The distributions of these randomly sampled subsets are shown in Fig. 16.
In addition,
CC-12M-CH and YFCC-15M-CI, These are classification subsets of CC-12M and YFCC-15M that have corresponding class labels of 1K ImageNet classes for each image. The curation process follows Fang et al. [27], except that we allow class name matches to be not case-sensitive. In comparison to LAIONet, is simply substring matching without filtering. The resulting datasets are at a scale of 3.48M
and 2.90M PEMCC-15M-CIiy respectively.
C.6 Evaluation setting
Unless otherwise specified, the evaluation of models is all performed on ImageNet validation split. For CLIP, the default zero-shot classification setting applied. That is, each class is embedded as an average vector of text features produced using 80 class templates provided in [68]. Then for both CLIP and SL., the predicted class that of the nearest neighbor class prototype.
C.7 Computing resources
Experiments are conducted NVIDIA A100 GPUs. Each CLIP and SL training experiment runs on 4 GPUs in parallel, and there are roughly 400 experiments (data points) for the controlled study.
D Details about DINO experiments
D.1 Preliminaries
Self-supervised learning from pseudo-labels. It natural to extend SL to self-supervised settings for representation learning, long pseudo-labels are available. Earlier work [8] applies k-means clustering to deep features and takes cluster assignments as pseudo-labels. Following works [2, 10] reform pseudo-labeling as optimal transport and solve it with the Sinkhorn Knopp algorithm. This is then simplified by DINO [11] with centering and sharpening operations on the model's predictions, and extended soft labels (thus called self- eeff-disilllation instead of self-labeling).
Knowledge DIstillation with NO labels (DINO). DINO [11] is a discriminative self-supervised visual pre-training method. The pretext task is formulated self-distillation: enforcing the student model's predictions to be close to teacher models' soft pseudo labels. The input to two models are random augmented views of the same image, and the teacher model updated as the exponential moving average of the student model (also called "mean teacher"). DINO learns set of prototypes (feature vectors) as the classification head, and is used by student and teacher models produce logits and pseudo labels. Since the prototypes resemble classification head, the aforementioned vocabulary subsampling strategy can also be similarly applied to DINO.
D.2 Training details
The training details follow the suggested practices of DINO [11] for training ResNets. That is, train using SGD optimizer with a base learning rate of 0.03, and fixed weight decay of 0.0001. The scale of global crops is (0.14, 1), and the scale of local crops is (0.05, 0.14). Other apeer-parameters are kept as default. We use the ResNet-50 backbone with the same structure as Radford et al. [68], and train for 100 epochs with batch size of 1024.
The last layer of DINO's projection head is equivalent to a set of prototypes, thus it is natural to integrate the techniques experimented to be valid on classification models. We keep the total number of prototypes to 65536 as default.
For vocabulary sampling-bass based DINO, we subsample the same set of prototypes for the teacher and student models and compute the self-distillation loss on this restricted prototype set. The vocabulary (prototype set) is shared in mini-batch, and different across training iterations.
25

---

D.3 Transfer learning details
Datasets and metrics, We test models' transfer learning performance on the benchmark initially proposed in [40], and adopt the implementation from [23]. The datasets in this benchmark in- clude: Food- 101d-101 [7], CIFAR 10/100 [42], Birdsnap [5], SUN397 [90], Stanford Cars [41]. FGVC Aircraft [54], PASCAL VOC 2007 [24), Describable Textures (DTD) [16], Oxford-IIIT Pets [65], Caltech- 101 [28], and Flowers- 102 [59]. The evaluation metric is mostly top-1 accuracy, with excep- tions of mean per-class accuracy on FGVC Aircraft, Oxford-IIIT Pets, Caltech-101, and Flowers-102,
and 11 1ppontt mAP PASCAL VOC 2007.
Linear probing. Image features are extracted from the backbone of the teacher model following [11]. Then following [23], we train an lg-regularized multinomial logistic regression classifier on frozen features extracted from the backbone. The model is optimized using L-BFGS on the softmax cross- entropy objective. No data augmentation is applied, and the images the short size using bicubic resampling and center-cropped to 224 x 224. The ez-regularization are searched from 45 logarithmically spaced values between 10-6 and 10'.
resized to 224 pixels along for
E Extended results
E.1 Examples of class distribution and CLIP performance
In Fig. 17, we provide an example of the distribution of subsampled classes and per-class zero-shot accuracy of CLIP (ViT-B/32) pre-trained on LAION-400M and MetaCLIP-400M accordingly. The head classes are easy be found the web, e.g., "T-shirt", "mobile phone", "throne", and "goose", etc. In contrast, the tail classes are dominated by fine-trained biological concepts, ranging from "barn spider". "earth star fungus", to "gyromitra", Collecting such data is hard and requires expert knowledge. Despite this, we find both models can achieve good performance on some tail classes.
Prebrain cass hrquency (ranked by LAION 400M, carple)
20¹
LAION-40OM
MetaCLIF-4DOM
Sasaaaa pan pied accura % licegld
30t
WWW
800
-60
10t
307-
20
vey ooe da to vse tite
aie Tatiit pornen
prond Eool
freond
Sertin
Grette swiss countaain Suunt phraui
honter
joy terried
spoler Meankay Doj
hen of weeds cottiee
come yrue
Permant Sharmaired
her pardin mod
Gedfrey's
Entietasche velow
Figure 17: Examples of the distribution of subsampled classes (bar plot), and per-class zero-shot accuracy (line plot) CLIP (ViT-B/32) pre- pre-trained accordingly (. LAION-400M and MetaCLIP. 400M). Both models show weak correlation between class frequency and accuracy.
E.2 Extension of Fig. 1b with per-model results
In supplement to the analysis in Fig. 1b where results of CLIP are averaged by the dataset it trains on, we provided more detailed results of CLIP in Fig. 18. Besides zero-shot classification results on ImageNet [18], Fig. 18 also provides results evaluated ImageNetV2 [70]. Results are consistent.
26

---

ImageNet
ImageNetV2
Dataset
0.2
0.2
CC-12M
Corr. (freq. vs. #pred.)
Corr. (freq, vs. #pred.)
YFCC-15M
LAION-400M
MetaCLIP-40OM
0.1
0.1
LAION-2B
MetaCLIP-2.5B
Backbone
ResNet
0.0
0.0
ConvNeXt
VIT
Acc (top-1,
0.0
0.2
0.4
0.0
0.2
0.4
Corr. (freq. vs. acc.)
Corr. (freq. vs. acc.)
Figure 18: An overview of the correlation between open-source CLIP models' per-class accuracy, and prediction distribution with pre-training data's class frequency. The weak correlation to sample frequency is consistent whether evaluated on ImageNet [18] or ImageNetV2 [70].
E.3 Extension of Fig. 3 with language pre-training
supplementary to the analysis in Fig. 3, which is conducted under the setting that models are trained from scratch. Here we also provide the results that all models are trained using frozen CLIP text encoders/heads in Fig. 19. We find that the results are generally consistent with those in the main paper. In addition, we find language pre- pre-training provides shortcut to models and allows them to leverage language supervision (CLIP) and dehiased pretext tasks (SL. with higher effectiveness. This is supported by the sharper slopes in (a, blue line) and (b, green line) in comparison to Fig. 3.
Prediction Bias Perspective
Overall Accuracy Perspective
Model
x
better
better
Correlation (treq. #pred.))
smaller vocabulary
+
CLIP Text Type Templates
0.5
55
Synset Templates
0.4
Overall BEC [top-1,
50
Deseription Only
Only
45
Tags Only
0.3
40
Title Description
Tags Desorittion
Title Tags
Full Text
higher text descriptiveness
SL Varabulary Size
0.1
25
100
X 1000
0.0
20
200
0.25
0.30
0.35
0.40
0.45 0.50 0.55
0.25
0.30
0.35
0.40
0.45
0.50
0.55
Correlation (freq. vs. acc.)
Correlation (freq, vs. acc.)
Figure 19: Results on IN-Caps about caption diversity and vocabulary size. Both CLIP and SL use frozen text encoders/prototypes from pre-trained CLIP. The trends are mostly consistent with Fig. In addition, the models using template-basee supervision are (a) less biased and (b) show better accuracy than the training-from foom--rrraccc counterparts Fig. indicating the knowledge in language pre-training to be obtained by CLIP. This also holds true for SL and natural language-supervise CLIP, as supported by shaper slopes in (a, blue line) and (b, green line).
E.4 Extended visualizations of CLIP's multi-modal feature space
In supplement of Fig. 7b, we also plot the vision feature centers and corresponding sample features of some classes Fig. 20. Results are produced by a CLIP ViT-B/32 model pre-trained on LAION- 400M, and obtained by inferencing are plotted separately due to the modality gap (despite being the same feature space) [47]. Fig. 20a shows the features of images from some subsampled classes, and corresponding vision feature centers.
the ImageNet validation split. Note that vision and text features
27

---

Hebd
samples
Tail
(a) CLIP vision samples and centers.
(b) CLIP vision
(c) CLIP text
Figure 20: t-SNE visualization of samples encoded by CLIP vision/text encoders in the multi-modal feature space (on ImageNet validation set). (a) Images encoded by CLIP vision encoder, and their class-wise mean features. Classes are subsampled. (b) Vision feature centers of all ImageNet classes. (c) Class templates encoded by CLIP text encoder, the same as Fig. 7b. Vision and text features are plotted separately due to the modality gap (despite being in the same feature space) [47].
In coherence to results in Fig. 7a.2, there is not a clear tendency on whether head or tail classes form compactor clusters. addition, Fig. 20b and Fig. 20c show the vision and text feature centers of all ImageNet-IK IK classes, where head and tail classes are highlighted. The vision feature centers are produced by averaging sample features by classes, and the text feature centers used by CLIP, as described in Appx. C.6. The margins between tail classes encoded by the vision encoder are notably smaller. In contrast, tail class centers produced by the text encoder are better separated. This phenomenon might be connected with the modality gap [47]. and is of research value for future explorations.
of the classifier
E.5 Original numeric data of DINO transfer learning results
In Tab. 2, we provide the original numeric data used to obtain Fig. 10 for reference.
Table Linear probing evaluation results of DINO variants pre-trained on LAIONet for 100 epochs. Extreme data imbalance makes LAIONet much harder for DINO to learn transferrable representations, and vocabulary subsampling strategy effectively helps DINO overcome such defects.
Dataset
Voel
Airer
Birds
C101
Cars
CF10
CF100
DTD Flower
Food
Pets
SUN
VOC
Avg
Results of vanilla DINO
ImageNet 65536
27.0
37.1
82.3
23.6
86,4
62.9
68.7
80.8
55.8
66.4
57.0
81.6
60.8
LAIONet
16384
29.7
28.2
78.2
22.5
83.6
60.7
67.9
78.3
49.5
55.0
55.9
76.1
57.1
LAIONet 65536
26.7
24.6
77.8
24.6
83.5
60.0
67.1
79.3
48.6
55.5
55.4
77.3
56.7
Results of DINO vocahulary sampling
(65536
prototypes in total)
LAIONet
1024
30.8
27.2
78.6
23.8
83.9
61.4
68.1
80.5
50.7
57.7
56.0
77.2
58.0
LAIONet 4096
30.3
30.1
78.9
24.8
84.6
63.4
69.5
77.7
53.3
61.0
78.6
LAIONet 16384
32.2
31.2
79.4
25.2
85.4
63.9
70.2
79.1
54.3
62.2
57.7
79.0
60.0
E.6 Zooming in at the class distributions (linear scale)
To provide a clearer image of the imbalanced class distribution of pre-training datasets, we show zoomed-in version of Fig. la with linear scale in Fig. 21. Also, we see that MetaCLIP does successfully alleviate the dominance of head classes. But note that unfortunately. all datasets are still extremely imbalanced, and how improve models' robustness still to be explored.
28

---

02
04
8.6
9.8
1,0
1.2
1.4
0.25
0.30
0.75
1.00
1.25
1.50
135
X1.0
50
125
McIa LAJON-400M
MetaCLLP
YPCC-15M
LAION-28
200
1.0
3.5
x10
110
410
225
300
375
475
550
05
18
1.5
20
100
35
4.0
0.5
18
15
20
25
3.0
410
X10
X10
625
723
R00
800
X30
875
900
975
Figure 21: A zoom-in version of Fig. la showing class frequencies (linear scale) ranked by LAION- 400M. An imbalanced class distribution is shared across datasets.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
This paper's contributions and scope are reflected.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Discussed Sec. 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
30

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Full implementation details provided in Appxs. B to D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [Yes]
Justification:
Code and data can be accessed via attps://github.com ctt.:/ccccCCCCMMCc
clip- beyond-tail.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
iccy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/ cec/public/guuii
wwwsssslicccyyii for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Full implementation details provided in Appxs. B to D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Figures are plotted with 95% confidence intervals or standard deviations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
32

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Provided in Appx. C.7.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttts://eeurpp.... http httpppl//tthccsuuideeee.//.ppe ines?
Answer: [Yes]
Justification:
This paper conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
An impact statement is provided in Sec. 5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The assets cited and corresponding licenses are respected.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
34

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "RSw5Ymp8Hk", "reviewer": "Reviewer_tULz", "summary": "CLIP models are known to exhibit good robustness and high quality representations compared to supervised models. In this paper, the authors study CLIP models through the lens of the distribution of concepts in the underlying training datasets. The authors find that CLIP appears to demonstrate better robustness to data imbalance compared to supervised models. The authors conduct a detailed study the analyze various factors influencing robustness, and show that applying a dynamic classification task inspired by CLIP training improves robustness in supervised models.", "questions": "This paper describes several of the key mechanisms that influence model robustness. Besides the proposed training technique for supervised models, what recommendations would the authors propose for improving the next generation of self-supervised/CLIP models?", "limitations": "The authors have a section that describes the limitations of the work (Appendix F)", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is well motivated and original. The unusual robustness of CLIP is well-known, but its mechanism are not well understood\n- The problem is clearly stated and the experimental analysis is clear.\n- The datasets used are well-motivated and likely provide high quality data (especially ImageNet-Captions) for their experiments\n- The experimental setup is sound, i.e. models are trained with the same backbones, data augmentation, and standard training techniques.\n- Figures are presented well\n- The paper is overall well-written", "weaknesses": "- A major experimental claim is that CLIP models exhibit a lower correlation between ImageNet class accuracy and frequency. However, this claim makes the assumption that the frequency of these classes in the training data is accurately measured, which may not be true for LAIONet, which is too large to verify. If rare classes are undercounted, which could affect the correlation.\n- (Line 140) The statement alf a batch size 1024 training run having something like 600 imagenet classes is probably roughly true, although large models trained in industry clusters likely have a much larger batch size, and closer to 1000 classes.\n- When applying the proposed class sub-sampling training method (Figure 10, section 4.2), the results are only stated qualitatively as narrowing the gap. However, a quantitative analysis would be more appropriate. In general, this proposed training technique seems very interesting, and I would appreciate seeing more significant analysis of how this could be used.\n- The authors state that their method “uncovers the mechanisms behind CLIP’s generalizability’ (Line 12). However, while I agree that their experiments find quality evidence for several key relationships that are involved in generalizability/robustness, it is too generous to say that they have fully uncovered these mechanisms, or that they have been characterized in detail. I believe the authors should soften these claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer tULz,\n\nThank you for your time and efforts in helping us improve our paper! We hereby answer your questions and resolve your concerns as follows:\n\n> W1: Frequency of classes in LAIONet may not be accurately measured\n\n1. Thanks for pointing this out! This is why we tried to study as many factors on IN-Caps as possible. However, it is still expected that we verify these factors on other datasets to better support our findings (as requested by Reviewer XjUF), and provide a complete discussion on factors not supported by IN-Caps (eg, data distribution and data scale), thus facing an inevitable trade-off. Accurate concept frequency estimation of web datasets is an emerging topic and we believe it is valuable for future research.\n2. Regarding LAIONet, the authors are aware of the label noise and explored applying a filter. They empirically found this to be effective in overcoming such noise (see Fig. 11 in LAIONet’s arXiv preprint). When replicating LAIONet, we are conservative on the choice of the filtering threshold, trying to make the selected images have highly confident labels.\n3. We also explored an even simpler strategy: set-based substring matching (detailed in Sec. B.2). We applied this strategy to estimate the frequency in large-scale datasets (eg, LAION-400M). In comparison to advanced estimation techniques using ChatGPT and Llama2 (Parashar et al.), our method shows very consistent results (correlation of 0.8-0.9, see Sec. A.4 for details).\n\n> W2: Industry-trained CLIP have much larger batch sizes, thus all imagenet classes might be covered\n\nPlease note that the industry trains on web datasets that cover way more concepts. Our setting of batch size 1024 considers IN-Caps that cover 1K concepts. LAION-400M could cover tens of thousands of long-tail distributed concepts. When sampling a batch of 32K images from these 400M images, it is likely that only a portion of 1K ImageNet classes would be retrieved.\n\n> W3: Quantitative analysis of Fig. 10 would be interesting\n\n1. Thanks for acknowledging this analysis! For numbers in this figure before subtraction, please check Tab. 2 on page 22. Regarding the appropriate choice of vocabulary size, this figure suggests there is a sweet spot smaller than the total prototype number (eg, around 16384 for 65536 prototypes). Also, the lighter orange results (using 16384 prototypes w/o subsampling) show worse results, verifying the effectiveness of our strategy.\n2. We recently noticed a study [a] that shows the learnable prototypes of DINO (akin to the classifier of SL) may be biased to imbalanced data and many of them collapse. Our strategy might help alleviate such collapse and encourage a more balanced utilization of these prototypes. We agree this finding is interesting and valuable for future research in clustering-based SSL, including SwAV, DINO, iBOT, DINOv2, etc.\n4. We also provide DINO results trained on IN-Caps in the attached PDF (Fig. R6) to better support the findings.\n5. We also would like to emphasize that our experiments of vocabulary subsampling are intended to verify the underlying rebalancing/debiasing mechanism of CLIP can be transferred to, other domains (SL and SSL). We do not intend to propose a novel method, but to point out problems to solve (model-side debiasing is widely needed when training on web data). We call for, and it is very likely better methods for debiasing large-scale trained SSL/LLM/MLLM to be developed in the future.\n\nReference:\n\n[a] Govindarajan et al., On Partial Prototype Collapse in Clustering-Based Self-Supervised Learning\n\n> W4: One claim should be softened\n\nThanks for reminding us! This sentence was intended to emphasize we study CLIP’s robustness/generalization to a specific property – data imbalance. We realize that this claim can be vague and strong, and will soften the statement and reduce the use of “generalization”. Also, we are considering to simplify the title to “What makes CLIP more robust to pre-training data imbalance”.\n\n> Q1: Recommendations for next-gen SSL/CLIP models\n\nIn contrast to most previous efforts that study CLIP’s robustness to OOD evaluation data, we focus on long-tailed pre-training data. From this point, we indeed have some general messages to share with the community:\n\n1. Data matters. Agreeing with a common trend in ML research, improving the quality of training data is as important as designing better models. Scaling up the data also helps.\n2. Data is not the silver bullet. Robustness does not always come for free as data scales, and model-side debiasing is needed. CLIP can learn robust features from long-tailed data, and the robustness consistently improves as data scales up. In contrast, SL is always clearly biased despite being trained on larger-scale datasets.\n3. Robustness to pre-training data imbalance is also something lacking in modern LLM/MLLMs, despite being trained on massive-scale data.\n   1. In Sec. 10 of [b], the authors show that if 1/8 “useful data” is mixed with 7/8 “junk data”, LLM model capacity degrades by 20x, and teaching the model to identify them effectively improves performance. Despite there are extensive efforts in data curation for LLMs (eg, data rebalancing), the large data corpus still can be long-tailed/biased, and more attention is needed.\n   2. In Fig. 3 of [c], the authors show that LLaVA’s poor classification performance is strongly correlated to the class frequency in training data, while not for CLIP.\n   3. These two findings indicate that CLIP’s robustness to pre-training data imbalance might not apply to generative models.\n\nReference:\n\n[b] Allen-Zhu et al., Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws\n\n[c] Zhang et al., Why are Visually-Grounded Language Models Bad at Image Classification?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OoAit6Our8", "reviewer": "Reviewer_XjUF", "summary": "Motivated by the novel observation on CLIP's robustness against data imbalance, this paper provides extensive analysis on factors that determine such robustness, including language supervision, classification templates, data scaling, distribution, and open-world concepts. The paper shows that such factors are transferrable to supervised or self-supervised learning to reach CLIP-level robustness.", "questions": "- How did you calculate the ImageNet-class frequency in web-scale datasets in Fig.1 (a)?\n- Regarding weakness 1., experiments on ablating the factors directly onto the large-scale-trained models and test their correlation as done in Fig. 2 should be included.\n- Legend colors do not match to those in Figure 9 (b).\n- Please more provide details of Section 4.2., such as, what models authors compare, and what models the different colors in Fig. 10 indicate.", "limitations": "See weakness.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It is novel observation that CLIP shows robustness against data imbalance.\n- The factors, including data scale and dynamic classification templates, that determine the robustness have practical insights in data imbalance problems.", "weaknesses": "1. The importance of data scale predominates other factors:\n- in ablation studies for data scaling, the setup with frozen CLIP text encoder header only shows the reasonably low correlations (still, much higher than large-scale examples shown in Fig.1) among other setups. The similar results are observed in the ablation studies in different distributions (Fig. 5-b) as well as extreme case (Fig. 8), where given the same distribution, the setup with frozen CLIP text encoder shows superior correlations. Are these results indicating that the large-scale data pretraining predominates the CLIP's robustness on data imbalance? In other words, given large-scale data, e.g., metaCLIP-2.5B, how ineffective are those factors studied in the paper? Are they still influential or negligible?  \n- In Figure 5, I find it difficult to clearly understand regarding distribution shifts. It needs more details regarding the distribution shifts and the meaning of the dashed line. How are the results calculated and what is the take-away message?\n- In Figure 6, the CLIP with YFCC-15M shows stronger correlations than that with YFCC-15M-cls, which cannot be explained by the authors’ claim that open-world datasets improve robustness.\n\n2. In section 4.1, additional results on SL using init head + sub voc. are necessary to fairly ablate the factors for showing transferability to SL.\n\n3. It is difficult to clearly understand the setup in Section 4.2. It lacks detailed explanation on setups for comparison, and the measures only include downstream task performance, not the correlations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer XjUF,\n\nThank you so much for your thorough review and kind suggestions! We answer your questions as follows:\n\n> W1&Q2: Given large-scale data, how ineffective are other factors studied in the paper; Fig. 2 on large-scale datasets should be included\n\nNice suggestion! We agree that scaling up data is one of the most effective factors, but it is only helpful for CLIP, and not for SL. Also, scaling up data could introduce heavy-tailed data distribution, and models can be thus more biased. To present a more complete picture, we also conducted the experiments in Fig. 2 on LAIONet and YFCC. The results are as follows:\n\n| Corr. (freq. vs. acc.)  | IN-Caps | LAIONet (match freq.) | LAIONet (full) | YFCC-15M | LAION-400M |\n| ----------------------- | :-----: | :-------------------: | :------------: | :------: | :--------: |\n| SL                      |  0.52   |         0.44          |      0.63      |   0.64   |     -      |\n| SL w/ sub. voc.         |  0.39   |         0.27          |      0.43      |   0.44   |     -      |\n| CLIP w/ class templates |  0.50   |         0.34          |      0.43      |   0.36   |     -      |\n| CLIP w/ captions        |  0.28   |         0.26          |      0.40      |   0.34   |    0.12    |\n\nLAIONet variants are as in our paper: the match-freq version matches the scale (449K) and class distribution of IN-Caps, and the full version is of high imbalance and larger scale (3.26M). Unlike other web data like YFCC, LAIONet is filtered to consist of 1K ImageNet classes.\n\n1. Comparing between rows, the model-side factor (sub. voc.) is consistently effective, and the benefit of descriptive captions gradually narrows as data scales up sufficiently.\n2. Comparing IN-Caps and LAIONet (match freq.), the distribution shift forms a regularization and all models are less biased.\n3. Comparing LAIONet (match freq.) and LAIONet (full), despite scaling up from 0.45M to 3.3M, more severe data imbalance (see Fig. 2 right) makes all models more biased.\n4. Comparing LAIONet (full), YFCC-15M, and LAION-400M, CLIP consistently benefits as data scales up, while SL cannot. This indicates to benefit from data factors, model factor is also important.\n\n> W2: Details of Fig. 5 are needed; how are results calculated and what is the take-away message?\n\nThanks for pointing this out and we have polished Fig. 5 to make it more reader-friendly, please see the attached PDF (Fig. R1). The takeaway message is: distribution shift and higher image diversity in web data help improve model robustness, while extreme data imbalance in them harms robustness.\n\n> W3: Fig. 6 shows a counter-trend to our claim\n\nThanks for reminding us! We checked with the plotting code and identified a bug in indexing. We have updated the figure in the attached PDF.\n\n> W4: Additional results on SL using rand init head + sub voc. is needed for fair comparison\n\nPlease note that the existing comparison with frozen head is already a fair comparison and shows the effectiveness of vocabulary subsampling. Under the 0-shot tail (open-world) setting, a model without a pre-trained head has no knowledge about the mapping to novel classes, and thus cannot perform better than random guessing. We tried the setting suggested by the reviewer, and the best results in tail classes were smaller than 1% accuracy. Despite this, we expect the strategy to work under normal (non-extreme) data imbalance settings, eg, as shown in Fig. 3b, comparing darkest and lighter crosses.\n\n> W5&Q4: Setup of Sec. 4.2 need further clarifications\n\nThanks for pointing this out. We provided more detailed settings of Sec. 4.2 in Sec. D, especially D.2. We will add a reference to the appendix to make it clearer. We compare DINO variants trained on LAIONet to the original DINO trained on ImageNet, in terms of transfer learning performance (also see Tab. 2 on page 22). Orange refers to original/vanilla DINO, and blue refers to DINO+vocabulary subsampling. A lighter color refers to a smaller prototype number.\n\n> Q1: How do we obtain ImageNet class frequency in web-scale datasets\n\nWe provided details about the estimation process in Sec. B.2, and the models considered in Sec. B.3. We implemented a simple set-based substring matching method for efficient estimation on large-scale datasets. Despite being simple, we find it strongly correlates to results estimated using more advanced techniques (discussed in Sec. A.4).\n\n> Q3: Legend colors do not match to those in Fig. 9 (b)\n\nWe have revised this figure following your suggestion (see attached PDF), thanks!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oCVtgHR0gy", "reviewer": "Reviewer_iwVX", "summary": "This paper explores the remarkable robustness of CLIP training compared to supervised training when dealing with real-world imbalanced datasets. This advantage can be attributed to dynamic classification using a subset of classes. Furthermore, the paper shows that robustness and discriminability improve with denser language supervision, larger data scales, and broader open-world concepts. The experiments are mainly conducted on ImageNet-captions and LAIONet to validate the main claims.", "questions": "please refer to the weakness part.", "limitations": "The limitations are discussed in this paper. No potential negative social impact.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper reveals the underlying distribution (using ImageNet concepts) of several widely-used vision-language pretraining datasets.\n- The issue of vision-language learning under imbalanced data distribution is critical.\n- This paper presents the finding that CLIP training is more robust than supervised learning and explains the underlying mechanism behind this phenomenon.", "weaknesses": "-\tThis paper explores vision-language pretraining (VLP) under imbalanced distributions primarily from a data perspective, focusing on language supervision, data scaling, and open-world concepts. These factors, known to enhance robust performance, have been discussed in several previous studies.\n-\tOther factors beyond data may also affect the robustness of VLP under imbalanced distributions. For instance, from a loss perspective, comparing the learning abilities of CLIP loss and cross-entropy loss in supervised settings, particularly under imbalanced conditions, needs discussion.\n-\tIn the language supervision section, a more comprehensive experiment would compare self-supervised contrastive learning methods, as they share the same contrastive objective. This comparison would isolate the differences between contrastive and cross-entropy losses, focusing solely on variations in language supervision.\n-   It seems that the concept frequency estimation in this paper may suffers from noise, multi-label structure. \n-   The evaluation is mainly focusing on classification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer iwVX, your questions are answered below:\n\n> W1: The factors studied are primarily from a data perspective and known to enhance robust performance\n\n1. To the best of our knowledge, most existing works study CLIP’s robustness to OOD evaluation data, while we are the first to study CLIP’s robustness to *imbalance pre-training* data from these perspectives. We believe this viewpoint is timely given the current research trend of brute-force scaling up, without a detailed study of factors in the pre-training data and how they interact with model factors.\n2. Please note that not all data factors we studied could enhance robust performance. Eg, we also find that more severe data imbalance in web data can harm robustness (Fig. 5).\n3. We also note that the benefits of larger-scale data do not come for free, it also requires a suitable model. Eg, CLIP’s robustness consistently improves as data scales up, while not for traditional SL, and even not for modern generative models (LLM & MLLM, see our response to reviewer tULz’s Q1).\n\n> W2: Other factors beyond data should be considered, eg, from a loss perspective\n\nActually, this is exactly what motivated us to identify the factor “dynamic classification as pretext task” (Sec. 3.3). We tried to align every detail between SL and CLIP, and finally found applying CLIP loss to SL to be a detrimental factor. We then dug into the mechanisms of CLIP loss, and found CE+vocabulary subsampling to have the same effect.\n\nIn addition, we also experimented on factors like the architecture of the vision backbone and text backbone, vision pre-training, stronger data augmentation, larger batch size, and test-time prompts, and did not find noticeable effects. These factors are thus not presented in the paper. A discussion on the motivation behind the factors we considered is provided in Sec. A.6.\n\n> W3: Self-supervised learning methods should also be discussed to show the effect of language supervision\n\nThanks for raising this concern. We additionally trained a self-supervised learning method DINO on ImageNet-Captions, and used k-NN to evaluate its classification performance. In comparison to SL and CLIP, it further verifies the effectiveness of language supervision.\n\n| Model                    | SL   | DINO (k-NN) | CLIP |\n| ------------------------ | :--: | :---------: | :--: |\n| Corr. (freq. vs. acc.)   | 0.52 | 0.35        | 0.28 |\n| Corr. (freq. vs. #pred.) | 0.57 | 0.20        | 0.08 |\n\n> W4: It seems the concept frequency estimation may suffer from noise, multi-label structure\n\n1. This is why we tried to study as many factors on IN-Caps as possible. However, it is still expected that we verify these factors on other datasets to better support our findings (as requested by Reviewer XjUF), and provide a complete discussion on factors not supported by IN-Caps (eg, data distribution and data scale), thus facing an inevitable trade-off. Accurate concept frequency estimation of web datasets is an emerging topic and we believe it is valuable for future research.\n2. Regarding LAIONet, the authors are aware of the label noise and explored applying a filter. They empirically found this to be effective in overcoming such noise (see Fig. 11 in LAIONet’s arXiv preprint). When replicating LAIONet, we are conservative on the choice of the filtering threshold, trying to make the selected images have highly confident labels.\n3. We also explored an even simpler strategy: set-based substring matching (detailed in Sec. B.2). We applied this strategy to estimate the frequency in large-scale datasets (eg, LAION-400M). In comparison to advanced estimation techniques using ChatGPT and Llama2 (Parashar et al.), our method shows very consistent results (correlation of 0.8-0.9, see Sec. A.4 for details).\n\n> W5: The evaluation is mainly focusing on classification\n\nPlease note that we study the effects of data imbalance in pre-training data, which raises challenges in accurate estimation of labels, control of ablating factors, and high training cost of each experiment (1-2 days on 4xA100 GPUs each experiment, more than 400 experiments in total). Evaluations on classification are a result of trading off these factors and presenting as comprehensive an analysis as possible.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fuJq3SvaEZ", "reviewer": "Reviewer_WXxS", "summary": "This author empirically studies the mechanisms behind CLIP's generalization ability from multiple aspects. Based on the findings, they further transfer the merits of CLIP to other scenarios like supervised learning and self-supervised learning.", "questions": "1. The methodology for handling training data during the random subsampling of vocabulary, as discussed in Section 3.3, remains ambiguous. With a batch size of 1024 and a reduced vocabulary size of 500, how are the remaining samples, whose labels do not fit within the reduced vocabulary, treated? This clarification is essential for understanding the implications of the approach on the learning process.\n2. Is there a correlation between smaller vocabulary sizes and smaller batch sizes in these experiments?\n3. The rationale behind the selection of specific thresholds (0.7, 0.75, 0.8, 0.82) in Figure 5 is unclear, particularly the choice of 0.82 over 0.85.", "limitations": "1. No code is provided at this stage.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed problem is worth exploring.\n2. The experiments are comprehensive.", "weaknesses": "1. The authors studied the robustness/generalization of CLIP from various points, but some points are known to the community. Sections 3.2 and 3.4 claim that higher caption density and larger data volumes in training data contribute to robustness in terms of imbalances. However, this observation appears to be well-understood in the community. It is generally accepted that high-quality, richly described data is preferable.\n2. I appreciate the presented comprehensive studies. However, the settings and conclusions in Section 3 are occasionally unclear. I recommend adding a brief conclusion to each subsection to enhance clarity and coherence.\n3. In Section 4.2, which examines subsampling vocabulary in self-supervised learning with DINO, the impact of this technique is only demonstrated on models with a default dimension of 65,536. It remains unclear if this approach would be equally effective on other self-supervised learning models that utilize significantly smaller default dimensions, such as 2,048.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer WXxS, thanks for helping improve our paper, and your concerns are responded to as follows:\n\n> L1: No code is provided at this stage\n\nWe have organized the code and provided it to the AC with a separate comment according to the rebuttal policy, please contact the AC for access. The code contains all code and data ranging from concept frequency estimation, dataset curation, and training and evaluating SL/CLIP/DINO models and their variants considered. We also provided notebooks to replicate all figures present in the paper. We will make it public to help future research.\n\n> W1: It is generally accepted that high-quality, richly described data is preferable\n\n1. We agree, but we also would like to note that we are the first to study the effect of these factors in terms of robustness to *imbalanced pre-training data*, contrasting existing works that focus on *OOD evaluation data*. We believe this viewpoint is timely given the current research trend that lacks a detailed study of factors in the pre-training data and how they interact with model factors.\n2. Please note that not all data factors we studied could enhance robust performance. Eg, we also find that more severe data imbalance in web data can harm robustness (Fig. 5).\n3. We also note that the benefits of larger-scale data do not come for free, it also requires a suitable model. Eg, CLIP’s robustness consistently improves as data scales up, while not for traditional SL, and even not for modern generative models (LLM & MLLM, see our response to reviewer tULz’s Q1).\n\n> W2: Settings and conclusions in Sec. 3 are occasionally unclear\n\nThanks for pointing this out. We put most details of the setting in the appendix due to the page limit. In the next version, we’ll incorporate more details in the main text and highlight the takeaway messages according to your suggestion.\n\n> W3: It remains unclear whether subsampling vocabulary helps other SSL methods\n\n1. We choose DINO because its pretext task involves pseudo-labeling with online-learned prototypes – each standing for a virtual class (called “last_layer” in the code). Its feature dimension is 2048 and the projector dimension is 256 by default, similar to other SSL methods. Other contrastive methods discriminate between randomly sampled images in training, and vocabulary subsampling is not applicable. \n2. We recently noticed a study [a] that shows the learnable prototypes of DINO (akin to the classifier of SL) may be biased to imbalanced data and many of them collapse. Our strategy might help alleviate such collapse and encourage a more balanced utilization of these prototypes.\n3. We agree this finding is valuable for extension to other SSL methods, especially clustering-based SSL, including SwAV, DINO, iBOT, DINOv2, etc.\n4. We also would like to emphasize that our experiments of vocabulary subsampling are intended to verify the underlying rebalancing/debiasing mechanism of CLIP can be transferred to other domains (SL and SSL). We do not intend to propose a novel method, but to point out problems to solve (model-side debiasing is widely needed when training on web data). We call for, and it is very likely better methods for debiasing large-scale trained SSL/LLM/MLLM to be developed in the future.\n\nReference:\n\n[a] Govindarajan et al., On Partial Prototype Collapse in Clustering-Based Self-Supervised Learning\n\n> Q1: When applying vocabulary subsampling, how are remaining samples whose labels do not fit treated\n\nWe ensure all GT labels in a batch are covered after subsampling. Please check Sec. C.3 for further details.\n\n> Q2: Is there a correlation between smaller vocabulary sizes and smaller batch sizes\n\nOur experiments kept the batch size unchanged. However, we do expect the optimal vocabulary to be a smaller value correlating with the batch size, thus rebalancing learning signals while forming meaningful classification tasks.\n\n> Q3: How are the thresholds in Fig. 5 selected\n\nThe choice is to ensure we can obtain images as much as the scale of ImageNet-Captions (0.45M). When using thresholds higher than 0.82, the remaining number of images is small, eg, 0.26M for threshold 0.85.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their time, valuable comments, and kind suggestions. We also appreciate that our work is recognized to have a "novel" observation [tULz, XjUF, iwVX], study a "critical" problem [XjUF, iwVX, WXxS], design "sound" [tULz] and "comprehensive" [WXxS] experiments, and present "very interesting" [tULz] findings that "have practical insights" [XjUF].

In the responses to each reviewer, we have tried our best to resolve misunderstandings, provide experimental support, and clarify details. These discussions and new results have inspired us to improve our work and revise unclear writings and figures. We will incorporate all these in the next version of our manuscript.

In the attached PDF, we provide some revised figures according to the suggestions. We also have sent an anonymized link to our code in a separate comment to the AC, and please contact the AC for access. The code contains all the details to replicate this project, ranging from concept frequency estimation, dataset curation and ablation, and training and evaluating SL/CLIP/DINO models and their variants considered. We also provided notebooks to replicate all figures present in the paper. We will make it public to facilitate future research.

Thank you again and looking forward to discussing with you in the next period!

Meta-review: Though the paper has mixed reviews, the majority of reviewers recommend acceptance and appreciate the detailed analysis of the robustness of CLIP-style models relative to their training data distributions and with respect to robustness for imbalanced data in particular. The AC team agrees that this analysis adds value to the community, and thus recommend acceptance.
Decision: Accept (poster)