What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights

Authors: xin  wen
bingchen  zhao
yilun  chen
jiangmiao  pang
xiaojuan  qi
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 19:07:30

Abstract: Severe data imbalance naturally exists among web-scale vision-language datasets. Despite this, we find CLIP pre-trained thereupon exhibits notable robustness to the data imbalance compared to supervised learning, and demonstrates significant effectiveness in learning generalizable representations. With an aim to investigate the reasons behind this finding, we conduct controlled experiments to study various underlying factors, and reveal that CLIP's pretext task forms a dynamic classification problem wherein only a subset of classes is present in training. This isolates the bias from dominant classes and implicitly balances the learning signal. Furthermore, the robustness and discriminability of CLIP improve with more descriptive language supervision, larger data scale, and broader open-world concepts, which are inaccessible to supervised learning. Our study not only uncovers the mechanisms behind CLIP's generalizability beyond data imbalance but also provides transferable insights for the research community. The findings are validated in both supervised and self-supervised learning, enabling models trained on imbalanced data to achieve CLIP-level performance on diverse recognition tasks. Code and data are available at: https://github.com/CVMI-Lab/clip-beyond-tail.

Main paper: 
 introduction: 
 The development of contrastive language-image pre-training (CLIP) [36, 44, 57, 68, 93] has demonstrated unprecedented success in learning generalizable representations, empowering zero-shot vision tasks and robustness to natural distributional shifts. This success can be primarily attributed to the effective use of large-scale uncurated image captioning datasets collected from the web. A recent trend involves delving into the distribution of these datasets and explicitly introducing interventions to the curation process to create better data for training [29, 91] . However, limited research has been conducted on analyzing the distribution of concepts/classes in these datasets and the behavior of CLIP under varying distributions. This work thus starts by presenting a concept-centric analysis of existing web-scale image-text datasets and models pre-trained accordingly (Fig. 1 ).Motivation. Our motivation for this study arises from an intriguing observation of CLIP's zero-shot performance on ImageNet: CLIP is notably more robust to pre-trained data imbalance than supervised learning. We examine various vision-language datasets at different scales, and analyze their distribution with respect to ImageNet classes. We find that image-text datasets share an extremely imbalanced class distribution (Fig. 1a ). Interestingly, we find that the zero-shot classification performance of trained CLIP models is more robust to this imbalance, especially compared to models obtained by supervised learning. This is evidenced by a weaker correlation between a class's performance and its frequency (Fig. 1b ). This trend is consistent across CLIP models and pre-training datasets and even holds true for smaller-scale datasets like CC-12M [12] . This phenomenon inspires us to study the underlying causes for CLIP's relative robustness toward data imbalance and what we can learn from.Our study and findings. To answer the question above, we conduct controlled experiments to analyze factors including supervision signal and pretext task (Fig. 3 ), data distribution (Fig. 4 ), scale (Fig. 5 ), and open-world concepts (Fig. 6 ). Our extensive studies have led us to the following findings:• Language supervision, particularly the texts with increased descriptiveness (informativeness), enhances both the robustness and discriminability of CLIP, and preserves more feature variation.• CLIP's pretext task forms dynamic classification problems, wherein only a subset of classes is present during training, effectively isolates biases to dominant classes, and balances learning signal.• Severe data imbalance in web datasets increases the risk of bias in models. However, distribution shift and higher data diversity in them can enhance robustness, albeit a trade-off in data efficiency.• CLIP's robustness and discriminability improve together with data scaling, benefitting from its ability to utilize open-world data, a privilege not accessible to supervised learning.Applications. Inspired by the findings of our study, we found that this robustness to data imbalance can be transferred to supervised and self-supervised learning models with simple techniques by making the classification task dynamic during training. Under extremely imbalanced data scenarios, we show that a vanilla classification model can also generalize well to tail (or even open-world) classes as well as CLIP via 1) fixing the classifier with class prototypes from pre-trained CLIP text encoder, and 2) training with randomly subsampled vocabulary (results in Fig. 8 , and analysis in Fig. 9 ). Beyond classification, we also show improved transferability on DINO [11] pre-trained on uncurated web data by simply randomly subsampling the prototypes in training (Fig. 10 ).Summary. Our study is one of the pioneering efforts to explore CLIP's robustness in the context of imbalanced data distributions. Our exploration provides a comprehensive analysis that uncovers the mechanisms contributing to CLIP's robustness against data imbalance. As we will demonstrate in this paper, the insights gained from our research are transferable to other domains, including supervised and self-supervised learning frameworks. [27] augments train images of ImageNet with texts by querying Flickr with image URLs. The texts include title, description, and tags. Bottom: LAIONet [77] is a filtered subset of LAION-400M [73] , obtained by matching ImageNet classes with captions and filtering by CLIP text encoder for disambiguation.2 Related work CLIP's distributional robustness. The debut of CLIP not only set the state-of-the-art performance on conventional image classification benchmarks but also demonstrated unprecedented robustness to challenging distribution shifts. Studies have shown that this robustness stems from the diverse training distributions CLIP has seen during training time [27, 69] . Also, it is shown that the data quality plays an important role in enhancing the distributional robustness of CLIP [58] . It may seem that CLIP obtains the improvement distributional robustness due to the similarity of pretraining data to the distribution shifted data, but [55] shows that it is not the case where even after pruning similar data, CLIP still obtains strong robustness, indicating generalizable representations are learned.Learning from uncurated data. Apart from robustness to distribution shifts, previous works have also delved into the nature of uncurated large-scale datasets [35, 49, 77, 91] . Studies have shown that self-supervised learning can produce more robust models than supervised learning on uncurated data [35, 49] . Moreover, focusing on learning of subsets of the entire dataset [9, 82] has shown to further enhance self-supervised learning from uncurated data. On the learning on uncurated data, the language information has shown to help learn good representations [71] . Balancing the concept distribution of uncurated data has shown to be a scalable way of learning good models [91] . However, the uncurated data is not all harmful for performance, the lower intra-class similarity of the data is shown to help preserve information/variation in representations [77] , but at low data efficiency [85] .Generalization of vision models. One of the main themes of computer vision research in the era of deep learning is the search for more generalizable models. Works have focused on self-supervised pretraining with only images, among which contrastive learning [13] and self-distillation [11, 61] are shown to be effective. With the introduction of large-scale image-text datasets [73, 74] , there is a huge interest in learning more generalizable vision representations from additional language supervision. While techniques for incorporating language supervision have been proposed [19, 36, 68, 72, 94] , further exploration of how semantic grounding help improves the generalization is needed [21] .To fully utilize language supervision, using synthetic data from large language models to improve language supervision is a newly emerged research area [25, 26] .3 What makes CLIP more robust to long-tailed pre-training data?In the following, we conduct a series of controlled experiments to systematically evaluate the role of various factors on the robustness of CLIP to data imbalance. These factors include supervision signal (Sec. 3.2), pretext task (Sec. 3.3), data distribution (Sec. 3.4), data scale (Sec. 3.5) , and open-world concepts (Sec. 3.6) . Moreover, we also provide some insights on CLIP's feature space in Sec. 3.7. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Discussed in Sec. 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
setting: 
 Datasets. Experiments in this study are conducted on variants of two image-text datasets: ImageNet-Captions [27] and LAIONet [77] to allow better data-centric control. An overview is shown in Fig. 2 . Both datasets provide images with their paired captions, and class labels on ImageNet. The captions of ImageNet-Captions are in the format of title, description, and tags (some can be missing for a specific image), which allows control of captions' descriptiveness. Images of LAIONet are drawn from LAION, which has a higher intra-class variability and is extremely imbalanced across classes. This makes it more challenging to train on and allows isolating the effect of data distribution.Models. We consider both CLIP and supervised learning (SL) with ResNet-50 as the backbone.Given that CNNs are generally considered less robust than ViTs [4] , this choice also enables us to infer the robustness of other models. For SL, we align most details with CLIP [68] to rule out the effect of irrelevant factors. E.g., we use the same weak data augmentation as CLIP, adopt a prototypical classification head (i.e., ℓ 2 -normalizing both features and classifier weights), and apply a learnable temperature to logits. The training schedules of CLIP and SL follow [15] and [27] , respectively. Models are fully trained from scratch by default. More details are provided in Appx. C.Metrics. We compute Spearman correlation coefficients [78] between class frequency and models' statistics (class-wise top-1 accuracy and number of samples predicted as each class). Besides, we also consider metrics from neural collapse literature [32, 63] for analyzing feature distribution. Formally, defining the global feature meanµ G = Avg i,c h i,c , class-level means µ c = Avg i h i,c , within- class covariance Σ W = Avg i,c (h i,c -µ c )(h i,c -µ c ) ⊤ , and between-class covariance Σ B = Avg c (µ c -µ G )(µ c -µ G )⊤ , the metrics are defined as:EQUATIONwhere † denotes the Moore-Penrose pseudoinverse, h i,c is the feature of the i-th example in class c, and C is the total number of classes. Intuitively, NC1 and NC2 measure the compactness and separation of clusters, respectively. NC1 approaches zero when the within-class variation of features becomes negligible, and NC2 converges to zero when classifiers reach maximal and equal margins (i.e., ETF structure) [63] . Note that these two metrics are originally defined as an average across classes, and it is simple to obtain per-class NC1 and NC2 metrics, measuring the variability of a specific class or its average margin to all other classes.Setting. We start by examining the impact of language supervision, the primary distinction between CLIP and other contrastive learning approaches. This is done by creating texts with roughly monotonic increasing descriptiveness given metadata of ImageNet-Captions. For the low-diversity texts, we create • synthetic class-centric texts using classification templates from CLIP [68] given class names or synset [56] . The • natural language-based texts are created by concatenating different types of captions (see Fig. 2 ), and the descriptiveness of language supervision is controlled by the number of text types used. More details are available in Appx. C.2.Results. Fig. 3 provide a comprehensive comparison between model variants from different perspectives. Restricting our view to CLIP models in the first two subfigures, • higher text descriptiveness results in improvements in both robustness and discriminability of CLIP, as shown by lower correlation (Fig. 3a ) and higher overall accuracy (Fig. 3b , y-axis). On the other hand, • relatively less descriptive texts show weaker results that are close to results of • templated-based CLIP (Fig. 3a ,x-axis). We see this as less descriptive texts could collapse to class-centric supervision without much additional variance. Despite this, predictions of • template-based CLIP are still notably less biased by pre-training data than ✖ SL (Fig. 3b ), indicating other factors may re-balance CLIP's predictions.Setting. We note that the pretext of Results. As shown in Fig. 3a , sampling a ✖ smaller vocabulary notably reduces SL's prediction bias, and obtains robustness similar to • template-based CLIP. Regarding the favorable vocabulary size, smaller ones are more effective in reducing prediction bias (Fig. 3a ), and intermediate ones also improve accuracy (Fig. 3b ). The preferred vocabulary size for ImageNet-Captions is around 100.Discussion. Our intuition of the phenomena above is that dynamic classification in some way achieves class-level re-balancing. When the ground truth (GT) corresponds to a tail class, a small vocabulary isolates the negative impact of most head classes, avoiding bias towards them and enabling the model to focus on classifying the tail class itself. Besides, it is worth noting that as demonstrated in [32, 50] , optimization continues after the model's predictions reach zero error, and seeks minimum intra-class variability and maximum inter-class margin (especially larger margin around head classes). Thus when the GT is a head class, this approach limits the number of negative classes and could prevent the model from excessively distorting the representations of them through over-optimization.Motivation. Motivated by the findings of [27] regarding the impact of image distribution on CLIP's robustness to natural distribution shifts, our study also examines its influence on robustness to data imbalance. As shown in [77] , a higher filter threshold leads to a more condensed image distribution, a result that is confirmed in Fig. 4a . We thus create LAIONet variants of different intra-class variations by adjusting this threshold. All variants in this section keep the data scale the same as ImageNet-Captions (0.45M). In addition, due to the disparity in class distribution between LAIONet and ImageNet-Captions, we also create a variant that aligns with the class frequencies of ImageNet-Captions ('=freq') while preserving web image distribution. This variant is sampled from the full version (3.26M) that uses a threshold of 0.7. More details about datasets are provided in Appx. C.5.A comparison between models trained on the aforementioned datasets is present in Fig. 4b . We find that web data is not naturally friendly for de-biasing, but could have made models more biased due to extreme data imbalance (comparing '=freq' with other columns). The distribution shift of web data could improve robustness if a • pre-trained text head is available (circles vs. squares, last column). If not, scaling may help. Moreover, results with smaller thresholds also turn out to be more robust, indicating that higher intra-class data diversity (smaller threshold) improves robustness. 3.5 Data scaling (also achievable via language pre-training)Motivation. We note that the correlations of CLIP in Fig. 3a ( x-axis) are still higher than that of open-source models in Fig. 1b . One key remaining factor is the scale of pre-training data (see Fig. 1b for large-scale results). Given that ImageNet-Captions is small-scaled (see Fig. 2 ), experiments following are conducted on LAIONet. See Appxs. C.4 and C.5 for more details about the setting. Train data scale (million) It is also unattainable by SL even using the same head. 3) Language pre-training using CLIP is more favorable for image-text tasks than pure language modeling (e.g., RoBERTa [51] ).Results. Fig. 5 presents the results obtained from uniformly subsampled subsets of LAIONet. These findings extend the scaling law: as data scales, ImageNet zero-shot accuracy (Fig. 5a ) and models' robustness to data imbalance (Fig. 5b ) improve simultaneously. We also provide a comparison between text encoders: • training from scratch, initializing with • pre-trained CLIP (frozen) or • frozen RoBERTa [51] , or • fine-tuning the text encoder together. • Frozen CLIP language head enables the vision model to leverage a well-established feature space as supervision, achieving better data efficiency (Fig. 5a ) and robustness to data imbalance (Fig. 5b ). • Fine-tuning CLIP text head results in over-fitting (similar results with • training from scratch), and • RoBERTa does not suit the contrastive task and adversarially affects performance. Further investigation through NC-based metrics shows •• frozen heads effectively preserves intra-class variation (Fig. 5c ), which is at risk of being lost when • fine-tuned. Both • frozen and • fine-tuned heads contribute to inter-class margins (Fig. 5d ), and if • randomly initialized, scaling training data still can achieve improved margins.Compared to ✖ SL, CLIP can better utilize web-crawled data and pre-trained text encoder (Fig. 5a ). But note that when evaluating close-set accuracy, the data efficiency of CLIP is still much lower than SL trained on classification datasets (e.g., ImageNet). Results. Cluster compactness (• NC1) does not show a strong correlation with CLIP's failures (Fig. 7a .1), and the frequent classes of LAION models tend to preserve more intra-class variation (Fig. 7a .2). Besides, there are some implications from the margin between class centers (•• NC2).For example, Fig. 7a .1 shows that the fail classes of smaller-scale models (12/15M) are hardly discriminative to most classes (• NC2 M ), while larger-scale models (≥ 400M) only fail on some nearest-neighbor classes (• NC2 nn M ). This indicates that the failing classes already have good separation from most other classes, and the confusion primarily comes from very few hard classes.Regarding the effects of data imbalance on CLIP (Fig. 7a .2), we find a strong connection to • NC2 W , denoting denser head and coarser tail classes in text space. t-SNE [86] of the class centers is provided in Fig. 7b for reference, and more visualizations of vision features can be found in Fig. 20 .Discussions. Though weakly correlated to the class frequency, CLIP's performance is still highly biased [87, 99] . If data imbalance is not the main cause, then what are other suspect of CLIP's failures? We hypothesize that ImageNet is intrinsically biased. The classes are not of equal difficulty [17] and some are even ambiguous [6, 39, 75] , e.g., "sunglass" vs. "sunglasses". In this case, it is possible for a model trained on the balanced ImageNet to be biased [17] , and some errors are unsolvable no matter how much training data is added. Besides, CLIP leverages open-world concepts in training, which are not counted for frequency but still could affect close-set performance. Moreover, such biases might be connected with CLIP's hallucination [31, 53, 92] . We believe these are valuable questions to be explored. In supplement to this discussion, we also discuss CLIP's bias measured on broader sets of concepts in Appx. A.2 and the effects of data imbalance on CLIP in Appx. A.5.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
acquiring clip-level generalization: 
 This section shows findings from CLIP's underlying mechanisms can be applied to both supervised learning (Sec. 4.1) and self-supervised learning (Sec. 4.2) under severe data imbalance.In quest of the limit of CLIP's robustness to pre-training data imbalance, we create an extreme case based on ImageNet-Captions: trimming the tail classes to only one shot, or even completely zero shot (i.e., an open-world setting). We then train models on this trimmed dataset, and evaluate performance on ImageNet regarding tail/other classes. As shown in Fig. 8 , at the scale of ImageNet-Captions (∼0.45M), • CLIP trained from scratch also fails on tail classes when trained under severe data imbalance. Despite this, by adopting a • pre-trained text encoder following Sec. To understand the underlying mechanisms, we present a case study on the affinity matrix between classifiers, and tail class accuracies under the zero-shot tail (50 classes) setting in Fig. 9 . The affinity matrices of the classification head (see Fig. 9a , we subsample 100 classes for visualization) demonstrate that the learned tail prototypes collapse to singularity, while the class prototypes from Our study provides a thorough analysis of this technique from another perspective, and we hope it can motivate future applications in this field.To show the universality of the aforementioned techniques, we also explore the application in improving self-supervised learning when pre-trained on imbalanced data. As discussed in [3, 61] , DINO's performance is sensitive to the imbalance in web-crawled pre-training data, and thus data deduplication is a crucial process in DINOv2 [61] . As discussed by a recent study [30] , the learnable prototypes of DINO (akin to the classifier of SL) may be biased to imbalanced data and many collapses (like Fig. 9a ). We hypothesize that applying subsampling to the prototypes may alleviate this phenomenon. Our intuition is that the operation resembles dropout and could encourage better utilization of the online-learned prototypes of DINO, thus improving representations learned from uncurated web data. Based on vanilla DINO [11] , we randomly subsample prototypes (instead of using them all) during the calculation of the self-distillation loss (see details in Appx. D). All models are pre-trained for 100 epochs on LAIONet, and evaluated on the transfer learning benchmark of [40] . Linear Probe Score (%) To rule out the influence of total vocabulary size (number of prototypes), we also train ■ vanilla DINO with reduced vocabulary (16384). This model is notably weaker than that trained with ■ subsampling (16384 for each training iter, 65536 in total), and supports the improvement's effectiveness.S t a n f o r d C a r s F G V C A i r c r a f t F l o w e r s 1 0 2 S U N 3 9 7 D T D C I F A R 1 0 C I F A R 1 0 0 P a s c a l V O C 2 0 0 7 C a l t e c h 1 0 1 F o o d 1 0 1 O x f o5 Limitations, future work, and broader impacts Limitations. Our study has focused on the robustness of CLIP-type models in relation to the data imbalance naturally raised from web data sources. We have demonstrated that our findings are transferrable to the supervised and self-supervised learning setting for classification tasks. However, we acknowledge that our estimation of image-text datasets' concept frequency is based on a simple rule-based pipeline, which could be prone to caption noise, multi-label, and ambiguity. Besides, CLIP models are not only employed for classification tasks, the study of leveraging CLIP for open-world detection or segmentation is the area our study does not cover. Additionally, given the nature of the web-based data sources used in our study, we acknowledge that the data may contain implicit bias or harmful information. We provide more discussions in Appx. A.Future work. Our findings cover insights in language supervision, pretext task, data scaling, and concept scaling, but only a small portion are validated in application. One direction for future work is to explore the use of language supervision and open-world data in recognition models. Besides, a recent work [43] finds Adam optimizer to outperform (stochastic) gradient descent on heavy-tailed data, which can be another factor in CLIP's robustness and is worth further exploration. On the other hand, we are interested in extending our discovery to the open-world detection and segmentation tasks to see if our findings still hold under these more challenging scenarios.Furthermore, as we have analyzed in our study, language supervision plays an important role in achieving such robustness to the data imbalance, thus we are also interested in studying whether or not similar traces of generalization exist in (multi-modal) large language models (e.g., Llama [83] , BLIP-2 [45] , LLaVA [48] , etc.). However, despite being trained on large-scale data with language supervision, recent works show that LLM/MLLMs still suffer from long-tailed training data [37, 46] , and their performance is highly correlated with the frequency that corresponding knowledge appeared in training [1, 95] . This indicates that generative models might be intrinsically more prone to long-tailed data than contrastive models like CLIP, and injecting rebalancing mechanisms into the generative process could be valuable for future explorations.Broader impacts. We provide an in-depth analysis of CLIP's robustness to data imbalance, which helps understand the effectiveness of CLIP. The techniques here are also shown to be effective for other domains (supervised learning and self-supervised learning) to overcome biases in tail underrepresented classes. Thus, we expect our work not to pose potential negative societal consequences but rather to improve society's overall equality and inclusiveness. 
concluding remarks: 
 Our work starts with the observation that although web-crawled datasets share an extremely imbalanced data distribution, CLIP is relatively more robust to it. Extensive studies on 1) language supervision, 2) pretext task, 3) web data distribution, 4) data scaling, and 5) open-world concepts reveal significant findings about the underlying mechanisms of this robustness. We have also demonstrated that these findings can be transferred to classification and self-supervised learning methods, yielding improved generalization under pre-training data imbalance. Our study uncovers key factors of CLIP's robustness to pre-training data imbalance, and provides new perspectives to understand its generalizability. The insights learned are validated on tasks from extremely long-tailed supervised learning to self-supervised learning on web-crawled data. While CLIP has been a game changer in these research fields, it has long been utilized as is. Our study, instead, delved into the mechanisms behind CLIP, providing an opportunity to improve downstream tasks by leveraging the underlying mechanisms rather than relying solely on the model itself, with greater flexibility and adaptability. and domain-specific ones (e.g., EuroSAT). These data might be relatively rare on the web or have significant gaps with other data, thus hard to benefit from scaling or generalization from existing data.Oxford-IIIT Pets [65] , Flowers-102 [59] , Places365 [96] , EuroSAT [34] , and Describable Textures (DTD) [16] . Pre-trained CLIP models' correlation statistics on these concept sets are as shown in Fig. 12 . Models pre-trained at scale (≥ 400M) remain robust on most datasets. However, some fine-trained (e.g., CUB and Flowers-102) and domain-specific (e.g., EuroSAT) datasets tend to be harder to learn and easier to bias. These data might be relatively rare on the web and can have significant gaps with other data formats (satellite images are relatively uncommon), thus hard to benefit from scaling or generalization from existing data.Fig. 1a in the main paper has illustrated qualitatively that the class distributions of large-scale image-text datasets are roughly shared (correlated). Here, we also provide quantitative results about the correlation coefficients between the class distribution of different image-text datasets Fig. 13 . Under most concept sets, the correlation is high and supports our claim: there exists a distributional convergence across large-scale image-text datasets. Results of MetaCLIP [91] variants are relatively less correlated, which might be due to the re-balancing operation in the curation process.Our estimation of concept frequency is based on a simple rule-based pipeline (see details in Appx. B.2), which could be prone to caption noise, multi-label, and ambiguity. A concurrent work by Parashar et al. [64] finds concept synonyms using ChatGPT [60] , and estimates the class frequencies of each caption using Llama 2 [84] . These advanced techniques may produce more accurate class frequencies. In Fig. 14 , we provide the correlation coefficients between our estimations and the results of [64] . The high correlation across most datasets implies an agreement and verifies the validity of our estimations.There is an exception for DTD [16] , in which class names are about descriptive textures. This is more abstract than natural concepts and can be more semantically ambiguous [64] , and require more sophisticated designs in frequency estimation.LAION-2B MetaCLIP-2.5B [64] . There is an agreement on most concept sets except DTD [16] , which is about descriptive textures and can be more semantically ambiguous [64] .As illustrated in Figs. 1b and 5 , the discriminability and robustness to pre-training data imbalance improve simultaneously as data scales up. But neither does it mean CLIP is unbiased (see discussions in Sec. 3.7), nor does it indicate CLIP is absolutely robust to data imbalance. In Fig. 15 , we plot binned results of CLIP following Parashar et al. [64] . Looking at the average trend, the tail classes are still of inferior performance. However, note that the standard deviation is high, indicating there are still many good-performing tail classes. Moreover, the figure also verifies CLIP is more robust than SL (Fig. 15a ), and the harm of data imbalance alleviates as data scales up (Fig. 15b ).The design of our study is largely influenced by [27] , which is among the first to study data's effect on CLIP's robustness. After ruling out the effects of language supervision and data distribution, we found there is still a notable gap between CLIP and SL in Fig. 3 . We then exhausted every factor we could to align details between models, and finally found the pretext task of dynamic classification to be a key factor, which could implicitly de-bias classifiers, and reproducible by applying vocabulary subsampling. Besides that, we also considered other factors like the architecture of vision backbone and text backbone, vision pre-training, stronger data augmentation, larger batch size, and test-time prompts, and did not find noticeable effects. Additionally, we looked into the properties of the dataset instead of models and found that web data had mixed effects. Further, we extend the scaling law of CLIP and find open-world data to be an effective factor. scales, including MS-COCO [14] , CC-3M [76] and 12M [12] , YFCC-100M [80] and 15M [68] , WIT [79] , SBU [62] , RedCaps [20] , LAION-400M [73] and 2B/5B [74] , and MetaCLIP [91] , etc. This work considers those with both metadata and pre-trained CLIP publicly available, i.e., CC-12M, YFCC-15M, LAION-400M/2B, and MetaCLIP-400M/2.5B.This study specifically examines the classes of ImageNet [18] , which encompasses 1K common object categories. To obtain the class distribution on image-text datasets, we follow the common practice [27, 77, 91] to query captions with class names and their WordNet [56] synset. In implementation, we also loosen the sub-string matching condition to set-level matching (overlooking the order of words) for a higher recall, and manually introduced negative words (e.g., 'vehicle', 'truck' for class 'ram', 'bird', and 'wing' for class 'crane') to reduce false positives. Besides, we normalize letters to lowercase, remove non-letter and non-number symbols, and lemmatize words to nouns. For MetaCLIP, which provides a readily available distribution of 500K concepts, we simply summed up the statistics of target concepts (classes). And for other datasets, we ran the search over all captioning data.The models are collected from the models of OpenCLIP [15] . We select models that have captions or metadata of the pre-training dataset publically available, and restrict the backbones to ResNet [33] , ConvNeXt [52] , and ViT [22] . The remaining set comprises 41 models covering different model architectures (6 ResNets, 8 ConvNeXts, and 27 ViTs), model scales (ResNet-50/101, ConvNeXt-B/L/XL, and ViT-B/L/H/G), data scales (from 12M to 2.5B), training schedules, and optimization techniques. An overview of the results of these models is provided in Fig. 18 .Our training settings follow the common practice in [27] , CLIP experiments utilize cross-entropy losses and the AdamW optimizer. The initial learning rate is set to 0.001, and a cosine-annealing learning rate schedule with 500 warmup steps is employed. The hyper-parameters for AdamW are β 1 = 0.9, β 2 = 0.999, and ϵ = 10 -8 . The batch size is set to 1024. Model training lasts for 32 epochs. We also tried training 90 epochs to match that of SL but found 32 epochs is enough for convergence and longer training has no notable benefit. SL models are trained using SGD with Nesterov momentum for 90 epochs. The weight decay is set to 0.0001, momentum to 0.9, and batch size to 256. The initial learning rate is 0.1 and is decayed by 0.1 at epochs 30, 50, and 70.To maximally align details with CLIP, both methods adopt the slightly modified ResNet structure as in [68] . The augmentation pipeline is also kept consistent: random resized crop to size 224 with a scale range of (0.9, 1.0), followed by normalization with a mean of (0.48145466, 0.4578275, 0.40821073), and a standard deviation of (0.26862954, 0.26130258, 0.27577711). Note that this data augmentation pipeline is notably weaker than those commonly used by SL.For • template-based captions, the caption of an image is generated using a randomly sampled template from 80 class templates provided in [68] , e.g., "a photo of a [class]". If synsets are used, the class name [class] is also randomly sampled from its synsets. For • natural-language captions, we refer to Fig. 2 (upper) for an example image and corresponding text metadata (including title, description, and tags). More examples can be found in Fig. 3 of [27] . The way captions are created is simply by concatenating metadata together with spacing. E.g., if the [title] is "A phone call and night", and the [description] is "I might have a thing with telephones. . . ", then the resulting caption is [title description]: "A phone call and night I might have a thing with telephones. . . ". This follows the practice of [27] , and is also the way CLIP [68] curates caption data from YFCC-15M.The training vocabulary refers to the label set that a model classifies at a specific training iteration. Given a mini-batch of samples, a minimal label set is formed as the union of all GTs in this mini-batch. If the expected vocabulary size is not met, we additionally sample classes from the remaining, and the probability a class is selected is determined by the frequency of the corresponding class in the pre-training data. Note that the sampling is performed at the class level, which differs from the sampling strategies in long-tail learning that are done at the sample level. We also tried uniform sampling, i.e., treating each class with equal probability, which yielded slightly weaker results.Discussions. For SL, vocabulary subsampling refers to randomly reducing the size of candidate classes (akin to dropout on the classification head) when classifying an image during training. 1) Regarding how it works, Fig. 3a (y-axis ) shows it effectively reduces the model's predictions' correlation to class frequencies, a key indicator of classifier's bias. 2) Regarding why this technique can de-bias classifiers, our intuition is that this plays a similar role to dropout: the classifier is regularized to put equal importance on all classes. Biases still exist in the subsampled classes, but the gradients cancel out each other during training. 3) Regarding why frequency-based sampling works better than dropping all classes with equal probability, we hypothesize that the dropping operation can de-bias the classifier regardless of how classes are selected, and sampling by frequency is more helpful for representation learning. The intuition comes from the finding in long-tail learning that resampling data by inverse frequency helps de-bias classifier, but harms representation learning [38, 97] .For CLIP experiments, the text encoder is trained from scratch by default. If the text encoder uses frozen CLIP, this means the text encoder is initialized by the pre-trained CLIP weights from [68] .During training, the parameters of the text encoder remain unchanged. In the CLIP init setting, after initialization, the text encoder is also fine-tuned in the training process. Further, for RoBERTa experiments, we follow the implementation of [15] and replace the text encoder with pre-trained RoBERTa [51] available on HuggingFace [89] . This is kept frozen during training, as we found fine-tuning it results in NaN loss.For SL experiments, we replace the commonly used linear classifier with a prototypical classifier to better follow CLIP's structure. This means the bias term in this linear layer is omitted, and both the feature from the backbone and the classifier's weight are ℓ 2 -normalized, thus weights in the linear layer can be viewed as a set of prototypes (feature vectors). To facilitate optimization, a learnable scaler with a maximum scale of 100 is added as CLIP [68] to upscale logits. For the setting using fixed prototypes obtained from CLIP, we format each class to a sentence using the template "a [class]", feed them to the text encoder of a pre-trained CLIP, and keep the output class-wise text features as SL model's classification head/prototypes.ImageNet-Captions subsets. Starting from the original ImageNet-Captions [27] , we take only image-text pairs that correspond to the 100 classes of Tian et al. [81] , thus obtaining a 100-class subset called ImageNet-Captions-100. Besides, we randomly sample from ImageNet-Captions and construct a subset that is of the same scale as ImageNet-Captions-100 but with the same number of classes as ImageNet-Captions. This subset is called ImageNet-Captions (10%). Note that it is of the same scale of ImageNet-Captions-100, and not necessarily 10% of ImageNet-Captions. Subsample Ratio 100% 50% 25%12.5% 6.25% 3.125% LAIONet variants. LAIONet [77] is a subset of LAION-400M [73] created by matching between ImageNet class synsets and captions. Items with low CLIP text similarity between the caption and class definition are filtered out to reduce label noise.Our reproduction sets 0.7 as the default threshold, and 3.26M images are successfully crawled. Experiments in Sec. 3.4 consider LAIONet variants filtered with different text-definition similarity thresholds: 0.7, 0.75, 0.8, 0.82, and the sizes of corresponding LAION-400M subsets are originally 3.26M, 1.93M, 0.88M, and 0.58M. We then randomly subsample them to be the same scale as ImageNet-Captions (0.45M). Besides, the variant that matches the class distribution of ImageNet-Captions is sampled from the 3.26M version, and the scale is also kept the same as ImageNet-Captions. In addition, experiments in Sec. 3.5 use LAIONet subsets randomly sampled from the 3.26M version (threshold set to 0.7), at the portion of 1 /1, 1 /2, 1 /4, 1 /8, 1 /16, and 1 /32, respectively. The distributions of these randomly sampled subsets are shown in Fig. 16 .CC-12M-Cls and YFCC-15M-Cls. These are classification subsets of CC-12M and YFCC-15M that have corresponding class labels of 1K ImageNet classes for each image. The curation process follows Fang et al. [27] , except that we allow class name matches to be not case-sensitive. In comparison to LAIONet, it is simply substring matching without filtering. The resulting datasets are at a scale of 3.48M (CC-12M-Cls) and 2.90M (YFCC-15M-Cls), respectively.Unless otherwise specified, the evaluation of models is all performed on ImageNet validation split.For CLIP, the default zero-shot classification setting is applied. That is, each class is embedded as an average vector of text features produced using 80 class templates provided in [68] . Then for both CLIP and SL, the predicted class is that of the nearest neighbor class prototype.Experiments are conducted on NVIDIA A100 GPUs. Each CLIP and SL training experiment runs on 4 GPUs in parallel, and there are roughly 400 experiments (data points) for the controlled study.Self-supervised learning from pseudo-labels. It is natural to extend SL to self-supervised settings for representation learning, as long as pseudo-labels are available. Earlier work [8] applies k-means clustering to deep features and takes cluster assignments as pseudo-labels. Following works [2, 10] reform pseudo-labeling as optimal transport and solve it with the Sinkhorn Knopp algorithm. This is then simplified by DINO [11] with centering and sharpening operations on the model's predictions, and extended to soft labels (thus called self-distillation instead of self-labeling).Knowledge DIstillation with NO labels (DINO). DINO [11] is a discriminative self-supervised visual pre-training method. The pretext task is formulated as self-distillation: enforcing the student model's predictions to be close to teacher models' soft pseudo labels. The input to two models are random augmented views of the same image, and the teacher model is updated as the exponential moving average of the student model (also called "mean teacher"). DINO learns a set of prototypes (feature vectors) as the classification head, and is used by student and teacher models to produce logits and pseudo labels. Since the prototypes resemble a classification head, the aforementioned vocabulary subsampling strategy can also be similarly applied to DINO.The training details follow the suggested practices of DINO [11] for training ResNets. That is, train using SGD optimizer with a base learning rate of 0.03, and fixed weight decay of 0.0001. The scale of global crops is (0.14, 1), and the scale of local crops is (0.05, 0.14). Other hyper-parameters are kept as default. We use the ResNet-50 backbone with the same structure as Radford et al. [68] , and train for 100 epochs with a batch size of 1024.The last layer of DINO's projection head is equivalent to a set of prototypes, thus it is natural to integrate the techniques experimented to be valid on classification models. We keep the total number of prototypes to 65536 as default.For vocabulary sampling-based DINO, we subsample the same set of prototypes for the teacher and student models and compute the self-distillation loss on this restricted prototype set. The vocabulary (prototype set) is shared in a mini-batch, and different across training iterations.Datasets and metrics. We test models' transfer learning performance on the benchmark initially proposed in [40] , and adopt the implementation from [23] . The datasets in this benchmark include: Food-101 [7] , CIFAR10/100 [42] , Birdsnap [5] , SUN397 [90] , Stanford Cars [41] , FGVC Aircraft [54] , PASCAL VOC 2007 [24] , Describable Textures (DTD) [16] , Oxford-IIIT Pets [65] , Caltech-101 [28] , and Flowers-102 [59] . The evaluation metric is mostly top-1 accuracy, with exceptions of mean per-class accuracy on FGVC Aircraft, Oxford-IIIT Pets, Caltech-101, and Flowers-102, and 11-point mAP on PASCAL VOC 2007.Linear probing. Image features are extracted from the backbone of the teacher model following [11] .Then following [23] , we train an ℓ 2 -regularized multinomial logistic regression classifier on frozen features extracted from the backbone. The model is optimized using L-BFGS on the softmax crossentropy objective. No data augmentation is applied, and the images are resized to 224 pixels along the short size using bicubic resampling and center-cropped to 224 × 224. The hyper-parameters for ℓ 2 -regularization are searched from 45 logarithmically spaced values between 10 -6 and 10 5 .In Fig. 17 , we provide an example of the distribution of subsampled classes and per-class zero-shot accuracy of CLIP (ViT-B/32) pre-trained on • LAION-400M and ■ MetaCLIP-400M accordingly.The head classes are easy to be found the web, e.g., "T-shirt", "mobile phone", "throne", and "goose", etc. In contrast, the tail classes are dominated by fine-trained biological concepts, ranging from "barn spider", "earth star fungus", to "gyromitra". Collecting such data is hard and requires expert knowledge. Despite this, we find both models can achieve good performance on some tail classes. In supplement to the analysis in Fig. 1b where results of CLIP are averaged by the dataset it trains on, we provided more detailed results of CLIP in Fig. 18 . Besides zero-shot classification results on ImageNet [18] , Fig. 18 also provides results evaluated on ImageNetV2 [70] . Results are consistent. The weak correlation to sample frequency is consistent whether evaluated on ImageNet [18] or ImageNetV2 [70] .In supplementary to the analysis in Fig. 3 , which is conducted under the setting that models are trained from scratch. Here we also provide the results that all models are trained using frozen CLIP text encoders/heads in Fig. 19 . We find that the results are generally consistent with those in the main paper. In addition, we find language pre-training provides a shortcut to models and allows them to leverage language supervision (CLIP) and debiased pretext tasks (SL) with higher effectiveness. This is supported by the sharper slopes in (a, blue line) and (b, green line) in comparison to Fig. In addition, the models using • template-based supervision are (a) less biased and (b) show better accuracy than the training-from-scratch counterparts in Fig. 3 , indicating the knowledge in language pre-training to be obtained by CLIP. This also holds true for SL and • natural language-supervised CLIP, as supported by shaper slopes in (a, blue line) and (b, green line).In supplement of Fig. 7b , we also plot the vision feature centers and corresponding sample features of some classes in Fig. 20 . Results are produced by a CLIP ViT-B/32 model pre-trained on LAION-400M, and obtained by inferencing on the ImageNet validation split. Note that vision and text features are plotted separately due to the modality gap (despite being in the same feature space) [47] . Fig. 20a shows the features of images from some subsampled classes, and corresponding vision feature centers. In coherence to results in Fig. 7a .2, there is not a clear tendency on whether head or tail classes form compactor clusters. In addition, Fig. 20b and Fig. 20c show the vision and text feature centers of all ImageNet-1K classes, where head and tail classes are highlighted. The vision feature centers are produced by averaging sample features by classes, and the text feature centers are as of the classifier used by CLIP, as described in Appx. C.6. The margins between tail classes encoded by the vision encoder are notably smaller. In contrast, tail class centers produced by the text encoder are better separated. This phenomenon might be connected with the modality gap [47] , and is of research value for future explorations.In Tab. 2, we provide the original numeric data used to obtain Fig. 10 for reference. To provide a clearer image of the imbalanced class distribution of pre-training datasets, we show a zoomed-in version of Fig. 1a with linear scale in Fig. 21 . Also, we see that MetaCLIP does successfully alleviate the dominance of head classes. But note that unfortunately, all datasets are still extremely imbalanced, and how to improve models' robustness to it is still to be explored. Justification: This paper's contributions and scope are reflected.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Full implementation details are provided in Appxs. B to D.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: Figures are plotted with 95% confidence intervals or standard deviations.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Provided in Appx. C.7.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: This paper conforms with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: An impact statement is provided in Sec. 5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper does not pose such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The assets are cited and corresponding licenses are respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.MetaCLIP [91] is relatively more balanced than other datasets due to concept re-balancing in curation. 
 
 Appendix:  


Decision: Though the paper has mixed reviews, the majority of reviewers recommend acceptance and appreciate the detailed analysis of the robustness of CLIP-style models relative to their training data distributions and with respect to robustness for imbalanced data in particular. The AC team agrees that this analysis adds value to the community, and thus recommend acceptance.

Reviews: review: paper_summary: paper_summary: CLIP models are known to exhibit good robustness and high quality representations compared to supervised models. In this paper, the authors study CLIP models through the lens of the distribution of concepts in the underlying training datasets. The authors find that CLIP appears to demonstrate better robustness to data imbalance compared to supervised models. The authors conduct a detailed study the analyze various factors influencing robustness, and show that applying a dynamic classification task inspired by CLIP training improves robustness in supervised models.
strength_weakness: strength_weakness: - A major experimental claim is that CLIP models exhibit a lower correlation between ImageNet class accuracy and frequency. However, this claim makes the assumption that the frequency of these classes in the training data is accurately measured, which may not be true for LAIONet, which is too large to verify. If rare classes are undercounted, which could affect the correlation.
- (Line 140) The statement alf a batch size 1024 training run having something like 600 imagenet classes is probably roughly true, although large models trained in industry clusters likely have a much larger batch size, and closer to 1000 classes.
- When applying the proposed class sub-sampling training method (Figure 10, section 4.2), the results are only stated qualitatively as narrowing the gap. However, a quantitative analysis would be more appropriate. In general, this proposed training technique seems very interesting, and I would appreciate seeing more significant analysis of how this could be used.
- The authors state that their method “uncovers the mechanisms behind CLIP’s generalizability’ (Line 12). However, while I agree that their experiments find quality evidence for several key relationships that are involved in generalizability/robustness, it is too generous to say that they have fully uncovered these mechanisms, or that they have been characterized in detail. I believe the authors should soften these claims.
questions: questions: This paper describes several of the key mechanisms that influence model robustness. Besides the proposed training technique for supervised models, what recommendations would the authors propose for improving the next generation of self-supervised/CLIP models?
limitations: limitations: The authors have a section that describes the limitations of the work (Appendix F)
paper_summary: paper_summary: Motivated by the novel observation on CLIP's robustness against data imbalance, this paper provides extensive analysis on factors that determine such robustness, including language supervision, classification templates, data scaling, distribution, and open-world concepts. The paper shows that such factors are transferrable to supervised or self-supervised learning to reach CLIP-level robustness.
strength_weakness: strength_weakness: 1. The importance of data scale predominates other factors:
- in ablation studies for data scaling, the setup with frozen CLIP text encoder header only shows the reasonably low correlations (still, much higher than large-scale examples shown in Fig.1) among other setups. The similar results are observed in the ablation studies in different distributions (Fig. 5-b) as well as extreme case (Fig. 8), where given the same distribution, the setup with frozen CLIP text encoder shows superior correlations. Are these results indicating that the large-scale data pretraining predominates the CLIP's robustness on data imbalance? In other words, given large-scale data, e.g., metaCLIP-2.5B, how ineffective are those factors studied in the paper? Are they still influential or negligible?  
- In Figure 5, I find it difficult to clearly understand regarding distribution shifts. It needs more details regarding the distribution shifts and the meaning of the dashed line. How are the results calculated and what is the take-away message?
- In Figure 6, the CLIP with YFCC-15M shows stronger correlations than that with YFCC-15M-cls, which cannot be explained by the authors’ claim that open-world datasets improve robustness.

2. In section 4.1, additional results on SL using init head + sub voc. are necessary to fairly ablate the factors for showing transferability to SL.

3. It is difficult to clearly understand the setup in Section 4.2. It lacks detailed explanation on setups for comparison, and the measures only include downstream task performance, not the correlations.
questions: questions: - How did you calculate the ImageNet-class frequency in web-scale datasets in Fig.1 (a)?
- Regarding weakness 1., experiments on ablating the factors directly onto the large-scale-trained models and test their correlation as done in Fig. 2 should be included.
- Legend colors do not match to those in Figure 9 (b).
- Please more provide details of Section 4.2., such as, what models authors compare, and what models the different colors in Fig. 10 indicate.
limitations: limitations: See weakness.
paper_summary: paper_summary: This paper explores the remarkable robustness of CLIP training compared to supervised training when dealing with real-world imbalanced datasets. This advantage can be attributed to dynamic classification using a subset of classes. Furthermore, the paper shows that robustness and discriminability improve with denser language supervision, larger data scales, and broader open-world concepts. The experiments are mainly conducted on ImageNet-captions and LAIONet to validate the main claims.
strength_weakness: strength_weakness: -	This paper explores vision-language pretraining (VLP) under imbalanced distributions primarily from a data perspective, focusing on language supervision, data scaling, and open-world concepts. These factors, known to enhance robust performance, have been discussed in several previous studies.
-	Other factors beyond data may also affect the robustness of VLP under imbalanced distributions. For instance, from a loss perspective, comparing the learning abilities of CLIP loss and cross-entropy loss in supervised settings, particularly under imbalanced conditions, needs discussion.
-	In the language supervision section, a more comprehensive experiment would compare self-supervised contrastive learning methods, as they share the same contrastive objective. This comparison would isolate the differences between contrastive and cross-entropy losses, focusing solely on variations in language supervision.
-   It seems that the concept frequency estimation in this paper may suffers from noise, multi-label structure. 
-   The evaluation is mainly focusing on classification.
questions: questions: please refer to the weakness part.
limitations: limitations: The limitations are discussed in this paper. No potential negative social impact.
paper_summary: paper_summary: This author empirically studies the mechanisms behind CLIP's generalization ability from multiple aspects. Based on the findings, they further transfer the merits of CLIP to other scenarios like supervised learning and self-supervised learning.
strength_weakness: strength_weakness: 1. The authors studied the robustness/generalization of CLIP from various points, but some points are known to the community. Sections 3.2 and 3.4 claim that higher caption density and larger data volumes in training data contribute to robustness in terms of imbalances. However, this observation appears to be well-understood in the community. It is generally accepted that high-quality, richly described data is preferable.
2. I appreciate the presented comprehensive studies. However, the settings and conclusions in Section 3 are occasionally unclear. I recommend adding a brief conclusion to each subsection to enhance clarity and coherence.
3. In Section 4.2, which examines subsampling vocabulary in self-supervised learning with DINO, the impact of this technique is only demonstrated on models with a default dimension of 65,536. It remains unclear if this approach would be equally effective on other self-supervised learning models that utilize significantly smaller default dimensions, such as 2,048.
questions: questions: 1. The methodology for handling training data during the random subsampling of vocabulary, as discussed in Section 3.3, remains ambiguous. With a batch size of 1024 and a reduced vocabulary size of 500, how are the remaining samples, whose labels do not fit within the reduced vocabulary, treated? This clarification is essential for understanding the implications of the approach on the learning process.
2. Is there a correlation between smaller vocabulary sizes and smaller batch sizes in these experiments?
3. The rationale behind the selection of specific thresholds (0.7, 0.75, 0.8, 0.82) in Figure 5 is unclear, particularly the choice of 0.82 over 0.85.
limitations: limitations: 1. No code is provided at this stage.

Comments: title: Message from the Authors
A TL;DR version of our response
Thank you!
Thank you for your positive feedback!
Thank you for your response.
Looking forward to having further discussions
Looking forward to having further discussions
Looking forward to having further discussions
comment: Dear Reviewer XjUF,

We sincerely appreciate the considerable time and effort you have dedicated to reviewing our paper. Your thought-provoking questions and insightful feedback have been instrumental in improving the quality of our work.

We believe we have provided sufficient evidence to address your concerns. Although you still have concerns about the CLIP head (**not our contribution but just an analysis tool**), we have done our best to address it by showing additional experiments and explaining the figures. But before elaborating on this issue, we would like to re-emphasize the main focus and contributions of the paper, as the discussions above might have fallen too much into detail and lost the big picture.

**What is this paper mainly about.**
We are one of the first works to dig into the concept distribution of web-crawled datasets and reveal the robustness of industry-trained CLIP to it. We conduct a series of systemic and controlled experiments to ablate different factors and identify the influential ones to CLIP. We further verify that these factors can be transferred to SL/SSL settings. Given that the community is switching from model-centric to data-centric, from lab-scale data to web-scale data, our work 1) identifies a key problem: web data is extremely imbalanced; 2) identifies a good property of CLIP: robustness to such pre-train data; 3) reveals mechanisms behind it: both model and data factors; and 4) provide transferrable insights to other domains (eg, both long-tail and open-world recognition works have been using CLIP head as is, and we highlight that introducing its pre-training pretext to downstream task is also necessary). We also provided all code and data to the AC, believing that releasing this project could greatly benefit future related research.

**The scaling law between data and robustness to data imbalance is one privilege of CLIP.**
While it is commonly believed that scaling up data is beneficial to model performance (eg, accuracy), this does not directly apply to models' robustness to data imbalance/bias. One example is the results of SL (as in Fig. 1b). More importantly, **it also does not apply to generative LLM/MLLMs that are also pre-trained on massive data** (detailed in our response to reviewer tULz Q1). Considering their vocabulary as a giant class set, the way they predict each token (autoregressively) is also the same way with SL (using CE loss) and could be a cause of bias. Instead, our work reveals that CLIP implicitly applies subsampling to its vocabulary and continues to benefit from larger-scale data. This means **CLIP's factors are not undermined by, but are the root causes of more-data-more-robustness to data imbalance**.

**Scaling up data is orthogonal to other factors and they can be mutual-complementary.**
We also would like to emphasize that the factors we discovered and not contradictory with the data scale, and this is also exactly the reason we included results using frozen CLIP heads (which means knowledge from large-scale data). 1) In the additional experiment we provided in the rebuttal, we extended Fig. 3 to the largest scale we could, showing that caption diversity and vocabulary subsampling are consistently helpful as data scales up. 2) The blue lines (w/ frozen CLIP head) are aimed at **showing model behavior when large-scale knowledge is readily available and only small-scale controlled data is at hand**, and the trends cohere with training-from-scratch results, parallelly defending our findings. 3) The gap between this setting is apparent, but note that these factors are **mutual-complementary** with data scale -- **imagine what happens if we synthesize a giant dataset with extreme data imbalance and massive duplicated low-quality data** -- representations learned could be worse than training on CIFAR10.

We sincerely hope that our responses have satisfactorily addressed your concerns. We remain open and enthusiastic about addressing any additional questions you may have, as your continued input would greatly contribute to further improving our work. We would like to thank you again for your devotion to reviewing our manuscript and will continue to improve its quality by all means. We also treasure this opportunity to share the problems found, the design of the study, and the findings with the NeurIPS community and sincerely hope these could be inspiring for future research.

Best regards,

Paper 4735 Authors
Dear Reviewer WXxS,

We sincerely appreciate the considerable time and effort you have dedicated to reviewing our paper. Your thought-provoking questions and insightful feedback have been instrumental in improving the quality of our work.

In response to your queries and concerns, we have carefully crafted detailed answers. We are keenly aware of the numerous submissions you are managing and deeply appreciate your attention to our research. Given the time constraints during this busy discussion period, we have summarized the main points of our rebuttal below. **This concise overview aims to provide a quick reference while respecting your valuable time.**

- Code with all implementation details is provided to the AC and will be publicly available very soon (L1).
- We study CLIP's robustness to **imbalanced pre-training data**, while prior works study robustness to **OOD evaluation data**, and prior conclusions do not directly apply to our setting; We also find data factors that harm robustness; We also identify important model factors that make CLIP stand out, and **even modern LLM/MLLMs trained with large-scale data suffer from imbalanced data** (W1).
- Vocabulary subsampling could be a general way to rebalance (pseudo-)labels in supervised and unsupervised settings. Still, in this section we intend to pose a general problem (**model-side debiasing is widely needed when training on web data**) rather than proposing novel solutions (W3).
- Details of settings are provided in the appendix and questions are explained in our response (W2 & Q1/2/3).

We sincerely hope that our responses have satisfactorily addressed your concerns. We remain open and enthusiastic about addressing any additional questions you may have, as your continued input would greatly contribute to further improving our work.
Dear Reviewer iwVX,

We appreciate your positive feedback and are truly delighted to see that our response has addressed your concerns and questions.

Thank you once again for dedicating your time and effort to reviewing our work and providing us with insightful suggestions!
Thanks for the detailed responses. Most of my concerns have been addressed. I appreciate the additional experiments w.r.t self-supervised learning method DINO on ImageNet-Captions. Overall, I notice the importance of this paper on understanding imbalanced multimodal learning, and would like to raise my score to '5'.
Dear Reviewer tULz,

We appreciate your positive feedback and are truly delighted to see that our response has addressed your concerns and questions. If all weakness points are resolved, would you mind considering a raise of your recommendation?

Thank you once again for dedicating your time and effort to reviewing our work and providing us with insightful suggestions!
Thank you for addressing in depth my feedback for the paper. I believe that the response satisfies my points.
Reviewer iwVX, do you have any additional questions or feedback?
Reviewer WXxS, do you have any additional questions or feedback?
Dear Reviewer WXxS,

Thank you so much for your time and efforts in reviewing our paper. Hope our rebuttal has addressed your concerns, and we will revise our paper based on your feedback. We are happy to discuss with you further if you have other concerns. Thanks for your time!

Best regards,

Paper 4735 authors
Dear Reviewer iwVX,

Thank you so much for your time and efforts in reviewing our paper. Hope our rebuttal has addressed your concerns, and we will revise our paper based on your feedback. We are happy to discuss with you further if you have other concerns. Thanks for your time!

Best regards,

Paper 4735 authors
Dear Reviewer tULz,

Thank you so much for your time and efforts in reviewing our paper. Hope our rebuttal has addressed your concerns, and we will revise our paper based on your feedback. We are happy to discuss with you further if you have other concerns. Thanks for your time!

Best regards,

Paper 4735 authors
Dear reviewer XjUF,

Thanks for your great efforts in reviewing our paper. We respect your opinions but thought it best to emphasize that:
- The aim of our paper is to answer **what makes existing industry-trained CLIP models robust to the data imbalance in their pre-training datasets**. We considered a total of 41 CLIP models in OpenCLIP, and their trend is presented in Fig. 1b as a motivating point. All of them are *trained from scratch*. Thus we already presented a complete study with the "from scratch" results. The head is not a valid factor to answer this question, as again, **all of them are trained from scratch**.
- The reason we additionally provided a parallel discussion on the head is simply to 1) show a practical trick for training CLIP on small-scale data, and 2) answer the question "When CLIP is trained with much more data, how valid are other factors?" by simulating large-scale trained-from-scratch CLIPs. Combining with the former, the takeaways of our paper include both tricks for training CLIP with restricted computation and more general guidance. For the latter, our study re-evaluates the factors studied and shows that they are still valid when the model has more knowledge.
- The fact that correlations with the head are much lower is expected -- it knows 400M data, compared to the 3M (at most) of training from scratch. Still, it is higher than that of Fig. 1b (trained from scratch on 400M data). When comparing from scratch and results with the head, control of variables is important. Again, **the head is not a valid factor of CLIP, and we just intended to show a practical trick to the community.** It is a very minor part of our paper and we also do not claim this trick as our contribution.
I greatly appreciate the authors' detailed responses for further clarification.

- The authors' responses clarified their intuition that several data factors show trends of improving robustness when independently considering models trained from "scratch"  from those with"Frozen CLIP Head". However, I still think it is true that "Frozen CLIP Head" w/o considering such data factors (e.g., sub voc, data scale) even provides substantial improvements to robustness compared to "scratch" with considering such factors. In other words, the contribution of such factors becomes highly marginal when comparing to that of "Frozen CLIP Head". This takes back to my original concern: _using “Frozen CLIP head” taken from pretrained model with large-scale data is the dominant factor that provides the low correlations_. 
- I greatly appreciate the authors' attempts for better understandability of the paper. However, I think the current paper requires more than minor level of revision for concise and clear presentation of the results as well as their insights, including critical error in Fig. 6 of the original submission.

Therefore, I lean to maintaining my original rating for now, but I may re-assess after discussions with other reviewers.
Dear reviewer XjUF,

Thanks for your responses and we would like to make some further clarifications:

> Is "Frozen CLIP head" taken from pretrained model with large-sacle data the dominant factor?

Concerning whether conclusions of Fig. 4b are dominated by “frozen CLIP head”:

1. It is important to emphasize that directly comparing the correlation values of models trained from scratch with those of a frozen pre-trained CLIP head is not fair, nor do we intend to make such a comparison in the figure. This is because the CLIP head is trained on 400M data points, while the other methods are trained on at most 3M data points. Due to limited computational resources, we could only scale our data to the million level for these studies. However, in Fig. 4b, we observe a clear trend: as the data scale increases, the correlation consistently decreases, indicating that data scale is indeed a significant factor in robustness.
2. Furthermore, in Fig. 1, where we scale the data from 3M to 2.5B, it becomes even more apparent that increasing the data amount leads to lower correlation values. It's important to note that *all models in Fig. 1 are trained from scratch*, meaning that the pre-trained head is not a contributing factor to the results in Fig. 1. Interestingly, the low correlation observed when using a frozen CLIP pre-trained model actually supports our hypothesis that robustness improves significantly with larger data scales. This implies that with substantial amounts of data, the trend toward increased robustness becomes more pronounced. 
3. The reason we incorporated frozen CLIP head into Fig. 4 is to illustrate that the feature space learned by CLIP from large-scale data can serve as a teacher to finetune models with a smaller scale of training data while achieving robustness *for free*. Still, even in this setting, it can be observed that robustness can be enhanced with more training data.

Concerning whether conclusions of Fig. 5b are dominated by “frozen CLIP head”:

4. In Fig. 5b, we use "frozen CLIP head" to 1) estimate the behaviors of a large-scale trained CLIP when only small-scale controlled data is available, and 2) verify how effective the data factors are given knowledge from large-scale data (exactly your major concern). The results from the "frozen CLIP head" and the "from scratch" setting should not be compared directly. Instead, they should be interpreted separately, as both approaches can independently verify the effect of data imbalance, diversity, and distribution shift. The results of Fig 5b show that tendencies of "from scratch" and "frozen CLIP head" are coherent, indicating these data factors are consistently effective, regardless of whether knowledge from large-scale data is available. Fig. 3 under this setting is also discussed in Fig. 17.

We apologize for the confusion and will revise the figure to clarify the presentation. Additionally, we will separate the study into individual figures to enhance clarity.

> Is “no distribution shift” equal to “=freq”?

Sorry for the confusion, the revised figure in our PDF response might be clearer than the original one. Results between LAIONet subsets have no distribution shift, and only difference in intra-class image diversity (if using different thresholds) or level of data imbalance (if using same threshold, different class distribution). "=freq" refers to the LAIONet variant that has the same class distribution with IN-Caps, it should be compared with IN-Caps and the only difference between them is distribution shift.

> For heavy-tailed classes in both 1 and 0-shot, is relying on large-scale pretrained CLIP head our main contribution?

No, using pre-trained "frozen CLIP head" is already a common practice in open-world recognition (more similar to the setting in Sec. 4.1) and long-tailed classification, and not our novel contribution. But note that, none of them considered introducing the training mechanisms of CLIP (vocabulary subsampling) to the downstream task. Our contribution is to show adding vocabulary subsampling is important to acquire CLIP knowledge when its head is reused.

> The authors' responses do not include the results of correlation

Please note that Sec. 4.1 considers a very different setting than Sec. 3, aiming to let models better acquire CLIP knowledge when pre-trained CLIP head is used, and not to study factors behind CLIP's robustness. In this setting, the expected property is higher tail/novel-class accuracy. Correlation values indeed can be computed, but note that many classes have the same 1/0 frequency, and the correlation is less indicative in this case.
I appreciate the authors’ rebuttal. Please find my responses below:

1. My major concern in W1 and Q1 was that most of the analysis, e.g., Figs 4-b, 5-b and 8, using “Frozen CLIP head” taken from pretrained model with large-scale data is the dominant factor that provides the low correlations, while other factors show minor impacts. Specifically, in Fig.4-b, only those with Frozen CLIP head shows meaningfully low correlations, while others fail to do so. In addition, in Fig 8, results with "Frozen CLIP head" show improved tail accuracy, while others fail. These results give an impression that leveraging the "Frozen CLIP head" is the major factor that suggests the robustness among various factors explored in the paper, which is not surprising. The authors' responses do not seem to fully address the issue.
2. Is “no distribution shift” equal to “=freq”? 
4. Thanks for the clarification. However, the authors' responses further strengthen my impression that this paper’s main contribution is heavily relying on large-scale pretrained CLIP head for heavy-tailed classes in both 1 and 0-shot.
5. The authors' responses do not include the results of correlation, which is the main focus of this paper.
3, 6, 7. Thanks for the clarification.