No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance

Authors: vishaal  udandarao
ameya  prabhu
adhiraj  ghosh
yash  sharma
philip h s torr
adel  bibi
samuel  albanie
matthias  bethge
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:12:04

Abstract: Web-crawled pretraining datasets underlie the impressive "zero-shot" evaluation performance of multimodal models, such as CLIP for classification and Stable-Diffusion for image generation. However, it is unclear how meaningful the notion of "zero-shot" generalization is for such multimodal models, as it is not known to what extent their pretraining datasets encompass the downstream concepts targeted for during "zero-shot" evaluation. In this work, we ask: How is the performance of multimodal models on downstream concepts influenced by the frequency of these concepts in their pretraining datasets?

We comprehensively investigate this question across 34 models and 5 standard pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M, LAION-Aesthetics), generating over 300GB of data artifacts. We consistently find that, far from exhibiting "zero-shot" generalization, multimodal models require exponentially more data to achieve linear improvements in downstream "zero-shot" performance, following a sample inefficient log-linear scaling trend. This trend persists even when controlling for sample-level similarity between pretraining and downstream datasets, and testing on purely synthetic data distributions. Furthermore, upon benchmarking models on long-tailed data sampled based on our analysis, we demonstrate that multimodal models across the board perform poorly. We contribute this long-tail test set as the Let it Wag! benchmark to further research in this direction. Taken together, our study reveals an exponential need for training data which implies that the key to "zero-shot" generalization capabilities under large-scale training data and compute paradigms remains to be found.

Main paper: 
 introduction: 
 Multimodal models like CLIP [98] and Stable Diffusion [104] have revolutionized performance on downstream tasks. CLIP is now the de facto standard for "zero-shot" image recognition [143, 74, 136, 49, 142] and image-text retrieval [47, 64, 25, 127, 139] , while Stable Diffusion is now the de facto standard for "zero-shot" text-to-image (T2I) generation [100, 18, 104, 42] . In this work, we investigate this empirical success through the lens of zero-shot generalization [70] , which refers to the ability of models to apply their learned knowledge to new unseen concepts (not seen during training). Accordingly, we ask: Are current multimodal models truly capable of "zero-shot" generalization?To tackle this question, we conduct a comparative analysis involving two main factors: (1) the performance of models across various downstream tasks, and (2) the frequency of test concepts within their pretraining datasets. We compile a comprehensive list of 4, 029 concepts 2 from 27 downstream generation prompts. For example, from, "A man is wearing a hat", we extract "man" and "hat" as concepts. We filter out ambiguous or irrelevant concepts that are present in less than five downstream evaluation samples. In sum, we collate 4, 029 concepts sourced from 17 classification, 2 image-text retrieval, and 8 text-to-image generation downstream datasets (see Tab. 1 and Sec. 3.1 for details).Concept Frequency from Captions. For efficient concept searches, we pre-index all captions from the pretraining datasets, i.e., construct a mapping from concepts to captions. We first use part-of-speech tagging to isolate common and proper nouns, and lemmatize them with SpaCy [58] (lemmatization helps standardize verbs, enabling the estimation of their frequencies too [65] ). These lemmatized terms are then cataloged in inverted unigram dictionaries, mapping each term to all sample indices in the pretraining dataset containing that term. To determine the frequency of a concept, we examine the concept's unigrams within these dictionaries. For multi-word concepts, we split them into their constituent unigrams, and then independently search for all unigrams before intersecting their hit lists to get a list of matched sample indices. The frequency of the concept in text captions is the count of these intersecting sample indices. This algorithm hence allows scalable O(1) search with respect to the number of captions for any concept in pretraining dataset captions.Concept Frequency from Images. Unlike text captions, we do not have a finite vocabulary for pre-indexing pretraining images. Instead, we collect all the 4, 029 downstream concepts and verify their presence in images using a pretrained image tagging model. We tested various open-vocabulary object detectors, multi-tagging models and image-text matching models, for this concept tagging task. We found that RAM++ [59] -an open-set tagging model that tags images based on a predefined list of concept descriptions, in a multi-label manner-performs the best. We automatically consider the relationship between concepts (like synonyms) and concept hierarchies [84] , since RAM++ uses descriptions generated by a language model (Appx. I.3) for each concept, to tag each image with certain concepts. This approach generates a list of pretraining images, each tagged with whether the downstream concepts are present or not, from which we can compute concept frequencies (Appx. I).Image-Text Matched Concept Frequencies. Finally, we combine the frequencies obtained from both text and image searches to compute matched image-text frequencies. This involves identifying pretraining samples where both the image and its associated caption correspond to the concept. By intersecting the lists from our image and text searches, we determine the count of samples that align in both modalities, offering a comprehensive view of concept representation in the pretraining datasets. This step is necessary as we observed significant image-text misalignment between concepts in pretraining datasets (see Tab. 3), hence captions may not reflect what is present in the image and vice-versa. This behaviour has also been alluded to in prior work investigating data curation strategies [78, 77, 134, 89] . We provide a more detailed analysis of image-text misalignment in Sec. 5. 
temporal filtering:: 
 We applied a filter to only retrieve images after January 2023 to minimize overlap with images used in the pretraining datasets of multimodal models. Note this helps mitigate but does not ensure that the overlap problem is resolved. 
comparing pretraining frequency & "zero-shot" performance: 
 Equipped with frequency estimates for downstream concepts, we now establish the relationship between image-text-matched pretraining concept frequencies and zero-shot performance across classification, retrieval, and generation tasks. We first detail our setup and then discuss key results.Table 1 : Pretraining and downstream datasets used in Image-Text (CLIP) experiments.Pretraining CC-3M [115] CC-12M [28] YFCC-15M [123] LAION-400M [110] ImageNet [36] SUN397 [133] UCF101 [116] Caltech101 [45] EuroSAT [56] CUB [131] Classification-Eval Caltech256 [50] Flowers102 [90] DTD [32] Birdsnap [16] Food101 [21] Stanford-Cars [66] FGVCAircraft [79] Oxford-Pets [93] Country211 [98] CIFAR-10 [68] CIFAR100 [68] Retrieval-Eval Flickr-1K [138] COCO-5K [75] Table 2 : Models used in text-to-image (T2I) experiments.M-Vader [15] DeepFloyd-IF-M [9] DeepFloyd-IF-L [9] DeepFloyd-IF-XL [9] GigaGAN [63] DALL•E Mini [35] DALL.E Mega [35] Promptist+SD-v1. 4 [53] Models Dreamlike-Diffusion-v1.0 [2] Dreamlike Photoreal v2.0 [3] OpenJourney-v1 [4] OpenJourney-v2 [5] SD-Safe-Max [104] SD-Safe-Medium [104] SD-Safe-Strong [104] SD-Safe-Weak [104] SD-v1.4 [104] SD-v1.5 [104] SD-v2-Base [104] SD-v2-1-base [104] Vintedois-Diffusion-v0.1 [7] minDALL.E [105] Lexica-SD-v1.5 [1] Redshift-Diffusion [6]We analyze two classes of multimodal models: Image-Text and Text-to-Image. For both, we detail the pretraining and testing datasets, along with their associated evaluation parameters.Datasets. We use 4 pretraining, 2 downstream retrieval, and 17 downstream classification datasets, covering a broad spectrum of objects, scenes, camera-types, and fine-grained distinctions (see Tab. 1).Note on Pretraining Dataset Diversity. Each analyzed pretraining dataset significantly differs in data collection, filtering, and cleaning operations. CC-3M [115] , originally intended to be used for training image captioning models, explicitly has no real-world entities or proper nouns present, and is cleaned only to have common nouns in its captions. CC-12M [28] and YFCC-15M [123] , collected from Flickr, have user-provided metadata. Finally, LAION-400M [110] and LAION-Aesthetics [111] contain raw images downloaded from Common-Crawl [101] with alt-texts as the captions, which can be inherently very noisy as they are uploaded by non-expert humans as a placeholder for images.Models. We test CLIP [98] models with both ResNet [54] and Vision Transformer [37] architecture, with ViT-B-16 [87] and RN50 [49, 88] trained on CC-3M and CC-12M, ViT-B-16, RN50, and RN101 [61] trained on YFCC-15M, and ViT-B-16, ViT-B-32, and ViT-L-14 trained on LAION400M [110] . We follow open_clip [61] , slip [87] and cyclip [49] for our implementation.Prompting. For zero-shot classification, we experiment with three prompting strategies: {class-name} only, "A photo of a {classname}" and prompt-ensembles [98] , which averages over 80 different prompt variations of {classname}. For retrieval, we use the image or the caption as input corresponding to I2T (image-to-text) or T2I (text-to-image) retrieval respectively.Metrics. We compute mean accuracy for classification tasks [98] . For retrieval, we measure Recall@1, Recall@5, and Recall@10 for both text-to-image and image-to-text retrieval tasks [98] .Datasets. Our pretraining dataset is LAION-Aesthetics [111] , with downstream evaluations done on subsampled versions of eight datasets: CUB200 [131] , Daily-DALLE [34] , Detection [31] , Parti-Prompts [140] , DrawBench [106] , COCO-Base [75] , Relational Understanding [33] and Winoground [124] . Please refer to HEIM [72] for more details on the evaluation datasets.Models. We evaluate 24 T2I models, detailed in Tab. 2. Their sizes range from 0.4B parameters (DeepFloyd-IF-M [9] and DALL•E Mini [35] ) to 4.3B parameters (DeepFloyd-IF-XL [9]). We include various Stable Diffusion models [104] as well as variants tuned for specific visual styles [6, 4, 5] .Prompting. Text prompts from the evaluation datasets are used directly to generate images, with 4 image samples generated for each prompt. This trend holds for both zero-shot classification (results averaged across 17 datasets) and image-text retrieval (results averaged across 2 datasets). ** indicates that the result is significant (p < 0.05 with a two-tailed t-test [118] ), and thus we show Pearson correlation (ρ) [73] as well.Figure 3 : Log-linear relationships between concept frequency and T2I aesthetic scores. Across all tested T2I models pretrained on LAION-Aesthetics, we observe a consistent linear relationship between aesthetic score (averaged across 8 datasets) on a concept and the log-scaled concept frequency.Metrics. Evaluation consists of image-text alignment and aesthetic scores. For automated metrics [72] , we use expected and max CLIP-score [57] to measure image-text alignment along with expected and max aesthetics-score [110] to measure aesthetics. To verify reliability of automated metrics, we compare them with human-rated scores (measured on a 5-point grading scale) for both image-text alignment and aesthetics [72] . To supplement the human-rated scores provided by HEIM [72] , we confirm our findings by performing our own small-scale human evaluation (Appx. C).We now probe the impact of pretraining concept frequency on "zero-shot" performance of models.Our main results, across various tasks and model types, are shown in Figs. 2 and 3 .Understanding the Plots. The plots in the main paper present text-image (CLIP) models' zeroshot classification results using accuracy and text-to-image retrieval performance using Recall@10. Similarly, we present T2I generative models' performance on image generation tasks using the expected aesthetics score. For the other aforementioned metrics for retrieval as well as other automated generation metrics along with human-rated scores, we find that they show similar trends, and we provide them for reference in Apps. B and C. For clarity, the data presentation is simplified from scatter plots to a cohesive line, similar to work from Kandpal et al. [62] and Razeghi et al. [102] . The x-axis is log-scaled, and performance metrics are averaged within bins along this axis for ease-of-visualization of the log-linear correlation. We removed bins containing very few concepts per bin by standard IQR removal [132] following Kandpal et al. [62] . We additionally compute Pearson correlation ρ [73] for each line and provide significance results based on a two-tailed t-test [118] .Key Finding: Log-linear scaling between concept frequency and zero-shot performance. Across all the 16 different plots, we observe a clear log-linear relationship between pretraining concept frequency and zero-shot performance. These plots vary in (i) model type (discriminative vs. generative), (ii) task (classification vs. retrieval), (iii) model architecture and parameter count, (iv) pretraining dataset (curation methods and scales), (v) evaluation metrics, (vi) prompting strategies, and (vii) concept frequencies isolated only from image or text caption (additional experiments for (v) are presented in Apps. B and C, for (vi) are presented in Appx. A, and for (vii) are presented in Appx. D). The observed log-linear scaling trend persists across all seven presented dimensions. In some plots, we notice a slight dip at the high-frequency concepts-we analyse this in greater detail in Appx. L. Thus, taken together, our results reveal data-hungry learning, i.e, a lack in current multimodal models' ability to learn concepts from pretraining datasets in a sample-efficient manner.We employ a pre-trained InceptionNet [119] to remove outliers from the entire image pool. We do this by taking all pairwise cosine-similarities between all images in the pool, and removing the images that are in the bottom 5% of the similarity values4 .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We incorporate statistics of compute and storage resources in Appx. P, which includes total disk space required, number of shards stored as well as the wall-clock runtimes (WCT) for RAM++ image tagging and text-index construction.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
stress-testing frequency-performance trends with distributional controls: 
 In this section, we perform two control experiments to account for different confounding distributional factors of pretraining datasets, to ensure the robustness of our log-linear frequency-performance scaling trends: (1) we control for sample-level similarity in distribution between pretraining and evaluation datasets [137, 81] , and (2) we investigate effects of pretraining data with radically different controlled concept distributions, with entirely synthetically-generated image-text pairs [52] .Motivation. Prior work has suggested that sample-level similarity between pretraining and downstream datasets impacts model performance [62, 81, 137, 102] . This leaves open the possibility that our frequency-performance results are simply an artifact of this factor, i.e., as concept frequency increases, it is likely that the pretraining dataset also contains more similar samples to the test sets. We hence investigate the frequency-performance trends when controlling for sample-level similarity.Setup. We use the LAION-200M [10] dataset for this experiment. We first verify that a CLIP-ViT-B-32 model pretrained on the LAION-200M dataset (used to study sample similarity in prior work [81] ) exhibits a similar log-linear trend between concept frequency and zero-shot performance. Then, we use the near_pruning method from Mayilvahanan et al. [81] to eliminate 50 million samples most similar to the test sets from the pretraining LAION-200M dataset. We provide details for this in Appx. G.1. This procedure removes the most similar samples between pretraining and test sets. We verify that this procedure influences the performance of the model drastically across our aggregate classification and retrieval tasks respectively, replicating the findings of Mayilvahanan et al. [81] .Key Finding: Concept Frequency is still Predictive of Performance. We repeat our analysis on models trained with this controlled pretraining dataset with 150M samples, and report results on the We showcase the distribution of pretraining frequencies of all concepts aggregated across all 17 of our downstream classification datasets. Across all the pretraining datasets, we observe very heavy tails. We normalize the concept frequencies and remove concepts with 0 counts for improved readability of the plots.same downstream classification and retrieval datasets, in Fig. 4 (left). Despite removing the most similar samples between pretraining and test sets, we still consistently observe a clear log-linear relationship between pretraining frequency of test set concepts and zero-shot model performance.Conclusion. This analysis reaffirms that, despite removing pretraining samples closely related to the downstream evaluation datasets, the log-linear relationship between concept frequency and zeroshot performance persists. Note that this is despite substantial decreases in absolute performance, highlighting the robustness of concept frequency as a performance indicator for CLIP models.Motivation. Sampling across real-world data might not result in significant differences in concept distribution, as we will later show in Sec. 5. Hence, we repeat our analysis on a synthetic dataset designed with an explicitly different concept distribution [52] . This evaluation aims to understand if pretraining concept frequency remains a significant performance predictor within a synthetic concept distribution, generalizing even for models pretrained on entirely synthetic images and text captions.Finding 1: Pretraining Datasets Exhibit Long-tailed Concept Distributions. Our analysis in Fig. 5 reveals an extremely long-tailed distribution of concept frequencies in pretraining datasets, with over two-thirds of concepts occurring at almost negligible frequencies relative to the size of the datasets (we highlight the "head" part of this distribution in boxes). Our observations extend the findings of past work that have noted the long-tailed distribution of large-scale language datasets [14, 26, 94, 146] .As we observed with the log-linear trend, this distribution directly reflects disparities in performance.Finding 2: Misalignment Between Concepts in Image-Text Pairs. Our concept frequency estimation pipeline enables us to investigate the alignment of concepts within paired pretraining image-text data. Perfect image-text alignment is defined as every image-text pair containing the same concepts. Previous studies have qualitatively discussed the problem of misalignment in large image-text datasets [77, 134, 78] . Our analysis enables us to quantify this misalignment degree-for each image-text pair in the pretraining dataset, we find concepts that are matched to the image and the text caption independently. If there are no intersecting concepts from the independent image and text hits, we mark that pair as misaligned (detailed algorithm shown in Appx. K). Tab. 3 shows the high degree of misalignment in all image-text pairs (5-36%). To the best of our knowledge, this is the first attempt to quantify the misalignment degree in pretraining image-text datasets explicitly. We release the precise misaligned image-text samples from pretraining datasets to enable better data curation.Finding 3: Concept Frequencies Across Datasets are Correlated. Despite vast differences in the size (3M-400M samples) and curation strategies of the pretraining datasets, we discovered a surprisingly high correlation in concept frequencies across them (see Tab. 4). This suggests that the internet, as the common source of these datasets, naturally exhibits a long-tailed distribution, influencing any dataset derived from it to display similar long-tailed behavior also. This result inspired "Let It Wag!".6 Testing the Tail: Let It Wag!Motivation. In the previous sections, we identified a consistent long-tailed concept distribution across pretraining datasets, highlighting the scarcity of certain concepts on the web. This observation forms the basis of our hypothesis that models likely underperform when tested against data distributions that are heavily long-tailed. To test this, we carefully curate 290 concepts identified as the least frequent across all pretraining datasets. This includes concepts like eggnog, wormsnake, and tropical kingbird. We then use these concepts to create an evaluation dataset, "Let It Wag!".Dataset Details. The "Let It Wag!" classification dataset comprises 130K test samples downloaded from the web using the method of Prabhu et al. [96] . The test samples are evenly distributed across 290 categories of long-tailed concepts. From the list of curated concepts, we download test set images, deduplicate them, remove outliers, and finally manually clean and hand-verify the class labels.Analysis Details. We run both classification and image generation experiments on "Let It Wag!". For classification, we evaluate 40 text-image (CLIP) models on the "Let It Wag!" classification dataset, using an ensemble of 80 prompts from Radford et al. [98] . For the generation task, we utilize SD-XL [95] , SD-v2 [104] , and Dreamlike-Photoreal-v2.0 [3] , to generate images for the long-tailed concepts.For each model, we run 50 diffusion steps, maintaining default settings for all other parameters.Text-Image Classification Results. We showcase the results of our long-tailed classification task in Fig. 6 -we plot results of all models on both "Let It Wag!" (y-axis) and ImageNet (x-axis). We observe that all models underperform by large margins on the long-tailed "Let It Wag!" dataset (upto 20% lower absolute accuracies compared to ImageNet). This performance drop-off generalises across all model scales and 10 different pretraining data distributions, reinforcing the notion that all web-sourced pretraining datasets are inherently constrained to be long-tailed. With that said, note that the higher capacity models (fitted line with slope=1.58 in Fig. 6 ) seem to be closing the gap to ImageNet performance, indicating improved performance on the long-tailed concepts.Ocarina Chuck-will's-widowFigure 7 : Qualitative results on "Let It Wag!" concepts demonstrate failure cases of T2I models on the long-tail. We created 4 prompts for each concept using Gemini [121] and GPT-4 [12] which are fed to 3 Stable Diffusion [104] models. Generations with red border are incorrect, green border are correct and yellow border are ambiguous. Despite advances in high-fidelity image generation, there is large scope for improvement for such long-tail concepts (quantitative results in Appx. N.1).. We provide a qualitative analysis on image generation for assessing T2I models on the rare "Let It Wag!"concepts in Fig. 7 . For enhancing image diversity, we generate prompts using Gemini [121] (top row of generated images) and GPT-4 [12] (bottom row of generated images). Green borders represent correct generations, red borders represent incorrect generations and yellow borders represent ambiguous generations. While descriptive prompting generally aids in improving the quality of generated images [53] , we still observe T2I models failing to comprehend and accurately represent many concepts in our "Let It Wag!" dataset. Some failure cases involve misrepresenting activities (such as Pizza Tossing or Cricket Bowling as shown in Fig. 29 ), generating the wrong concept (Chuck-will's-widow as shown in Fig. 7 (top)), as well as not comprehending the concept at all (Ocarina in Fig. 7 (bottom)). We hence show that Stable Diffusion models are prone to the long tail qualitatively-we also provide quantitative results in Appx. N.1.Across both the classification and generation experiments, we have showcased that current multimodal models predictably underperform, regardless of their model scale or pretraining datasets. This suggests a need for better strategies for sample-efficient learning on the long-tail.We employ a pre-trained InceptionNet [119] model to identify and remove duplicates. This step involves setting high thresholds for soft de-duplication (0.9 for common classes and 0.95 for fine-grained classes) to ensure only minor, precise exclusions. A threshold of 0.9/0.95 means that we consider images to be duplicates if the cosine similarity of that image's embedding (from InceptionNet) with any other image's embedding in the image pool is larger than 0.9/0.95. 
manual verification:: 
 Following the automated cleaning, we manually inspect and verify the accuracy of the remaining images for each class to ensure they meet quality standards.6. Second-level De-duplication with Perceptual Hashing: Post-verification, we use perceptual hashing [38] with a threshold of 10 bits to identify and remove duplicate images within each class, ensuring uniqueness across our dataset5 .7. Class Balancing: Finally, we balance the dataset to ensure an equal representation of classes. This process was followed for increased quality and reliability of our dataset for image recognition tasks. 
related work: 
 We discuss the most relevant prior works to ours here, and defer an extended literature review to Appx. F. Past works [98, 47, 88, 43, 76, 39, 82, 81, 67] have highlighted the importance of pretraining data for improved downstream model performance. Fang et al. [43] demonstrated that pretraining data diversity is key to CLIP's strong out-of-distribution generalisation. Nguyen et al. [88] extended this analysis to show that differences in data distributions can change model performance, enabling effective data mixing strategies for pretraining. Mayilvahanan et al. [81] complemented these works by showing that CLIP's performance is correlated with the similarity between pretraining and test datasets. Our findings further pinpoint that the frequency of concept occurrences is a key indicator of performance. This complements existing work in areas like question-answering [62] and numerical reasoning [102] in LLMs. Concurrent to our work, Parashar et al. [92] also explore the problem of long-tailed concepts in LAION-2B and how it affects CLIP performance, supporting our findings.In contrast to their work, our demonstration that the long tail yields a log-linear trend, explicitly indicates exponential sample inefficiency in pretrained multimodal models. Additionally, contrary to their work, we index both image and text modalities, as well as span across several scales of diverse pretraining datasets. Our frequency estimation procedure on both texts and images independently, enables us to provide a more finer-grained analysis of pretraining datasets than previously studied in the literature, like (1) quantifying the misalignment between images and text captions, (2) assessing the similarity of the different pretraining data concept distributions, and (3) doing a number of control experiments to thoroughly stress-test the robustness of our log-linear scaling results. 
conclusion: 
 In this work, we studied 5 pretraining datasets of 34 multimodal models, analyzing the distribution and composition of concepts within them, generating over 300GB of data artifacts that we publicly release.Our findings reveal that across concepts, significant improvements in zero-shot performance require exponentially more data, following a sample-inefficient log-linear scaling trend. This pattern persists despite controlling for similarities between pretraining and downstream datasets or even when testing models on entirely synthetic data distributions. Further, all tested models consistently underperformed on the "Let it Wag!" dataset, which we systematically constructed from our findings to test for long-tail concepts. This underlines a critical reassessment of what "zero-shot" generalization entails for multimodal models, highlighting the limitations in their current generalization capabilities.Part I Appendix  
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: For the Let It Wag! dataset we introduce, we take considerable steps to ensure responsible release, all of which are detailed in Appx. H.2. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We extend the zero-shot classification results from Fig. 2 in Fig. 8 with two different prompting strategies: the results in the main paper used the {classname} only as the prompts, here we showcase both (1) "A photo of a {classname}" prompting and (2) 80 prompt ensembles as used by Radford et al [98] . We observe that the strong log-linear trend between concept frequency and zero-shot performance consistently holds across different prompting strategies. CC-3M CC-12M YFCC-15M LAION-400M A photo of a {} Prompt Ensembles CC-3M CC-12M YFCC-15M LAION-400MWe supplement Fig. 2 in the main paper, where we showed results with the text-to-image (I2T) recall@10 metric. In Figs. 9 and 10, we present results for the retrieval experiments across all six metrics: I2T-Recall@1, I2T-Recall@5, I2T-Recall@10, T2I-Recall@1, T2I-Recall@5, T2I-Recall@10. We observe that the strong log-linear trend between concept frequency and zero-shot performance robustly holds across different retrieval metrics. CC-3M CC-12M YFCC-15M LAION-400M I2T Recall@1 CC-3M CC-12M YFCC-15M LAION-400M CC-3M CC-12M YFCC-15M LAION-400M I2T Recall@5 I2T Recall@10We extend the results from Fig. 3 with Figs. 11 to 15. As with Fig. 3 , due to the high concept frequency, the scaling trend is slightly less pronounced. Furthermore, we do see inconsistency in the trends for the human-rated scores retrieved from HEIM [72] , hence we perform our own small scale human evaluation to check them.Human Study with People Concepts. Given the societal relevance [24] , we decided to test Stable Diffusion [104] (v1.4) on generating public figures. We scraped 50,000 people from the "20230123all" Wikidata JSON dump by filtering for entities listed as "human" [8], and scraped a reference image for the human study for each person if an image was available. After computing concept frequency from LAION-Aesthetics text captions (using suffix array [71] ), we found that ≈10,000 people were present in the pretraining dataset. Note that to ensure the people's names were treated as separate words, we computed frequency for strings of the format " {entity} ". We then randomly sample 360 people (for which a reference image was available) normalized by frequency [23] for the human study. For generating images with Stable Diffusion, we used the prompt "headshot of {entity}", in order to specify to the model that "{entity}" is referring to the person named "{entity}" [51] .We assessed image-text alignment with a human study with 6 participants, where each participant was assigned 72 samples; for consistency, of the 360 total samples, we ensured 10% were assigned to 3 participants. Provided with a reference image, the participants were asked if the sample accurately depicts the prompt (see Fig. 16 ). Specifically, "Does the image accurately depict the above prompt?". Three choices were provided: "Yes" (score=1.), "Somewhat" (score=0.5), and "No" (score=0.). Accuracy was computed by averaging the scores.As can be seen in Fig. 17 , we observe a log-linear trend between concept frequency and zero-shot performance. Thus, we observe that the log-linear trend between concept frequency and zero-shot performance consistently holds even for T2I models.Note on Participant Acquisition. Experiment participants, who volunteered for the study, provided informed consent. IRB approval was not obtained. Across all tested models pretrained on the LAION-Aesthetics dataset, we observe a consistent linear relationship between T2I zero-shot performance on a concept and the log-scaled concept pretraining frequency.Figure 12 : Log-linear relationships between concept frequency and T2I human aesthetic scores. Across all tested models pretrained on the LAION-Aesthetics dataset, we observe a consistent linear relationship between T2I zero-shot performance on a concept and the log-scaled concept pretraining frequency.Figure 13 : Log-linear relationships between concept frequency and T2I human alignment scores. Across all tested models pretrained on the LAION-Aesthetics dataset, we observe a consistent linear relationship between T2I zero-shot performance on a concept and the log-scaled concept pretraining frequency.Figure 14 : Log-linear relationships between concept frequency and T2I Avg. CLIP scores. Across all tested models pretrained on the LAION-Aesthetics dataset, we observe a consistent linear relationship between T2I zero-shot performance on a concept and the log-scaled concept pretraining frequency. Figure 17 : Log-linear relationship between concept frequency and T2I human evaluation for text-image alignment for people concepts. We observe a consistent linear relationship between T2I zero-shot performance on a concept and the log-scaled concept pretraining frequency.In all the main performance-frequency plots we have presented until now, the concept frequencies were estimated using the intersection of the image-frequencies and the text-frequencies. Here, we showcase results with using them independently in Figs. 18 and  19 respectively. We note that both independent searching methods showcase log-linear trends as before confirming our main result. We observe that the strong log-linear trend between concept frequency and zero-shot performance robustly holds across concepts derived from image and text domains independently as well. We believe our main conclusions of exponential data inefficiency should hold regardless of the model architecture and the training objective for any VLM. However, to test this thoroughly, we investigated two methods that have been empirically shown to improve generalization capabilities of CLIP models: CyCLIP [49] and SLIP [87] . We use 4 different models, each trained with either CyCLIP/SLIP on three different datasets-we then plot our main log-linear scaling results similar to Fig. 2 for CyCLIP and SLIP models-these plots are in Fig. 20 . We observe for both SLIP and CyCLIP models, the log-linear scaling trends hold strong, with high Pearson correlation coefficients, further signifying the robustness of our main results. Hence, we emphasize that our main conclusions hold true even when considering multimodal models that explicitly introduce new training objectives with the aim of improving model generalization.Classification Supplementing the discussion in the main paper, we further provide a broad overview of the surrounding literature within which our paper is mainly positioned.Effect of Pre-training Data on Downstream Data. Several data-centric prior works [98, 47, 88, 43, 89, 76, 134, 135, 145, 117, 80, 99, 107, 108, 39, 27, 103] have highlighted the importance of pretraining data in affecting performance. Fang et al. [43] robustly demonstrated that pretraining data diversity is the key property underlying CLIP's strong out-of-distribution generalisation behaviour. Similarly, Berlot-Attwell et al. [17] showed that attribute diversity is crucial for compositional generalization [60] , namely systematicity [46] . Nguyen et al. [88] extended the Fang et al. [43] analysis to show that differences in data distributions can predictably change model performance and that this behaviour can lead to effective data mixing strategies at pretraining time. Mayilvahanan et al. [81] complemented this research direction by showing that CLIP's performance is correlated with the similarity between training and test datasets. Udandarao et al. [129] further showed that the frequency of certain visual data-types in the LAION-2B dataset was roughly correlated to the performance of CLIP models in identifying visual data-types. McCoy et al. [82] introduced the teleological approach to understanding model generalization, showing that LLMs are more reliable on tasks and input/output instances that are more probable based on their pretraining datasets. Our findings further pinpoint that the frequency of concept occurrences is a key indicator of performance. This complements existing research in specific areas like question-answering [62] and numerical reasoning [102] in large language models, where high train-test set similarity does not fully account for observed performance levels [137] . Concurrent to our work, Parashar et al. [92] also explore the problem of long-tailed concepts in the LAION-2B dataset and how it affects performance of CLIP models, supporting our findings. In contrast to their work, we look at count separately in image and text modalities, as well as across pretraining sets, and do a number of control experiments to thoroughly test the robustness of our result. Further, our frequency estimation procedure on text captions and images independently enables us to provide a more fine-grained analysis of the pretraining data distribution like quantifying misalignment between images and texts, and assessing similarity of the different pretraining data concept distributions. Finally, our demonstration that the long tail yields a log-linear trend explicitly indicates exponential sample inefficiency in large-scale pretrained models.Detailed Differences and Contributions compared to Parashar et al. [92] . We emphasise that we complement this prior work and point out that our work comprehensively tests the strength of the loglinear scaling trend across several datasets, spanning varying levels of data curation and dataset sizes. Further, we note that in Parashar et al. [92] , the estimated frequencies are computed using only the text captions of LAION-2B. These estimated frequencies are then used as the canonical frequencies for plotting the performance-frequency curves for all the tested models (despite these models being trained on different pretraining datasets other than LAION-2B). Our work strongly showcases why this apparent asymmetry in their frequency estimation methodology should work-from Tab. 4, we show that different VLM pretraining datasets are strongly correlated in their concept distributions. Hence, in spite of Parashar et al. [92] using only LAION-2B as their source dataset for frequency estimation, their results roughly hold true because of this strong correlation across pretraining datasets. Our methodology of incorporating both images and text captions when computing the frequency estimates is crucial for explaining this. Hence, we believe that our work comprehensively generalizes and explains the findings of prior work while also providing insights into the pretraining datasets (e.g., misalignment degree and correlation of concept distributions in datasets).Data-centric analyses. Our work also adds to the plethora of work that aims to understand and explore the composition of large-scale datasets, and uses data as a medium for improving downstream tasks. Prior work has noted the importance of data for improving model performance on a generalised set of tasks [47, 11, 41, 13, 114] . For instance, several works utilise retrieved and synthetic data for adapting foundation models on a broad set of downstream tasks [128, 55, 125, 22, 109, 144, 96] . Maini et al. [78] observed the existence of "text-centric" clusters in LAION-2B and measured its impact on downstream performance. Other work has seeked to target the misalignment problem that we quantified in Tab. 3 by explicit recaptioning of pretraining datasets [69, 29, 130, 141, 89, 18] . Further, studies have also shown that by better data pruning strategies, neural scaling laws can be made more efficient than a power-law [117, 10] . Prior work has also showcased that large-scale datasets suffer from extreme redundancy in concepts, and high degrees of toxic and biased content [40, 126] . Further research has showcased the downstream effects that such biases during pretraining induce in state-of-the art models [20, 112, 19, 48] . Our work tackles the issue of long-tailed concepts in pretraining datasets, and shows that this is an important research direction to focus efforts on.LAION-200M is a dataset obtained by deduplicating LAION-400M by pruning exact duplicates, near duplicates, and semantically similar samples within LAION-400M [10] . The control pretraining set is created by pruning 50 million highly similar samples from LAION in the order of decreasing perceptual similarity to datapoints in ImageNet-val set. We use the 150M pretraining set for obtaining the concept distribution. We evaluate the performance of a ViT-B-32 CLIP model trained on this dataset on our downstream tasks and present our analysis on those tasks.Based on our frequency estimation pipeline from Sec. 2, we carefully curate 290 of the least frequent concepts across LAION-400M pretraining dataset (out of the 4, 029). We then remove all the concepts that have 0 counts to ensure that our final dataset consists of concepts that have been detected atleast once in LAION-400M, this method has also been used in Kandpal et al. [62] to ensure robustness to noise in the estimation process. We then add them as our set of concepts in Let It Wag!. A few example concepts from our final list are: {Beechcraft_1900, Black_Rosy_Finch, Irish_Wolfhound, Japanese_Chin, Kentucky_Warbler, eastern_hog-nosed_snake, eel, eggnog, flatfish, isopod, kingsnake, ladle, lakeshore, letter_opener}. We release our full concept list publicly here.High-level insights about long-tail concepts. The broad categories of the most long-tailed concepts with a few examples for each are as follows (a majority of them are also highlighted in Figs. 28 We provide some further statistics of the test-set below. 1. Diverse Sourcing: We gather images from three different online sources-Flickr, DuckDuckGo, and Bing Search-to maximize the variety of our dataset while retaining very easy-to-classify images 3 .We detail why we use the RAM++ model [59] instead of CLIPScore [57] or open-vocabulary detection models [86] . Furthermore, we elaborate on how we selected the threshold hyperparameter used for identifying concepts in images.We provide some qualitative examples to illustrate why we chose RAM++. Our input images do not often involve complex scenes suitable for object detectors, but many fine-grained classes on which alongside CLIP, even powerful open-world detectors like OWL-v2 [86] have poor performance. We ablate the choice of the threshold we use for assigning concepts to images using the RAM++ model. For the given set of concepts, RAM++ provides a probability value (by taking a sigmoid over raw logits) for each concept's existence in a particular image. To tag an image as containing a particular concept, we have to set a threshold for deciding this assignment. We test over three thresholds: {0.5, 0.6, 0.7}, showcasing quantitative and qualitative results for all thresholds in Figs. 23 and 24 .We observe best frequency estimation results using the highest threshold of 0.7. This is due to the high precision afforded by this threshold, leading to us counting only the "most aligned images" per concept as hits. With lower thresholds (0.5, 0.6), we note that noisier images that do not align well with the concept can be counted as hits, leading to degraded precision and thereby poorer frequency estimation. To ensure that we do not incorrectly tag images with erroneous concepts, our primary objective is to optimize hit precision, even if it means occasionally missing out on tagging some images with correct concepts. Hence, we use 0.7 as the threshold for all our main results.In addition to providing a list of concepts to the RAM++ model, we also provide a set of GPT-4 generated responses that describe each concept (please refer to Tab. 5 for examples). This ensures that we adequately cover synonyms of concepts and take into account concept hierarchies [84] . This further improves tagging precision by using visual descriptions to better identify concepts (this has been shown to enhance performance in previous works [97, 83] . We open-source these descriptions. ImageNet [36] Tench A tench is a freshwater fish that typically has a greenish or brownish body with reddish fins, small scales, and a pair of barbels near its mouth. It can grow up to 70 cm long.SUN397 [133] Alley An alley is a narrow passageway or lane between or behind buildings, which is often used for access or for parking.UCF101 [116] Blowing_Candles A Blowing_Candles moment can be identified concisely as a moment or event where a person is blowing out the candles on a cake, typically at a birthday celebration.Caltech101 [45] Butterfly A butterfly is a small, flying insect known for its colorful and symmetrical wings. It has a slender body, antennae and three pairs of legs.Least_Auklet A Least Auklet is a small seabird with a black back and wings, white underparts, and a stubby orange bill. They also have white eye-rings and a small, rounded tail.EuroSAT [56] Pasture Land Pasture land can be concisely identified as an open or cleared land covered with grass, clover, or the like, suitable for grazing by livestock, with little to no trees and is primarily used for agricultural purposes.Flowers102 [90] pink primrose A pink primrose is a perennial flower featuring delicate, soft pink petals arranged around a yellow center, with bright green leaves at the base.DTD [32] bumpy A bumpy object has an uneven or rough surface with lots of small raised areas or protuberances Food101 [21] churro A churro is a long, thin, golden-brown pastry that is typically ridged and may be dusted with sugar.A 707320 is a model of the Boeing 707, which is a mid-size, long-range, narrow-body fourengine jet airliner.Stanford-Cars [66] 1993 Volvo 240 Sedan The 1993 Volvo 240 Sedan can be identified by these features:1. Manufacturer: Volvo 2. Production Year: 1993 3. Model: 240 4. Body Style: 4-door sedan 5. Engine: 2.3L 4-cylinder 6. Transmission: 5-speed manual or 4-speed automatic CIFAR100 [68] bowl A bowl is a round dish or container typically used to hold food, often deeper than a plate with a wide open top.A metro can be identified concisely as an urban railway system that operates within large cities, offering high-frequency services and utilizing multiple cars and stations.In all our main plots, we explicitly exclude zero-frequency concepts from our evaluations following Kandpal et al. [62] , since frequency estimation is potentially noisy, leading to low recall rates (also discussed in Appx. I). However, to verify if our log-linear trends still hold when including all the zero-frequency concepts, we re-plot all our main zero-shot classification results from Fig. 2 by including the ones which have zero-frequencies-Fig. 25 showcases these results. We find our main log-linear scaling trends are retained. To further corroborate this, we present average accuracies for concepts with frequency 0 and non-zero frequency bins in Tab. 6 below. We note that average performance for the 0-frequency concepts is significantly lower than other non-zero frequency concepts, especially when compared to very high-frequency concepts. This justifies our main claim that exponentially more data is needed per concept to improve performance linearly.Figure 25 : Main Results with 0-frequency concepts. We re-plot all our main classification results from Fig. 2 by including concepts which have zero-frequencies. Note that the 0-frequency points are assimilated into the 10 0 bin-in each plot, the 10 0 bin (leftmost) consists of about 60 -70% 0-frequency concepts. We find that our main log-linear scaling trends are retained.Table 6 : Performance per frequency bin. Here, we explicitly report the average classification performance of models trained on different pretraining datasets, per frequency bin (i.e., 0-frequency concepts only, concepts with frequencies in the range 1-10, 10-100 etc.). We note that average performance for the 0-frequency concepts is significantly lower than other non-zero frequency concepts, especially when compared to the performance of very high-frequency concepts.Dataset/Model Freq=0 Freq=1-10 Freq=10-100 Freq=100-1000 Freq=1000-10000 We provide some intuitions on why there are some drops in the trend at high frequencies for the CC-3M and CC-12M classification plots in Fig. 2 . We investigated which concepts occur at such high frequencies, specifically above 10 4 . From our analysis, we hypothesize two key reasons for these performance dips:• Concept ambiguity: We observe many concepts that are homonyms / polysemous (same spelling but different meaning i.e., can represent multiple concepts at once). Some examples are watch, bear, house, fly, bridge, cloud, park, face, bar, tower, wave, etc. • Broad concepts: A concept with a broader scope of definition supersedes a narrower one (concept 'dog' vs the specific breeds of dogs seen in ImageNet ('yorkshire terrier', 'boston terrier', 'scottish terrier', 'golden retriever', etc)). These concepts are too coarse-grained and hence can be visually represented by a diverse set of images. Performance variance of these concepts can be quite high based on the specific set of images given for testing.These ambiguities become more prevalent the more ubiquitous a concept is, which is directly tied to its frequency obtained from pretraining datasets. Some more examples for a deeper understanding of the diversity of concepts are: 'cucumber', 'mushroom', 'Granny Smith', 'camera', 'chair', 'cup', 'laptop', 'hammer', 'jeep', 'lab coat', 'lipstick', 'american-flag', 'bear', 'cake', 'diamond-ring', etc.We provide zero-shot classification plots for CC-3M, CC-12M, and LAION-400M in Fig. 27 , including 95% confidence intervals for each point. This approach follows the standard practice from works like Miller et al. [85] , Taori et al. [120] . Our plots show that the spread at higher frequencies is significantly larger than at moderate frequencies, following the analysis in Appx. L that higher frequency concepts are more ambiguous and polysemous. These results support the observed dips in accuracy at high-frequency points in the CC-3M and CC-12M plots in Fig. 2 . We provide additional quantitative and qualitative results in this section for T2I models evaluated on the "Let It Wag!" dataset.ClassificationWe analyse how state-of-the-art T2I models perform on the long-tailed concepts comprising the "Let It Wag!" dataset. As detailed in Sec. 6, we generate 4 images for each concept using Stable Diffusion XL [95] , Stable Diffusion v2 [104] and Dreamlike Photoreal [1] .Prompting Strategy. The prompting strategy (system role) used, adapted from Shahmohammadi et al. [113] , was:Follow my commands:1. I wish to generate text prompts about a given subject which I will use for image generation using off-the-shelf text-to-image models such as Stable Diffusion and DALL-E 3.2. Assume all the subjects are nouns.3. Follow a similar style and length of prompts as coco-captions. 4. Keep prompts concise and avoid creating prompts longer than 40 words. 5. Structure all prompts by setting a scene with at least one subject and a concrete action term, followed by a comma, and then describing the scene. For instance,"a view of a forest from a window in a cosy room, leaves are falling from the trees." Generate detailed prompts for the concepts in the order in which they are given. Your output should be just the prompts, starting with "1."With this pool of generated images, we conduct a controlled experiment on the long-tailed concepts using nearest-neighbor retrieval as the evaluation metric by querying a generated image and retrieving the top-k results from a gallery of images taken from the "Let It Wag!" dataset. The overall pipeline is as follows:Setup. We define the query and gallery set for head and tail concepts. For tail concepts, we sample the 25 concepts with the lowest frequency from the "Let It Wag!" dataset. For head concepts, we sample the 25 most frequent concepts for comparison. We use the same prompting strategy with the selected 25 concepts across all 3 T2I models. To create the gallery set, we randomly sample 100 images for each of these concepts. We use DINOv2 [91] ViT-S/14 as the feature extractor.Results. In Table 8 , we provide the Cumulative Matching Characteristic (CMC@k) results for all 3 T2I models used in our experiment. CMC@k was chosen as we are interested in measuring the performance delta between head and tail concepts for successful retrievals within the top-k retrieved real images for a given generated image. We observe a large performance gap between Head and Tail concepts, providing a quantitative evaluation of generation performance of T2I models.Table 8 : Generated-real retrieval scores. We compare retrieval results of DINOv2 ViT-S/14 when using generated images as query images. We report ∆ CMC@k results where k={1,2,5} between head and tail concepts. In Fig. 7 of the main text, we provide an initial insight into the qualitative performance of T2I models on "Let It Wag!" concepts. For ease of comprehension and comparison, we segregate concepts into 4 clusters: Aircraft (Fig. 28 ), Activity (Fig. 29 ), Animal (Fig. 30 ) and others (Fig. 31 ).Please note that we compress the aforementioned images to a lower quality due to the file size limitation of our submission. We will replace them with the original, high quality image files for the final version.Results. Fig. 28 shows T2I models having difficulty in representing an aircraft in its full form in a majority of cases in addition to misrepresenting the specific model in the generated images. Fig. 29 showcases the difficulty T2I models face when representing actions or activities from prompts. Fig. 30 exemplifies the same inability of T2I models to accurately represent animal species. Finally, the remainder of the query set is shown in Fig. 31 and includes the inability to classify and subsequently generate certain species of flowers and objects. We highlight a few limitations and open questions of our work, leading to some possible exciting avenues for future research.Understanding Image-Text Misalignments. One can explore the origins of misalignments between images and texts, such as the limitations of exact matching for concept identification in captions, inaccuracies from the RAM++ tagging model, or captions that are either too noisy or irrelevant. A few potential mitigating strategies are to explicitly recaption the images [29, 89] or to utilize the grounded concepts from the images as aditional feedback signal.Investigating Compositional Generalization. In our work, we only analyse concepts in isolation, and do not take into account the combination of concepts. "Zero-shot generalization" often refers to models' ability for compositional generalization (understanding new combinations of concepts not previously encountered). This is distinct from traditional zero-shot learning and presents an intriguing, yet unresolved challenge: analyzing compositional generalization from a data-centric perspective.Methods for Bridging the Generalization Gap. Addressing the challenges posed by the longtail distribution involves improving model generalization to overcome the limited improvement from pretraining we found in our study. Retrieval mechanisms can compensate for the inherent generalization shortcomings of pretrained models, providing a viable path to mitigating the effects of long-tailed pretraining data distributions.Towards a Theoretical Model for the Log-Linear Scaling Trends. Our experiments comprehensively showcase the log-linear scaling trend of model performance with pretraining concept frequency empirically, across several diverse pretraining datasets and models. However, our analysis lacks a detailed theoretical framework explaining why such a trend exists. Building such a framework can help get better intuitions about the underlying mechanics of data dependence in multimodal models, which could be crucial for developing more efficient training strategies or algorithms.On the Interaction of Model Scale and Concept Frequency. An important aspect of the current recipe for building robust foundation models is model scale. Despite investigating models across different scales, a key open question is what the effect of model scaling would be on the slope of the log-linear fit in our plots. Precisely studying the rate of change of the slope across model scales would enable making stronger claims on the optimal capacity-data-frequency tradeoffs.Potential Mitigating Solutions. While our paper does not propose specific solutions, we believe its primary contribution is in thoroughly highlighting the issues with current pretraining strategies for multimodal models across various datasets, pretraining methods, architectures, training objectives, and tasks. Additionally, by releasing the "Let it Wag!" testbed, we provide a straightforward test set for future research to build upon, aiming to improve the generalization of multimodal models to long-tail scenarios. However, we suggest a few potential methods that could be explored to enhance multimodal long-tail:• Retrieval Augmentation: Enhancing generalization to long-tail concepts can be achieved by utilizing the "world-knowledge" of LLMs to provide detailed descriptions for these concepts. This approach transforms the task from simply recognizing long-tail concepts by name to recognizing them by both names and descriptions. • Curriculum Learning: Our tested models used random IID sampling during training.However, research into better sequencing of data samples could potentially improve model generalization to long-tail concepts by inducing more transferable feature representations in VLMs. • Synthetic Data: Addressing the issue of long-tail concepts in web-sourced datasets may not be feasible by merely increasing data samples. There will likely always be low-data density regions in the pretraining data distribution. Using synthetic data, either through procedurally generated samples or text-to-image models, could be a viable mitigation strategy.We hope these suggestions provide valuable directions for future research and contribute to the development of multimodal models capable of better generalization.Our work uses large-scale image-text pretraining datasets and models. The broad societal implications of both of these artifacts have been comprehensively discussed in prior work [98, 19, 20] . By extensively studying the composition of these large-scale datasets via principled methods, our work tries to gain a better understanding of their composition. A key result from our work that has serious potential implications for the broader society is the poor performance of multimodal models on the long-tail. From Tab. 4 and Fig. 5 , it is clear that web-sourced datasets all exhibit the same long-tailed biases. This suggests that current models will predictably underperform on digitally marginalized communities and societies that are underrepresented on the web. Our results call for improved algorithms for training such multimodal models, such that they are more inclusive and performant on the long-tail. We also publicly release all of our data artifacts. Since the multimodal datasets we analyze in our work are extremely biased and can contain hateful, harmful and toxic content [20] , our publicly released data artifacts potentially reflect these biases too. However, we hope that, by facilitating analysis of such large-scale datasets via our artifacts, future research efforts focus on gaining a better understanding of how to make these datasets fairer and more inclusive.NeurIPS Paper Checklist • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: We uniformly conform to the Code of Ethics and, in particular, all data-related concerns about our "Let it Wag!" benchmark. We communicate the details of our benchmark with a license, allow access to research artifacts, make our work reproducible, and carefully consider all societal impacts and harmful consequences of our research output. Note that we use the LAION-400M and LAION-Aesthetics datasets. LAION-400M has not been shown to contain any harmful child-sexual abuse material (CSAM). However, LAION-Aesthetics is a subset of the LAION-5B dataset, which has been shown to contain CSAM [122] . We are in contact with the LAION-5B dataset authors, and will reproduce results on their cleaned set once released. However, note that our set of 4, 029 concepts do not contain any harmful content, and hence we are confident that our main results should hold even across the cleaned datasets.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the broader societal impacts of our work in Appx. S Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).we use the image sourcing pipeline of C2C[96]We use the fastdup library for outlier removal: https://github.com/visual-layer/fastdupWe use the imagededup library for de-duplication: https://github.com/idealo/imagededup 


Decision: This work conducted a systematic study on the zero-shot generalization capability of multimodal models such as CLIP for classification and Stable Diffusion for image generation. It receives unanimous positive scores after rebuttal discussions. The problem of understanding the limitations of VLMs is important, the study on the T2I models is insightful, and the dataset that aggregates the tailed concepts is valuable. AC agrees this is a solid contribution to the community and recommends acceptance. The authors should properly acknowledge and discuss prior work "The Neglected Tails in Vision-Language Models" as suggested by the reviewers.

Reviews: review: paper_summary: paper_summary: This work explores the extent to which zero-shot generalisation really occurs in large-scale in model that were trained on web-scale datasets. The approach taken relies on identifying concepts that are present in train and test data and evaluating concept frequencies and per-concept performance. From extensive experiments, the authors find that test performance of the models correlates strongly with the frequency of concepts seen during training.
strength_weakness: strength_weakness: * My general feel after reading the paper is that the issue of zero-shot performance of model trained on web-scale data is not as bad as the paper (abstract) makes it out to be. For example, from Fig 6 we see that ImageNet and long-tail (e.g. Let It Wag!) benchmarks actually tend to agree as model performance improves. Similarly, we see from Fig 2 that larger models tend to have better performance even for concepts with low train dataset frequencies.
* Presentation of the results is not always clear. For example, what are the different pannels in Fig 3 or Fig 5?
* Results presented in the paper do not really study zero-shot performance. Specifically, in Fig 2 or 3 there appears to be no evaluated points with freq = 0. As such, the paper really studies few-shot performance as a function of the number of training examples.
* Methodologically, I have an issue with way concept frequency in images was estimated - it relies on a pre-trained model (RAM++) to tag/classify images. This model itself likely suffers from training datasets biases and thus could miss or over- or under-estimate image concepts. This limitation also puts into questions the image-caption misalignment results presented in Tab 3.
questions: questions: See weaknesses.
limitations: limitations: See weaknesses, specifically whether zero-shot performance is really being evaluated in the paper, and how concept frequencies were determined for images.
paper_summary: paper_summary: This paper examines the relationship between the frequency of concepts in pretraining data and the performance of downstream tasks associated with those concepts. Extensive experimental results reveal a log-linear relationship, suggesting that exponential increases in data are necessary to improve zero-shot model performance. Additionally, the authors reconfirm the long-tailed distribution of concepts in well-known pretraining datasets, highlighting the challenge of handling rare concepts in foundation models. Based on this analysis, the authors provide a benchmark to evaluate model performance on tail concepts.
strength_weakness: strength_weakness: Nothing I can think of.
questions: questions: - I am curious about the phenomenon where the average accuracy drops after the concept frequency of 10^4 in Figure 2. Could the authors provide some examples and hypotheses about why this happens?
- How much variance is there in each point of Figure 2? I am curious about how much the tendency varies depending on the difficulty of the concept, its size, or other aspects.
- Does zero-shot performance predict few-shot or linear probing performance? I wonder if zero-shot models still learn important features for rare concepts, but these features are dominated by class imbalance in zero-shot tasks.
- Typically, how long does it take to train a model on each dataset? (GPU hours)
limitations: limitations: The authors adequately addressed the limitations with future directions in Appendix N.
paper_summary: paper_summary: The authors evaluate CLIP's zero-shot performance and its correlation to the pretraining data. Their experiments, based on the open LAION datasets, show that the limitations of Vision-Language Models (VLMs) in downstream tasks such as image generation and zero-shot recognition are linked to the frequency of concepts in the data.

VLMs like CLIP are pivotal for downstream multimodal applications, and understanding their biases is crucial for developing more robust systems. While the paper demonstrates the limitations of these foundational models, it currently focuses more on breadth rather than presenting in-depth insights.

I would be willing to increase my score if the authors could address my questions and fix some of the references.

[1] Parashar, Shubham, et al. "The Neglected Tails in Vision-Language Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Schuhmann, Christoph, et al. "Laion-5b: An open large-scale dataset for training next generation image-text models." Advances in Neural Information Processing Systems 35 (2022): 25278-25294.
strength_weakness: strength_weakness: 1. I feel the authors are somewhat aggressive in their claim of a 'constant linear relationship' as stated in Fig 2's caption. There are noticeable dips in accuracy that are not well explained. Given that the authors can identify false positives using their frequency measurement pipeline and are confident about the frequency of a given concept, a true linear correlation should not show these dips. This observation is not adequately addressed by the authors.

2. The authors' claim that their work is the first to establish that pretraining datasets of VLMs are long-tailed seems overzealous. Prior work has demonstrated this fact [1], and the correlation between zero-shot accuracy and pretraining frequency has been established. Additionally, [1] analyzes both LAION-400M and LAION-2B, a superset of LAION-400M [2], whereas the biggest dataset evaluated in this work is LAION-400M. Furthermore, [1] has been incorrectly cited in the paper.

3. The authors present a challenging new benchmark but provide very limited information on how these concepts were collected. There could be multiple ways to sample 290 classes, which could lead to worse performance for CLIP compared to popular benchmarks like ImageNet.

4. Additionally, the authors could have provided some high-level information about which concepts are relatively less present, such as snakes, birds, airplanes, etc. This information could offer insights toward improving CLIP.
questions: questions: 1. Can the authors provide insights about the classes used to create the dataset? Specifically, what was (a) the most frequent concept, (b) the least frequent concept, and (c) the average frequency of a concept? This would help understand the structure of the dataset, and how tailed the dataset is.

2. Can the authors present the standard deviation of each frequency bin? It would be interesting to see the variance as the frequency increases, as this may help explain the dips. CLIP might show less variance for relatively frequent concepts.
limitations: limitations: 1. While the authors highlight a significant problem, they do not propose any solutions to mitigate it. Offering potential solutions could have strengthened the paper.

2. The paper covers a broad range of topics, but it would benefit from a stronger focus on the generation aspect and dataset. Previous literature [1] has indicated that zero-shot recognition of CLIP models depends on the distribution of concepts in the pretraining data. The value of this work lies in highlighting that generative systems also face this issue. Emphasizing this aspect would make the paper easier to follow and enhance its impact.

3. Lastly, while analyzing the smaller YFCC or CC datasets is commendable, I feel that modern VLMs are pretrained on a scale of billions. Therefore, presenting this analysis on these smaller datasets has limited practical impact compared to insights derived from larger datasets like LAION or DataCOMP. Additionally, YFCC and CC datasets have experienced data loss over time, further diminishing their relevance.
paper_summary: paper_summary: This paper investigates the "zero-shot" performance of multimodal models like CLIP and Stable-Diffusion. By analyzing 34 models and 5 pretraining datasets, the study finds that these models require exponentially increasing amounts of data to achieve linear improvements in performance. Additionally, the study suggests that the key to effective "zero-shot" generalization with large-scale training data and compute paradigms remains elusive.
strength_weakness: strength_weakness: -  Please unify the description of the table, above or below the table.

- This work does not point out correlations or differences with prior work, preventing me from assessing the novelty of this work.

-  Qualitative visualizations and relevant theoretical analysis are lacking, although the perspective of the work is easy to understand.
questions: questions: - Does this work conflict with research into improving the generalization of multimodal models? I think the author should discuss it.

- Explanation of Figures 24-27 is insufficient.

- What is the motivation behind this work? What implications does it have for future research?

- To validate the points made in this paper, should other tasks beyond classification and retrieval be evaluated?
limitations: limitations: - See Weaknesses and Questions.

Comments: title: Response to Reviewer WCHb
Rebuttal to reviewer WCHb contd. [2/2]
Rebuttal to reviewer WCHb contd. [1/2]
comment: We would like to thank the reviewer for deeply engaging with our work and helping us improve the overall quality of the paper.
We would like to thank the reviewer for deeply engaging with our work and helping us improve the overall quality of the paper. We will most definitely take into account all the points raised by the reviewer (regarding related work, discussing high-frequency dips, and statistics of Let-It-Wag!) and update the pertinent sections in the revised version of the paper.
Thank you for addressing my concerns; I am satisfied with the authors' responses.

If the authors can incorporate the feedback from this rebuttal, such as updating the related work, discussing the dips in high-frequency classes, and including details for the 'Let It Wag' dataset, the paper would be a strong addition to NeurIPS 2024. Foundational models like CLIP have gained significant popularity due to their zero-shot recognition capabilities, so identifying their limitations is crucial.

In light of this discussion and the comprehensive answers provided, I am increasing my score from 6 to 7. I appreciate the authors' efforts and hope they will incorporate the suggested changes.
Thanks for highlighting this, we've included the results for models pretrained on LAION-400M in the table below, we hope this sufficiently addresses the reviewer's concerns.

|Model/Dataset|ImageNet Acc|Let-It-Wag! Acc (290 classes)|Let-It-Wag-filtered! Acc (151 classes)|
|-|-|-|-|
|ViT-B-32/LAION-400M|60.24|34.02|32.88|
|ViT-B-16/LAION-400M|67.03|40.53|39.14|
|ViT-L-14/LAION-400M|72.75|48.10|46.59|
I noticed that the above results don't include those for LAION-400M. Since the analysis is based on LAION-400M, including these results for a more comprehensive overview might be helpful.
Thank you for following up, and highlighting what should be resolved in the revision—we will most certainly fix the references, will point to our ambiguity analysis in the main text, ensure our contributions wrt to [1] are adequately addressed in the literature review, and altogether will include all rebuttal insights within the appendix of the revised paper. 

Regarding creating a filtered version of the Let-It-Wag! dataset, as suggested, we keep only those classes that have a frequency of less than or equal to 1000 instances per class. This filtered dataset contains 151 classes compared to the original 290 classes in Let-It-Wag!. We then re-ran a diverse set of zero-shot CLIP/SigLIP models on this Let-It-Wag-filtered dataset, and showcase the comparison results to both ImageNet and the original Let-It-Wag! dataset in the table below:

|Model/Dataset|ImageNet Acc|Let-It-Wag! Acc (290 classes)|Let-It-Wag-filtered! Acc (151 classes)|
|-|-|-|-|
|RN50/CC-3M|20.09|3.74|2.56|
|RN50/CC-12M|33.14|8.92|5.95|
|RN-50/openai|59.84|32.93|30.18|
|ViT-B-32/openai|63.34|33.52|32.90|
|ViT-B-32/datacomp|69.18|46.91|49.21|
|ViT-B-16/CC-3M|17.11|3.01|2.42|
|ViT-B-16/CC-12M|37.39|11.49|7.88|
|ViT-B-16/openai|68.36|37.86|37.77|
|ViT-B-16/datacomp|73.48|52.90|56.00|
|ViT-B-16/WebLI|78.49|54.64|49.69|
|ViT-L-14/openai|75.53|45.32|44.49|
|ViT-L-14/datacomp|79.21|63.04|65.70|
|ViT-H-14/DFN|83.45|71.91|71.80|
|ViT-L-16/WebLI|82.07|61.51|56.13|
|ViT-So400m/WebLI|82.03|67.33|63.61|

We observe that almost all models underperform on the Let-It-Wag-filtered dataset compared to the Let-It-Wag! dataset. One interesting point to note here is that all exceptions were models trained on DataComp-1B---they perform ~2-3% worse on Let-It-Wag! compared to the Let-It-Wag! filtered dataset. Given DataComp’s focus of optimizing data curation, this perhaps suggests that research into better data curation could be a viable route to enabling progress on the long-tailed concepts, however this warrants a more concrete treatment which could be an interesting direction for follow-up work to explore. With that said, **all performance numbers remain significantly lower than ImageNet performance across all models, which underscores that models still struggle to perform well on concepts that are long-tailed in web data-distributions**. We will highlight these results and interpretations in the revision.

We hope this sufficiently addresses the reviewer's concerns and are happy to provide further clarifications if required.
Thank you for your response—it has addressed most of my questions. Here are my thoughts on each point:

1. I think it's important to note the ambiguity problem as a limitation of the current frequency measurement pipeline. This can help offer some explanation of the dips observed and clear some questions a reader may have when observing the dips. 

2. It seems there's still some confusion regarding [1], which also analyzes LAION-400M, not just LAION-2B, as I mentioned in my review. Since your work is closely related to previous research, it's crucial to provide a clear comparison that differentiates your contributions from existing work. I suggest enhancing the current literature review section to better highlight these novel aspects, as the current writing doesn't achieve this.

3. Thank you for detailing the dataset curation process. **However, I'm curious if you considered reducing the benchmark size by sampling only classes with fewer than 1,000 instances. If so, did this result in further accuracy degradation?**

4. It would be beneficial to include the dataset statistics and high-level insights presented in the rebuttal within the appendix of the submission.

I have asked one additional question above and will wait for the authors' reply.
Thank you for your further investigation and thorough response! My questions are nicely addressed, and I really enjoyed reading your paper. I am going to keep my score as it is, since I already gave a high score from the beginning.
> **L1: mitigating solutions?**

Thank you for this insightful question. While our paper does not propose specific solutions, we believe its primary contribution is in thoroughly highlighting the issues with current pretraining strategies for multimodal models across various datasets, pretraining methods, architectures, training objectives, and tasks. Additionally, by releasing the "Let it Wag!" testbed, we provide a straightforward test set for future research to build upon, aiming to improve the generalization of multimodal models to long-tail scenarios.
However, we suggest a few potential methods that could be explored to enhance multimodal long-tail:

- **Retrieval Augmentation:** Enhancing generalization to long-tail concepts can be achieved by utilizing the "world-knowledge" of LLMs to provide detailed descriptions for these concepts. This approach transforms the task from simply recognizing long-tail concepts by name to recognizing them by both names and descriptions.
- **Curriculum Learning:** Our tested models used random IID sampling during training. However, research into better sequencing of data samples could potentially improve model generalization to long-tail concepts by inducing more transferable feature representations in VLMs.
- **Synthetic Data:** Addressing the issue of long-tail concepts in web-sourced datasets may not be feasible by merely increasing data samples. There will likely always be low-data density regions in the pretraining data distribution. Using synthetic data, either through procedurally generated samples or text-to-image models, could be a viable mitigation strategy.

We hope these suggestions provide valuable directions for future research and contribute to the development of multimodal models capable of better generalization. We will add these points to the future works section in the appendix.

> **L2: stronger focus on generation and Let-It-Wag! dataset**

We thank the reviewer for raising this point. We would like to respectfully argue that we believe our image-text contributions are equally as important to the other contributions, please see response to W2. Re. focus on Let-It-Wag! dataset, we agree that it would be prudent to emphasize its construction/properties further. We will make these points clearer in the paper and hope this clarifies the issue.

> **L3: usefulness of insights gained from CC/YFCC experiments?**

We thank the reviewer for raising this important point. 

- **CC/YFCC datasets used for important VLM pretraining methods.** While we agree that smaller datasets like CC-3M/YFCC-15M might not yield insights which are practically relevant for SoTA performance, we still believe that it is an important validation to perform these experiments at that scale. We note that lots of high impact work on CLIP-like models have been empirically validated by pretraining on datasets like CC-3M/CC-12M/YFCC-15M, including [CyCLIP](https://arxiv.org/abs/2205.14459), [DeCLIP](https://arxiv.org/abs/2110.05208), [OpenCLIP](https://github.com/mlfoundations/open_clip), [SLIP](https://arxiv.org/abs/2112.12750), [FILIP](https://arxiv.org/abs/2111.07783), [MERU](https://arxiv.org/abs/2304.09172) and [LaCLIP](https://arxiv.org/abs/2305.20088).
 
- **Expansive insights spanning a range of dataset scales.** Further, we note that each of these datasets from small-scale CC-3M to the large scale LAION-400M employ different data curation strategies, and by showcasing that (1) *all datasets have very similar pretraining concept distributions*, and (2) *models trained on any of these datasets showcase the same consistent log-linear scaling trends*, we have empirically showcased the robustness of our main exponential data inefficiency result, and uncovered several important properties of VLM pretraining datasets.

- **Significant compute resources.** Finally, we would like to point out the significant compute resources it would take to conduct our analysis on even larger-scale datasets like DataComp-1B or LAION-2B—in appx L tab 8, we report the storage and compute costs required for conducting all our experiments—at the LAION-400M scale, we would need almost 10TB of disk space and 2000GPU / 7000CPU hours (this is roughly in the order-of-magnitude required for pretraining CLIP models themselves, see [Cherti et al](https://arxiv.org/abs/2212.07143) tab 18). 

- **Testing on other web-scale datasets with Let-It-Wag!** Given our results’ robustness across several scales, we have strong reasons to believe that our result will continue to generalize even for larger-scale datasets and models. One point of evidence to further bolster this is that we tested models trained on larger-scale datasets including DataComp-1B, DFN-1B and WebLI-10B, in our analysis on the Let-It-Wag! Dataset in fig 6. We however do not see any significant deviations for models trained on these datasets that would lead us to believe that they showcase different characteristics to the datasets we analysed in our work.
> **W3: info on Let-It-Wag! construction**

Thank you for pointing out this concern, we apologise for the lack of details regarding the construction of the Let-It-Wag! Benchmark. We provide some details here and have also added this information to the appx.

For curating the set of 290 concepts, we collate our frequency estimates from the LAION-400M dataset for all the 4,029 concepts we consider. We then remove all the concepts that have 0 counts to ensure that our final dataset consists of concepts that have been detected atleast once in LAION-400M, this method has also been used in [[3](https://arxiv.org/pdf/2211.08411)] to ensure robustness to noise in the estimation process. For each extracted concept, we then apply the image download and manual cleaning procedure described in appx G.2. Finally, we removed those concepts that had a low number of total images left post the cleaning process, and retained all the other concepts. We then sorted these concepts in ascending order of their estimated concept frequencies, and retained the top 290. These concepts are used as the final list of classes for our Let-It-Wag! benchmark.

> **W4: high-level insights about long-tail concepts**

Thank you for this question. The broad categories of the most long-tailed concepts with a few examples for each are as follows (a majority of them have been highlighted already in Figs 24-27 in the appendix):
- **Birds:** Western_Scrub_Jay, Cassins_Finch, Prairie_Warbler, Red_eyed_Vireo, Veery
- **Animals:** flatworm, Tibetan_Mastiff, Scottish_Terrier, vine_snake, newt
- **Aircrafts:** A300B4, A310, Falcon_900, DHC-8-300, MD-11 
- **Objects:** guillotine, letter_opener, ladle, dust_jacket
- **Plants and fungi:** mexican_aster, gyromitra, great_masterwort, thorn_apple, cape_flower
- **Misc:** consomme, stratified_texture, eggnog

Additionally, we will release the full list of these concepts as the classes of the Let-It-Wag! benchmark in the appx.

> **Q1: stats about Let-It-Wag?**

Thank you for raising this question. We provide the required stats below.
- **Most frequent concepts:** partridge (count=9489), Bank Swallow (count=9489), eel (7907)
- **Least frequent concepts:** Red-necked Grebe (count=0), SR-20 aircraft (count=0), Globe-flower (count=0)
- **Median frequency of concepts:** 97.5
- **Mean frequency of concepts:** 1096.2

We also show the full histogram of concept frequencies for the 290 concepts in the Let-It-Wag! dataset in the uploaded rebuttal pdf. From the histogram, it is evident that most of the concepts in Let-It-Wag! have frequency less than 2000. About half of the concepts in Let-It-Wag! (~140) have a frequency less than 1000. Hence, this histogram sufficiently establishes that our **Let-It-Wag! dataset truly captures the long tail**.

> **Q2: variance in fig 2 points**

Thank you for the insightful question. We provide zero-shot classification plots for CC-3M, CC-12M, and LAION-400M in the uploaded rebuttal PDF, including 95% confidence intervals for each point. This approach follows the standard practice from works like [[1](https://arxiv.org/abs/2107.04649), [2](https://arxiv.org/abs/2007.00644)]. Our plots show that **the spread at higher frequencies is significantly larger than at moderate frequencies**, corroborating the finding that **higher frequency concepts are more ambiguous and polysemous**. These results **support the observed dips in accuracy at high-frequency points**. We will include these plots in the appendix and discuss this in the paper.