Context: Here is the text of a work.

—BEGINNING OF WORK—

No “Zero-Shot” Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance
Abstract

Web-crawled pretraining datasets underlie the impressive “zero-shot" evaluation performance of multimodal models, such as CLIP for classification/retrieval and Stable-Diffusion for image generation. However, it is unclear how meaningful the notion of “zero-shot" generalization is for such multimodal models, as it is not known to what extent their pretraining datasets encompass the downstream concepts targeted for during “zero-shot" evaluation. In this work, we ask: How is the performance of multimodal models on downstream concepts influenced by the frequency of these concepts in their pretraining datasets?

We comprehensively investigate this question across 34 models and five standard pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M, LAION-Aesthetics), generating over 300GB of data artifacts. We consistently find that, far from exhibiting “zero-shot” generalization, multimodal models require exponentially more data to achieve linear improvements in downstream “zero-shot" performance, following a sample inefficient log-linear scaling trend. This trend persists even when controlling for sample-level similarity between pretraining and downstream datasets [75], and testing on purely synthetic data distributions [48]. Furthermore, upon benchmarking models on long-tailed data sampled based on our analysis, we demonstrate that multimodal models across the board perform poorly. We contribute this long-tail test set as the Let it Wag! benchmark to further research in this direction. Taken together, our study reveals an exponential need for training data which implies that the key to “zero-shot” generalization capabilities under large-scale training paradigms remains to be found.
\doparttoc\faketableofcontents
            \faGithub github.com/bethgelab/frequency_determines_performance
              \faDatabase huggingface.co/datasets/bethgelab/let-it-wag
1 Introduction

Multimodal models like CLIP [87] and Stable Diffusion [92] have revolutionized performance on downstream tasks—CLIP is now the de-facto standard for “zero-shot" image recognition [129, 68, 122, 45, 128] and image-text retrieval [44, 60, 23, 113, 125], while Stable Diffusion is now the de-facto standard for “zero-shot" text-to-image (T2I) generation [89, 16, 92, 40]. In this work, we investigate this empirical success through the lens of zero-shot generalization [65], which refers to the ability of the model to apply its learned knowledge to new unseen concepts. Accordingly, we ask: Are current multimodal models truly capable of “zero-shot” generalization?

To address this, we conducted a comparative analysis involving two main factors: (1) the performance of models across various downstream tasks and (2) the frequency of test concepts within their pretraining datasets. We compiled a comprehensive list of 4,02940294,0294 , 029 concepts1 from 27 downstream tasks spanning classification, retrieval, and image generation, assessing the performance against these concepts. Our analysis spanned five large-scale pretraining datasets with different scales, data curation methods and sources (CC-3M [103], CC-12M [26], YFCC-15M [109], LAION-Aesthetics [99], LAION-400M [98]), and evaluated the performance of 10 CLIP models and 24 T2I models, spanning different architectures and parameter scales. We consistently find across all our experiments that, across concepts, the frequency of a concept in the pretraining dataset is a strong predictor of the model’s performance on test examples containing that concept. Notably, model performance scales linearly as the concept frequency in pretraining data grows exponentially i.e., we observe a consistent log-linear scaling trend. We find that this log-linear trend is robust to controlling for correlated factors (similar samples in pretraining and test data [75]) and testing across different concept distributions along with samples generated entirely synthetically [48].

Our findings indicate that the impressive empirical performance of multimodal models like CLIP and Stable Diffusion can be largely attributed to the presence of test concepts within their vast pretraining datasets, thus their reported empirical performance does not constitute “zero-shot” generalization. Quite the contrary, these models require exponentially more data on a concept to linearly improve their performance on tasks pertaining to that concept, highlighting extreme sample inefficiency.

In our analysis, we additionally document the distribution of concepts encountered in pretraining data and find that:

    •

    Concept Distribution: Across all pretraining datasets, the distribution of concepts is long-tailed (see Fig. 5 in Sec. 5), which indicates that a large fraction of concepts are rare. However, given the extreme sample inefficiency observed, what is rare is not properly learned during multimodal pretraining.
    •

    Concept Correlation across Pretraining Datasets: The distribution of concepts across different pretraining datasets are strongly correlated (see Tab. 4 in  Sec. 5), which suggests web crawls yield surprisingly similar concept distributions across different pretraining data curation strategies, necessitating explicit rebalancing efforts [11, 121].
    •

    Image-Text Misalignment between Concepts in Pretraining Data: Concepts often appear in one modality but not the other, which implies significant misalignment (see Tab. 4 in Sec. 5). Our released data artifacts can help image-text alignment efforts at scale by precisely indicating the examples in which modalities misalign. Note that the log-linear trend across both modalities is robust to this misalignment.

To provide a simple benchmark for generalization performance for multimodal models, which controls for the concept frequency in the training set, we introduce a new long-tailed test dataset called “Let It Wag!”. Current models trained on both openly available datasets (e.g., LAION-2B [99], DataComp-1B [44]) and closed-source datasets (e.g., OpenAI-WIT [87], WebLI [28]) have significant drops in performance, providing evidence that our observations may also transfer to closed-source datasets. We publicly release all our data artifacts (over 300GB), amortising the cost of analyzing the pretraining datasets of multimodal foundation models for a more data-centric understanding of the properties of multimodal models in the future.

Several prior works [87, 44, 78, 41, 79, 70] have investigated the role of pretraining data in affecting performance. Mayilvahanan et al. [75] showed that CLIP’s performance is correlated with the similarity between training and test datasets. In other studies on specific areas like question-answering [58] and numerical reasoning [90] in large language models, high train-test set similarity did not fully account for observed performance levels [123]. Our comprehensive analysis of several pretraining image-text datasets significantly adds to this line of work, by (1) showing that concept frequency determines zero-shot performance and (2) pinpointing the exponential need for training data as a fundamental issue for current large-scale multimodal models. We conclude that the key to “zero-shot" generalization capabilities under large-scale training paradigms remains to be found.
2 Concepts in Pretraining Data and Quantifying Frequency
Refer to caption
Figure 1: Concept Extraction and Frequency Estimation Pipeline. (left) We compile 4,02940294,0294 , 029 concepts from 17171717 classification, 2222 retrieval, and 8888 image generation prompt datasets. (right) We construct efficient indices for both text-search (using standard unigram indexing (1)) and image-search (using RAM++ [56] (2)); intersecting hits from both gives us (3) the image-text matched frequencies per concept.

In this section, we outline our methodology for obtaining concept frequencies within pretraining datasets. We first define our concepts of interest, then describe algorithms for extracting their frequencies from images and text captions of pretraining datasets. Finally, we discuss how to aggregate them to calculate matched image-text concept frequencies. For a schematic overview of our methods, see Fig. 1.

Defining Concepts. We define “concepts” as the specific objects or class categories we seek to analyze in the pretraining datasets. For zero-shot classification tasks, these concepts are the class names, such as the 1,00010001,0001 , 000 classes in ImageNet [34] (e.g., “tench”, “goldfish”, “stingray”). For image-text retrieval and image generation tasks, concepts are identified as all nouns present in the test set captions or generation prompts, respectively. For example, in the caption, “A man is wearing a hat”, we extract “man” and “hat” as relevant concepts. We additionally filter out nouns that are present in less than five downstream evaluation samples to remove ambiguous or irrelevant concepts. Across all our experiments, we collate a list of 4,02940294,0294 , 029 concepts sourced from 17171717 classification, 2222 retrieval, and 8888 image generation downstream datasets (see Tab. 1 for details).

Concept Frequency from Text Captions. To enable efficient concept searches, we pre-index all captions from the pretraining datasets, i.e., construct a mapping from concepts to captions. We first use part-of-speech tagging to isolate common and proper nouns and subsequently lemmatize them to standardize word forms [61] with SpaCy [55] . These lemmatized nouns are then cataloged in inverted unigram dictionaries, with each noun being the key and all the indices in the pretraining data samples containing that noun being its values. To determine the frequency of a concept, particularly those composed of multiple words, we examine the concept’s individual unigrams within these dictionaries. For multi-word expressions, by intersecting the lists of sample indices corresponding to each unigram, we identify the samples that contain all parts of the concept. The frequency of the concept in the text captions is the count of these intersecting sample indices. Our frequency estimation algorithm hence allows scalable 𝒪⁢(1)𝒪1\mathcal{O}(1)caligraphic_O ( 1 ) search with respect to the number of captions for any given concept in the pretraining dataset captions.

Concept Frequency from Images. Unlike text captions, we do not have a finite vocabulary for pre-indexing pretraining images, and thus cannot perform 𝒪⁢(1)𝒪1\mathcal{O}(1)caligraphic_O ( 1 ) concept lookup. Instead, we collect all the 4,02940294,0294 , 029 downstream concepts and verify their presence in images using a pretrained image tagging model. We tested various open-vocabulary object detectors, image-text matching models and multi-tagging models. We found that RAM++ [56]—an open-set tagging model that tags images based on a predefined list of concepts in a multi-label manner—performs the best. This approach generates a list of pretraining images, each tagged with whether the downstream concepts are present or not, from which we can compute concept frequencies. We provide qualitative examples along with design choice ablations in Appx. F.

Image-Text Matched Concept Frequencies. Finally, we combine the frequencies obtained from both text and image searches to calculate matched image-text frequencies. This involves identifying pretraining samples where both the image and its associated caption correspond to the concept. By intersecting the lists from our image and text searches, we determine the count of samples that align in both modalities, offering a comprehensive view of concept representation across the dataset. We note that this step is necessary as we observed significant image-text misalignment between concepts in the pretraining datasets (see Tab. 4), hence captions may not reflect what is present in the image and vice-versa. This behaviour has also been alluded to in prior work investigating pretraining data curation strategies [72, 71, 120, 79]. We provide more detailed analysis on image-text misalignment in Sec. 5.
3 Comparing Pretraining Frequency & “Zero-Shot" Performance

Having obtained frequency estimates for our downstream concepts, we now establish the relationship between image-text matched pretraining concept frequencies and zero-shot performance across classification, retrieval, and generation tasks. We first detail our experimental approach and then discuss key results.
3.1 Experimental Setup
Table 1: Pretraining and downstream datasets used in Image-Text (CLIP) experiments.
Dataset Type 	Datasets
Pretraining 	CC-3M [103]  CC-12M [26]  YFCC-15M [109]  LAION-400M [98]
	ImageNet [34]  SUN397 [119]  UCF101  [104]  Caltech101 [43]  EuroSAT [52]  CUB [117]
Classification-Eval 	Caltech256 [46]  Flowers102 [80]  DTD [30]  Birdsnap [15]  Food101 [19]  Stanford-Cars [62]
	FGVCAircraft [73]  Oxford-Pets [83]  Country211 [87]  CIFAR-10 [63]  CIFAR100 [63]
Retrieval-Eval 	Flickr-1K [124]  COCO-5K [69]

We analyze two classes of multimodal models: Image-Text and Text-to-Image. For both, we detail the pretraining and testing datasets, along with their associated evaluation parameters.
3.1.1 Image-Text (CLIP) Models

Datasets. Our evaluation consists of 4 pretraining datasets, 2 downstream retrieval datasets, and 17 downstream classification datasets, presented in Tab. 1, covering a broad spectrum of objects, scenes, and fine-grained distinctions.

Models. We test CLIP [87] models with both ResNet [50] and Vision Transformer [35] architecture, with ViT-B-16 [77] and RN50  [45, 78] trained on CC-3M and CC-12M, ViT-B-16, RN50, and RN101 [57] trained on YFCC-15M, and ViT-B-16, ViT-B-32, and ViT-L-14 trained on LAION400M [98]. We follow open_clip [57], slip [77] and cyclip [45] for all implementation details.

Prompting. For zero-shot classification, we experiment with three prompting strategies: {classname} only, “A photo of a {classname}" and prompt-ensembles [87], which averages over 80808080 different prompt variations of {classname}. For retrieval, we use the image or the caption as input corresponding to I2T (image-to-text) or T2I (text-to-image) retrieval respectively.

Metrics. We compute mean zero-shot classification accuracy for classification tasks [87]. For retrieval, we assess performance using traditional metrics for both text-to-image and image-to-text retrieval tasks [87] (Recall@1, Recall@5, Recall@10).
3.1.2 Text-to-Image Models
Table 2: Models used in text-to-image (T2I) experiments.
Category 	Models
	M-Vader [14]  DeepFloyd-IF-M [9]  DeepFloyd-IF-L [9]  DeepFloyd-IF-XL [9]
	 GigaGAN [59]  DALL·E Mini [33]  DALL.E Mega [33]  Promptist+SD-v1.4 [49]
Models 	Dreamlike-Diffusion-v1.0 [2]  Dreamlike Photoreal v2.0 [3]  OpenJourney-v1 [4]  OpenJourney-v2 [5]
	SD-Safe-Max [92]  SD-Safe-Medium [92]  SD-Safe-Strong [92]  SD-Safe-Weak [92]
	SD-v1.4 [92]  SD-v1.5 [92]  SD-v2-Base [92]  SD-v2-1-base [92]
	Vintedois-Diffusion-v0.1 [7]  minDALL.E [93]  Lexica-SD-v1.5 [1]  Redshift-Diffusion [6]

Datasets. Our pretraining dataset is LAION-Aesthetics [99], with downstream evaluations done on subsampled versions of eight datasets as released by HEIM [67]: CUB200 [117], Daily-DALLE [32], Detection [29], Parti-Prompts [126], DrawBench [94], COCO-Base [69], Relational Understanding [31] and Winoground [110]. Please refer to HEIM [67] for more details on the evaluation datasets used.

Models. We evaluate 24 T2I models, detailed in Tab. 2. Their sizes range from 0.4B parameters (DeepFloyd-IF-M [9] and DALL·E Mini [33]) to 4.3B parameters (DeepFloyd-IF-XL [9]). We include various Stable Diffusion models [92] as well as variants tuned for specific visual styles [6, 4, 5].

Prompting. Text prompts from the evaluation datasets are used directly to generate images, with 4 image samples generated for each prompt.

Metrics. Evaluation consists of image-text alignment and aesthetic scores. For automated metrics [67], we use expected and max CLIP-score [54] to measure image-text alignment along with expected and max aesthetics-score [98] to measure aesthetics. To verify reliability of the automated metrics, we compare them with human-rated scores (measured on a 5-point grading scale) for both image-text alignment and aesthetics [67]. To supplement the human-rated scores provided by HEIM [67], we confirm our findings by performing a small-scale human evaluation as well (see Appx. C).
Refer to caption
Figure 2: Log-linear relationships between concept frequency and CLIP zero-shot performance. Across all tested architectures (RN50, RN101, ViT-B-32, ViT-B-16, ViT-L-14) and pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M), we observe a consistent linear relationship between CLIP’s zero-shot performance on a concept and the log-scaled concept pretraining frequency. This trend holds for both zero-shot classification (results averaged across 17 datasets) and image-text retrieval (results averaged across 2 datasets). ** indicates that the result is significant (p<0.05𝑝0.05p<0.05italic_p < 0.05 with a two-tailed t-test.), and thus we show pearson correlation (ρ𝜌\rhoitalic_ρ) as well.
Refer to caption
Figure 3: Log-linear relationships between concept frequency and T2I aesthetic scores. Across all tested T2I models pretrained on the LAION-Aesthetics dataset, we observe a consistent linear relationship between zero-shot performance on a concept and the log-scaled concept pretraining frequency.
3.2 Result: Pretraining Frequency is Predictive of “Zero-Shot" Performance

We now probe the impact of concept frequency in pretraining datasets on the zero-shot performance of image-text models. We utilize the matched image-text concept frequencies for estimating frequency of concepts during pretraining. Our findings, illustrated comprehensively in Figs. 2 and 3, demonstrate the effect concept frequency has on model performance across various tasks and model types.

Understanding the Plots. The plots in the main paper present text-image (CLIP) models’ zero-shot classification results using accuracy and text-to-image retrieval performance using Recall@10. Similarly, we present T2I generative models’ performance on image generation tasks using the expected aesthetics score. For the other aforementioned metrics for retrieval as well as other automated generation metrics along with human-rated scores, we find that they show similar trends, and we provide them for reference in Apps. B and C. For clarity, the data presentation is simplified from scatter plots to a cohesive line similar to work from Kandpal et al. [58] and Razeghi et al. [90]. The x-axis is log-scaled, and performance metrics are averaged within bins along this axis for ease-of-visualization of the log-linear correlation. We removed bins containing very few concepts per bin by standard IQR removal [118] following Kandpal et al. [58]. We additionally compute the pearson correlation ρ𝜌\rhoitalic_ρ for each line and provide significance results based on a two-tailed t-test [106].

Key Finding: Log-linear scaling between concept frequency and zero-shot performance. Across all 16 plots, we observe a clear log-linear relationship between concept frequency and zero-shot performance. Note that these plots vary in (i) discriminative vs. generative model types, (ii) classification vs. retrieval tasks, (iii) model architecture and parameter scales, (iv) pretraining datasets with different curation methods and scales, (v) different evaluation metrics, (vi) different prompting strategies for zero-shot classification, and (vii) concept frequencies isolated only from image or text domains (additional experiments which show variation along (v) are presented in Apps. B and C, across (vi) are presented in Appx. A, and across (vii) are presented in Appx. D). The observed log-linear scaling trend persists across all seven presented dimensions. Thus, our results clearly reveal data hungry learning, i.e, a lack in current multimodal models’ ability to learn concepts from pretraining datasets in a sample-efficient manner.
4 Stress-Testing the Concept Frequency-Performance Scaling Trend

In this section, we seek to isolate the effect of concept frequency on zero-shot performance by controlling a widely known influential factor [123, 75]: similarity in distribution between pretraining and downstream test data. Additionally, we aim to validate our hypothesis further by examining the relationship between concept frequency and downstream performance on models trained on pretraining data with synthetically controlled concept distributions, images and captions.
4.1 Controlling for Similar Samples in Pretraining and Downstream Data
Refer to caption
Figure 4: Stress-testing the log-linear scaling trends. We provide further evidence that the log-linear relationship between performance and concept frequency holds across different scenarios: (left) we control for the effect of “similarity” between the downstream test sets and pretraining datasets, and (right) we conduct experiments on an entirely synthetic pretraining distribution with no real-world text-captions or images.

Motivation. Prior work has suggested that sample-level similarity between pretraining and downstream datasets impacts model performance [58, 75, 123, 90]. This leaves open the possibility that our frequency-performance results are simply an artifact of this factor, i.e., as concept frequency increases, it is likely that the pretraining dataset also contains more similar samples to the test sets. We hence investigate whether concept frequency remains predictive of downstream performance after controlling for sample-level similarity.

Setup. We use the LAION-200M [10] dataset for this experiment. We first verified that a CLIP-ViT-B-32 model trained on LAION-200M dataset (used to study sample similarity in prior work [75]) exhibits a similar log-linear trend between concept frequency and zero-shot performance. Then, we use the near_pruning method from Mayilvahanan et al. [75] to eliminate 50 million samples most similar to the test sets from the pretraining LAION-200M dataset. We provide details for this in Sec. E.1. This removes the most similar samples between pretraining and test sets. We verify that this procedure influences the performance of the model drastically in performance across our aggregate classification and retrieval tasks respectively, replicating the findings of Mayilvahanan et al. [75].

Key Finding: Concept Frequency still Predictive of Performance. We repeat our analysis on models trained with this controlled pretraining dataset with 150M samples, and report results on the same downstream classification and retrieval datasets in LABEL:{fig:ablation-prasanna-and-synthclip} (left). Despite the removal of the most similar samples between pretraining and test sets, we still consistently observe a clear log-linear relationship between pretraining frequency of test set concepts and zero-shot performance.

Conclusion. This analysis reaffirms that, despite removing pretraining samples closely related to the test sets, the log-linear relationship between concept frequency and zero-shot performance persists. Note that this is despite substantial decreases in absolute performance, highlighting the robustness of concept frequency as a performance indicator.
4.2 Testing Generalization to Purely Synthetic Concept and Data Distributions

Motivation. Sampling across real-world data might not result in significant differences in concept distribution, as we will later show in Sec. 5. Hence, we repeat our analysis on a synthetic dataset designed with an explicitly different concept distribution [48]. This evaluation aims to understand if pretraining concept frequency remains a significant performance predictor within a synthetic concept distribution, generalizing even on models pretrained on entirely synthetic images and captions.

Setup. The SynthCI-30M dataset [48] introduces a novel concept distribution, generating 30 million synthetic image-text pairs. Utilizing the publicly available data and models from this benchmark, we explore the relationship between concept frequency and model performance in this synthetic data regime.

Key Finding: Concept Frequency is still Predictive of Performance. We report results on models trained with their controlled dataset in LABEL:{fig:ablation-prasanna-and-synthclip} (right). We still consistently observe a clear log-linear relationship between concept frequency and zero-shot performance.

Conclusion. This consistency highlights that concept frequency is a robust indicator of model performance, extending even to entirely synthetically constructed datasets and pretraining concept distributions.
Refer to caption
(a) Text search counts
Refer to caption
(b) Image search counts
Refer to caption
(c) Image-text search counts
Figure 5: Concept distribution of pre-training datasets is highly long-tailed. We showcase the distribution of pre-training frequencies of all concepts aggregated across all our downstream classification datasets. Across all three pre-training datasets, we observe very heavy tails. We normalize the concept frequencies and remove concepts with 0 counts for improved readability.
5 Additional Insights from Pretraining Concept Frequencies

We now present notable observations concerning the distribution of downstream concept frequencies across text, image, and text-image matched modalities in pretraining datasets.

Finding 1: Pretraining Datasets Exhibit Long-tailed Concept Distribution. Our analysis in Fig. 5 reveals an extremely long-tailed distribution of concept frequencies in pretraining datasets, with over two-thirds of concepts occurring at almost negligible frequencies relative to the size of the datasets. Our observations support the findings of past work that have noted the long-tailed distribution of large-scale language datasets [24, 84, 132]. As we observed with the log-linear trend, this distribution directly reflects disparities in performance.

Finding 2: Misalignment Between Concepts in Image-Text Pairs. We investigated the alignment of concepts within paired pretraining image-text data. Perfect image-text alignment is defined as every image-text pair containing the same concepts. Previous studies have qualitatively discussed the problem of misalignment in large image-text datasets [71, 120, 72]. Our analysis enables us to quantify this misalignment degree—for each image-text pair in the pretraining dataset, we find the concepts that are matched to the image and the text caption independently. If there are no intersecting concepts from the independent image and text hits, we count that pair as misaligned (detailed algorithm provided in Appx. G).  Tab. 4 shows the high degree of misalignment in all image-text pairs. To the best of our knowledge, this is the first attempt to explicitly quantify the degree of misalignment in pretraining image-text datasets. We release the precise misaligned image-text pairs in the pretraining datasets to enable better data curation.

Finding 3: Concept Frequencies Across Datasets are Correlated. Despite vast differences in the size (ranging from 3M to 400M samples) and curation strategies of the datasets analyzed, we discovered a surprisingly high correlation in concept frequencies across them, as presented in Tab. 4. This consistency suggests that the internet, as the common source of these datasets, naturally exhibits a long-tailed distribution, influencing any dataset derived from it to also display similar long-tailed behavior. This result inspired the “Let It Wag!” dataset.
Table 3: For each pretraining dataset, we present the number of misaligned image-text pairs and the misalignment degree: the fraction of misalignment pairs in the dataset.
[Uncaptioned image] Figure 6: Large-drops in accuracy on “Let It Wag!”. Across all 40 tested models, we note large performance drops compared to ImageNet. Further, we note that the gap in performance seems to be decreasing for higher-capacity models as demonstrated by the large positive slope (1.58) for the larger models.
Dataset/ 	Number of 	Misalignment
Misalignment 	Misaligned pairs 	Degree (%)
CC3M 	557,683 	16.81%
CC12M 	2,143,784 	17.25%
YFCC15M 	5,409,248 	36.48%
LAION-A 	23,104,076 	14.34%
LAION400M 	21,996,097 	5.31%
Correlations 	CC3M 	CC12M 	YFCC15M 	L400M
CC3M 	1.00 	0.79 	0.96 	0.63
CC12M 	– 	1.00 	0.97 	0.74
YFCC15M 	– 	– 	1.00 	0.76
L400M 	– 	– 	– 	1.00
Table 3: For each pretraining dataset, we present the number of misaligned image-text pairs and the misalignment degree: the fraction of misalignment pairs in the dataset.
Table 4: We compute correlation in concept frequency across pretraining datasets. Despite significant differences in scale and curation, we consistently observe strong correlation.
6 Testing the Tail: Let It Wag!

Motivation. From the previous sections, we have identified a consistent long-tailed concept distribution, highlighting the scarcity of certain concepts on the web. This observation forms the basis of our hypothesis that models are likely to underperform when tested against data distributions that are heavily long-tailed. To test this, we carefully curate 290 concepts that were identified as the least frequent across all pretraining datasets. This includes concepts like an A310 aircraft, a wormsnake, and a tropical kingbird. We then use these concepts to create a classification test set, “Let It Wag!”.

Dataset Details. The “Let It Wag!” classification dataset comprises 130K test samples downloaded from the web using the method of Prabhu et al. [86]. The test samples are evenly distributed across 290 categories that represent long-tailed concepts. From the list of curated concepts, we download test set images, deduplicate them, remove outliers, and finally manually clean and hand-verify the labels.

Analysis Details. We ran both classification and image generation experiments on “Let It Wag!”. For classification, we assessed the performance of 40 text-image (CLIP) models on the “Let It Wag!” classification dataset, using an ensemble of 80 prompts from Radford et al. [87]. For the generative task, we utilized SD-XL [85] , SD-v2 [92], and Dreamlike-Photoreal-v2.0 [3] to generate images for the long-tailed concepts. For each model, we ran 50 diffusion steps, maintaining default settings for all other parameters.

Text-Image Classification Results. We showcase the results of our long-tailed classification task in Fig. 6—we plot results of all models on both “Let It Wag!” (y-axis) and ImageNet (x-axis). We observe that all models underperform by large margins on the long-tailed “Let It Wag!” dataset (upto 20% lower absolute accuracies compared to ImageNet). This performance drop-off generalises across all model scales and 10 different pretraining data distributions, reinforcing the notion that all web-sourced pretraining datasets are inherently constrained to be long-tailed. With that said, note that the higher capacity models (fitted line with slope=1.58 in Fig. 6) seem to be closing the gap to ImageNet performance, indicating improved performance on the long-tailed concepts.

T2I Generation Results. We provide a qualitative analysis on image generation for assessing T2I models on rare concepts in Fig. 7. For diversity, we generate prompts using Gemini [108] (top row of generated images) and GPT4 [12] (bottom row of generated images). Green borders represent correct generations, red borders represent incorrect generations and yellow borders represent ambiguous generation. While descriptive prompting generally aids in improving the quality of generated images [49], we still observe T2I models failing to comprehend and accurately represent many concepts in our “Let It Wag!” dataset. Some failure cases involve misrepresenting activities (such as Pizza Tossing or Cricket Bowling as shown in Fig. 24), generating the wrong concept (Chuck-will’s-widow as shown in  Fig. 7 top), as well as not comprehending the concept at all (Ocarina in Fig. 7 bottom). We can see that Stable Diffusion models are prone to the long tail qualitatively—we also provide quantitative results in Sec. H.1.

Conclusion. Across both the classification and generation experiments, we have showcased that current multimodal models predictably underperform, regardless of their model scale or pretraining datasets. This suggests a need for better strategies for sample-efficient learning on the long-tail.
Refer to caption
Figure 7: Qualitative results on the “Let It Wag!” dataset categories demonstrate failure cases of state-of-the-art T2I models on long-tailed concepts. In our experiments, we create 4 text prompts for each category using Gemini [108] and GPT4 [12] which are fed to 3 Stable Diffusion [92] models. Generation with red border is incorrect, with green border is correct and with yellow border is ambiguous. We observe that despite advances in high-fidelity image generation, there is scope for improvement for such concepts.
7 Related Work

Effect of Pre-training Data on Downstream Data. Several data-centric prior works [87, 44, 78, 41, 79, 70, 120, 121, 131, 105, 74, 88, 95, 96, 37, 25, 91] have highlighted the importance of pretraining data in affecting performance. Fang et al [41] robustly demonstrated that pretraining data diversity is the key property underlying CLIP’s strong out-of-distribution generalisation behaviour. Nguyen et al [78] extended this analysis to showcase that differences in data distributions can predictably change model performance, and that this behaviour can lead to effective data mixing strategies at pretraining time. Mayilvahanan et al. [75] complemented this research direction by showing that CLIP’s performance is correlated with the similarity between training and test datasets. Udandarao et al. [114] further showed that the frequency of certain visual data-types in the LAION-2B dataset was roughly correlated to the performance of CLIP models in identifying visual data-types. Our findings further pinpoint that the frequency of concept occurrences is a key indicator of performance. This complements existing research in specific areas like question-answering [58] and numerical reasoning [90] in large language models, where high train-test set similarity does not fully account for observed performance levels [123]. Concurrent to our work, Parashar et al. [82] also explore the problem of long-tailed concepts in the LAION-2B dataset and how it affects performance of CLIP models supporting our findings. In contrast to their work, we look at count separately in image and text modalities, as well as across pretraining sets, and do a number of control experiments to thoroughly test the robustness of our result. Finally, our demonstration that the long tail yields a log-linear trend explicitly indicates exponential sample inefficiency in large-scale pretrained models.

Data-centric analyses. Our work also adds to the plethora of work that aims to understand and explore the composition of large-scale datasets, and uses data as a medium for improving downstream tasks. Prior work has noted the importance of data for improving model performance on a generalised set of tasks [44, 11, 39, 13, 102]. For instance, several works utilise retrieved and synthetic data for adapting foundation models on a broad set of downstream tasks [115, 51, 111, 20, 97, 130, 86]. Maini et al. [72] observed the existence of “text-centric” clusters in LAION-2B and measured its impact on downstream performance. Other work has seeked to target the misalignment problem that we quantified in Tab. 4 by explicit recaptioning of pretraining datasets [64, 27, 116, 127, 79, 16]. Further, studies have also shown that by better data pruning strategies, neural scaling laws can be made more efficient than a power-law [105, 10]. Prior work has also showcased that large-scale datasets suffer from extreme redundancy in concepts, and high degrees of toxic and biased content [38, 112]. Further research has showcased the downstream effects that such biases during pretraining induce in state-of-the art models [18, 100, 17]. Our work tackles the issue of long-tailed concepts in pretraining datasets, and shows that this is an important research direction to focus efforts on.
8 Conclusions and Open Problems

In this work, we delved into the five pretraining datasets of 34 multimodal vision-language models, analyzing the distribution and composition of concepts within, generating over 300GB of data artifacts that we publicly release. Our findings reveal that across concepts, significant improvements in zero-shot performance require exponentially more data, following a log-linear scaling trend. This pattern persists despite controlling for similarities between pretraining and downstream datasets or even when testing models on entirely synthetic data distributions. Further, all tested models consistently underperformed on the “Let it Wag!” dataset, which we systematically constructed from our findings to test for long-tail concepts. This underlines a critical reassessment of what “zero-shot” generalization entails for multimodal models, highlighting the limitations in their current generalization capabilities. We highlight a few exciting avenues for future research to bridge these gaps or obtain further insights:

Understanding Image-Text Misalignments. One can explore the origins of misalignments between images and texts, such as the limitations of exact matching for concept identification in captions, inaccuracies from the RAM++ tagging model, or captions that are either too noisy or irrelevant.

Investigating Compositional Generalization. The term “zero-shot generalization” often refers to models’ ability for compositional generalization, meaning the ability to understand new combinations of concepts not previously encountered. This is distinct from traditional zero-shot learning and presents an intriguing, yet unresolved challenge: analyzing compositional generalization from a data-centric perspective.

Methods for Bridging the Generalization Gap. Addressing the challenges posed by the long-tail distribution involves improving model generalization to overcome the limited improvement from pretraining we found in our study. Retrieval mechanisms can compensate for the inherent generalization shortcomings of pretrained models, providing a viable path to mitigating the effects of long-tailed pretraining data distributions.
Acknowledgements

The authors would like to thank (in alphabetic order): Jonathan Roberts, Karsten Roth, Mehdi Cherti, Prasanna Mayilvahanan, Shyamgopal Karthik and Thao Nguyen for helpful feedback and providing access to various resources throughout the project. YS would like to thank Nicholas Carlini, Daphne Ippolito, Katherine Lee, Matthew Jagielski, and Milad Nasr. AP is funded by Meta AI Grant No. DFR05540. VU and YS thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS). VU also thanks the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program for support. PT thanks the Royal Academy of Engineering for their support. AB acknowledges the Amazon Research Award. SA is supported by a Newton Trust Grant. MB acknowledges financial support via the Open Philanthropy Foundation funded by the Good Ventures Foundation. This work was supported by the German Research Foundation (DFG): SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms, TP4, project number: 276693517 and the UKRI grant: Turing AI Fellowship EP/W002981/1. MB is a member of the Machine Learning Cluster of Excellence, funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC number 2064/1 – Project number 390727645.
References

    [1]

    George Kingsley Zipf. Human behavior and the principle of least effort: An introduction to human ecology. Ravenio books, 2016.

\doparttoc\faketableofcontents

Part I Appendix
\parttoc
Appendix A Concept Frequency is Predictive of Performance Across Prompting Strategies

We extend the zero-shot classification results from Fig. 2 in Fig. 8 with two different prompting strategies: the results in the main paper used the {classname} only as the prompts, here we showcase both (1) “A photo of a {classname}” prompting and (2) 80 prompt ensembles as used by Radford et al [87]. We observe that the strong log-linear trend between concept frequency and zero-shot performance consistently holds across different prompting strategies.
Refer to caption
Figure 8: Log-linear relationships between concept frequency and CLIP zero-shot performance. Across all tested architectures (RN50, RN101, ViT-B-32, ViT-B-16, ViT-L-14) and pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M), we observe a consistent linear relationship between CLIP’s zero-shot classification accuracy on a concept and the log-scaled concept pretraining frequency. This trend holds for both “A photo of a {classname}” prompting style and 80 prompt ensembles [87]. ** indicates that the result is significant (p<0.05𝑝0.05p<0.05italic_p < 0.05 with a two-tailed t-test.), and thus we show pearson correlation (ρ𝜌\rhoitalic_ρ) as well.
Appendix B Concept Frequency is Predictive of Performance Across Retrieval Metrics

We supplement Fig. 2 in the main paper, where we showed results with the text-to-image (I2T) recall@10 metric. In Figs. 9 and 10, we present results for the retrieval experiments across all six metrics: I2T-Recall@1, I2T-Recall@5, I2T-Recall@10, T2I-Recall@1, T2I-Recall@5, T2I-Recall@10. We observe that the strong log-linear trend between concept frequency and zero-shot performance robustly holds across different retrieval metrics.
Refer to caption
Figure 9: Log-linear relationships between concept frequency and CLIP I2T retrieval performance. Across all tested architectures (RN50, RN101, ViT-B-32, ViT-B-16, ViT-L-14) and pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M), we observe a consistent linear relationship between CLIP’s retrieval performance (measured using image-to-text metrics) on a concept and the log-scaled concept pretraining frequency. ** indicates that the result is significant (p<0.05𝑝0.05p<0.05italic_p < 0.05 with a two-tailed t-test.), and thus we show pearson correlation (ρ𝜌\rhoitalic_ρ) as well.
Refer to caption
Figure 10: Log-linear relationships between concept frequency and CLIP T2I retrieval performance. Across all tested architectures (RN50, RN101, ViT-B-32, ViT-B-16, ViT-L-14) and pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M), we observe a consistent linear relationship between CLIP’s retrieval performance (measured using text-to-image metrics) on a concept and the log-scaled concept pretraining frequency. ** indicates that the result is significant (p<0.05𝑝0.05p<0.05italic_p < 0.05 with a two-tailed t-test.), and thus we show pearson correlation (ρ𝜌\rhoitalic_ρ) as well.
Appendix C Concept Frequency is Predictive of Performance for T2I Models

We extend the results from Fig. 3 with Figs. 11, 12, 13, 14 and 15. As with Fig. 3, due to the high concept frequency, the scaling trend is weaker. Furthermore, we do see inconsistency in the trends for the human-rated scores retrieved from HEIM [67], hence we perform a small scale human evaluation to check them.

Given the societal relevance [22], we decided to test Stable Diffusion [92] (v1.4) on generating public figures. We scraped 50,000 people from the “20230123-all” Wikidata JSON dump by filtering for entities listed as “human” [8], and scraped a reference image for the human study for each person if an image was available. After computing concept frequency from LAION-Aesthetics text captions (using suffix array [66]), we found that ≈\approx≈10,000 people were present in the pretraining dataset. Note that to ensure the people’s names were treated as separate words, we computed frequency for strings of the format “ {entity} ”. We then randomly sample 360 people (for which a reference image was available) normalized by frequency [21] for the human study. For generating images with Stable Diffusion, we used the prompt “headshot of {entity}”, in order to specify to the model that “{entity}” is referring to the person named “{entity}” [47].

We assessed image-text alignment with a human study with 6 participants, where each participant was assigned 72 samples; for consistency, of the 360 total samples, we ensured 10% were assigned to 3 participants. Provided with a reference image, the participants were asked if the sample accurately depicts the prompt. Three choices were provided: “Yes” (score=1.), “Somewhat” (score=0.5), and “No” (score=0.). Accuracy was computed by averaging the scores.

As can be seen in Fig. 16, we observe a log-linear trend between concept frequency and zero-shot performance. Thus, we observe that the log-linear trend between concept frequency and zero-shot performance consistently holds even for T2I models.
Refer to caption
Figure 11: Log-linear relationships between concept frequency and T2I Max aesthetic scores. Across all tested models pretrained on the LAION-Aesthetics dataset, we observe a consistent linear relationship between T2I zero-shot performance on a concept and the log-scaled concept pretraining frequency.
Refer to caption
Figure 12: Log-linear relationships between concept frequency and T2I human aesthetic scores. Across all tested models pretrained on the LAION-Aesthetics dataset, we observe a consistent linear relationship between T2I zero-shot performance on a concept and the log-scaled concept pretraining frequency.
Refer to caption
Figure 13: Log-linear relationships between concept frequency and T2I human alignment scores. Across all tested models pretrained on the LAION-Aesthetics dataset, we observe a consistent linear relationship between T2I zero-shot performance on a concept and the log-scaled concept pretraining frequency.
Refer to caption
Figure 14: Log-linear relationships between concept frequency and T2I Avg. CLIP scores. Across all tested models pretrained on the LAION-Aesthetics dataset, we observe a consistent linear relationship between T2I zero-shot performance on a concept and the log-scaled concept pretraining frequency.
Refer to caption
Figure 15: Log-linear relationships between concept frequency and T2I Max CLIP scores. Across all tested models pretrained on the LAION-Aesthetics dataset, we observe a consistent linear relationship between T2I zero-shot performance on a concept and the log-scaled concept pretraining frequency.
Refer to caption
Figure 16: Log-linear relationship between concept frequency and T2I human evaluation for text-image alignment for people concepts. We observe a consistent linear relationship between T2I zero-shot performance on a concept and the log-scaled concept pretraining frequency.
Appendix D Concept Frequency is Predictive of Performance across Concepts only from Image and Text Domains

In all the main performance-frequency plots we have presented until now, the concept frequencies were estimated using the intersection of the image-frequencies and the text-frequencies. Here, we showcase results with using them independently in Figs. 18 and 17 respectively. We note that both independent searching methods showcase log-linear trends as before confirming our main result. We observe that the strong log-linear trend between concept frequency and zero-shot performance robustly holds across concepts derived from image and text domains independently as well.
Refer to caption
Figure 17: Log-linear relationships between image concept frequency and CLIP performance. Across all tested architectures (RN50, RN101, ViT-B-32, ViT-B-16, ViT-L-14) and pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M), we observe a consistent linear relationship between CLIP’s zero-shot accuracy and retrieval performance on a concept and the log-scaled concept pretraining frequency (searched using only pretraining images). ** indicates that the result is significant (p<0.05𝑝0.05p<0.05italic_p < 0.05 with a two-tailed t-test.), and thus we show pearson correlation (ρ𝜌\rhoitalic_ρ) as well.
Refer to caption
Figure 18: Log-linear relationships between text concept frequency and CLIP performance. Across all tested architectures (RN50, RN101, ViT-B-32, ViT-B-16, ViT-L-14) and pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M), we observe a consistent linear relationship between CLIP’s zero-shot accuracy and retrieval performance on a concept and the log-scaled concept pretraining frequency (searched using only pretraining text captions). ** indicates that the result is significant (p<0.05𝑝0.05p<0.05italic_p < 0.05 with a two-tailed t-test.), and thus we show pearson correlation (ρ𝜌\rhoitalic_ρ) as well.
Appendix E Experimental Details
E.1 Setup of Mayilvahanan et al. [75]

LAION-200M is a dataset obtained by deduplicating LAION-400M by pruning exact duplicates, near duplicates, and semantically similar samples within LAION-400M [10]. The control pretraining set is created by pruning 50 million highly similar samples from LAION in the order of decreasing perceptual similarity to datapoints in ImageNet-val set. We use the 150M pretraining set for obtaining the concept distribution. We evaluate the performance of a ViT-B/32 CLIP model trained on this dataset on our downstream tasks, and present our analysis on those tasks.
E.2 Let It Wag!: Test Set Curation

To ensure our datasets are thoroughly cleaned and diverse, we follow a meticulous process:

1. Diverse Sourcing: We gather images from three different online sources—Flickr, DuckDuckGo, and Bing Search—to maximize the variety of our dataset, while retaining very easy-to-classify images2.

2. Temporal Filtering: We applied a filter to only retrieve images after January 2023 to minimize overlap with images used in the pre-training of Vision-Language Models (VLMs). Note this helps mitigate but does not ensure the overlap problem is resolved.

3. Outlier Removal: We employ a pre-trained InceptionNet [107] to remove outliers from the entire image pool. We do this by taking all pairwise cosine-similarities between all images in the pool, and removing the images that are in the bottom 5% of the similarity values3.

4. Initial De-duplication with an InceptionNet: We employ a pre-trained InceptionNet [107] model to identify and remove duplicates. This step involves setting high thresholds for soft de-duplication (0.9 for common classes and 0.95 for fine-grained classes) to ensure only minor, precise exclusions. A threshold of 0.9/0.95 means that we consider images to be duplicates if the cosine similarity of that image’s embedding (from InceptionNet) with any other image’s embedding in the image pool is larger than 0.9/0.95.

5. Manual Verification: Following the automated cleaning, we manually inspect and verify the accuracy of the remaining images for each class to ensure they meet quality standards.

6. Second-level De-duplication with Perceptual Hashing: Post-verification, we use perceptual hashing [36] with a threshold of 10 bits to identify and remove duplicate images within each class, ensuring uniqueness across our dataset4.

7. Class Balancing: Finally, we balance the dataset to ensure an equal representation of classes.

This process was followed for increased quality and reliability of our dataset for image recognition tasks.
Appendix F Why and How Do We Use RAM++?

We detail why we use the RAM++ model [56] instead of CLIPScore [53] or open-vocabulary detection models [76]. Furthermore, we elaborate on how we selected the threshold hyperparameter used for identifying concepts in images.
F.1 Why RAM++ and not CLIP or open-vocabulary detectors?

We provide some qualitative examples to illustrate why we chose RAM++. Our input images do not often involve complex scenes suitable for object detectors, but many fine-grained classes on which alongside CLIP, even powerful open-world detectors like OWL-v2 [76] have poor performance.
Refer to caption
Figure 19: Qualitative Results comparing OWL-v2, RAM++ and CLIP. We show qualitative examples across three different models: OWL-v2, RAM++ and CLIP on fine-grained concepts.
F.2 How: Optimal RAM++ threshold for calculating concept frequencies

We ablate the choice of the threshold we use for assigning concepts to images using the RAM++ model. For the given set of concepts, RAM++ provides a probability value (by taking a sigmoid over raw logits) for each concept’s existence in a particular image. To tag an image as containing a particular concept, we have to set a threshold deciding this assignnment. We test over three thresholds: {0.5, 0.6, 0.7}, showcasing quantitative and qualitative results for all thresholds in Figs. 21 and 20.
We observe best frequency estimation results using the highest frequency of 0.7. This is due to the high precision afforded by this threshold, leading to us counting only the “most aligned images” per concept as hits. With lower thresholds (0.5, 0.6), we note that noisier images that do not align well with the concept can be counted as hits, leading to degraded precision and thereby poorer frequency estimation. Hence, we use 0.7 as the threshold for all our main results.
Refer to caption
Figure 20: Qualitative Results with different RAM++ thresholds. We show qualitative examples across three different thresholds: {0.5, 0.6, 0.7} for estimating concept frequency using the RAM++ model. We note that the significantly better concepts identified by the higher threshold (0.7) compared to the lower thresholds (0.5, 0.7). The images are sourced from the CC-3M dataset.
Refer to caption
Figure 21: Effect of different thresholds for determining concept frequency using RAM++. We test three different thresholds: {0.5, 0.6, 0.7} for estimating concept frequency using the RAM++ model. We note that the correlations are significantly stronger with a threshold of 0.7—this is justified by the higher precision of image sample hits at a higher threshold (0.7). Comparatively, lower thresholds (0.5, 0.7) lead to noisier images being counted as hits, hence reducing the hit precision for determining frequency. ** indicates that the result is significant (p<0.05𝑝0.05p<0.05italic_p < 0.05 with two-tailed t-test.), and thus we show pearson correlation (ρ𝜌\rhoitalic_ρ) too.
Appendix G Details about Misalignment Degree Results

In Tab. 4 in the main paper, we quantified the misalignment degree, and showcased that a large number of image-text pairs in all pretraining datasets are misaligned. In Alg. 1, we describe the method used for quantifying the misalignment degree for each pretraining dataset. We also showcase some qualitative examples of a few image-text pairs from the CC-3M dataset that are identified as misaligned using our analysis.
Data: Pretraining dataset 𝒟={(i1,t1),(i2,t2),…,(iN,tN)}𝒟subscript𝑖1subscript𝑡1subscript𝑖2subscript𝑡2…subscript𝑖𝑁subscript𝑡𝑁\mathcal{D}=\{(i_{1},t_{1}),(i_{2},t_{2}),\dots,(i_{N},t_{N})\}caligraphic_D = { ( italic_i start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_t start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) , ( italic_i start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , italic_t start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) , … , ( italic_i start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT , italic_t start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT ) }, Image Index Iimgsubscript𝐼imgI_{\text{img}}italic_I start_POSTSUBSCRIPT img end_POSTSUBSCRIPT, Text Index Itextsubscript𝐼textI_{\text{text}}italic_I start_POSTSUBSCRIPT text end_POSTSUBSCRIPT
Result: mis_degree
mis_degree←0←mis_degree0\textit{mis\_degree}\leftarrow 0mis_degree ← 0 for (i,t)∈𝒟𝑖𝑡𝒟(i,t)\in\mathcal{D}( italic_i , italic_t ) ∈ caligraphic_D do
       img_concepts←Iimg⁢[i]←img_conceptssubscript𝐼imgdelimited-[]𝑖\text{img\_concepts}\leftarrow I_{\text{img}}[i]img_concepts ← italic_I start_POSTSUBSCRIPT img end_POSTSUBSCRIPT [ italic_i ] // extract all concepts from this image
       text_concepts←Itext⁢[t]←text_conceptssubscript𝐼textdelimited-[]𝑡\text{text\_concepts}\leftarrow I_{\text{text}}[t]text_concepts ← italic_I start_POSTSUBSCRIPT text end_POSTSUBSCRIPT [ italic_t ] // extract all concepts from this text caption
       hits←set_intersection⁢(img_concepts,text_concepts)←hitsset_intersectionimg_conceptstext_concepts\text{hits}\leftarrow\texttt{set\_intersection}(\text{img\_concepts},\text{% text\_concepts})hits ← set_intersection ( img_concepts , text_concepts ) if 𝑙𝑒𝑛⁢(ℎ𝑖𝑡𝑠)=0𝑙𝑒𝑛ℎ𝑖𝑡𝑠0\text{len}(\text{hits})=0len ( hits ) = 0 then
             mis_degree←mis_degree+1←mis_degreemis_degree1\textit{mis\_degree}\leftarrow\textit{mis\_degree}+1mis_degree ← mis_degree + 1
       end if
      return m⁢i⁢s⁢_⁢d⁢e⁢g⁢r⁢e⁢e/N𝑚𝑖𝑠_𝑑𝑒𝑔𝑟𝑒𝑒𝑁{{mis\_degree}}{/}{N}italic_m italic_i italic_s _ italic_d italic_e italic_g italic_r italic_e italic_e / italic_N
end for
Algorithm 1 Extracting misalignment degree from pretraining datasets
Refer to caption
Figure 22: Qualitative examples of misaligned image-text pairs identified. We present 4 samples from the CC3M pretraining dataset that are identified as misaligned by our analysis. Here, the text captions clearly do not entail the images, and hence do not provide a meaningful signal for learning.
Appendix H T2I Models: Evaluation

We provide additional quantitative and qualitative results in this section for T2I models evaluated on the “Let It Wag!” dataset.
H.1 Quantitative Results by Retrieval

We provide further analysis on how state-of-the-art T2I models perform on the long-tailed concepts comprising the “Let It Wag!” dataset. As detailed in Sec. 6, we generate 4 images for each concept using Stable Diffusion XL [85], Stable Diffusion v2 [92] and Dreamlike Photoreal [1].

Prompting Strategy. The prompting strategy (system role) used, adapted from Shahmohammadi et al. [101], was:
Follow my commands: 1. I wish to generate text prompts about a given subject which I will use for image generation using off-the-shelf text-to-image models such as Stable Diffusion and DALL-E 3. 2. Assume all the subjects are nouns. 3. Follow a similar style and length of prompts as coco-captions. 4. Keep prompts concise and avoid creating prompts longer than 40 words. 5. Structure all prompts by setting a scene with at least one subject and a concrete action term, followed by a comma, and then describing the scene. For instance,“a view of a forest from a window in a cosy room, leaves are falling from the trees.” Generate detailed prompts for the concepts in the order in which they are given. Your output should be just the prompts, starting with “1.”

With this pool of generated images, we conduct a controlled experiment on the long-tailed concepts using nearest-neighbor retrieval as the evaluation metric by querying a generated image and retrieving the top-k results from a gallery of images taken from the “Let It Wag!” dataset. The overall pipeline is as follows:

Setup. We define the query and gallery set for head and tail concepts. For tail concepts, we sample the 25 concepts with the lowest frequency from the “Let It Wag!” dataset. For head concepts, we sample the 25 most frequent concepts for comparison. We use the same prompting strategy with the selected 25 concepts across all 3 T2I models. To create the gallery set, we randomly sample 100 images for each of these concepts. We use DINOv2 [81] ViT-S/14 as the feature extractor.

Results. In Table 5, we provide the Cumulative Matching Characteristic (CMC@k) results for all 3 T2I models used in our experiment. CMC@k was chosen as we are interested in measuring the delta between head and tail concepts for successful retrievals within the top-k retrieved real images for a given generated image. We observe a large performance gap between Head and Tail concepts, providing a quantitative evaluation of generation performance of T2I models.
Table 5: Generated-real retrieval scores. We compare retrieval results of DINOv2 ViT-S/14 when using generated images as query images. We report Δnormal-Δ\Deltaroman_Δ CMC@k results where k={1,2,5} between head and tail concepts.
Model 	ΔΔ\Deltaroman_ΔCMC
	k=1 	k=2 	k=5
Stable Diffusion XL 	13.0 	16.0 	16.8
Stable Diffusion v2 	11.0 	10.0 	10.4
Dreamlike Photoreal 	8.0 	9.0 	9.4
H.2 Qualitative Results

In Fig. 7 of the main text, we provide an initial insight into the qualitative performance of T2I models on “Let It Wag!” concepts. For ease of comprehension and comparison, we segregate concepts into 4 clusters: Aircraft (Fig. 23), Activity (Fig. 24), Animal (Fig. 25) and others (Fig. 26).

Results. Fig. 23 shows T2I models having difficulty in representing an aircraft in its full form in a majority of cases in addition to misrepresenting the specific model in the generated images. Fig. 24 showcases the difficulty T2I models face when representing actions or activities from prompts. Fig. 25 exemplifies the same inability of T2I models to accurately represent animal species. Finally, the remainder of the query set is shown in Fig. 26 and includes the inability to classify and subsequently generate certain species of flowers and objects.
Refer to caption
Figure 23: Qualitative results on the Aircraft cluster.
Refer to caption
Figure 24: Qualitative results on the Activity cluster.
Refer to caption
Figure 25: Qualitative results on the Animal cluster.
Refer to caption
Figure 26: Qualitative results for other selected failure cases.
Appendix I Classification Results: Let It Wag!

Here, we present the raw accuracy values of the 40 tested models on both Let It Wag! and ImageNet in Tab. 6. For reference, we also report the datasets these models were trained on and the number of parameters for each model. We see clear drops in performance compared to ImageNet, across model sizes, architectures and pretraining datasets.
Table 6: Full results dump on Let It Wag! and ImageNet.
Pretraining Dataset 	Model 	Num. Parameters (in millions) 	ImageNet Acc. 	Let It Wag! Acc.
CC-3M [103] 	RN50 	102.01 	20.09 	3.74
	ViT-B-16 	149.62 	17.10 	3.01
CC-12M [26] 	RN50 	102.01 	33.14 	8.92
	ViT-B-16 	149.62 	37.39 	11.49
YFCC-15M [109] 	RN50 	102.01 	31.88 	13.15
	RN101 	119.69 	34.04 	15.19
	ViT-B-16 	149.62 	37.88 	19.25
OpenAI-WIT [87] 	RN50 	102.01 	59.82 	31.93
	RN101 	119.69 	62.28 	31.88
	ViT-B-32 	151.28 	63.32 	33.52
	ViT-B-16 	149.62 	68.34 	37.85
	ViT-L-14 	427.62 	75.54 	45.31
WebLI [28] 	ViT-B-16 	203.79 	78.49 	54.63
	ViT-L-16 	652.15 	82.07 	61.50
	SO400M 	877.36 	83.44 	67.32
DataComp [44] 	ViT-B-32 	151.28 	69.18 	46.90
	ViT-B-16 	149.62 	73.48 	52.89
	ViT-L-14 	427.62 	79.21 	63.04
DataComp-DFN [42] 	ViT-B-16 	149.62 	76.24 	56.59
	ViT-H-14 	986.11 	83.44 	71.91
CommonPool [44] 	ViT-B-32 	151.28 	23.04 	7.73
	ViT-B-16 	149.62 	57.77 	20.97
	ViT-L-14 	427.62 	76.37 	46.96
LAION-400M [98] 	ViT-B-32 	151.28 	60.23 	32.88
	ViT-B-16 	149.62 	67.02 	39.13
	ViT-L-14 	427.62 	72.74 	46.59
LAION-2B [99] 	ViT-B-32 	151.28 	66.55 	41.79
	ViT-B-16 	149.62 	70.22 	44.21
	ViT-L-14 	427.62 	75.25 	51.03
	ViT-H-14 	986.11 	77.92 	58.98
	ViT-g-14 	1366.68 	78.46 	59.01
	ViT-bigG-14 	2539.57 	80.09 	63.54
MetaCLIP-400M [121] 	ViT-B-32 	151.28 	65.58 	40.50
	ViT-B-16 	149.62 	70.80 	46.50
	ViT-L-14 	427.62 	76.20 	52.78
MetaCLIP-FullCC [121] 	ViT-B-32 	151.28 	67.66 	43.84
	ViT-B-16 	149.62 	72.12 	49.32
	ViT-L-14 	427.62 	79.17 	57.48
	ViT-H-14 	986.11 	80.51 	62.59
SynthCI-30M [48] 	ViT-B-16 	149.62 	30.67 	9.15


—END OF WORK—

ELI5 this work at the level of a high school student, focusing only on the text-to-image results.