SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CLIP score and LAION filtering. We experiment with CLIP score filtering (also employed by LAION), where we take only examples having cosine similarity scores between CLIP image and text embeddings that exceed a pre-defined threshold. We investigate a range of thresholds and two OpenAI CLIP models for computing the scores: the ViT-B/32 model (as in LAION) and the larger ViT-L/14. We also combine CLIP score thresholds and cld3 English filtering to reproduce the LAION-2B filtering scheme. Table 16 in Appendix P.1 summarizes the different CLIP score configurations.
Text-based filtering. We select examples that contain text overlapping with ImageNet class names, which serve as a proxy for relevance to downstream tasks. Specifically, we select English captions (according to fasttext) that contain words from ImageNet-21K or ImageNet-1K [37] class synsets.
1More precisely, given a batch of data  $\{(x_1,y_1),\dots,(x_B,y_B)\}$  with images  $x$  and captions  $y$ , we train the image encoder  $g$  and text encoder  $v$  with the loss  $\ell = \frac{1}{2}\sum_{i = 1}^{B}\frac{\sigma_{ii}}{\sum_{j = 1}^{B}\sigma_{ij}} +\frac{1}{2}\sum_{i = 1}^{B}\frac{\sigma_{ii}}{\sum_{j = 1}^{B}\sigma_{ji}}$ , where  $\sigma_{ij} = \exp \langle g(x_i),h(y_j)\rangle$ . We also use a learnable temperature parameter as in Radford et al. [111].
Table 3: Zero-shot performance for select baselines in the filtering track. On all scales, filtering strategies lead to better performance than using the entire, unfiltered pool. The intersection between imaged-based and CLIP score strategies performs well on most tasks and scales. For all metrics, higher is better (see Appendix O for details).  $\cap$  denotes the intersection of filtering strategies.
Scale	Filtering strategy	Dataset size	Samples seen	ImageNet	ImageNet dist. shifts	VTAB	Retrieval	Average over 38 datasets
small	No filtering	12.8M	12.8M	0.025	0.033	0.145	0.114	0.132
Basic filtering	3M	12.8M	0.038	0.043	0.150	0.118	0.142
Text-based	3.2M	12.8M	0.046	0.052	0.169	0.125	0.157
Image-based	3M	12.8M	0.043	0.047	0.178	0.121	0.159
LAION-2B filtering	1.3M	12.8M	0.031	0.040	0.136	0.092	0.133
CLIP score (L/14 30%)	3.8M	12.8M	0.051	0.055	0.190	0.119	0.173
Image-based ∩ CLIP score (L/14 30%)	1.4M	12.8M	0.039	0.045	0.162	0.094	0.144
medium	No filtering	128M	128M	0.176	0.152	0.259	0.219	0.258
Basic filtering	30M	128M	0.226	0.193	0.284	0.251	0.285
Text-based	31M	128M	0.255	0.215	0.328	0.249	0.307
Image-based	29M	128M	0.268	0.213	0.319	0.256	0.312
LAION-2B filtering	13M	128M	0.230	0.198	0.307	0.233	0.292
CLIP score (L/14 30%)	38M	128M	0.273	0.230	0.338	0.251	0.328
Image-based ∩ CLIP score (L/14 30%)	14M	128M	0.297	0.239	0.346	0.231	0.328
large	No filtering	1.28B	1.28B	0.459	0.378	0.426	0.419	0.437
Basic filtering	298M	1.28B	0.516	0.423	0.446	0.480	0.458
Text-based	317M	1.28B	0.561	0.465	0.465	0.352	0.466
Image-based	293M	1.28B	0.572	0.454	0.483	0.479	0.476
LAION-2B filtering	130M	1.28B	0.553	0.453	0.510	0.495	0.501
CLIP score (L/14 30%)	384M	1.28B	0.578	0.474	0.538	0.466	0.529
Image-based ∩ CLIP score (L/14 30%)	140M	1.28B	0.631	0.508	0.546	0.498	0.537
xlarge	No filtering	12.8B	12.8B	0.723	0.612	0.611	0.569	0.621
LAION-2B filtering	1.3B	12.8B	0.755	0.637	0.624	0.620	0.636
CLIP score (L/14 30%)	3.8B	12.8B	0.764	0.655	0.643	0.588	0.650
Image-based ∩ CLIP score (L/14 30%)	1.4B	12.8B	0.792	0.679	0.652	0.608	0.663
Image-based filtering. We select a subset of examples whose visual content overlaps with ImageNet classes. After applying English language (fasttext) and caption length filtering, we cluster the image embeddings extracted by the OpenAI ViT-L/14 model for each image into 100K groups using Faiss [75]. We then find the nearest neighbor group for every ImageNet training example, and keep examples belonging to these groups. We apply this procedure using either ImageNet-21K (14M images) or ImageNet-1K (1.2M images), forming two subsets.
4.2 BYOD baselines
We experiment with multiple external data sources, including four moderately sized datasets (10 to 58M samples) studied by Nguyen et al. [101]—CC12M [24], YFCC15M [140, 111], RedCaps [38] and Shutterstock [101]—and the larger LAION-2B [129]. Additional experiments, along with more details about the data sources are provided in Appendix P.2. We consider these data sources as they are and do not perform additional preprocessing. We also present experiments combining some of the data sources (using only the external datasets, or in addition to data from our pool).
5 Results and discussion
5.1 Building better datasets
Main results. Our key results are in Table 3. Most notably, the intersection between image-based filtering and CLIP score filtering excels on most tasks. The exception is at the small scale and for retrieval datasets. Furthermore, other filtering strategies like basic, CLIP score, image-based, text-based filtering show better downstream performance when compared to no filtering. A much larger suite of experiment results can be found in Appendix R.
DATACOMP leads to better image-text datasets. We hope DATACOMP catalyzes the search for the next generation of multimodal datasets. We contribute DATACOMP-1B, which is the output of the Image-based  $\cap$  CLIP score (L/14  $30\%$  ) baseline filter at the xlarge scale of the filtering track.
${}^{2}$  Cherti et al. [28] also observe that models rank differently on classification and retrieval tasks.
Figure 2: Performance of random subsets (dotted line) and CLIP score filtering (solid line) when varying the subset size. When taking random subsets, larger subsets are always better. For CLIP score filtering, subsets with intermediate size perform best.
Our dataset is comprised of 1.4B samples, which not only is smaller than the LAION-2B dataset with 2.3B samples, but also comes from a smaller pool. Nevertheless, a CLIP L/14 trained on DATACOMP-1B outperforms the LAION-2B competitor by 6.1 percentage points on ImageNet (see Table 1). Moreover, training on DATACOMP-1B improves ImageNet accuracy by 3.7 percentage points over OpenAI's ViT-L/14 trained with the same compute budget. Additionally, even if we restrict ourselves to 400M samples, we can still find a subset of DATACOMP-1B that outperforms OpenAI's ViT-L/14, as seen in Table 24. These results demonstrate the impact that DATACOMP can make and provide a foundation upon which participants can build.
External data sources can improve performance. Appendix P.2 Table 18 shows results for several baselines in the BYOD track. We find several instances where adding external data sources improves performance over using just data from COMMONPOOL. For example, at the large scale, combining CLIP-filtered data from COMMONPOOL with external data from CC12M [24], YFCC15M [140, 111], RedCaps [38] and Shutterstock [101] boosts ImageNet accuracy by 4.3 percentage points. See Appendix P.2 for more experiments and details.
Trade-off between data diversity and repetition. In Figure 2, we see that randomly selecting subsets of the pool has little effect and degrades performance substantially when only small fractions are used. When filtering with CLIP scores, the optimal training set comes from selecting  $\sim 30\%$  of the pool with the highest scores. The difference in performance trends between random subsets and CLIP score filtering highlights the importance of filtering strategies for selecting samples.
5.2 DATACOMP design analyses
COMMONPOOL and LAION are comparable with the same filtering. To validate our pool construction, we show that we can build datasets comparable to LAION-2B by employing their filtering technique on our pool. LAION-2B selects all samples where the caption is in English and the cosine similarity score from a trained ViT-B/32 CLIP model is above 0.28. We compare this filtering approach on our pool using the same number of samples, 130M samples at the large scale. We find that the different data sources perform comparably:  $55.3\%$  vs  $55.7\%$  accuracy on ImageNet, and 0.501 vs 0.489 average performance over our evaluation sets using our pool and LAION-2B, respectively.
Consistency across scales. We find that the ranking between filtering strategies is typically consistent across different scales. This is illustrated in Figure 3, which shows that the baselines at small and medium scales are positively correlated. Moreover, as shown in Appendix Table 22, the rank correlations of performance is high, between 0.71 and 0.90 for different scale pairs.
Consistency across training changes. DATACOMP fixes the training procedure, so a natural question is whether better datasets from DATACOMP are better outside of DATACOMP. While DATACOMP-1B is trained at the x1arge scale, we show in Appendix Table 23 that even when substituting the ViT-L/14
Figure 3: Correlation between small and medium scale baselines. Smaller scales can serve as useful guides for larger scales. Results for additional scales are shown in Appendix Figure 22.
for a ViT-B/16 or ViT-B/32, training on DATACOMP-1B outperforms training on OpenAI's WIT and LAION-2B. Additionally, we found that modifying hyperparameters such as training steps and batch size minimally affects the relative ordering of different data curation methods on downstream performance. Details on hyperparameter ablations are in Appendix L.
5.3 Evaluation trends
ImageNet accuracy is indicative, but not the complete picture. Similarly to Kornblith et al. [84], in Appendix Figure 25 we find that ImageNet performance is highly correlated with the average performance across all datasets we study, with an overall correlation of 0.99. However, ImageNet performance is not representative of all evaluation tasks, as the correlation between ImageNet accuracy and accuracy on other individual datasets varies substantially, in some cases even exhibiting a negative correlation, as discussed in Appendix R.
Robustness and fairness. While typical models trained on a target task suffer large performance drops under data distribution shift, zero-shot CLIP models are known to exhibit strong performance across many distributions [111]. In Appendix Figure 26, we show that CLIP models trained with data from our pool are more robust to distribution shift than ImageNet-trained models from Taori et al. [139]'s testbed. Examining geographic diversity, we find that our models are better than ImageNet-trained models, but fall short of models fine-tuned on diverse curated datasets (see Appendix Figure 21). We also perform a face classification analysis and identify demographic biases in our models: notably, the BYOD datasets we consider can increase the risk of misclassification. See Appendix Q for more fairness and diversity analyses.
6 Limitations and conclusion
In terms of societal risks, creating an index of image-text pairs from the public internet can be problematic. The internet contains unsafe, toxic, and sensitive content, which ideally should not percolate into machine learning datasets. Though we take steps to remove NSFW content and blur human faces to protect privacy, we hope future work will further explore the biases and risks from COMMONPOOL and DATACOMP-1B. We see several additional directions for future work, including 1) Curating more data sources. 2) Improved data filtering algorithms. 3) Further supervision signals (e.g., image captions coming from captioning models). 4) Additional input modalities (e.g., video, 3D objects). 5) Broader evaluations for vision-and-language and robotics tasks.
Overall, we see DATACOMP as a first step towards improving training datasets, and hope our new benchmark will foster further research. By providing a controlled experimental setting, DATACOMP enables researchers to iterate on dataset design on rigorous empirical foundations. We open-source all of our code, data, and infrastructure, and hope these resources will help the community build the next generation of multimodal datasets.
Note that unlike Kornblith et al. [84] we evaluate zero-shot performance rather than transfer learning.
SYG and JH are supported by NSF Graduate Research Fellowships. GS is supported by the Onassis Foundation - Scholarship ID: F ZS 056-1/2022-2023. GD has been supported by the Onassis Fellowship (Scholarship ID: F ZS 012-1/2022-2023), the Bodossaki Fellowship and the Leventis Fellowship. This research has been supported by NSF Grants AF 1901292, CNS 2148141, DMS 2134012, TRIPODS II-DMS 2023166, Tripods CCF 1934932, IFML CCF 2019844 and research gifts by Western Digital, WNCG IAP, UT Austin Machine Learning Lab (MLL), Cisco, the Len Blavatnik and the Blavatnik Family Foundation, the Stanly P. Finch Centennial Professorship in Engineering, Open Philanthropy, Google, Microsoft, and the Allen Institute for AI.
We would like to thank Amro Abbas, Danny Bickson, Alper Canberk, Jessie Chapman, Brian Cheung, Tim Dettmers, Joshua Gardner, Nancy Garland, Sachin Goyal, Huy Ha, Zaid Harchaoui, Ari Holtzman, Andrew Hundt, Andy Jones, Adam Klivans, Ronak Mehta, Sachit Menon, Ari Morcos, Raviteja Mullapudi, Jonathon Shlens, Brandon McKinzie, Alexander Toshev, David Grangier, Navdeep Jaitly, Kentrell Owens, Marco Tulio Ribeiro, Shiori Sagawa, Christoph Schuhmann, Matthew Wallingford, and Ross Wightman for helpful feedback at various stages of the project. We are particularly grateful to Daniel Levy and Alec Radford for early encouragement to pursue this project and feedback on the experimental design.
We thank Stability AI and the Gauss Centre for Supercomputing e.V.⁴ for providing us with compute resources to train models. We are thankful for the compute time provided through the John von Neumann Institute for Computing (NIC) on the GCS Supercomputer JUWELS Booster [78] at Jülich Supercomputing Centre (JSC), and for storage resources on JUST [50] granted and operated by JSC, as well as computing and storage resources from the Helmholtz Data Federation (HDF).
<sup>4</sup>https://gauss-centre.eu
[157] Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, and Alexander Ratner. WRENCH: A comprehensive benchmark for weak supervision. In NeurIPS, 2021. URL https://openreview.net/forum?id=Q9SKS5k8io.
[158] Jieyu Zhang, Cheng-Yu Hsieh, Yue Yu, Chao Zhang, and Alexander Ratner. A survey on programmatic weak supervision, 2022. https://arxiv.org/abs/2202.05433.
[159] Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/regression by conditional adversarial autoencoder. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017. https://arxiv.org/abs/1702.08423.
[160] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krahenbuhl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In European Conference on Computer Vision (ECCV), 2022. https://arxiv.org/abs/2201.02605.