DataComp: In search of the next generation of multimodal datasets
Authors: Samir Yitzhak Gadre
Gabriel Ilharco
Alex Fang
Jonathan Hayase
Georgios Smyrnis
Thao Nguyen
Ryan Marten
Mitchell Wortsman
Dhruba Ghosh
Jieyu Zhang
Eyal Orgad
Rahim Entezari
Giannis Daras
Sarah M Pratt
Vivek Ramanujan
Yonatan Bitton
Kalyani Marathe
Stephen Mussmann
Richard Vencu
Mehdi Cherti
Ranjay Krishna
Pang Wei Koh
Olga Saukh
Alexander Ratner
Shuran Song
Hannaneh Hajishirzi
Ali Farhadi
Romain Beaumont
Sewoong Oh
Alex Dimakis
Jenia Jitsev
Yair Carmon
Vaishaal Shankar
Ludwig Schmidt
Venue: NeurIPS 2023 Datasets and Benchmarks Oral (2023)
Keywords: CLIP
zero-shot
data curation
vision-and-language
datasets
pre-training
benchmark
TL;DR: A image-text dataset benchmark where model training is fixed and participants iterate on data curation strategies.

Abstract: Multimodal datasets are a critical component in recent breakthroughs such as CLIP, Stable Diffusion and GPT-4, yet their design does not receive the same research attention as model architectures or training algorithms. To address this shortcoming in the machine learning ecosystem, we introduce DataComp, a testbed for dataset experiments centered around a new candidate pool of 12.8 billion image-text pairs from Common Crawl. Participants in our benchmark design new filtering techniques or curate new data sources and then evaluate their new dataset by running our standardized CLIP training code and testing the resulting model on 38 downstream test sets. Our benchmark consists of multiple compute scales spanning four orders of magnitude, which enables the study of scaling trends and makes the benchmark accessible to researchers with varying resources. Our baseline experiments show that the DataComp workflow leads to better training sets. Our best baseline, DataComp-1B, enables training a CLIP ViT-L/14 from scratch to 79.2% zero-shot accuracy on ImageNet, outperforming OpenAI's CLIP ViT-L/14 by 3.7 percentage points while using the same training procedure and compute. We release \datanet and all accompanying code at www.datacomp.ai.

In search of the next generation of multimodal datasets
Samir Yitzhak Gadre Gabriel Ilharco*1, Alex Fang"!, Jonathan Hayase',
Georgios Smyrnis", Thao Nguyen', Ryan Marten 7.9, Mitchell Wortsman', Dhruba Ghosh Jieyu Zhang', Eyal Orgad Rahim Entezari"o, Giannis Daras
Sarah Pratt Vivek Ramanujan', Yonatan Bitton Kalyani Marathe Stephen Mussmann', Richard Vencu", Mehdi Cherti Ranjay Krishna', Pang Wei 1.1 Olga Saukh Alexander Ratner Shuran Song?, Hannanch Hajishirzi Ali Farhadi', Romain Beaumont Sewoong Oh Alex Dimakis Jenia Jitsey 6,8
Yair Carmon', Vaishaal Shankar*, Ludwig Schmidt
Abstract
Multimodal datasets are a critical component in recent breakthroughs such as CLIP, Stable Diffusion and GPT-4, yet their design does not receive the same research attention as model architectures or training algorithms. To address this shortcoming in the machine learning ecosystem, we introduce DATACOMP, a testbed for dataset experiments centered around new candidate pool of 12.8 billion image-text pairs from Common Crawl. Participants in our benchmark design new filtering techniques or curate new data sources and then evaluate their new dataset by running our standardized CLIP training code and testing the resulting model on 38 downstream test sets. Our benchmark consists of multiple compute scales spanning four orders of magnitude, which enables the study of scaling trends and makes the benchmark accessible to researchers with varying resources. Our baseline experiments show that the DATACOMP workflow leads to better training sets. Our best baseline, DATACOMP- IB, enables training a CLIP ViT-L/14 from scratch to 79.2% zero-shot accuracy ImageNet, outperforming OpenAl's CLIP ViT-L/14 by 3.7 percentage points while using the same training procedure and compute. We release DATACOMP and all accompanying code www. datacomp. ai.
Introduction
Recent advances in multimodal learning such CLIP [111], DALL-E [115, 116), Stable Diffusion [123]. Flamingo [8], and GPT-4 [103] offer unprecedented generalization capabilities in zero-shot classification, image generation, and in-context learning. While these advances use different algorithmic techniques. e.g., contrastive learning, diffusion, or auto-regressive modeling, they all on common foundation: large datasets containing paired image-text examples. For instance, CLIP's training set contains 400 million image-text pairs, and Stable Diffusion was trained on the two billion examples from LAION-20 [129]. This new generation of image-text datasets is 1,000 times larger than previous datasets such as ImageNet, which contains 1 1.2M images [37, 126].
Despite the central role of image-text datasets, little is known about them. Many state-of-the-ar datasets are proprietary, and even for public datasets such as LAION-2B [129], it is unclear how design choices such the data source or filtering techniques affect the resulting models. While there are thousands of ablation studies for algorithmic design choices (loss function, model architecture, etc.), datasets often treated as monolithic artifacts without detailed investigation. Moreover,
Equal contribution. randomly ordered. Correspondence to contact@datacomp ai. Unniversity of Washington *Columbia University "Tel Aviv University "Apple 'UT Austin "LAION 7A12 *Juelich Supercomputing Center, Research Center Juclich University of Illinois Urbana- Shaana-Caampain Graz University of Technology Hebrew University GGoogl Research Snorkel Al
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Table I: Zero-shot performance of CLIP models trained on different datasets. DATACOMP-IB. assembled with simple filtering procedure on image-text pairs from Common Crawl, leads to model with higher accuracy than previous results while using the same number of muutippy-accumuuan
operations (MACs) or less during training. See Section 3.5 for details on the evaluation datasets.
# samples
Train compute
ImageNet
Dataset
Dataset size
Architecture
seen
(MACs)
accuracy
OpenAI's WIT [111]
0.4B
13B
ViT-L/14
1.1 x 1021
75.5
LAION-400M [128, 28]
0.4B
13B
ViT-L/14
1.1 x 10
72.8
LAION-2B [129, 28]
2.3B
13B
VIT-L/14
1.1 x 1021
73.1
LAION-2B [129, 28]
2.3B
34B
ViT-H/14
6.5 x 10
78.0
LAION-2B [129, 28]
2.3B
34B
ViT-g/14
9.9 x 10a1
78.5
DATACOMP-IB (ours)
1.4B
13B
ViT-L/14
1.1 x 1021
79.2
datasets currently lack the benchmark-driven development process that has enabled a steady stream of improvements on the model side and isolates data enhancements from changes to the model. These issues impede further progress in multimodal learning, as evidenced by recent work showing that public datasets currently do not match the scaling behavior of proprietary alternatives [28].
In this paper, we take a step towards a more rigorous dataset development process. Our first and central contribution is DATACOMP, a new benchmark for multimodal dataset design. DATACOMP flips the traditional benchmarking paradigm in machine learning where the dataset is fixed and researchers propose new training algorithms. Instead, we hold the entire training code and computational budget constant so that participants innovate by proposing new training sets. To evaluate the quality of training set, we score the resulting model with a testbed of 38 classification and retrieval tasks such as ImageNet [37], V2 [121], DTD [30], EuroSAT [63]. SUN-397 [146], and MSCOCO [26].
DATACOMP focuses on two key challenges that arise when assembling large training datasets: what data sources to train on, and how to filter a given data source. Each challenge corresponds to one track in our benchmark. To facilitate the filtering track, our second contribution is COMMONPOOL, a dataset of 12.8B image-text pairs collected from Common Crawl and currently the largest public image-text dataset. We release CommonPool as an index of image url-text pairs under CC-BY-4.0 license, and apply content checks in its construction to remove unsafe or unwanted content. In the filtering track, the goal of participants to find the best subset of COMMONPOOL train on, In the second track, Bring Your Own Data (B YOD). participants may leverage any data source, as long as it does not overlap with our evaluation testbed.
Our third contribution is an investigation of scaling trends for dataset design. In particular, DATACOMP contains four scales, where we vary the training budget and the candidate pool size from 12.8M to 12.8B samples (see Table 2). Expressed in GPU hours, the cost of single training run ranges from 4 to 40,000 GPU hours the A100 cluster we used for development. The different scales enable researchers with different resources to participate in our benchmark. Moreover, our results show that the ranking of filtering approaches is largely consistent across scale.
Our fourth contribution is over three hundred baseline experiments, including techniques such as querying captions for relevant keywords, filtering based image embeddings, and applying a threshold on CLIP scores. A key result from our baselines experiments is that smaller, more stringently filtered datasets can lead to models that generalize better than larger datasets coming from the same pool. At the 12.8B scale, our best filtering baseline increases Image.ett zero-shot accuracy by 6.9 percentage points (pp) relative the unfiltered pool (see Table 3). For the B BOD track, our initial experiments show that 109M additional data points (less than 1 1% of the 12.8B pool) improve the CLIP-filtered subsets of COMMONPOOL by up to 1.2 pp ImageNet accuracy (see Table 18).
Finally, our fifth contribution is DATACOMP-IB, a new state-of-the-ar multimodal dataset. We obtain DATACOME IB by combining our two most promising filtering baselines. DATACOMP-IB enables training a CLIP VIT-L/14 model to an ImageNet zero-shot accuracy of 79.2% (see Table 1), corresponding 9x computational cost reduction when compared to larger CLIP ViT-g/14 model trained on LAION-2B for about 3x longer. Moreover, our model outperforms OpenAl's original CLIP V by 3.7 percentage points, while using the same compute budget.
To make DATACOMP a shared environment for controlled dataset experiments, we publicly release our candidate pool url index, tooling for assembling these pools, our filtering baselines, and our

---

Choose scale
Select data
Train
Evaluate
Filtering track
CommonFool
Cand date dataset
Train CLIP model with fixed architecture and hyper-parameters
Evaluate the model on downstream tasks
small, medium, large or xlarge
External data sources
Bring your own data track
Figure 1: DATACOMP participant workflow, A) Choose scale based on resource constraints. B) Design dataset, in either the filtering or BYOD track. C) Train CLIP model on the designed dataset using fixed architecture and hyperparameters (Section 3.4). D) Evaluate the trained model on a suite of diverse downstream tasks (Section 3.5).
code for training and evaluating models at www www. datacomp ai. We believe that our infrastructure will help put research dataset design rigorous empirical foundations, draw attention to this understudied research area, and lead to the next generation of multimodal datasets.
2 Related Work
We review the most closely related work and include additional related work in Appendix C.
The effects of data curation. Classical work considers dataset cleaning and outlier removal [74, 152, 124, 125] to discard samples that may lead to undesirable model bias. A related line of work develops coreset selection algorithms [61, 7, 46, 11, 94, 145, 32], which aim to select data subsets that lead to the same performance as training on the entire dataset. These techniques appear to scale poorly to larger data regimes [51, 6]. More recent efforts in subset selection often operate on already curated datasets [98, 141, 130, 16, 33, 106] (e.g.. CIFAR-10, ImageNet) or on smaller data regimes (e.g., YFCC-15M [111, 140]). These settings often do not reflect newer training paradigms that involve (1) noisy image-text pairs instead of category labeled images and (2) large scale datasets (c.g.. billions of samples). While data-centric investigations have led to community competitions like DCBENCH [43] and DATAPERF [97]. existing benchmarks have likewise operated at small data scales [100] compared datasets like LAION-2B [129], which contains over two billion images. DATACOMP bridges this gap by aligning data-centric investigation with large scale image-text training.
There has also been renewed interest in dataset pruning and deduplication. Sorscher al. [135] show that data pruning can improve traditional scaling trends on ImageNet, but do not consider image-text training or larger datasets. Raffel et al. [113] remove sentence redundancies when creating the C4 corpus. Subsequent work further demonstrated the benefits of deduplication for better language modeling [90]. Radenovic et al. [1 [1] introduce CAT filtering for image-text datasets rule-based system to retain high quality samples. Abbas et al. [6] propose SemDeDup, which starts with the CAT-filtered LAION-440M subset, further employing clustering to remove semantic duplicates. DATACOMP facilitates data-centric investigation at an even larger scale (i.e., 12.8B sample scale) and provides a common experimental setting for fair comparison amongst dataset creation algorithms.
Large-scale multimodal datasets. Datasets have been instrumental to building multimodal models like CLIP [111], Flamingo [8], Stable Diffusion [123], DALL-E [115, 116] and GPT-4 [103]. These methods succeeded by training large, heterogencous datasets rather than solely through advanced modelling techniques. For example, OpenAl's CLIP trains on 400M image-text pairs from the web, roughly 300x the size of ImageNet [37]. Prior work on scaling image-text datasets also provides promising trends with respect to zero-shot model performance [73, 107]. Additional large scale datasets like FILIP-300M [149], FLD-900M [153], and PaLI-10B [25] were constructed to train multimodal models. However, many datasets used to train such models (including the dataset for OpenAl's CLIP) proprietary, making hard to conduct data-centric investigations.
Even for public image-text datasets like SBU [104], Flickr30k [151], MS-COCO [26], TaiSu [92], Conceptual Captions [131], CC12M [24], RedCaps [38], WIT [136], Shutterstock [101], YFCC-
3

---

Table 2: Experimental configurations, with compute in multiply-aceumullte operations (MACs).
Scale
Model
Train compute (MACs)
Pool size and # samples seen
small
VIT-B/32
9.5 x 10th
12.8M
medium
ViT-B/32
9.5 x 10
128M
large
ViT-B/16
2.6 x 1049
1.28B
xlarge
VIT-L/14
1.1 x 1021
12.8B
100M [140], COYO-700M [20], LAION-400M [128], or LAION-2B [129] little is known about what constitutes a good image-text dataset. Preliminary analysis suggests that different image-text data sources lead CLIP models with different properties [101]. However, previous work is limited to smaller scale data (10-15M examples). Birhane et al. [15] examine LAION-400M and find NSFW imagery and racial slurs, centering the dangers web-scale multimodal datasets. To combat toxicity, we preprocess our pool to remove NSFW content and blur human faces detected in images. For more details on our safety preprocessing see Section 3.2, Appendices E and G.
3 The DATACOME benchmark
DATACOMP is meant to facilitate data-centric experimentation. While traditional benchmarks emphasize model design, DATACOMP is centered around dataset development, where the resulting datasets can be used to train high accuracy models. We focus large image-text datasets and quantify dataset submission by training a CLIP model on it from scratch [111] and evaluating on 38 downstream image classification and retrieval tasks. We additionally have three secret test sets, which will be released after year, to guard against overfitting. To facilitate such investigations, we provide candidate pool of uncurated image-text pairs sourced from the public internet. Our benchmark offers two tracks: one where participants must filter samples from the pools we provide, and another where participants can use external data. Moreover. DATACOMP is structured to accommodate participants with diverse levels of computational resources: each track is broken down into four scales with varying compute requirements. We now discuss high-level design decisions, construction of a 12.8B image-text data pool to facilitate the competition, benchmark tracks, model training, and evaluation.
3.1 Competition design
Overview. In many areas of machine learning, larger datasets lead to better performing models [87, 79, 73, 107, 66, 28, 19, 111, 1 112]. Hence comparing only datasets with the same size is natural starting point. However, this approach is flawed as controlling the dataset size ignores critical curation constraints:
candidate pool size (i.e., number of image-text pairs to harvest) and training compute. For instance, assembling a dataset like LAION-2B consists of identifying data sources (e.g., Common Crawl or Reddit) and filtering the data source. Notably, the final dataset size is design choice and is only upper-bounded by the data sources. Hence, the true data constraint is the size of the reservoir of samples: candidate pool to be filtered. To make DATACOMP realistic benchmark, we therefore fix the candidate pool in the filtering track, but give participants control over the training set size.
Compute cost is another relevant constraint. To put datasets of different size on equal footing, we specify the total number of training samples seen. Consider the 12.8B compute scale and filtered datasets A and B, with 6.4B and 3.2B image-text pairs respectively. At this scale, we train by making two passes over A, while making four passes over B. key result from our experiments is that smaller, more stringently filtered datasets can lead to models that generalize better.
Competition tracks. Two key procedures assembling training dataset are filtering data source [128, 129, 20] and aggregating data sources [36, 37J. To reflect this structure, DATACOMP has two tracks: filtering, where participants select a subset of the samples from COMMONPOOL. and Bring Your Own Data (BOO)), where participants use any source of data. Key decisions for each tracks are described Sections 3.2 and 3.3, respectively. For full competition track rules see Appendix A.
Competition compute scales. To facilitate study of scaling trends and accommodate participants with various computational resources, we structure DATACOMP using four scales of compute: small, medium, large and xlarge. Each new scale increases the number of samples seen during training by

---

10x (from 12.8M to 12.8B samples seen). and the pool we provide by the same factor (from 12.8M samples 12.8B samples). Table gives the experimental configuration used for each scale. For the small scale, our runs took 4 hours on an A A10 GPU, and for the xlarge scale 81 hours on 512 GPUs.
3.2 COMMONPOOL generation, for the filtering track
We construct large-scale pool of image-text pairs, COMMONPOOL, from Common Crawl [3]. CommonPool is distributed as an image url-text pair index under a CC-BY-4.0 license. Our pool construction pipeline has four steps: url extraction and data download, NSFW detection, evaluation set deduplication, and face blurring. We additionally provide per sample metadata (e.g.. CLIP features). Starting from the xlarge COMMONPOOL, we take successive random subsets create large, medium, and small COMMONPOOL (e.g., medium is a subset of large).
Extracting urls and dowloading data. We first cc2dataset [1], which utilizes Apache Spark [155], to extract pairs of image urls and nonempty alt-text from all Common Crawl snapshots from 2014 to 2022. We then deduplicate the url-text pairs and randomly shuffle. This step results in ~88B possible samples. Not all samples are downloadable; other samples are not suitable due to NSFW content or overlap with our evaluation sets. We attempt download 440B samples using img2dataset [5] resulting in 166.8 image-text pairs. For more details, see Appendix D.
Safety preprocessing. Since Common Crawl is snapshot of the internet, require strict preprocessing to remove unsafe content. We use Detoxify [60] to prune samples that contain unsafe text (e.g., obscene, sexually explicit, or threatening language), We also discard samples with explicit visual content. To do so, we train a classifier on CLIP ViT-L/14 [111] features, using the NSFW dataset used in LAION-SB [129]. We validate our classifier against the Google commercial image safety API. See Appendix E for details. Around 19% image-text pairs are considered NSFW, taking the pool of ~16.8B downloads to 11366 samples.
Evaluation set deduplication. To prevent accidental overfitting to certain test sets in our evaluation suite, we perform thorough hear-duplicate removal between the candidate pool and our evaluation sets, using a sfate-of-the-rt image deduplication model [150]. Appendix F contains additional details. The model flags ~3% of the 16.8B images near-duplicates, reducing the ~13.6 pool to 133.B samples. From here we select a random subset to get the xlarge pool of 12.8B samples.
Face detection & blurring. To protect the privacy of individuals, we detect and blur faces from images in our pool using face detector [53]. As observed by Yang et al. [148], obfuscating faces has little impact on model performance, as we also observe in our experiments (Appendix G).
Pool metadata. To bootstrap participants we distribute metadata for each sample in COMMONPOOL (e.g., image url, alt-text, original image resolution, CLIP features, and CLIP similarity scores). Following Carlini al. [22], release SHA256 hashes for each image to guard against data poisoning in subsequent COMMONPOOL downloads. For additional details see Appendix H. We open-source our metadata processing pipeline dataset2metadtta [4].
3.3 The bring your own data (BYOD) track
While COMMONPOOL can used to study different filtering techniques, state-of-the-art models often train on data from different sources. For instance, the Flamingo model [8] uses both multimodal massive web (M3W) and ALIGN datasets [73]. facilitate non-proprietary research on curating data from many sources,
instantiate separate DATACOMP track to allow participants combine multiple data streams. For example, participants could construct a training set from CC12M [24], YFCC100M [140], and data sources they label themselves. In Section 4.2 and Appendix P.2 describe our exploration using existing public, image-text datasets. These datasets are acquired from their respective sources and not re-release as part of DATACOMP.
3.4 Training
We create a common experimental setting that enables comparable experiments by fixing the training procedure. We closely follow the CLIP training recipe proposed by Radford et al. [111]: training
5

---

models from scratch with contrastive objective over images and captions. Given a set of image- caption pairs, we train an image encoder and a text encoder such that the similarity between the representations of images and their corresponding text is maximized relative to unaligned pairs." For each scale, we fix the model architecture and hyperparameters (see Table 2). We pick Vision Transformers (ViTs) [39] as the image encoder, considering the better scaling trends observed by Radford et al. [11I] compared to ResNets [62]. Models are trained for a fixed number steps
determined by the scale (Table 2), using the OpenCLIP repository [69]. See Appendix N for details.
3.5 Evaluation
We evaluate on a suite of 38 image classification and retrieval tasks. We also study two additional fairness tasks, detailed in Section 5 and Appendix Q. As discussed in Section 3.2, we remove test set images from DATACOMP to avoid contamination. Image classification datasets range from satellite imagery recognition to classifying metastatic tissues. In total have (with some overlap): 22 of the datasets evaluated in Radford et al. [111], 6 ImageNet distribution shifts (i.e., ImageNet-Sketer [143], ImageNet-V2 [121], ImageNet-A [65], ImageNet-O [65], ImageNet-R [64], and ObjectNet [13]), 13 datasets from VTAB [156], and 3 datasets from WILDS [83, 127]. Retrieval datasets include Flickr30k [151], MSCOCO [26], and the WinoGAViL commonsense association task [17]. To aggregate results over all evaluation tasks, we average the preferred metric for cach task.
DATACOMP adopts zero-shot evaluation protocol: models are tested without training the evaluation tasks. This approach is computationally efficient and measures a model's 's ability to perform well without any additional training. We find a strong rank correlation (>0.99) between performance in linear probe zero-shot settings (Appendix Figure 16). Additional details are in Appendix O.
4 Baselines
4.1 Filtering baselines
We study six simple filtering methods for the filtering track; see Appendix P.1 for further details.
No filtering. We simply use the entire pool as the subset, without any filtering. Since each pool size equal the sample budget, training consists of one pass over the data.
Random subsets. To isolate the effects of increasing the compute budget from increasing the dataset size, we form subsets consisting of I %,, 10%, 25%, 50% and 75% of the pool chosen at random.
Basic filtering. We consider many simple filtering operations inspired by Schuhmann al. [128] and Byeon et al. [20]: filtering by language (English captions, using either fasttext [77] or cld3 [2]); filtering by caption length (over two words and five characters); and filtering by image size (smaller dimension above 200 pixels and aspect ratio below three). We also experiment with combining language and caption length filtering and combining language, caption length, image size fitering. Unless otherwise specified, "basic" refers fasttext English, caption length, and image size filtering.
CLIP score and LAION filtering, We experiment with CLIP score filtering (also employed by LAION), where take only examples having cosine similarity scores between CLIP image and text embeddings that exceed a pre-defined threshold. We investigate a range of thresholds and two OpenAl CLIP models for computing the scores: the VIT-B/32 model (as in LAION) and the larger ViT-L/14. We also combine CLIP score thresholds and cld3 English filtering to reproduce the LAION-2B filtering scheme. Table 16 in Appendix P.I summarizes the different CLIP score configurations.
based filtering, We select examples that contain text overlapping with ImageNet class names, which serve as a proxy for relevance to downstream tasks. Specifically, we select English captions (according to fasttext) that contain words from ImageNet-21K or ImageNet-1K
[37] class synsets.
Moore precisely, given a batch data (111)11,, (FB,yB)) yB)} with images x and captions y, we train an where the image encoder g and text encoder with the loss 1/2 =.e Σ mi D. E/-r
di exp (9(x:), 1o(y))). We also a learnable temperature parameter in Radford et al. [111].

---

Table 3: sero-shot performance for select baselines in the filtering track. On all scales, filtering strategies lead to better performance than using the entire, unfiltered pool. The intersection between aaaged-based and CLIP score strategies performs well most tasks and scales. For all metrics, higher is better (see Appendix O for details). n denotes the intersection of filtering strategies.
Scale
Filtering stralegy
Dataset
Samples
ImageNet
ImageNet dist. shifts
VTAB
Retrieval
Average over 38 datasets
size
seen
No filtering
12.8M
1288
0.025
0.033
0.145
0.114
0.132
filtering
3M
12.8M
0.118
0.142
Text-hased
3.2M
12.8M
0.046
0.082
0.169
0.125
0.157
small
Image-based
3M
12.8M
0.043
0.047
0.178
0.121
0.159
LATON 2B filtering
1.3M
12.8M
0,031
0.040
0.136
0.092
0.133
CL.IP score (1/14 30%)
3.8M
12.8
0.051
0.055
0.190
0.119
0.173
Image-based In CLIP score (L/14 30%)
1.4M
12.8M
0.039
0.045
0.162
0.094
0.144
No fitterng
128M
128M
0.176
0.152
0.259
0.219
0,258
fllteing
30M
128M
0.226
0.193
0.284
0.251
0,285
Text-bused
31M
128M
0.255
0.215
0.328
0.249
0.307
medium
Image based
29M
128M
0.268
0.213
0.319
0.256
0.312
LAION-2B filtering
13M
128M
0,230
0.198
0.307
0.233
0,292
CLIP score (L/14 30%)
38M
128M
0,273
0.230
0.338
0.251
0.328
Image based score (L/14 30%)
0,297
0.239
0.231
0.328
No filtering
1,28B
1.28B
0.459
0.378
0.419
0.437
Rasic filtering
298M
1.28B
0.516
0.423
0.446
0,480
0.458
Text-hased
317M
1,28B
0,561
0,465
0.465
0.352
0.466
large
Image-based
293M
1.288
0,572
0.454
0.483
0.479
0.476
LATON-2B filtering
130M
1.288
0.553
0.453
0.510
0.495
0.501
CLIP score (L/14 30%)
384M
1.288
0,578
0.474
0.538
0.466
0,529
Image-based CLIP (L/14
140M
1,288
0.631
0.508
0.546
0.498
0.537
No filterig
12.88
12.88
0.723
0.612
0.611
0.569
0.621
LAION-2B liltering
1.38
12.88
0.755
0.637
0.624
0.620
0.636
xlarge
CL.P scure (L/14 30%
3.8B
12.88
0.764
0.655
0.643
0.588
0.650
Image based (1 CLIP score (1/14 30%)
1.4B
12.88
0.792
0.679
0.652
0.608
0.663
Image-based filtering. We select a subset examples whose visual content overlaps with ImageNet classes. After applying English language (fasttext) and caption length filtering, cluster the image embeddings extracted by the OpenAl ViT-L/14 model for each image into 100K groups using Faiss [75]. We then find the nearest neighbor group for every ImageNet training example, and keep examples belonging to these groups We apply this procedure using either ImageNet-21K (14M images) ImageNet-1K (1.2M images), forming subsets.
4.2 BYOD baselines
We experiment with multiple external data sources, including four moderately sized datasets (10 to 58M samples) studied by Nguyen et al. [101]-CCI2M [24], YFCCI5M [140, 111], RedCaps [38] and Shutterstock [101]-and the larger LAION-2B [129]. Additional experiments, along with more details about the data sources are provided in Appendix P.2. We consider these data sources they are and do not perform additional preprocessing. We also present experiments combining some of the data sources (using only the external datasets, in addition to data from pool).
Results and discussion
5.1 Building better datasets
Main results. Our key results are in Table Most notably, the intersection between image-based filtering and CLIP score filtering excels on most tasks. The exception is at the small scale and for retrieval datasets.? Furthermore, other filtering strategies like basic, CLIP score, image- taxt- filtering show better downstream performance when compared to no filtering. A much larger suite of experiment results can be found in Appendix R.
DATA COMP leads to better image-text datasets. We hope DATACOMP catalyzes the search for the next generation of multimodal datasets. We contribute DATACOMP-IB, which is the output of the Image-based n CLIP score (L/14 30%) baseline filter at the xlarge scale of the filtering track.
Cherti al. [28] also observe that models rank differently on classification and retrieval tasks.
7

---

ImageNet
Average over 38 datasets
0.6
Average performance
0.5
ImageNet accuracy
small scale
0,4
0.4
medium scale
large scale
0.3
CLIP score (L/14)
0.2
CLIP score (B/32)
0.2
Rand. subset
0.0
0.0
0.5
1.0
0.0
0.5
1.0
Fraction the pool used for training
Fraction of the pool used for training
Figure 2: Performance of random subsets (dotted line) and CLIP score filtering (solid line) when varying the subset size. When taking random subsets, larger subsets always better. For CLIP score filtering, subsets with intermediate size perform best.
Our dataset is comprised of 1.4B samples. which not only is smaller than the LAION-2B dataset with 2.3B samples, but also comes from smaller pool. Nevertheless, CLIP L/14 trained on DATACOMP-IB IB outperforms the LAION-2B competitor by 6.1 percentage points on ImageNet (see Table 1). Moreover, training on DATACOMP-II improves ImageNet accuracy by 3.7 percentage points over OpenAl's ViT-L/14 trained with the same compute budget. Additionally, even if we restrict ourselves to 400M samples, we can still find subset DATACOMP- 1B that outperforms OpenAl's ViT-L/14, seen in Table 24. These results demonstrate the impact that DATACOMP can make and provide a foundation upon which participants can build.
External data sources can improve performance. Appendix P.2 Table 18 shows results for several baselines in the BYOD track. We find several instances where adding external data sources improves performance over using just data from COMMONPOOL. For example, at the large scale, combining CLIP-filterd data from COMMONPOOL, with external data from CC12M [24], YFCC15M [140, 111], RedCaps [38] and Shutterstock [101] boosts ImageNet accuracy by 4.3 percentage points. See Appendix P.2 for more experiments and details.
Trade-off between data diversity and repetition. In Figure 2, we see that randomly selecting subsets of the pool has little effect and degrades performance substantially when only small fractions are used. When filtering with CL.P scores, the optimal training set comes from selecting ~30% of the pool with the highest scores. The difference performance trends between random subsets and CLIP score filtering highlights the importance of filtering strategies for selecting samples.
5.2 DATACOMP design analyses
COMMONPOOL. and LAION are comparable with the same filtering. To validate our pool construction, we show that we can build datasets comparable to LAION-2B by employing their filtering technique on our pool. LAION-2B selects all samples where the caption is in English and the cosine similarity score from trained ViT-B/32 CLIP model is above 0.28. We compare this filtering approach on our pool using the same number samples, 130M samples at the large seale. We find that the different data sources perform comparably: 55.3% vs 55.7% accuracy on ImageNet, and 0.501 vs 0.489 average performance over our evaluation sets using our pool and LAION-2B -B, respectively.
Consistency across scales. We find that the ranking between filtering strategies is typically consistent across different scales. This is illustrated Figure 3, which shows that the baselines at small and medium scales are positively correlated. Moreover, as shown in Appendix Table 22, the rank correlations of performance is high, between 0.71 and 0.90 for different scale pairs.
Consistency across training changes. DATACOMP fixes the training procedure, so natural question whether better datasets from DATACOMP are better outside of DATACOMP. While DATACOMP-1D trained at the xlarge scale, show in Appendix Table 23 that even when substituting the ViT-L/14

---

ImageNet
Average over 38 datasets
0.3
Avg. performance (medium)
0.35
ImageNet acc. (medium)
0.30
Basic
0.2
CLI score
0.25
Image based No filtering
0.20
0.1
+
Rand. subset
0.15
Text-based
0.10
0.0
0.00
0.02
0.04
0.075
0.100 0.125 0.150 0.175
ImageNet acc. (small)
Avg. performance (small)
Figure 3: Correlation between small and medium scale baselines. Smaller scales serve useful guides for larger scales. Results for additional scales are shown in Appendix Figure 22.
for a VIT-B/16 ViT-B/32, training DATACOMP-IB outperforms training on OpenAl's WIT and LAION-2B. Additionally, we found that modifying hyperparameters such as training steps and batch size minimally affects the relative ordering of different data curation methods downstream performance. Details on hyperparameter ablations are in Appendix L.
5.3 Evaluation trends
ImageNet accuracy is indicative, but not the complete picture. Similarly to Korbblith et al. [84], in Appendix Figure 25 we find that ImageNet performance is highly correlated with the average performance across all datasets we study, with overall correlation of 0.99. However, ImageNet performance not representative of all evaluation tasks, as the correlation between ImageNet accuracy and accuracy on other individual datasets varies substantially, in some cases even exhibiting a negative correlation, as discussed Appendix
Robustness and fairness. While typical models trained on a target task suffer large performance drops under data distribution shift, zero-shot CLIP models are known exhibit strong performance across many distributions [111]. In Appendix Figure 26, we show that CLIP models trained with data from our pool are more robust distribution shift than ImageNet-trained models from Taori et al. [139]'s testbed. Examining geographic diversity, we find that our models are better than ImageNet-trained models, but fall short of models fine-tuned on diverse curated datasets (see Appendix Figure 21). We also perform face classification analysis and identify demographic biases in our models: notably, the BYOD datasets we consider can increase the risk of misclassificcttin. See Appendix for more fairness and diversity analyses.
Limitations and conclusion
In terms of societal risks, creating an index image-text pairs from the public internet can be problematic. The internet contains unsafe, toxic, and sensitive content, which ideally should not percolate into machine learning datasets. Though we take steps to remove NSFW content and blur human faces to protect privacy, we hope future work will further explore the biases and risks from COMMONPOOL. and DATACOMP-1B: We see several additional directions for future work, including Curating more data sources. 2) Improved data filtering algorithms. Further supervision signals (c.g., image captions coming from captioning models). 4) Additional input modalities (c.g., video, 3D objects). Broader evaluations for vision-and languuag and robotics tasks.
Overall, we see DATACOMP as a first step towards improving training datasets, and hope our new benchmark will foster further research. By providing controlled experimental setting, DATACOMP enables researchers to iterate on dataset design on rigorous empirical foundations. We open- open-source all of our code, data, and infrastructure, and hope these resources will help the community build the next generation multimodal datasets.
Note that unlike Korblith et al. [84] evaluate zero-shot performance rather than transfer learning.

---

Acknowledgements
SYG and JH are supported by NSF Graduate Research Fellowships. GS supported by the Onassis Foundation Scholarship ID: F ZS 156-1/2022-2022 GD has been supported by the Onassis Fellowship (Scholarship ID: ZS 012-1/2022-221 20222...... the Bodossaki Fellowship and the Leventis Fellowship. This research has been supported by NSF Grants AF 1901292, CNS 2 2148141, DMS 2134012, TRIPODS II-MMS 20231 1623166, Tripods CCF 1934932, IFML CCF 2019844 and research gifts by Western Digital, WNCG IAP, UT Austin Machine Learning Lab (MLL), Cisco, the Len Blavatník and the Blavatnik Family Foundation, the Stanly P. Finch Centennial Professorship in Engineering, Open Philamthropy. Google, Microsoft, and the Allen Institute for AL.
We would like thank Amro Abbas, Danny Bickson, Alper Canberk, Jessie Chapman. Brian Cheung. Tim Dettmers, Joshua Gardner, Nancy Garland, Sachin Goyal, Huy Ha, Zaid Harchaoui, Ari Holtzman, Andrew Hundt, Andy Jones, Adam Klivans, Ronak Mehta, Sachit Menon, Ari Morcos, Raviteja Mullapudi, Jonathon Shlens, Brandon McKinzie, Alexander Toshev, David Grangier, Navdeep Jaitly, Kentrell Owens, Marco Tulio Ribeiro. Shiori Sagawa, Christoph Schuhmann, Matthew Wallingford, and Ross Wightman for helpful feedback at various stages of the project. We are particularly grateful to Daniel Levy and Alec Radford for early encouragement to pursue this project and feedback on the experimental design.
We thank Stability AI and the Gauss Centre for Supercomputing e. V. for providing us with compute resources to train models. We are thankful for the compute time provided through the John von Neumann Institute for Computing (NIC) on the GCS Supercomputer JUWELS Booster [78] at Jülich Supercomputing Centre (ISC), and for storage resources on JUST [50] granted and operated by JSC, as well computing and storage resources from the Helmholtz Data Federation (HDF).
"atpp:///guss-commrr..o
10

---

References
[1] httpa//www..pp............/..//a
[2] CLD3. httppp///gtttto..ooogggg///////
[3] Common Crawl. https://ccommonccaaa...rgg
[4] dataset2metadata. httprpaw.............co.oooooooa
[5] i http::////ggggioooo////////////a
[6] Amro Abbas, Kushal Tirumala, Dániel Simig. Surya Ganguli, and Ari S Moreos. Semdedup: Data-efficient learning at web-scale through semantic deduplication, 2023. attps://arxx:
org/ hts/bb//23033 09540.
[7] Pankaj Agarwal, Sariel Har-Peled, and Kasturi R. Varadarajan. Approximating extent measures of points. Journal of the ACM (JACM), 2004. attps://doi. org/ 10///00.11 1145/ 100873..
[8] Bean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurlPS), 2022. https://openrett wwtpperreeee.neee/...eeee www.teeenuulliee hit.uunddddbbbbmmaaaa....
[9] Abhijeet Awasthi, Sabyasachi Ghosh, Rasna Goyal, and Sunita Sarawagi. Learning from rules generalizing labeled exemplars. In International Conference on Learning Representations (ICIR), 2020. http:::////peeevvieeee
[10] Stephen H Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alex Ratner, Braden Hancock, Houman Alborzi, Rahul Kuchhal, Christopher Ré, and Rob Malkin. Snorkel drybell: A case study in deploying weak supervision at industrial scale. Special Interest Group on Management of Data (SIGMOD), 2019. https: // //arxiv.orr
11///112..10 00417.
[11] Olivier Bachem, Mario Lucic, and Andreas Krause. Coresets for nonparametric estimation the case of dp-means. In International Conference on Machine Learning (ICML), 2015.
https://proceedinga. mlr. press/v37 2////bachemis html.
[12] Peter Bandi, Osear Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 17 challenge. IEEE Transactions on Medical Imaging, 2018. https://pubmed: ncbi..c..... nlm. nih, gov/3071602//.
[13] Andrei Barbu, David Mayo, Julian Alverio. William Luo, Christopher Wang. Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buce E. Fox, and R. Garnett (cds.), Advances in Neural Information Processing Systems (NeurIPS), volume 32. Curran Associates, Inc., 2019. https: httts:///proceednggs. neurips.. wweurips.ccppppeer//eee wttpc//ppppe/////e/c//////////// http:pppppppp......a..aaaaaaaaaa
Paper. .pff. [14] Sara Beery, Elijah Cole, and Arvi Gjoka. The iwildcam 2020 competition dataset, 2020. https://arxiv. wwwg/a//////////oo/o///////////e 10340.
[15] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pomographyy, and malignant stereotypes. 2021. 01963.
[16] Vighnesh Birodkar, Hossein Mobahi, and Samy Bengio. Semantic redundancies in image- classification datasets: The 10% you don't need. arXiv preprint arXiv: 1449, 2019.
https://arxiv org/abs//90000o 1991.
[17] Yonatan Bitton, Nitzan Bitton Guetta, Ron Yosef, Yuval Elovici, Mohit Bansal, Gabriel Stanovsky, and Roy Schwartz. WinoGAViL: Gamified association benchmark to challenge vision-and-language models, 2022. httpp://aat http://aaxvvvvoo org/ abs/ab02 2207 12576.
[18] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-min
discriminative
components with random forests. European Conference on Computer Vixion (ECCV), 2014. https://link www.nng....i........... com www.//oo///cc/ciiee 100.001000777778800000 311---33111111 ---1999199999---1111
11

---

[19]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhimi Agarwal, Ariel Herbert- Voss,, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems (NeurIPS), 2020. attppe//prooeeedings neurips cc/ Paper pdf.
paper
[20] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. https://githan com/kaaaobrain/ coyo-dataset, 2022.
[21] Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. International Conference on Learning Representations (ICIR), 2023. sstps://arxiss org/ abs/2210. 14891.
[22] Nicholas Carlini, Matthew Jagielski, Christopher A Choquette- Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning web-scale training datasets is practical, 2023. pttpps///arxii
aaa/ab//// /2002 10149.
[23] Stephanie C. Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2022. https: //tp///aaxiii org/ abaaabs/// haa/abb//222../. 200/2220000055
[24] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021. attps://arxin: www.aarwww....abbaaeaera 222002 08981.
[25] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali: A jointly-scaled multilingual language-image model. In International Conference on Learning Representations (ICLR), 2022. https:// /ttp::///axvvev/e org/ wwgg/aa///// abs//2////222// 06794.
[26] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server, 2015, https://arxiv. .rg/ abs/1504 00325.
[27] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedingy of the Institute of Electrical and Electronics Engineers (IEEE), 2017. https://ieeeexllore ieee..eeeeeo.. org/ absttact// document/7891544.
[28] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reprodueible scaling laws for contrastive language-image learning, 2022. https: http//axvvvv///////ee//aaa org/ 22122. 0111..77140 07143.
[29] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018. https: //arxiv. org/
[30] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Conference on Computer Vision and Pattern Recognition (CVPR), 2014. dttpa://openaceen wew.eaaccceeeeeeet...eeee.e com/ content_cvvr__0000 201 html.
[31] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2011. sctps://proceedingg mlr, www.s./pe/////ee.... vrees/vv5. vv1/ coates cott/////////e//1eeo/ 1a. html.
12

---

[32] Michael B. Cohen, Cameron Musco, and Christopher Musco. Input sparsity time low-rank approximation via ridge leverage score sampling. In ACM-SIAM Symposium on Discrete Algorithms,, 2017. attps://dl acm. org/doi/ /10 00555/3039666. 3039801.
[33] C Coleman, C Ych, S Mussmann, B Mirzasoleiman, P Bailis. P Liang, Leskovec, and M Zaharia. Selection via proxy: Efficient data selection for deep learning. In International Conference on Learning Representations (ICLR), 2020. https: http///xxxvvv.o/// org/ wtrggabb/19966 11829.
[34] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Annual Meeting of the Association for Computational Linguistics (ACL), 2019. https:/ http:://xxxxvv.roo www.///g////o/o//eo/o////eeeeee/ 19111 02116.
[35] R Dennis Cook. Detection of influential observation in linear. Technometrics, 19(1):15-18, 1977.
[36] Achal Dave, Tarasha Khurana, Pavel Tokmakov, Cordelia Schmid, and Deva Ramanan. Tao: A large-scale benchmark for tracking any object. In European Conference on Computer Vision (ECCV), 2020.
/ttps://arxiv... /twpg/////////aaee 200./000556......
[37] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition (CVPR), 2009. https: http://eeeeepppeeeeee ieee. htt////taac///cccmeeeett 52008448.
[38] Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: Web-curated image- text data created by the people, for the people, 2021. https://arxiv org org hatgg/ab///// /bs/2
11 11431.
[39] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference Learning Representations (ICLR), 2021. https://openreviin www.:eeeeoommmddiiioooo.... diiiumidadddicbyyyy.....
[40] Matthijs Douze, Giorgos Tolias, Ed Pizzi, Zoë Papakipos, Lowik Chanussot, Filip Radenovic, Ismail Elezi, Ondrej Chum, and Cristian
Tomas Jenicek, Maxim Maximov, Laura Canton-Ferrer. The 2021 image similarity dataset and challenge, 2021. attps://arxxs org/ hts//bb//106 09672.
[41] Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with v-usable information. In International Conference on Machine Learning (ICML), 2022.
https://arxiv. org/ warg//b///aa abs/2110...1...///////////////// 221//111110.0.........
[42] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results, 2007. http://wwe pascal-network. wwwoooogglla/eooo challenges/ htt0//oc2007/wooo
/et///indxx///////// html.
[43] Sabri Eyuboglu, Bojan Karlaš, Christopher Ré, Ce Zhang, and James Zou. debench: benchmark for data-centric ai systems. In Proceedings of the Sixth Workshop on Data Management for End-To Toad-To-End Machine Learning, 2022. https: //dp:///dlaa acm. org/doi. /w///o/iabbbb////
10. 1145/3533022 3533310.
[44] Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). In International Conference on Machine Learning (ICML), 2022.
https://arxiv. org/ /b///////// aab//2255 01397.
[45] Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories. Conference on Computer Vision and Pattern Recognition (CVPR) Workshop, 2004. https://ieeeepplore. 1384978.
[46] Dan Feldman, Matthew Faulkner, and Andreas Krause. Scalable training of mixture models via coresets. In Advunces in Neural Information Processing Systems (NeulPS), 2011. https://proceedingg: neurips wwwpscccpppppppppp peaper_filesppperrre file fine///6665004444477777777666222 Paper pdf..
13

---

[47] Daniel Y. Fu, Mayee F. Chen, Frederic Sala, Sarah M. Hooper, Kayvon Fatahalian, and Christopher Ré. Fast and three-rious: Speeding up weak supervision with triplet methods, In International Conference on Machine Learning (ICML., 2020. https:// //arxivv. org/ abs/ 2002.11955.
[48] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012. https://1seexploree ieee. /tt//aaattt//ccmmeettt//// /ttp///occceeeeeeeeeeeee 6244074
[49] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pp. 2242-2251. PMLR, 2019.
[50] Stephan Graf and Olaf Mextorf. Just: Large-scale multi-tier storage infrastructure at the jülich supercomputing centre. Journal of large-scale research facilities JLSRF, 2021. https: //jlsrf. //llsrforgg////////..........l.l index. phpppppppss// /tt///rricceeeeeeeee/eeeeeeeeeet 180.
[51] Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepeore: A comprehensive library for coreset selection in deep learning, 2022. https://arxiv orgg 2204./04929. 08499.
[52] Han Guo, Nazneen Fatema Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. Fastif: Scalable influence functions for efficient model interpretation and debugging. 2020. https: 15781.
// swwrii..org//bs...
[53] Jia Guo, Jiankang Deng. Alexandros Lattas, and Stefanos Zafeiriou. Sample and computation redistribution for efficient face detection. In International Conference on Learning Representations (ICLR), 2021. https://arxiv: org/ httpa/aa//2//.00 04714.
[54] Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[55] Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2018.
tatp:://aalaanthllan
org/ wwwg///---// 018-2017.
[56] Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi. Simfluence: Modeling the influence of individual training examples by simulating training
runs, 2023. https: http/vaxvvvvoooe///////a abs/ / 2.0/33333..00 08114.
[57] Frank R Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, 1974. https: http://w..ssttto org/ htggstaable22225666
[58] Xiaochuang Han, Byron C Wallace, and Yulia Tsvetkov. Explaining black box predictions and unveiling data artifacts through influence functions, 2020. 06676.
[59] A. Hanna, Emily L. Denton, Andrew Smart, and Jamila Smith-Loud. Towards critical race methodology in algorithmic fairness. In Conference on Fairness, Accountability, and Transparency (FAceT)), 2020. https://arxiv org/ abss/1922. 03593.
[60] Laura Hanu and Unitary team. Detoxify, 2020. detoxify.
[61] Sariel Har-Peled and Soham Mazumdar. On coresets for k-means and k-median clustering. In Symposium on Theory of Computing (STOC), 2004. https://dol org/10. 10. 1145/1007355. 1007400.
[62] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016. https://arxiv. org/ herg/bss/155 httpaa///////.///////e./
[63] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019. attps://arxiv. org/ abaaaabs/...//////// 00029.
[64] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distributiio generalization. ICCV, 2021. https: aaxii org/ abs///0ss2000.... 16241.
14

---

[65] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Conference on Computer Vision and Pattern Recognition (CVPR).
2021. https://arras http:://axvvvvvoeee abs/1907 0tt///99700077777 .07144.
[66] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models, 2022. https://arxie: http://aaxvvv... org/ wwt.s/.rrg/ab///////// ab202233. 15556.
[67] Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld. Knowledge-busee based weak supervision for information extraction of overlapping relations. In Annual Meeting of the Association for Computational Linguistics (ACL), 2011. https:
/aaclanthology.... hwwphhhooogg.og///111-ee1 org/P11-1055.
[68] Andrew Hundt, William Agnew, Vicky Zeng, Severin Kacianka, and Matthew Gombolay. Robots enact malignant stereotypes. In Conference on Fairness, Accountability, and Transparency (FAceT'), 2022. https://arxix: org/ abs//20//22222.. 11569.
[69] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP, July 2021. https://doo orgoorg//// 2rgg/0..0 5.01/
zenodo. 5143773.
[70] Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannanch Hajishirzi, Simon Kornblith, Ali Farhadi, and I Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. In Advances in Neural Information Processing Systems (NeurlPSS) 2022.
https://arXte org/ abs/ 2200 05592.
[71] Andrew lyyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry, Datamodels: Predicting predictions from training data, 2022. https: /tttps://axivv... org/abs/ 2202.00622
[72] Tanuj Jain, Christopher Lennan, Zubin John, and Dat Tran. Imagededup, 2019. https: //github. tamm/idall//aa hoom///eeao/mmgeedduuu imt.ededug........
[73] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference Machine Learning (ICML), 2021. https:///araee
org/ 2202.. 05918.
[74] Mon-Fong Jiang. Shian-Shyong Tseng, and Chih-Ming Su. Two-phase clustering process for outliers detection. Pattern recognition letters, 2001. https::///// h/wp://wwwww//// sciencedirect.. com/ siience/article//bbeee http:/pp.111111185550000333333
[75] Jeff Johnson. Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 2019. btttps://arztee org/ abs/
[76] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. Conference Computer Vision and Pattern Recognition (CVPR), 2017. https://arxiv.com wwwgg///////////ooo/ee/ee/ogo//e att////2..1....//////////// 06890.
[77] Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. Conference of the European Chapter of the Association for
Computational Linguistics (EACI), 2017. https: http///axvvvv./ www.wiivvorroooooooooooooo abs/abb0/ 01759.
[78] Juelich Supercomputing Center.
JUWELS Booster Supercomputer, 2020.
http://www.........eetettetteeee http://pss/ueeeeeooccosse iew.aiiin..taaiiio html#
hardware-conf hardvarre-ooffguuratiio the -heerrtttt-----------tttt-te--ee name booster-module.
[79] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. https://arxiv..ot/ wttpg///g//.....//oooo
08361
[80] Kimmo Karkkainen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In JEFF/CVE Winter Conference on Applications of Computer Vision, 2021. https tttpp::/arxse /arxivivvv..///////a/////aa///a/ org/ argg/bs///e abs/1908....1/.. 04913.
15

---

[81] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning (ICML), 2017. https://arxiv. org/ abs/ / 1703. 04730.
[82] Pang Wei Koh, Kai-Siang Ang, Hubert Teo, and Percy S Liang. On the accuracy of influence functions for measuring group effects. Advances in Neural Information Processing Systems (NeurIPS), 2019. petts://arxiin http://awwwvv....aaaaaa//ass htggabs/1995 13289.
[83] Pang Wei Koh, Shiori Sagawa, Henrik Marklund. Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao. Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML), 2021. https: //tpvv/aavvvv////// org/ abtp//a...222/.///// 07421.
[84] Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Conference on Computer Vision and Pattern Recognition (CVPR), 2019. https: //arxiv. org/ http//as/1885..0888 08974.
[85] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In International Conference on Computer Vision Workshops (ICML), 2013. https://wwwt
http:://wwwccvcfoooooe cv-founnation
pttpaccppaccce/////e cttpes/coctttttteeeeeeeeeeeeeeee _icc__cc_____ _occ__sorsshps___ Object_Repreeeeeeeeeeeeeeeeeeeee
1113//19/html/Krausse _________bba_
2012_000111 _C_CV_ppper. .ttml.
[86] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, wtwtrrii//eaannn-feeeeeeeee
2009. https://www.cr wwronto..oooooooooooooooooooo.o.
2009- 1009-TR.yp pdf.
[87] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2012.
neurips cc/paper_filleeeeepeeeeeee fiaeerrfflllpapape
2012/file
pdf.
[88] Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi. Adversarial filters of dataset biases. In International Conference on Machine Learning (ICML), 2020. https: htt///axxvvv//////////eeeeaaa.va org.
www/aa/g///////oooooooooocaaaea/ aabb/20022 04108.
[89] Yann LeCun. The MNIST database of handwritten digits, 1998. http://yann. har.laaannnnn.....mmmmmm lecun. com/ exdb/nnist/.
[90] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Annual Meeting of the Association for Computational Linguistics (ACL), 2021. https:/ /ate://arzze http/ariiv..r.. org/ wwg//ba///////ooeeoooooeeeeeee/g aba/2107 06499.
[91] Li and Nuno Vasconcelos. Repair: Removing representation bias by dataset resampling. In Conference Computer Vision and Pattern Recognition (CVPR), 2019. https://arxit: org/ aass//9/: 07911.
[92] Yulong Liu, Guibo Zhu, Bin Zhu, Qi Song, Guojing Ge, Haoran Chen, GuanHui Qiao, Ru Peng, Lingxiing Wu, and Jinqiao Wang. Taisu: A 166m large-scale high-quality dataset for chinese lasion-language pre-training. In Advances in Neural Information Processing Systems (NeurIPS), 2022. https: https://proceedinge neurips cc/paper _ files/ pttes/paper/022/ f11e:/2888r0000ccccffffaaaa22222 Papeee--9-6----------re Paper- Saapeer-attstss....m mand__Bcchmaas...
pdf.
[93] Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. Conference on Computer Vision and Pattern Recognition (CVPR), 2022. https: http///xxvvvvoo/ orrxxvvoor///oo
.03545.
[94] Mario Lucie, Matthew Faulkner, Andreas Krause, and Dan Feldman. Training gaussian mixture models scale via coresets. Journal of Machine Learning Research (JMLR), 2018.
http://jmii wttgpapers////ooooooot.....ssset waaesss/vvvvii 10/1//555666 html.
[95] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft, 2013. https: / wwtps/iivvoggabbba wwtp/o//////////ooooe 5151.
[96] Gideon S Mann and Andrew McCallum. Generalized expectation criteria for semi-supervised learning with weakly labeled data. Journal of Machine Learning Research (JMLR), 2010. attps://wwi jmlr, org/ htt///pper////// wwteers/viinmaannn Oa. html.
16

---

[97] Mark Mazumder, Colby Banbury, Xiaozhe Yao. Bojan Karlaš, William Gaviria Rojas, Sudnya Diamos, Greg Diamos, Lynn He, Douwe Kiela, David Jurado, David Kanter, Rafael Mosquera. Juan Ciro, Lora Aroyo, Bilge Acun, Sabri Eyuboglu, Amirata Ghorbani, Emmetr Goodman, Tariq Kane, Christine R. Kirkpatrick, Tzu-Sheng Kuo, Jonas Mueller, Tristan Thrush, Joaquin Vanschoren, Margaret Warren, Adina Williams, Serena Yeung, Newsha Ardalani, Praveen Paritosh, Ce Zhang, James Zou, Carole-Jean Wu, Cody Coleman, Andrew Ng. Peter Mattson, and Vijay Janapa Reddi. Dataperf: Benchmarks for data-centric ai development, 2022. https: https///arxvv.ooo aba/2207 10062.
[98] Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning (ICML), 2020.
attps://arxiv.
a90/ 01827.
[99] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems (NeurIPS) Workshops, 2011. https://storags .d........ff..............m
googleapis
wwogleapis.com/pubbb.... toooo-ttols-puull
[100] Andrew Ng. Dillon Laird, and Lynn He. Data-centric ai competition, 2021. https://
sepps-ddeelearnnnngg ai github..
[101] Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. In Advances in Neural Information Processing Systems (NeurIPS), 2022. https://openreevie net/ Drrumidd-LTCBaFWWee
[102] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, 2008.
-eee. ht////cmmmn///5611444////
[103] OpenAI. Gpt-4 technical report, 2023. https: http:::://xxxivorr//b/baa wwwg/a//g/////o/o 087744.
[104] Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. Im2text: Describing images using 1 million captioned photographs. In Advances in Neural Information Processing Systems (NeurIPS), 2011. https: http:pppppr.ipp..pppppppppppp.pp .df..
[105] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In Conference on Computer Vision and Patlern Recognition (CVPR), 2012. https: // //ieeexplore. iieepplooeeieeee....... .org doo.ueett/24500
[106] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding important examples early in training. In Advances in Neural Information Processing Systems (NeurIPS), 2021. attps://aat //tp:///a/vv///////ee org/ /a/a//////10// .07055.
[107] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingxing Tan, and Quoe V. Le. Combined scaling for zero-shot transfer learning, 2021. autps://arxiss
http://rwwv.oo////////// abs 2211
[108] Vinay Uday Prabhu and Abeba Birhane. Large image datasets: A pyrrhic win for computer vision? In Winter Conference on Applications of Computer Vision (WACV), 2020. https: //arxiv. org/ abs////s......//////////// 16923.
[109] Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems (NeurIPS), 2020. bttps://aazxe
hts//ba/20000 2040400022222..22.
[110] Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende. Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan. Filtering, distillation, and hard negatives for vasion-language pre-training. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023. https: //axxiv////vv..////a/ .rg/ abs//a/1..222222
[111] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Knueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML.), 2021. https: // /arxiv.. org/abs/
17

---

[112] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. 2022. https:// 04356.
pexiv.org//
[113] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research (JMLR). 2020. https:
perxiv.org. www.aa////1// 10683.
[114] Vikram V. Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron B. Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. Beyond web-scraping: Crowd-sourcing a geodiverse datase, 2023. https://arxis: org httg:/ab/2330..0000
[115] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Πla Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning (ICML), 2021. https : /ttp///wxvvvv//////////////a.avt org/ abs/2102. 2b022.0222
[116] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical rext-conditional image generation with clip latents, 2022. https:/ /atps://arxiv..o http//axivvvorree 06125.
[117] A. J. Ratner, B. Hancock, J. Dunnmon, F. Sala, S. Pandey, and C. Ré. Training complex models with multi-task weak supervision. In Association for the Advancement of Artificial Intelligence (AAAT), 2019. attps://arxxx wtr.aiiowv/.aiaaaaaooooooooooltt org/arg/ 1111112800000....1
[118] Alexander Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. In Advances in Neural Information Processing Systems (NeurIPS), 2016. attps://arzae /ttp::/.ww.w..oo/////./// aa//abb////
07723.
[119] Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. In Very Large Data Bases Conference (VLDB), 2017. https: /ttp:v/:////vv//aa http://.gggg/bbbbbbb/eooee 17111 10160.
[120] Christopher Ré. Overton: data system for monitoring and improving machine-learned products. In 10th Conference on Innovative Data Systems Research. CIDR 2020, Amsterdam, The Netherlands, January 12-15, 2020, Online Proceedings. www.cidrdb.org. 2020. URL /ttp://ccdrdoe org/ http//idr2222//aapp http://pppppppp///2 http://3-reeri220.....1 pdf.
[121] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In International Conference on Machine Learning (ICMI), 2019. /tst::/pprocee http::////ccecogss/p//ee mlr. waa.r.pees//vv9 http://ehhtttt html.
[122] William Gaviria Rojas, Sudnya Diamos, Keertan Ranjan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2022. httpss://pppt http:///ppprreeee.eeeee/ee www.:/rut///o/mm///mmmm gnfYsaveOU4.
[123] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. https://arxiv.. httpae////wvvvoooo/bbbbeeee atgg//bb/2112.... 10752.
[124] Peter J Roussceuw and Mia Hubert. Robust statisties for outlier detection. Wiley interdisciplinarry reviews: Data mining and knowledge discovery, 2011. http://i2pc.
pdf.
[125] Peter J Rousseeuw and Mia Hubert. Anomaly detection by robust statisties. Wiley interdisciplinary reviews: Data mining and knowledge discovery, 2018. https://wires.
onlinelibrary
com/ doi/ /10. 100 1002 wwt.mmwmmmmmmmmmmmmmm 1236.
[126] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015. https: : /arxiviivvv...i//aaaaaaiaa. org/ hat///b//1400 0575.
[127] Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga Henrik Marklund, Sara Beery, Etienne David, Ian Stavness, Wei Guo, Jure Leskovee, Kate Saenko, Tatsunori Hashimoto, Sergey Levine, Chelsea Finn, and Percy Liang. Extending the wilds benchmark for unsupervised
18

---

adaptation. In International Conference on Learning Representations (ICLR), 2022. https: // //arxivv.coo argg/bs//1122 05090..
[128] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki, LAION-400M: Open dataset of clip-filtered 400 million image-text pairs, 2021. https://arxiv: org/abs/ #111.02114
[129] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-SB: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2022. https: https:// /opprreveeeeeeeeeeeeeeee net/ #rrum?id=MBY7Avar
[130] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. International Conference on Learning Representations (ICLR), 2018.
https://openreviex net
-R-
[131] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Anmual Meeting of the Association for Computational Linguisties (ACL), 2018. https: / attps://aclanttooogg aclanthoonn.oo...... org/P18 1188/1111111 1238/.
[132] Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and-language tasks?, 2021.
https://arxiv org/ h/r///bs/2// /2107 .03383.
[133] Changho Shin, Winfred Li, Harit Vishwakarma, Nicholas Roberts, and Frederic Sala. Universalizing weak supervision. In International Conference Learning Representations (ICLR), 2022. bttpp:://ppereee
[134] Haoyu Song, Li Dong, Weinan Zhang, Ting Liu, and Furu Wei. CLIP models are few- shot learners: Empirical studies on VQA and visual entailment. In Annual Meeting of the Association for Computational Linguistics (ACL), 2022. https:/ sstps://aclanttologg. org/ 2022.ac1-loma 421.
421.
[135] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Advances in Neural Information Processing Systems (NeurIPS), 2022. https: / http////peeeeeeeeeeeee www.torummiiiiiio UmvS1P-PyV.
[136] Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. Wit: Wikipedia- baseed image text dataset for multimodal multilingual machine learning. In 44th International ACM SIGIR Conference on Research and Development Information Retrieval, 2021. https://arrse org/ abs/2103. 01913.
[137] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In International Joint Conference on Neural Networks (IJCN), 2011. nttps://ieeexpi
ieee. .eee.org/ org/
document/ /ecument/6033395. 6033395.
[138) Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. https:
org/2020.
22200.022 emnlp-maii 746.
[139] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. In Advances in Neural Information Processing Systems (NeurIPS), 2020. https://dl. acm. org/ /aii//b///ii 20bs/10.30 35555/3495724.349999
[140] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The new data in multimedia research. Communications of the ACM, 2016.
https: arxiv. org/ abs 1503 01817.
19

---

[141] Mariya Toneva, Alessandro Sordoni. Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations (ICLR), 2015 8.
https://arziv.org/ http://.rrgabbb/br.babeee aas// 05159.
[142] Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling, Rotation equivariant CNNs for digital pathology, 2018. sttps://aries org/ http//aa/180..... 03962.
[143] Haohan Wang, Songwei Ge. Zachary Lipton, and Eric Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS), 2019. attps://arze
org/abs///a aabs/1905 13549.
[144] Ryan Webster, Julien Rabin, Loic Simon, and Frederic Jurie. On the de-duplication of laion-2b, 2023. https: /
https://arxiv.. http://aaxvvv... wtw.aiiwwii.oa..aaaooooooooo org/ abs//3/s//333/ 12733.
[145] Kai Wei, Rishabh lyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International Conference Machine Learning (ICML), 2015. https:
//proceedings.at mlr. press/v37//st //3////7//we/// hhtll.
[146] Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision (IJCV), 2016. https://link: springer http://innk..ringeer....
s11263- 014-0748-y
[147] Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng. and Olga Russakovsky. Towards fairer datasets: filtering and balancing the distribution of the people subtree the imagenet hierarchy. In Conference on Fairness, Accountability, and Transparency (FAceT), 2020. https://
arxiv.org/ org/ at/////s///1/////////// 1912/ 07726.
[148] Kaiyu Yang, Jacqueline H Yau, Li Fei-Fei, Jia Deng, and Olga Russakovsky, A study of face obfuscation in ImageNet. In International Conference on Machine Learning (ICML), 2022.
Attps://arae attpa.axiiiie/.ii wrr.i.ogg//aa...aaoooo.... aba/2103 06191.
[149] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grai Fine-grained interactive language- pre-training. In International Conference on Learning Representations (ICLR), 2022. https:
//arxiv.org org/ /a//ab///1111 22781/177777..
[150] Shuhei Yokoo. Contrastive learning with large memory bank and negative embedding subtraction for accurate copy detection, 2021. /ttps://arxiv.ot
04323.
[151] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2014. https://
aclanthology. oow//oo-//0///00 0014-100//
[152] Dantong Yu, Gholamhosein Sheikholeslami, and Aidong Zhang. Findout: Finding outliers in very large datasets. Knowledge and information Systems, 2002. https://lind htpps///nissssssiinerr..... com/
[153] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: new foundation model for computer vision, 2021. https: / /ttp:////xxxivieeeeeeea httpr/ii...rg//bb/111111
[154] Man-Ching Yuen, Irwin King, and Kwong-Sak Leung. A survey of crowdsourcing systems. In SocialCom. IEEE, 2011. https://ieeexplooe ieee. org/ http///cmmentttt/ 66112213.
[155] Matei Zaharia, Reynold S Xin, Patrick Wendell, Tathagata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng. Josh Rosen, Shivaram Venkataraman, Michael Franklin, et al. Apache spark: unifted engine for big data processing. Communications of the ACM, 2016.
https:
//d1. acm owwo/ooo////ooooooooooooo/oo dt////..1111111 10. 14552934664
[156] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, André Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. The visual task adaptation benchmark, 2019. http:
// //arxiv.. org/ abs att/ 1910.00000 04867.
20

---

[157] Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang. and Alexander Ratner. WRENCH: A comprehensive benchmark for weak supervision. In NeurIPS, 2021.
URI. peatts::/openreviiiee
new/forumm wit.umndddd9sssssiii..
[158] Jieyu Zhang, Cheng-Yi Yu Hsieh, Yue Yu, Chao Zhang, and Alexander Ratner. A survey on programmatic weak supervision, 2022. https: : / //tp///avvvv///////////eae/aaava http://wwww.....////////o///
05433.
[159] Zhifei Zhang. Yang Song, and Hairong Qi. Age progression/regresiinn by conditional adversarial autoencoder. Conference on Computer Vision and Pattern Recognition (CVPR),
2017. https://araa org/ has///0//7 1etp//277..00.3... 08423.
[160] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. European Conference on Computer Vision (ECCV), 2022. https: //axxiv.... http://awww...aaaaaa//ee http//bs//200.....m
21


Reviews:
[{"review_id": "Z3f3mdQnBq", "reviewer": "Reviewer_TaGm", "summary": "", "questions": "", "limitations": "Licensing. I hate to sound like a broken record on this topic.", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Data centric approach is good.\n2. Big and comprehensive paper, with a lot packed into it.\n3. Love the attention to multi-modal data, definitely an under-explored area\n4. Cool results", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ZLFIHQXRGE", "reviewer": "Reviewer_pyn4", "summary": "", "questions": "", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "(1) The proposed CommonPool is the largest openly available dataset for training vision-and-language models.\n\n(2) The data collection and filtering process has been comprehensively discussed to support community investigation and further improvement.\n\n(3) Experiments support that CLIP models trained on DataComp-1B exceed the performance of original CLIP models trained on the private dataset.\n\n(4) Over 300 baseline experiments are conducted to ensure the quality of their dataset and the validity of their conclusions.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "bcPchGC6bD", "reviewer": "Reviewer_N9Hd", "summary": "", "questions": "", "limitations": "Yes. The limitations section of the paper already addressed possible impacts.", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "### Strengths\n\nThe topic that this paper addresses is becoming increasingly relevant with the advent of new foundation models and open-source models. A lot of papers have investigated model architecture, training methods, and optimization techniques, yet only a few focus on datasets. The recent trend in scaling up models requires massive amounts of data to be put into the models, but the quality of this data has received less attention. For example, from the perspective of language models, we know that high-quality data can lead to better models [1, 2]. This paper focuses on visual language models, in particular CLIP [3], to investigate the quality and importance of the data, something that feels missing from the literature. \n\nThe evaluation conducted in this paper is extensive, containing over 300 baselines over four different scaling power and seven filtering methods. The results leads to introduction of `DataComp-1B` dataset that is not only smaller than `LAION-2B` [4] but also lead to training CLIP-L/14 model that outperforms its LAION variant with $6.1$pp.\n\nThe amount of consideration put into this paper is exemplary. Each section in the main text is accompanied by a corresponding section in the supplementary material, which provides detailed explanations and further insights. Additionally, the paper provides an analysis of fairness and bias in the trained models which is extremely valuable. \n\n### References: \n- [1] - Zhou, Chunting, et al. \"Lima: Less is more for alignment.\" arXiv preprint arXiv:2305.11206 (2023).\n- [2] - Mukherjee, Subhabrata, et al. \"Orca: Progressive learning from complex explanation traces of gpt-4.\" arXiv preprint arXiv:2306.02707 (2023).\n- [3] - Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" International conference on machine learning. PMLR, 2021.\n- [4] - Schuhmann, Christoph, et al. \"Laion-5b: An open large-scale dataset for training next generation image-text models.\" Advances in Neural Information Processing Systems 35 (2022): 25278-25294.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "fpx8NmXZuF", "reviewer": "Reviewer_GJCx", "summary": "", "questions": "", "limitations": "1, The training target is only suitable for  understanding  tasks, the resulted dataset may not be the best for generative tasks.\n2, Only filtering-based methods are included.  However,  In some works, researchers  used a captioning model to augment image captions, like ** LIU, Yulong, ZHU, Guibo, ZHU, Bin, et al. TaiSu: A 166M Large-scale High-Quality Dataset for Chinese Vision-Language Pre-training. Advances in Neural Information Processing Systems, 2022, vol. 35, p. 16705-16717. **", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "This work provide a public environment and massive resources  for the exploration of  the construction of an effective multi-modal dataset.  \nResults for over three hundred baseline experiments are provided, and evaluation is conducted on 38 downstream tasks.\n ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: - This paper presents DataComp, a new image-text dataset and benchmark designed to offer new avenues for research and evaluation. 
- Both reviewers and the Area Chair agree that this work promises significant contributions to the vision-language research community. Not only does it provide a large-scale dataset, but it also includes a comprehensive benchmark and leaderboard, as well as an extensive set of baseline experiments. 
- The authors have engaged in the discussion with reviewers to address questions and concerns, including dataset licensing and specific requirements for image generation, etc. 
- Considering the reviews and the potential impact of the paper, the Area Chair strongly recommends acceptance of the paper.
Decision: Accept (Oral)