Data Filtering Networks
Authors: Alex Fang
Albin Madappally Jose
Amit Jain
Ludwig Schmidt
Alexander T Toshev
Vaishaal Shankar
Venue: ICLR 2024 poster (2024)
Keywords: datasets
dataset filtering
dataset curation
large scale machine learning
Primary area: datasets and benchmarks
TL;DR: We build state of the art datasets using neural networks

Abstract: Large training sets have become a cornerstone of machine learning and are the foundation for recent advances in language modeling and multimodal learning. While data curation for pre-training is often still ad-hoc, one common paradigm is to first collect a massive pool of data from the Web and then filter this candidate pool down to an actual training set via various heuristics. In this work, we study the problem of learning a *data filtering network* (DFN) for this second step  of filtering a large uncurated dataset. Our key finding is that the quality of a network for filtering is distinct from its performance on downstream tasks: for instance, a model that performs well on ImageNet can yield worse training sets than a model with low ImageNet accuracy that is trained on a small amount of high-quality data. Based on our insights, we construct new data filtering networks that induce state-of-the-art image-text datasets. Specifically, our best performing dataset DFN-5B enables us to train state-of-the-art models for their compute budgets: among other improvements on a variety of tasks, a ViT-H trained on our dataset achieves 83.0% zero-shot transfer accuracy on ImageNet, out-performing larger models trained on other datasets such as LAION-2B, DataComp-1B, or OpenAI’s WIT. In order to facilitate further research in dataset design, we also release a new 2 billion example dataset DFN-2B and show that high performance data filtering networks can be trained from scratch using only publicly available data.

Published as a conference paper at ICLR 2024
DATA FILTERING NETWORKS
Alex Fang*1.2
Albin Madappally Jose
Amit Jain'
Ludwig Schmidt2 Alexander Toshey! Apple University of Washington
Vaishaal Shankar'
ABSTRACT
Large training sets have become cornerstone of machine learning and are the foundation for recent advances in language modeling and multimodal learning. While data curation for pre-training is often still ad-hoc, one common paradigm is to first collect a massive pool of data from the Web and then filter this candidate pool down to actual training set via various heuristics. In this work, we study the problem of learning a data filtering network (DFN) for this second step filtering large uncurated dataset. Our key finding is that the quality of network for filtering is distinct from its performance on downstream tasks: for instance, a model that performs well on TetgeNet can yield worse training sets than model with low ImageNet accuracy that is trained on small amount of high-quality data. Based on our insights, we construct new data filtering networks that induce state- of of-the-art art image-text datasets. Specifically, our best performing dataset DFN-5B enables us to train state-of- of-th-of- CLIP models for their compute budgets: among other improvements on variety of tasks, a ViT-H trained on our dataset achieves 84.4% zero-shot transfer accuracy on ImageNet, out-perforning models trained on other datasets such as LAION-2B, DataComp-1B, or OpenAl's WIT. In order to facilitate further research in dataset design, we also release a new 2 billion example dataset DFN-2B and show that high performance data filtering networks can be trained from scratch using only publicly available data.
INTRODUCTION
Carefully curated datasets have driven progress in machine learning for decades, from early pattern recognition experiments in Bell Labs recent developments like GPT-4 Stable Diffusion, and CLIP
(Highleyman & Kamentsky 1959,
LeCun et al. 1989 1998 Deng et al. 2009; Krizhevsky et al.)
2009 2012] Radford et al.) 2019 2021: [0222] OpenAl 2023 3. Despite their crucial role, datasets
themselves are rarely the subject of active research (Sambasivan et 2021).
Current approaches to improving performance on machine learning tasks have focused on scaling model capacity or training data volume. While scaling laws (Hestness et al.) 2017: Kaplan et al., 2020 Aghajanyan et al. 2023] Cherti al. (023) have elucidated the relationship between model size, data size, and performance, little formal guidance exists how to scale these quantities. On the model side, experimentation is straightforward with enough compute, permutations of width, depth, normalization and training hyperparameters can be rigorously evaluated, leading to consistent modeling improvements over the years (Touvron et al. 2023a Elsen et al. 2023).
The dataset side is unfortunately murkier. Most large-scale training sets not released, leaving the community to attempt open reproductions (Schuhmann et al., 2021 2022] Byeon et al.) 2022
Gao et al. 2020); however, these often one-off efforts without the iterative refinement that mod- els enjoy. Recent efforts like DataPerf, DataComp and MetaCLIP (Mazumder et al., 2022 Gadre
al., 2023 Xu et al. 2023) help bridge the gap by providing consistent dataset evaluation and
reproduction frameworks.
We argue dataset design leverage the same tools model design. Almost all large-scale dataset construction can be broken down into two phases: uncurated data collection and dataset filtering. We focus our work on the latter, with the assumption that large uncurated dataset exists. We show data filtering networks (DFNs) neural networks designed to filter data- can induce massive, high-quality pre-training datasets. Unlike previous techniques relying on domain-speciff heuristics,
Work done while Apple

---

Published as a conference paper at ICLR 2024
Accuracy vs Compute Tradeoff
ImageNet Zero Shat Transfer Accuracy
85
80
75
Datasel
Model
DC-18
VIT-B/16
DFN
VIT-L/14
VIT-G/14
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2,00
2.25
2.50
2.75
3.00
3.25
3.90
Total Training GMACS
Figure 1: Compute scaling behavior of training CLIP models on various datasets. DFN-2B, the sub- of CommonPool (DataComp-1223)) 12.8B) chosen by best performing data filtering networks, out- performs all other datasets including OpenAl's WIT and the previous state-of-the-ar CLIP training dataset DataComp-
Our ViT-L outperforms ViT-G trained on LAION with 18 x more compute. Similarly, our ViT-B/16 outperforms OpenAl's ViT-L/14 trained with 4x more compute. Our ViT- H/14 achieves 84. 4.4%% on ImageNet, per-perforning any model in its compute class. All DFN-trained models were trained DFN-2B, except for the VīT-H which was trained on DFN-5B. Both datasets were induced by the same DFN. note cost training DFN was omitted from this plot, which corresponds to less than 50 th of total CLIP training cost.
DFNs paired with a large unfiltered image-text pool produce billion-scale sate-offthe-ee datasets algorithmically. We demonstrate DFNs can be efficiently trained from scratch and improved with the same techniques standard ML models.
The contributions of this work are as follows. First, we characterize the properties of data filtering networks that lead to high-quality datasets. We ablate properties of data filtering networks from supervision signal to training data quality. We find that a small contrastive image-text model trained on only high-quality data is sufficient construct tate--ofttheaat datasets.
Second, we use these properties to train DFNs and construct datasets that induce Contrastive Image- Text Pre-trained (CLIP) models that achieve high accuracy and present better compute accuracy tradeoff than any existing dataset in the literature as show in Figure VIT-L/14 or 12.8B examples seen our DFN induced dataset DFN-28 -B to 81.4 ImageNet zero- shot transfer accuracy, outperforming the previous best ViT-L trained on DataComp- 1B by over percentage points. We further train ViT-H/14 ageNet zero-shot transfer accuracy. We show that models trained on these datasets show consistent improvements on many tasks, including zero-shot classification, retrieval, and visual question an- swering, and maintain the favorable robustness properties of CLIP models.
In particular we train a
larger DFN induced dataset DFN-5B 84.4 Im-
Lastly, the above insights can be used recipe construct high-quality datasets from scratch by using only public data thus making strides towards democratization of large high-qualig datasets. In addition, we release DFN-2B for the community enable research on large image-text models.
BACKGROUND AND RELATED WORK
2.1 CONTRASTIVE IMAGE LANGUAGE PRE- TRE-TRIINING (CLIP)
CLIP has altered the of cheaply available image-alt-text datasets by demonstrating the practical- ity of large-scale training on web-scraped image-text pairs to build state-of-the-art image represen- tations. CLIP consists of separate vision and text encoders, and uses contrastive loss during training to push the representations of related images and text pairs together, and unrelated pairs apart. Cru-
Most large public image-text datasets including LATON-SB and DataComp- IB are built using OpenAl's CLIP model
2

---

Published as a conference paper at ICLR 2024
cial to this process is large dataset of aligned image-text pairs images paired with semantically relevant text. The release of CLIP was followed by several other image-text models such as ALIGN, BASIC, LiT and Open-CLene -LIP all of which will refer to in this work as CLIP models (Jia et al.
2021 Pham et al., 2023 Zhai et al. 2022b IIharco et al. 2021). CLIP models generally come in 3 canonical sizes of vision transformer: VIT-B/32, VIT-B/16 and ViT-L/144; since then, the open source community has extended these to 3 larger variants ViT-H/14, ViT-g/14 and ViT-G/14 (Dosovitskiy) 2022a). Generally the larger models exhibit better zero-shot generalization and transfer properties. CLIP models have been trained on variety of datasets from OpenAl's WiT, Google's 's WebLI and JFT-3B, LAION, COYO and DataComp-12
(tt al., 2020; Zhai al.,
Prior work has also studied how to fine-tune CLIP models to improve performance in targeted direc- tions. CLIP models can fine-tuned on image classification tasks by using templates to transform labels to text (Fang et al., 2022 Goyal et al., 2022). Additionally, practitioners often use weight ensembling to preserve robustness properties of the pre-trained model while reaping the benefits of fine-tuning (Wortsman et al. 2022). We take advantage of these techniques in order to improve the filtering models we train in this work.
2.2
DATASET CONSTRUCTION
Prior to CLIP, datasets most commonly used in computer vision were supervised with human labels (Deng et al. 2009 Krizhevsky et al. 2009). Though these older dataset construction pipelines were quite intricate and did not scale beyond few million examples, they share some similarity with current constructions. Classical datasets such as ImageNet and CIFAR started with a large roughly curated pool of images paired with metadata, and used humans to either label or filter the data.
Modern dataset pipelines have a similar procedure but at a much higher scale. The initial pool of images can contain up to 100 billion images, and the dataset filtering is purely automated, often with a set of rules and heuristic filtering stages (Jia et al.] (021). Past work in natural language processing has used binary filters as an initial step to remove low quality documents (Wenzek al.) 2019 Brown et al. 2020). but contain multiple components to their filtering pipelines.
One of the first publicly available web- scale image-text datasets is LAION. LAION-400M and LAION-2B were constructed by collecting image-text pairs from Common Crawl, filtering by En- glish, and keeping pairs whose image and text are well aligned. This alignment is performed using procedure known as CLIP filtering, which uses existing image-text model (in LAION's case OpenAl CLIP ViT-B/32), and removes samples whose cosine similarity between image and text are below some threshold. We show pseudocode of the basic CLIP filtering operation in Appendix HH
While CLIP filtering is convenient it is dependent on a existing trained CLIP model, and perhaps limited on the top-line performance of any model trained using it as filter. For example, despite LAION-2B being five times larger than OpenAl's dataset, models trained on it could only match OpenAl's ImageNet zero-shot performance with significantly larger compute budget.
To better facilitate the study of image-text datasets, researchers created the DataComp benchmark (Gadre al. 2023). The benchmark provides 12.8 billion image-text pairs from Common Crawl so that researchers can study the effect of various data filtering techniques. DataComp fixes the computational budget used to train the resulting models, fixing the compute budget of the largest scale to match the cost of training OpenAl's ViT-L/14 CLIP model. These models are then evaluated on suite of 38 downstream tasks, which includes ImageNet and distribution shifts, VTABB and retrieval tasks. We use this benchmark as our primary method of evaluating the datasets created by our data filtering networks.
The authors of DataComp also released a baseline dataset, DataComp-1B 1B (DC-1B) that improved upon LAION-5B, by combining CLIP filtering with an ImageNet based clustering approach to im- prove dataset quality on variety of benchmarks. However this dataset still relies the OpenAI CLIP model for CLIP filtering and imposes a costly ImageNet specific clustering step in the pipeline.
Recent work (Xu et al.) 2023) has demystified the CLIP dataset construction process and demon- stracted high quality dataset construction is possible by simple keyword based sampling and global balancing. While their work does create competitive datasets, the reliance sampling heuristics from the original CLIP paper (Radford et al., 202T) allows for accurate dataset reproduction, our work focuses on improving model performance using dataset construction.
3

---

Published as a conference paper at ICLR 2024
1. Train
2. Filter
3. 3.Train
HQ Data
DFN
Uncurated Dota
DFN Induced Dataset
Final Network
Figure 2: high level overview of our pipeline for constructing datasets using DFNs
DATA FILTERING NETWORKS
The core object we study in this work a data filtering network (DFN). this section we define DFNs and introduce their evaluation setup.
3.1 DEFINITIONS
Since our ultimate goal is to build functions that filter potentially trillions examples efficiently, restrict the scope of our study DFNs that only applied pointwise to elements of larger data pool. Thus, processing data pool with a DFN, defined in pseudocode as follows
def apply_dfn (dfn, data_pool): return [x for in data_pool if (x)
lends itself to parallelization and thus efficient application. For a given DFN and pool, we refer to the data pool we train the DFN on as a filter dataset. Furthermore, we refer the dataset constructed by filtering the pool with the DFN the induced dataset. We then refer to a model trained (only) on that dataset the induced model.
As introduced in Section 2.2 a common choice for DFN is CLIP trained image-text model. Thus, DFN can not only be used induce dataset but also be applied common evaluation problems such zero-shot ImageNet classification. Inversely, CLIP model can be both used for general recognition as well as as a DFN. When we use a CLIP model as DFN, we define its fil- tering performance the performance of the induced model, evaluated on standard benchmarks, e.g. ImageNet top-1.
3.2 DATA FILTERING NETWORKS EVALUATION SETUP
address how evaluate DFNs. In our context, the quality of a DFN is determined by the strength of models it can induce. We build on the evaluation framework proposed by DataComp (Gadre al. 2023). DataComp provides multi-scale evaluation frame- work for datasets by measuring CLIP model zero-shot performance. It provides 4 nested unfiltered image-text pair pools of increasing size. this work, use the medium (128M datapoints), large (1.28B datapoints) and xlarge( I2.8B datapoints) pools. We also follow the DataComp guidelines of model hyperparameters for each of these pools, which are ViT-B/32 for medium, VIT-B/16 for large and ViT-L/14 for XL. Exact hyperparameters can be found in Table 6 We additionally expand our DFN to a larger pool of 42B images by combining 30B non-DataComp web-scraped images with the DataComp XL pool. We denote the dataset induced using this pool and our DFN as DFN-5B, which we use to train ViT-H/14 model.
With these definitions in place,
For evaluation use 38 zero-shot classification and retrieval tasks in the DataComp benchmark. We denote the average performance on these benchmarks simply as "Average" performance, but we also track various subsets: ImageNet performance (IN), ImageNet distribution shift performance (IN shifts), Visual Task Adapation Benchmark (VTAB), Retrieval performance (COCO, Flickr, Wino- GAViL).
Our actual training runs on both Nvidia A100s and TPU v4s. We use OpenClip and AXleamn to train CL.P models on GPUs and TPUs respectively (Ilharco et al. 2021 Apple 2023
4

---

Published as a conference paper at ICLR 2024
Model Performance Filtering Performance
Dataset
network trained on the fiitered data
SCI2K+CC3M-SS
ImageNet accuracy (top- 1) of the
HOTTP 33M (Ours)
HQTTP-350M (Ours)
LAION-28
OpenAU WITT000
CommonPool
DataComp-18
Steps
10
0.26
38
138
0.24
Model
VIT-8/32
VIT-/14
0.22
0.45
0.50
0.53
0.60
0.65
0.70
0.75
0.80
ImageNet accuracy (top- filtering network
Figure 3: Filtering strength is uncorrelated with image task performance. The models are trained using CLIP, and the number of samples seen and the training data are displayed on the right hand side. Filtering performance is measured by filtering on DataComp medium.
3.3 UNDERSTANDING DATA FILTERING NETWORKS
As open source CLIP-models improve standard vision metrics such ImageNet, the question arises whether we can replace the OpenAl CLIP model used in the dataset construction process with one of these better models. We even hope of recursively applying this process continuously train better models that can be used as better filtering models that once again yield even better models. Unfortunately, this does seem be true. Figure 3 shows that ImageNet performance of CLIP models is not correlated with filtering performance. To measure filtering performance, we create dataset by using the CLIP model apply CLIP filtering on DataComp's medium raw pool, and measure ImageNet performance of models trained on the induced dataset. It is especially striking that model with 30% less ImageNet performance than OpenAl's CLIP models can be as good when used filtering model.
We find that data quality is key to training good filtering models. demonstrate this, we start with high-quality pool of 10 million samples from Conceptual 12M (CC12M), and gradually replace with unfiltered data from Common Crawl until this pool only contains Common Crawl. We train DFNs these data mixes, and use these DFNs to CLIP filter separate pool of 128 million Common Crawl samples from DataComp's 's medium scale. In Figure 4 we measure the ImageNet performance of both the DFNs and the induced models trained on datasets generated by each of the DFNs. While the ImageNet performance of the DFNs degrade steadily as they are trained on larger fractions of filtering networks decreases immediately when the high-quality
unfiltered data, their performance pool is "poisoned" with even a small portion of unfiltered data. Once the filtering training pool is poisoned, the dataset induced by the DFN is only slightly better than unfiltered data.
Table 1: Filtering Performance of various filtering models, after filtering DataComp medium scale (ViT-B/32, 128M samples seen). We present results on ImageNet top-1 as well as "Average" set of tasks (see Sec.
3.2 for details.)
DFN Type
Filter Dataset
ImageNet
Average
No Filter Baseline
None
0.176
0.258
ResNet-34 Image Binary Filter
ImageNet
0.242
0.292
OpenAl ViT-B/32 Image Binary Filter
ImageNet
0.266
0.295
ResNet-34 Image Binary Filter
CC12M
0.203
0.257
OpenAl ViT-B/32 Image Binary Filter
CC12M
0.218
0.276
MBAE VIT-B/16
CC12M
0.237
0.297
CLIP ViT-B/32
CC12M
0.289
0.335
Next, we explore using filtering models beyond CLIP models. While DFNs can use any model that be reduced binary function, intuitively it makes sense CLIP models. By filtering with
5

---

Published as a conference paper at ICLR 2024
Effect of Data Quality on Model versus Filtering Performance
0.30
ImageNet accuracy (top-11
0.25
0.20
Model Performance
Filtering Induced Model Performance
DataComp Medium Filter Baseline
0.35
0.30
0.2
0.6
1.0
Poisoning
Figure 4: Data quality determines the filtering performance of models. We create these filter training datasets of various quality by having set pool size of 10 million samples, and interpolating between CC-12M (high quality) and CommonPool (low quality). We then train models induced by the DFN filtering DataComp medium.
a similarity score between the image and text, we encourage keeping samples where the image and text are aligned.
In order to verify this intuition we consider a few other options to produce a DFN. One is to train a binary classifier that can distinguish between ImageNet or CC12M data as positives and Common Crawl as negatives. We consider both ResNet (He et al., 2016) as well as frozen OpenAl CLIP embeddings for this filter. Another option is to use M3AE (Geng et al. 2022) trained on CC12M as a DFN that takes into account both images and text. We can use reconstruction loss as the filtering criterion, as it is reasonable proxy for how similar samples are the high-quality data used train the filtering model.
The filtering performance of all these options, including CLIP models, are summarized in Table I where the CL.IP model outperform the other backbones. A key difference between the binary classi- fier and CLIP filters that the binary filter makes an explicit assumption on what qualifies as good distribution, while CLIP filters more flexible. Although the M3AE and CLIP filtering models both are trained on CC12M and examine both modalities, M3AE performs much worse, potentially due combination of CLIP encouraging image-text alignment and the difficulty of text recon- struction from just CC12M. We conclude that CLIP models are the most practical and performant models for image-text DFNs.
CREATING BETTER DATA FILTERING NETWORKS
Equipped with better understanding of CLIP models data filtering networks, aim create better data filtering networks. DFNs can be trained and modified the same ways as standard machine learning models. We start by training CLIP model we can fine-tune the filtering network on subsequent datasets that we want to do especially well on. We use weight ensembling reduce overfitting on the fine-tuned datasets. Standard machine learning techniques such as augmentation, using a different initialization, and training for more steps with larger batch size seem to improve the filtering model. We demonstrate the effect of these interventions in Table On the other hand, using a different model size seems to have limited benefits. while model ensembling increases filtering costs without bringing gains. Compared to previous datasets such as DataComp-11 1B (DC-1B) which involved combining CLIPP filtering with clustering- based heuristics, DFNs simplify the data filtering process into single pipeline while also reducing computational costs.
high-quality dataset, and then
To create our best DFN, we train a VIT-B/32 CLIP model on High-Quality Image-Text Pairs (HQITP-350M), which is high-quality dataset of 357 million image-text samples with human- verified captions. This dataset similar to the HQITP-135M used in Ranasinghe et al. (2023), but expanded to 357M examples. We initialize the weights with OpenAl's checkpoint. We then fine- tune on the combined MS COCO training set, Flickr30k training set, and ImageNet 1k with OpenAI templates as the captions. We additional augmentation at both training and fine-tuning time.

---

Published as a conference paper at ICLR 2024
Table 2: Standard interventions used to improve models can be used on DFNs to induce stronger datasets, leading to better models. DFNs are used to filter and train DataComp large (ViT-B/16, 1.28B samples seen).
Intervention
IN
IN Shifts
VTAB
Retrieval
Average
0.620
0.493
0.534
0.515
0.536
Augmentation
V
0.626
0.501
0.534
0.516
0.542
2.56B
4096
0.626
0.506
0.536
0.511
0.545
Samples Seen Batch Size
5.12B / 8192
0.624
0.508
0.551
0.517
0.550
x
0.624
0.508
0.551
0.517
0.550
Fine-tune
/
0.678
0.540
0.555
0.534
0.560
0.674
0.535
0.533
0.529
0.548
OAI-Init
0.678
0.540
0.555
0.534
0.560
Table 3: Training on DFN-2B produces state-of-the-art CLIP models. Here we evaluate on the
DataComp benchmark, comparing against LAION-2B, DC-1B, MetaCLIP and OpenAl WIT-400M. Additional comparisons can be found on the DataComp leaderboard.
Dataset
DataComp Scale
IN
IN Shifts
VTAB
Retrieval
Average
DC-IB
medium
0.297
0.239
0.346
0.231
0.328
DFN-2B
medium
0.371
0.298
0.388
0.288
0.373
DC-1B
large
0.631
0,508
0.546
0.498
0.537
DFN-2B
large
0.678
0.540
0.555
0.534
0.560
LAION-2B
xlarge
0.731
0.603
0.586
0.589
0.601
OpenAl WIT-400M
xlarge
0.755
0.649
0.586
0.543
0.617
DC-1B
xlarge
0.792
0.679
0.652
0.608
0.663
DFN-2B
xlarge
0.814
0.688
0.656
0.649
0.669
LAION-2B
ViT-G/14-224px
0.801
0.691
0.646
0.635
0.667
DC- IB (CLIPA-V2)
ViT-G/14-224px
0.831
0.740
0.645
0.631
0.684
MctaCLIP
ViT-H/14-336px
0.805
0.700
0.640
0.652
0.667
WebLI
ViT-SO/400M-3844p
0.831
0.734
0.648
0.698
0.692
DFN-2B
ViT-L/14-224px
0.822
0.679
0.664
0.666
0.678
DFN-5B
ViT-H/14-224px
0.834
0.713
0.675
0.684
0.698
DFN-5B
ViT-H/14-378px
0.844
0.738
0.685
0.695
0.710
Additional training details can be found in Appendix B We create dataset DFN-2B by apply- ing this DFN on DataComp's full 12.8 billion sample CommonPool, with a threshold equivalent to taking the top 15% of samples.
Our DFN induces datasets that achieve srate-of-the-ee results on medium, large, and xlarge scales in DataComp. In particular xlarge, train ViT-L/14 on DFN-2B for 12.8B samples seen to achieve 81 .4% zero-shot accuracy on ImageNet, and 0.669 average over 38 DataComp evaluation 3 in terms of ImageNet zero-shot improvement, this is 2.2% im- provement over DC-1B, 5.9% improvement over OpenAl WIT-400M, and a 8.3% improvement over LAION-222 These improvements DataComp evaluation suite in distribution shifts, retrieval, VTABB and average performance. We also train ViT-L/14 on DFN-2B for 39B samples seen. Lastly, we train DFN-5B on a ViT-H/14 for 39B samples seen at 224 x 224 resolution, and 5B samples at 378 x 377 resolution achieving 84.4% zero-shot transfer accuracy on ImageNet, and 0.710 average on the DataComp evaluation suite. We find that models trained our DFN produced datasets outperform all other models on the evaluation suite regardless of pre-traini dataset: MetaClip, WebLI or DataComp- IB (Xu et al. 2023 Zhai et al. 2022a Gadre et al. 2023), archiectural improvements such as -hape-optimized ViTss (Alabdul- mohsin et al. (202). a more performant sigmoid loss (Zhai et al. 2023), or pre- pre-training performance optimizations such those in Li al. (202bb).
datasets. As shown in Table
beyond ImageNet, as we can see similar trends across the

---

Published as a conference paper at ICLR 2024
Table 4: High-quality data is best used to train the filtering model rather than the end model. Training
DFNs with HQITP-350M induces dataset that outperforms the dataset induced by worse DFN combined with SQITP-350M. Additional experiments with ViT-B/16 are in Appendix Table TO
Dataset
Model
IN
IN Shifts
VTAB
Retrieval
Average
OAI ViT-B/32 Induced Dataset
ViT-L/14
0.774
0.654
0.643
0.616
0.654
HQITP-350M
DEN without FT Induced Dataset
ViT-L/14
0.787
0.670
0.654
0.631
0.666
DFN-2B
ViT-L/14
0.814
0.688
0.656
0.649
0.669
DFN-2B HQITP-350M
ViT-L/14
0.813
0.691
0.662
0.656
0.670
Robustness under distribution shift
Average over 5 shifts (top-1. %)
75
ImageNet Classification
Linear fit (ImageNet Classification)
65
CLIP zero-shot
Linear fit (CLIP zero-shot)
Nol
Model trained dataset induced DFN
no
DFN no FT Induced
DFN no FT Induced + IN1k
DFN FT IN1K Retr
DFN on Retr Induced
55
65
75
85
ImageNet (top-1, %)
Figure 5: Datasets induced by DFNs can be robust to distribution shift. DFNs can be fine-tuned to maintain robustness of induced datasets, unlike directly training on ImageNet (IN). DFNs performing distillation because induced datasets lead to higher performing models than the original DFN. Distribution shifts used are IN-V2. ObjectNet,, IN-Sketch, IN-R, and IN-A (Recht al. 2019
Barbu et al.) 2019
Wang et al. 2019
Hendrycks et al. 2021a
Creating better datasets not only improves model performance, but also improves model efficiency. Performance that was once only achievable by larger models can be matched with smaller model trained on a better dataset. Our ViT-L/14 trained on DFN-2B surpasses ViT-G/14 trained on LAION-2B for 34B samples seen by 1.5% zero-shot accuracy on ImageNet, and by 0.002 aver- age performance, despite using 16x less computational cost Similarly, we can train ViT-B/16 on DFN-2B for 12.8B samples seen achieve competitive performance with OpenAl's VIT-I/14, representing a 4x computational cost reduction.
The key training good DFNs is using high-quality data for training the filtering network. Col- lecting verified high-quality data is expensive, as it often requires human annotations, and is thus difficult to scale to large quantities. But given a sizable high-quality dataset, we can explore if there are benefits to directly training it instead of using it to train DFN. In Table we compare models trained on datasets induced by our DFNs with a model trained on HQITP-350M, combined with the dataset induced by CLIP filtering CommonPool with OpenAl's VIT-B/32 Models trained DFN induced datasets outperform the baseline on all major categories within the DataComp evaluation suite. Furthermore, training the combination of HQITP-350M and DFN-2B seems have little improvement when compared to just training on DFN-2B. By training a DFN instead of directly training on high-quality data, we demonstrate a successful recipe for leveraging high-quality data for creating large-scale high-quality datasets.
We can also explore the differences between fine-tuning DFN and directly training on the fine- tuning dataset. In Figure 55 and Table 7 we compare models trained on a dataset induced by baseline DFN, dataset induced by the baseline DFN fine-tuned on ImageNet and a dataset induced by the baseline DFN without fine-tuning on ImageNet combined with ImageNet. While the model
ccllullation does not take into account patch dropout used to train VIT-G/14 on LAION-2B
8

---

Published as a conference paper at ICLR 2024
that directly trains on ImageNet has much higher performance on ImageNet and ImageNet- V2, it does not improve upon the baseline for the ObjectNet, ImageNet-Skettde and ImageNet-R. On the other hand, the DFN fine-tuned on ImageNet induces dataset that improves over the baseline ImageNet and all of its distribution shifts. Fine-tuning on DFNs acts as a regularizer to induce datasets similar the fine-tuning dataset, while maintaining strong robustness properties that come with drawing from a more distributionally diverse candidate pool.
4.1 BETTER DATASETS BEYOND VISION TASKS: VQA
Just like how machine learning models would ideally generalize across many tasks, would also like our datasets to generalize across diverse tasks. We show that our datasets not only lead to better models when evaluated on vision tasks, but also lead to better visual question answering (VQA) models. We train a BLIP2 model (Li et al. (2233a) which takes as input CLIP visual encoder and is trained for zero-shot VQA on COCO and Visual Genome, measure zero-shot VQA performance on VQVA2, GQA, and OK VKA (Goyal et al. 20117] Hudson & Manning Marino et al. 2019). We compare the performance on BLIP2 between the standard OpenAl ViT-L visual encoder and the ViT-L trained on DFN-2B. The DFN-2B model consistently outperforms the OpenAl VIT-L encoder and is competitive with a much larger EVA ViT-g model trained on LAION-2H
Table 5: Performance of BLIP-2 variants with different visual encoder training datasets. The DFN- 2B trained ViT-L provides consistent improvements across multiple zero-shot VQA tasks.
Visual Encoder Training Dataset
Architecture
VQAv2 Acc. (%)
GQA Acc. (%)
OKVQ Acc. (%)
OAI-WIT-400M
ViT-L
45.5
30.0
19.1
DFN-2B
ViT-L
48.3
31.3
21.9
LAION-2B
ViT-g
48.7
31.1
24.5
4.2 PUBLICLY REPRODUCIBLE DENS
Scientific research benefits from results that be reproduced by anyone from scratch. Though OpenAl's internal dataset and HQITP-350M are not publicly accessible, we demonstrate that a com- public data sources. We train ViT-B/32 on Conceptual Caption 12M, Conceptual Captions 3M. and Shutterstock 15M (Changpinyo et al. 2021 Sharma et al. 2018 Nguyen al. 2023). As shown in Appendix Table II]. this DEN surpasses OpenAl's ViT-B/32 in terms of filtering performance at DataComp's 's medium, large, and xlarge scales. Additionally, this DFN can modified as described in the previous section further improve filtering performance.
petitive DFN can be trained
5 DISCUSSION
The simplicity of the data filtering network pipeline makes it flexible tool integrate into existing workflows. As DFNs operates on individual samples, this approach scales linearly with candidate pool size, enabling the creation of datasets orders of magnitude larger than those that introduce in this work. Additionally, the DFNs we train in this work are relatively small neural networks which allows for filtering be directly integrated into training procedures of much larger networks for minimal marginal cost. DFNs can then filter batches of raw data that are then trained on, reducing the need for complex data pre-pree pre-procsssing procedures.
As useful as DFNs are in building performant models in this work, there are still many unanswered questions address in future work. We still do not know exactly how optimize directly for dataset quality, and thus opt for weak proxies such as alignment. It is not even clear what that proxy would be for other domains where DFNs could be applied such as speech, text or video data. We hope that these open questions and the bridge DFNs build between modeling work and dataset work can lead to fruitful new avenues of research.
PEVA's ViT-g has additional pre-training procedure trained on ImageNet-21k, COCO, Objects365 and Conceptual Captions 12M

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We would like to thank Bowen Zhang, Ruoming Pang, Brandon McKinzie, Mitchell Wortsman, Gabriel Ilharco, Ross Wightman, Achal Dave, Josh Susskind, Alaaeldin Ali, Fartash Faghri, Preetum Nakkiran, and Chen Huang for helpful feedback at various stages of the project.
Bowen and Ruoming were invaluable for helping us up AxLearn and answering countless ques- tions when we ran into various errors. Brandon pointed us in the direction of the HQITP datasets that were crucial for our best results, and also helped us with AxLearn issues. Mitchell's s OpenClip experience helped us set hyper-parameters for our largest scale runs. Gabriel helped debug webdataset related dataloader bug. Ross caught bug in our final high resolution model that led to modest performance improvement. Achal provided prrrameprrmeters and instructions for the BLIP2-based VQA experiments. Alaaeldin, Chen, Fartash, Josh, and Preetum provided helpful comments on the manuscript.
REFERENCES
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for gen- erative mixed-modal language models. arXiv preprint arXiv:2301.03722 2023.
Ibrahim Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting vit in shape: Scaling laws for compute-optimal model design. arXiv preprint arXiv:2305. 2023.
Apple. axlearn, July 2023.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the lim- its of object recognition models. In H. Wallach, H. Larochllle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems (NewrIPS), vol- ume 32. Curran Associates, Inc., 2019.
www.reerrsp... wpw.rrrpss.oom wttpppcccccpaa.. capear Paper.pdr
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Minwoo Bycon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Sachoon Kim. Coyo-700m: Image-text pair dataset. htttp://github.co com/ kat.kkaaaaabaa/a/aaaa
coyo-dataset] 2022.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3568-556,, 2021.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-imant learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-2829 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Puttern Recognition (CVPR), 2009. https::
ieee. org wttp://///..aaaaeee///eee///e//e
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. /929, 2020.
Erich Elsen, Augustus Odena, Maxwell Nye, Sagnak Taşıllar, Tri Dao. Curtis Hawthorne, Deepak Moparthi, and Arushi Somani. Releasing Persimmon- -8B 2023, URL https: / /www. adept. a ai/blog/pes pers mmon 8b)
10

---

Published as a conference paper at ICLR 2024
Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre- training (clip), 2022.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint urXiv:2304. 14108, 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling, arXiv preprint arXiv:2101.0002 2020.
Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurams, Sergey Levine, and Pieter Abbeel. Multimodal masked autoencoders learn transferable representations. arXiv preprint arXiv:2205. 14204, 2022.
Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models, 2022.
Yash Goyal, Tejas Khot, Douglas Summers-Stur Dhruv Batra, and Devi Parikh. Making the v vin vqa matter: Elevating the role of image understanding in visual question answering. In Prrrre Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
https://arxiv.cm 21tp://wwww...raaa/111.2.2..2.. 03385
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of cittrf-ddistrbbttion generalization. ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Conference Computer Vision and Pattern Recognition (CVPR), 2021b.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.
W. H. Highleyman and L. A. Kamentsky. A generalized scanner for pattern- and character- recognition studies. In Papers Presented at the the March 3-5, /959, Western Joint Computer Conference, IRE-AIEE-ACCE ACM '59 (Western), pp. 291-294, New York, NY. USA, 1959, Associ- ation for Computing Machinery. ISBN 9781450378659, doi: 10.1145/1457838. 1457894. URL /ttts::/doi..rog 4// 1457894
Drew A. Hudson and Christopher D. Manning, Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi. Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. URL https: /twp///c.i// org wtt.:////............... 5281/ Zzeodoo. 5143773
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision, 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint urXiv:2001.08361 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009. https:/ //www//www//////////////////. es. toronto. >xii//eeaniig--etturee.. 2009- TR. pdf

---

Published as a conference paper at ICLR 2024
Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2012. https:/
co/ppp Filees/paerr/2222/11ee
c3 c339862d3b9d 2ew.2//0c22cob666666664444422222 Paper pdff
Y. LeCun, Boser, S. Denker, Henderson, R. E. Howard, W. Hubbard. and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4):541- 551, 1989. doi: 10.1162/neco. 198.tecc.....9.... 1989.1.4.544
Yann LeCun, Yann LeCun, and Yann LeCun. The mnist database of handwritten digits, 1998. http://yann lecun. com/ wxdb/mmistt//////omemmc
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023a.
Xianhang Li, Zeyu Wang, and Cihang Xie. Clipa-v2: Scaling clip training with 81. laccuracy within $10,000 budget; an extra $4,000 unlocks $1.82023b.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Mark Mazumder, Colby Banbury, Xiaozhe Yao, Bojan Karlaš, William Gaviria Rojas, Sudnya Di- amos. Greg Diamos, Lymn He, Douwe Kiela. David Jurado, David Kanter, Rafael Mosquera, Juan Ciro, Lora Aroyo, Bilge Acun, Sabri Eyuboglu, Amirata Ghorbani, Emmett Goodman, Tariq Kane, Christine R. Kirkpatrick, Tzu-Sheng Kuo, Jonas Mueller, Tristan Thrush, Joaquin Vanschoren, Margaret Warren, Adina Williams, Serena Yeung, Newsha Ardalani, Praveen Par- itosh, Ce Zhang, James Zou, Carole-Jean Wu, Cody Coleman, Andrew Ng, Peter Mattson, and Vijay Janapa Reddi. Dataperf: Benchmarks for data-centric ai development, 2022. https:
//axxiv.or//aaa abw 22200000622
Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip, 2023.
OpenAl. Gpt-4 technical report, 2023.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, Mingxing Tan, and Quoc V. Le. Combined scaling for zero-shot transfer learning, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh. Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. Interna- tional Conference on Machine Learning (ICMI), 2021. /cttss///axxiv.orrg/b/////
00C20
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356, 2022.
Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander Toshev, and Jonathon Shlens. Perceptual grouping in contrastive vasion-language models, 2023.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classi- fiers generalize to ImageNet? In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Interna- tional Conference on Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pp. $389-5400, PMLR, 09-15 Jun 2019. http: http:///oocceeiigg..o mlr.
.tt..rress.././/mec.m.t...... press/
w97/recht 1 9a. html
12

---

Published as a conference paper at ICLR 2024
Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. "everyone wants to do the model work, not the data work". Data cascades in high-stakes ai. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-13, 2021.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M Open dataset of clip-filtered 400 million image-text pairs, 2021. https://arxiv.orr
http://aaxvv.v. wtt.://rrggabbbbl..../
02114
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmar- czyk, and Jenia Jitsev. LAION-5B: An open large-scale dataset for training next genera- tion image-text models. In Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2022. https: fttps:///penzeviers net www.:nettfforommm 1d Cd-M3Y74vmMcc
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings ACI., 2018.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykow. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Advances in Neural Information Processing Systems (NeurIPS), pp. 10506-10518 2019.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Conet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv: 19100399, 2019.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael GontijoLoppss Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models, 2022.
Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data, 2023.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers, 2022a.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. 2022b.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, 2023.
13

---

Published as a conference paper at ICLR 2024
A TRAINING HYPERPARAMETERS
Table 6: We follow the hyperparameter settings of the DataComp paper for the medium, large and
xlarge pool
Dataset
Model
Pool size and # seen samples
Batch Size
Max LR
Weight Decay
Warmup
Beta2
DataComp-medium
ViT-B/32
128M
4096
5e-4
0.2
500
Dat
ViT-B/16
1.28B
8192
5e-4
0.2
500
DataComp-xlarge
ViT-L/14
12.8B
90112
le-3
0.2
10000
0,95
DFN-5B-pool
ViT-H/14
39B
79872
2e-3
0.1
10000
0.95
B DFN HYPERPARAMETERS
DFNs trained for ablations use DataComp large scale hyperparameters with a ViT-B/32 instead of ViT-B/16. Final DFNs that induce DC-2B train for 5.12B samples, 16,384 batch size, and 2,000 steps of warmup.
C ROBUSTNESS OF USING IMAGENET AT FILTERING VS. TRAINING TIME
Table Fine-tuning a DFN on ImageNet induces datasets with nice robustness properties that are lost when directly training on ImageNet. Ran at DataComp large scale (ViT-B/16, 1.28B samples).
Dataset
IN
IN-V2
ObjectNet
IN-Sketch
IN-R
IN-A
VTAB
Baseline DFN
0.624
0.547
0.511
0.510
0.724
0.257
0.551
Baseline DFN FT on ImageNet
0.678
0.594
0.536
0.536
0.743
0.284
0.555
Baseline DFN IN
0.757
0.652
0.509
0.512
0.703
0.272
0.543
D FULL EXPERIMENTAL EVALUATION & MODEL RELEASE
Below we provide links to checkpoints and detailed evaluation results of models in Table on each of the 38 DataComp evaluation datasets
Model Link
ImageNet
Average
DENSB-CLIP-VT-H--144
0.844
0,710
DINSD-CLIP-VI-H--
0.834
0.698
CLIP--CCII-VILLLIII
0.814
0.669
DFNZB-CLIP- V-1-B-16
0.762
0.609
Table 8: Links to checkpoints and detailed evaluation results
14

---

Published as a conference paper at ICLR 2024
E
FIGURES MEASURING AVERAGE PERFORMANCE INSTEAD OF IMAGENET
Model Performance vs. Filtering Performance
0.36
Dataset
network trained on the fitered data
CC12K+CC3M-SS13
HQTTP-135M (Ours)
Average performance of the
0.34
HQTTP-350M (Ours)
OpenAA
OpenAI-WITADON WIT40OM
0.32
Commonnuul
DstaComp-18
Steps
0.30
18
58
0.20
34B
Model
0.26
VIT-8/32
0.55 Average performance of the filtering network
0.45
0.65
Figure 6: Average accuracy version of Figure
Effect Data Quality Model versus Filtering Performance
0.34
0.30
0.28
Model Performance
Filtering Induced Model Performance
0.24
DataComp Medium No Filter Baseline
0.20
Peisoning Ratio
Figure 7: Average accuracy version of Figure
15

---

Published as a conference paper at ICLR 2024
F
LOG-SCALE PLOT OF FIGURE
Accuracy vs Compute Tradeoff
ImageNet Zero- Shof Transfer Accuracy
75
Datasel
Model
LAION 28
B/32
DC-1B
VIT-B/16
DFN
VIT-U/14
GAI-WIT-40OM
VIT:H/14
83
VIT-G/14
10
10
10"
Total Training GMACS (Log Scale)
Figure 8: Compute scaling behavior of training CLIP models various datasets (log scale)
G ADDITIONAL TABLES
Table 9: We can produce high-quality DFNs completely from scratch. Specifically, we do not use any OpenAI CLIP models for results in this table. We also use HQITP-135M for DFN training, subset of HQRTP-350M that we use in the rest of the paper
DataComp
Dataset
Shifts
VTAB
Retrieval
Average
Scale
Induced by DEN HQITP-135M with FT on INIk, no OAI-Init, no Aug.. samples IB, BS 4096
xlarge
0,805
0.665
0.641
0.639
0.663
Table 10: High-quality data is best used train the filtering model rather than the end model. Training DFNs with HQRTP-350M induces a dataset that outperforms the dataset induced by worse DFN combined with HQITP-350M.
Dataset
Model
IN
IN Shifts
VTAB
Retrieval
Average
OAI VIT-B/32 Induced Dataset
VIT-B/16
0.706
0.572
0.582
0.575
0.596
HQITP-35H -35M
DEN without FT Induced Dataset
VIT-B/16
0.729
0.599
0.604
0.597
0.612
DFN-2B
VIT-B/16
0.762
0.623
0.598
0.611
0.609
OAI VIT-B/32 Induced Dataset
VIT-L/14
0.774
0.654
0.643
0.616
0.654
+ HQITP-350M
DEN without FT Induced Dataset
ViT-L/14
0.787
0.670
0.654
0.631
0.666
DFN-2B
VIT-L/14
0.814
0.688
0.656
0.649
0.669
DFN-2B HQRTP-350M
VIT-L/14
0.813
0.691
0.662
0.656
0.670
16

---

Published as a conference paper at ICLR 2024
H CLIP FILTERING PSEUDOCODE
We show pseudocode of the basic CLIP filtering operation below.
def clip_filter (image, text, threshcld=0.33 compute image and text representations
encode encode_image (image_
text_features compute anignment
(text_input)
dot_product image_features.1 @ text_features
norm_a image_features norm((
norm text text_ maaauures.nomm.
similarity = hot_proouctt / (norn_a * norm_b) b)
# filter by alignment
return similarity threshold
I PUBLICLY REPRODUCIBLE DFNs
Table 11: DFNs are trained with a ViT-B/32, then used to filter DataComp pools. Conceptual 12M, Conceptual Captions 3M, and Shutterstock 15M are publicly available datasets, demonstrating that large-scale high-quality datasets can be constructed with only publicly available resources.
DataComp
DFN Training Data
IN
IN Shifts
VTAB
Retrieval
Average
Scale
CC12M + CC3M SS15M
medium
0.307
0.253
0.359
0.274
0.349
OpenAl WIT-400M
medium
0.285
0.240
0.355
0.253
0.338
CC12M CC3M SS15M
large
0.591
0.481
0.522
0.503
0.532
OpenAl WIT-400M
large
0.578
0,466
0.525
0,475
0.527
CC12M + CC3M SS15M
xlarge
0.770
0.656
0.663
0.624
0.658
OpenAl WIT-400M
xlarge
0.764
0.640
0.628
0,599
0.638
17


Reviews:
[{"review_id": "4bwpBfnKTi", "reviewer": "Reviewer_NYkb", "summary": "The paper compares the performance of (primarily) various configurations of CLIP models to filter image-text datasets for subsequently pretraining vision transformers. The authors make interesting empirical observations regarding the same, and are able to achieve an impressive accuracy vs. compute tradeoff on image-net using their best configuration of data filtering networks.", "questions": "- Can you highlight the cost of filtering in e.g. Figure 1?\n- (Not relevant to the final rating) What do you think about applying these data filtering networks to other domains, e.g., text, or speech, or music?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- SoTA compute vs. accuracy trade-off on image-net.\n- Thorough & insightful analyses / recipes to understand the effect of different kinds of confounders while training data-filtering networks on downstream performance.\n- Expected public release of the best-performing filtered datasets.", "weaknesses": "- The proposed approach is still a heuristic for filtering data, and is limited to pointwise filtering for the sake of parallelizability.\n- The cost of filtering: (1) training the data filtering networks; and (2) linear inference over the entire dataset; might be too much to be amortized or ignored in the final compute vs. accuracy trade-off (more in questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SrL5UMcjWl", "reviewer": "Reviewer_Q5jf", "summary": "The paper studies the problem of how to filter large-scale web-scrawled data into the training data for training CLIP models. The paper proposes a data filtering network (DFN) for this task. \n* Training DFN: DFN is a small CLIP model pretrained on a small but “high-quality” dataset based on the authors’ insight: (1) the performance of DFN itself is not correlated to the induced model trained on DFN’s filtered data (Figure 3) and (2) Induced model’s performance degrades as the quality of data drops (Figure 4).\n* Using DFN for filtering (inference): DFN filters out image-text pairs whose cosine similarities are lower than a threshold (Section 2).\n\nBased on the insight above, the paper curates the High-Quality Image-Text Pairs (HQITP-350M) dataset containing 357 million image-text samples with human-verified captions to train DFNs. In the experiments, the paper shows that CLIP models trained on DFN-induced datasets (1) achieve state-of-the-art performance, (2) improve model efficiency (i.e., training a smaller model that achieves similar performance with previous larger models), and (3) improve VQA performance. The paper will release the DFN-2B dataset filtered by DFN to facilitate future research.", "questions": "In the rebuttal, I expect the authors to address my concerns regarding the following:\n1. Details of collecting the HQITP-350M dataset.\n2. More discussion on dataset quality", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper studies an important problem of filtering training data for CLIP.\n2. The paper provides insight into the problem—high-quality pretraining data for DFN matters.\n3. The proposed method achieves state-of-the-art performance with better model efficiency.\n4. The paper will release the DFN-2B dataset, which is a good contribution to the research community.\n5. The paper is well-written and easy to follow.", "weaknesses": "### Major Concern\n\n**[Details of Curating HQITP-350M]** The paper does not provide many details about how to create the high-quality HQITP-350M dataset for training DFNs. This is really important because it is the key for training a good DFN as the paper claims. For example, how does the human verification process work? For example, do human annotators make a binary decision of whether image and text are matched or give a score on the scale of 0 to 1? If it is the latter one, how to threshold the scores? How many annotators are requested? What is the source dataset for curating the HQITP-350M dataset? How many image-text pairs are filtered out by humans?\n\n### Minor Concern\n\n**[Definition of Dataset Quality]** The paper defines the “quality” of CLIP’s training data as the image-text similarity. I wonder if the authors consider other aspects to define the data quality, such as the quality of images and the quality of captions [1].\n\n### Minor suggestions\n\nFor evaluating the robustness against distributional shifts, the paper can also discuss or include more evaluation on newer OOD ImageNet benchmarks: ImageNet-D, ImageNet-X, and ImageNet-W.\n\nMissing citations to OOD ImageNet datasets used in the paper: ImageNet-Sketch [5],  ImageNet-A [6], ImageNet-V2 [7], ObjectNet [8], ImageNet-R [9] (caption in Figure 5).\n\n### References\n\n[1] Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt, “Improving Multimodal Datasets with Image Captioning,” in NeurIPS Dataset and Benchmark Track, 2023.\n\n[2] Evgenia Rusak, Steffen Schneider, George Pachitariu, Luisa Eck, Peter Vincent Gehler, Oliver Bringmann, Wieland Brendel, and Matthias Bethge, “If your data distribution shifts, use self-learning,” TMLR, 2022.\n\n[3] Badr Youbi Idrissi, Diane Bouchacourt, Randall Balestriero, Ivan Evtimov, Caner Hazirbas, Nicolas Ballas, Pascal Vincent, Michal Drozdzal, et al., “ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations,” in ICLR, 2023.\n\n[4] Zhiheng Li, Ivan Evtimov, Albert Gordo, Caner Hazirbas, Tal Hassner, Cristian Canton Ferrer, Chenliang Xu, and Mark Ibrahim, “A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others,” in CVPR, 2023.\n\n[5] Haohan Wang, Songwei Ge, Eric P. Xing, and Zachary C. Lipton, “Learning Robust Global Representations by Penalizing Local Predictive Power,” in NeurIPS, 2019.\n\n[6] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song, “Natural Adversarial Examples,” in CVPR, 2021.\n\n[7] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar, “Do ImageNet Classifiers Generalize to ImageNet?,” in ICML, 2019.\n\n[8] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz, “ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models,” in NeurIPS, 2019.\n\n[9] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, et al., “The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization,” in ICCV, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "agUwY6QfNL", "reviewer": "Reviewer_2tx6", "summary": "This paper's focus is on dataset filtering for contrastive language-image pre-training (CLIP). The discussion on data-filtering networks (DFN) is motivated by the use of pretrained CLIP models (referred to as the DFN) to filter datasets by discarding image-caption pairs whose embeddings (as produced by the DFN) are not similar to each other. The paper's main observation is that accuracy of the DFN on a given task doesn't predict how good a subset it selects (even for that task). The paper then uses this to motivate the creation of new DFNs that are trained on high quality image-caption data and then fine-tuned on \"important\" datasets. The paper confirms the effectiveness of the approach by demonstrating competitive performance on ImageNet Zero-Shot, ImageNet Distribution Shift, VTAB, Retrieval etc when filtering DataComp's medium, large and xlarge data.", "questions": "1) Is there any explanation for why a DFN with higher ImageNet accuracy is not always better at selecting data for ImageNet?\n\n2) Could the authors go into greater depth about the impacts of fine-tuning i.e. is the key component of training a good DFN fine-tuning on the evaluation datasets. If so, how do the authors see these models as generalizing on unseen tasks? \n\n3) Is there a trade-off in accuracy on some datasets? For one of the models, could we see the per dataset accuracy for VTAB using different DFNs and see if there are trade-offs in which datasets improve and which don't?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. The problem of better understanding and improving DFNs is extremely interesting. \n\n2. The observation that a DFN with good accuracy on a given task doesn't necessarily induce a good dataset on that task is very interesting.\n\n3. Several large-scale experiments are conducted to explore the effectiveness of the approach.", "weaknesses": "It seems like the improvement in performance over other DFNs is primary in ImageNet (and dist. shift) Zero-Shot as well as Retrieval. It seems like without \"finetuning\" on MS COCO training set, Flickr30k training set, and ImageNet 1K training set with ImageNet templates as captions, these performance improvements nearly disappear. While the main paper does ablate over fine-tuning, it doesn't highlight just how significant fine-tuning on these particular datasets is (primarily because these are the datasets we wish to evaluate on). \n\nPersonally, I don't find the conclusion that training a DFN on high-quality image-caption datasets allows the induced subsets to be higher quality very surprising or interesting. I do think a fine-grained study into specific properties of the pretraining data (e.g. similarity with downstream tasks' data, diversity of data, data imbalance) can make this work interesting and help further our understanding of DFNs. In it's current form, apart from the observation on ImageNet accuracy of DFN and that of the model trained on the induced dataset, I don't see what the contribution of this work is towards understanding DFNs. \n\nWhile I recognize, the improvement in accuracy on ImageNet, Retrieval etc., the importance of fine-tuning the DFN on the evaluation datasets leads to me believe that this approach may not generalize to other downstream tasks (as evidenced by the < 1% improvement on VTAB).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mYF66WIPs3", "reviewer": "Reviewer_ZvvG", "summary": "The paper proposes a contrastive loss trained model for data filtering", "questions": "1. How many runs are conducted to get the quantitative experimental results in the paper (e.g., Table 1-6)? Are the standard deviations sufficiently smaller than the differences between different settings?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The finding is interesting that a model's data filtering ability is not correlated with its image task performance. Actually, which is also discussed by [1].\n2. The finding is interesting and useful that data quality determines the trained model's filtering performance.\n3. The proposed data filtering network achieves impressive results on data filtering.\n\n[1] Yu, Haichao, et al. \"The Devil is in the Details: A Deep Dive into the Rabbit Hole of Data Filtering.\" arXiv preprint arXiv:2309.15954 (2023).", "weaknesses": "1. The HQITP-350M dataset is the key to the success of the DFN, so it is disappointing that it cannot be publicly accessible by the research community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)