Effective pruning of web-scale datasets based on complexity of concept clusters
Authors: Amro Kamal Mohamed Abbas
Evgenia Rusak
Kushal Tirumala
Wieland Brendel
Kamalika Chaudhuri
Ari S. Morcos
Venue: ICLR 2024 poster (2024)
Keywords: pruning
large-scale
data curation
concept-based
LAION
DataComp
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a pruning method where we aim to obtain optimal dataset coverage by assessing sample complexity; we report SotA results on the DataComp Medium benchmark and outperform regular OpenCLIP training on LAION with significantly less data.

Abstract: Utilizing massive web-scale datasets has led to unprecedented performance gains in machine learning models, but also imposes outlandish compute requirements for their training. In order to improve training and data efficiency, we here push the limits of pruning large-scale multimodal datasets for training CLIP-style models. Today’s most effective pruning method on ImageNet clusters data samples into separate concepts according to their embedding and prunes away the most proto- typical samples. We scale this approach to LAION and improve it by noting that the pruning rate should be concept-specific and adapted to the complexity of the concept. Using a simple and intuitive complexity measure, we are able to reduce the training cost to a quarter of regular training. More specifically, we are able to outperform the LAION-trained OpenCLIP-ViT-B/32 model on ImageNet zero-shot accuracy by 1.1p.p. while only using 27.7% of the data and training compute. On the DataComp Medium benchmark, we achieve a new state-of-the-art ImageNet zero-shot accuracy and a competitive average zero-shot accuracy on 38 evaluation tasks.

Published as a conference paper at ICLR 2024
EFFECTIVE PRUNING OF WEB-SCALE DATASETS BASED ON COMPLEXITY OF CONCEPT CLUSTERS
Amro Abbas"1, Evgenia Rusak Kushal Tirumala", Wieland Brendel
Kamalika Chaudhuri Ari S. Morcos
amrokamal @00mmmil com, evgenia. rusakeuni-tuuebiiei
de
University of Tübingen, Germany' Meta AI (FAIR)²
ELLIS Institute Tübingen Max-Planck Institute for Intelligent Systems* Tübingen Al Center University of California San Diego DatologyAl'
ABSTRACT
Utilizing massive web-scale datasets has led to unprecedented performance gains in machine learning models. but also imposes outlandish compute requirements for their training. In order to improve training and data efficiency, we here push the limits of pruning large-scale multimodal datasets for training CLIP-style models. Today's s most effective pruning method on ImageNet clusters data samples into separate concepts according to their embedding and prunes away the most proto- typical samples. We scale this approach to LAION and improve it by noting that the pruning rate should be concept-specific and adapted to the complexity of the concept. Using simple and intuitive complexity measure, the training cost to a quarter regular training. By filtering from the LAION dataset, find that training on smaller of high-quality data lead to higher performance with significantly lower training costs. More specifically, we are able outperform the LAION-trained OpenC1.IP-ViT-BB22 model ImageNet zero-shot accuracy by 1.Ip.p. while only using 27.7% of the data and training compute. Despite a strong reduction in training cost, we also see improvements on ImageNet dist. shifts, retrieval tasks and VTAB. On the DataComp Medium benchmark, we achieve a new state-of-the-a ImageNet zero-shot accuracy and a competitive average zero-shot accuracy on 38 evaluation tasks.
able to reduce
INTRODUCTION
OpenAl CLU (WUS-1000
100.0%
[LAIDN-D6M]
LAION CAT Esaiplas)
12.98
Num, of Dxemp leseep Seen
Scaling the model and the training dataset
Sepi
size has been shown increase perfor-
Taining nngg Cont
69.3%
8.98
mance across a wide range of tasks (Djo-
SemDoDupe
longa et al. 2021; Zhai al. 2022
(186M Esampled
[185M Es Examples)
41.6%
5,38
Kolesnikov et al. 2020; Taori et al. 2020
11124 Stamples
Foundation Models (Bommasani et
27. 7%
20.8%
184M Examptes)
2.78
3.68
2021 such as CL.IP (Radford et al. 2021b).
DinoV2 (Oquab et 2023), L.aMA and
61
63
64
65
66
Topl ImageNeet
LLaMA-2 (Touvron
al. 2023a b)
Eva Fang et al. (2023) have revolutionized the Deep Learning field and sparked inter- est beyond the academic realm with their unprecedented capabilities in vision and language. However, training (foundation) models on larger datasets incurs high com- putational and environmental costs which are out of reach for most academic labs.
Figure 1: With our approach, we outperform training the full LAION-400M, dataset (64.1% 63.0%) for CLIP-ViT-B/33 models while significantly reducing the training cost 27.7%. We filter from the LAION- CAT-440M by first deduplicating it to 277M exam- ples using the SemDeDup method and then applying Density-Bast -Basdd Pruning (DBP) to get datasets of sizes 84M, 12M, and 166M examples.
'Equal contribution. *Wor done during an Al residency (Amro) / research internship (Evgenia) at Meta AI (FAIR). Work done while at Meta AI (FAIR). Code g thub. com / amro kama / ve.
pruning

---

Published as a conference paper at ICLR 2024
In contrast the highly curated ImageNet dataset (Deng al., 2009), web-scale datasets such as LAION (Schuhmann et al., 2022) are noisy. and filtering out less informative data can strongly improve data efficiency and speed up learning. For example, discarding images below certain CLIP-score, which is the cosine similarity between image and caption embeddings, has been shown to improve data efficiency. The original LAION dataset used CLIP-score value of 0.3 as of the steps to create LAION-400M (Schuhmann et al. 202 In the recently proposed benchmark DataComp (Gadre al. 2023), which aims to find optimal data for broad range of downstream tasks, CLIP-score filtering has emerged as a strong baseline (Gadre et al. (023).. Apart from CLIP-score filtering, other works assessed the complexity and the action-content of individual captions, and removed images containing (parts of) the caption via text-spotting (Radenovic et al.) 2003). or used the CLIP-score to gauge the importance of the captions within the image before removing them
MMain et al. 2023).
So far, other works on pruning of large-scale datasets have focussed on assessing the quality of individual data samples. We argue that the marginal importance of data point depends on other data points in its vicinity, such that optimal dataset coverage allows to discard more data points from denser regions, while keeping more data points from sparser regions. In order to achieve this, Supervised-Prooonee
begin by scaling the simple and theoretically motivated (SSP-Pruning. [Sorscher et al., (2022) web-scale datasets. To recap SSP-Prunine (2022) proposed to cluster the embeddings of a pretrained model with k-means, then ranked all samples by their distance to the nearest cluster centroid and pruned the dataset by discarding the most prototypical examples. With their approach, SSorchhrr et al. (2022) outperformed all other pruning methods on ImageNet. Since SSP Pruning has been shown scale to large language models (Tirumala et al. 2023). we take this method as the most promising technique on ImageNet, and investigate which steps are necessary to scale
Pruning method (Sorscher al.)
CLIP training on LAION; we also modify the pruning criterion by considering the complexity of concepts in the dataset. On a high level, we wish to have approximately the same sample density across the whole embedding space, thus, we call our method Density-Ban Based- Prrsinng (DBP).
Our contributions are:
We scale SSP- ESP-Pruning to web-scale datasets which is non-trivial task involving a dedu- plication step, investigate how the complexity of different concepts within a dataset can be used for pruning, and report further improvements over regular SSP-Pruning.
We demonstrate that the pruning criterion we developed on LAION also transfers to the DataComp benchmark (Gadre et al., (023). and beat the current state of the art reported in the literature in most categories.
We show empirically that training on smaller high-quality data can result in a better model with significantly lower training cost.
2 RELATED WORK
2.1 DATA CURATION IN SUPERVISED LEARNING
Our work is related to coreset selection which focuses on identifying a highly informative subset of large dataset improve training efficiency. Usually, samples which are considered to be harder based on some scoring criterion are kept while casier examples are discarded. Criteria for ranking the samples are based on (dynamic) model uncertainty (Gal et al. 2017 Coleman al. 2019 He et al.) 2023). distance of samples to score medians (Xia et al.) 2022). the average L2 norm of the error vector (Paul et al. 202T), the degree of forgetting over the course training (Toneva et al. 2018). the degree of memorization (Feldman & Zhang, 2020) [Felmman 2020). and many others. Killamsetty et al. (2021) propose an iterative bi-level optimization to select the coreset from a large pool of unlabeled data which would result in minimum labeled set loss when trained upon in a semi-supervised manner.
2.2 CONTRASTIVE IMAGE- LANGUAGE PRETRAINING
Combining caption supervision with training of large models lcrge-scale datasets has transformed the field of computer vision, and models trained with CLIP (Radford et al. 20216) or ALIGN (Jia et al. 2021) have shown exceptional performance across a range of down-stream tasks, such image
2

---

Published as a conference paper at ICLR 2024
generation RRamesh et al. 2022), image segmentation (Xu et al.
2023b) text-to-image synthesis LLi
ct al. 2022b), video understanding (Xu et al. (2022). and others. The open-source projects OpenCLIP (llharco al. 202T) and LAION-20 2B (Schuhmann al. 2022) have democratized research on large- scale multimodal models and have been crucial to make such progress possible. Still, while training of large-scale models on large-scale datasets is possible in theory, it remains prohibitively expensive for most academic labs in practice: For example, training of the ViT-L/14 model (Dosovitskiy et al.
2020) with OpenCLIP took 400 A10 (40 GB) GPUs for around 127 hours.
2.3 DATA CURATION SCALE
There exist different strategies to make CLIP training more efficient. We split the data curation methods based on the way they filter the data into three categories, although overlaps exist.
Redundancy Reduction This category of methods aims to reduce data redundancy by removing duplicates as in Abbas et al. (2023); ; Webster al. (2023). These methods consider the similarity between examples in the data population and remove samples whose similarity falls below a pre- defined threshold. This results in more balanced data and saves training costs spent on training on semantically similar examples.
Matching Score Filtering This category of methods ranks the individual examples using an Image- Text matching (ITM) score computed using pre-trained model like CLIP (Radford et al., 2021b) or BLIP (Li et al., 2022a). A simple and strong baseline for ITM filtering the CLIP-score which the cosine similarity of image and text token embeddings of pretrained CLIP model. The LAION-400M dataset itself has been filtered using the CL.IP-score such that image-caption pairs were discarded their CLIP-score was below 0.3 (Schuhmann al.) 2021b). The CLIP-score is also strong bascline on subsets of all scales in the DataComp benchmark (Gadre et al. 2023).
Improving the data quality It has been shown that data efficiency can be improved by diversifying (Santurkar et al. 2022) or denoising the captions (Nguyen et al.) 2023) or by using shorter image/ text token sequences for larger image/ text encoders during CLIP training (Li et al. 2023). Xu et al. (2023a) incorporates a data objective into their training pipeline and dynamically selects data during training. Radenovic al. (023) remove examples with short captions and examples with low caption complexity. In addition, they remove examples that contain part of the caption as text the image prevent the model from spotting the caption from the image instead of learning visual semantics. While text-spotting filtering removes images that contain text, their CLIP-score values tend to be high. To resolve this problem. Maini et al. (2023) introduce T-MARS, data filtering technique which aims to compute more accurate CLIP-score values, by simply masking the text (if it exists) from all images before computing the CLIP-score values. |Wang al. (2023) propose multi-step algorithm which clusters the image embeddings, randomly samples from the clusters, and finally refines the captions of the retained samples. In contrast to their approach, we use cluster complexity to determine the number of examples to pick from each clusters; further, we pick the hardest examples from each cluster instead of random ones. In our experiments, we found that both choices improve performance.
METHODS
Our filtering pipeline has 3 stages: deduplication, CLIP-score filtering, and -ansity-Based--e Pruning.
Deduplication. We find that clusters web-scale datasets are dominated by duplicates, not allowing us to meaningfully interpret the distance to a cluster centroid as sample difffculty. Therefore, we first deduplicate the dataset using the SemDeDup method proposed by Abbas et al. (022). see Appendix A for details.
CLIP-score filtering In CLIP-score filtering, one calculates image and caption embeddings using strong pretrained CLIP model and removes examples below a certain cosine similarity (0.3 in LA0ON-400M, Schuhmann et al. 2021a) or picks a portion of the dataset with the highest cosine similarity. CLIP-score filtering removes low quality samples where the images and the captions do not match and is an integral part of many state-of-the-ar pruning methods, such as Maini et al. (2023); Radenovic al. (2023)
3

---

Published as a conference paper at ICLR 2024
Color Bar dd inter"d intra
high dane
low dinter
high Oura
low dyaa
0.0
inter
04
Oiter
0.
0.6
Chris
0.1
d intra
Figure 2: We determine the complexity of concepts within dataset by examining the clusters in the embedding space of a pretrained model. We characterize the clusters with their inter-cluster (left) and intra-cluster distance (middle). We find that clusters with small inter-cluster distance tend to show similar concepts and have low variability among each other. Further, we observe that dense clusters show higher similarity among their samples. Thus, to obtain more diverse dataset with high variability and low redundancy, we need to sample more from clusters with high inter-cluste distance and high intra-cluster distance. The scatter plot (right) shows the distribution of dintra over dinter on LAION-50M for 500 clusters.
Density-Based Pruning (DBP) Sorscher et al. 2022 proposed a self-supervised pruning metric for ImageNet (SSP-Pruning. where more protorypical examples are removed. We build our Density- Based Pruning (DBP) method on top of SSP-Pruning. Following |Sorscher et al. [022] we embed the data using a pretrained vision model and then cluster the data in the embedding space using k k-means clustering into & clusters. Then, considering the cluster centroid prototype, the method ranks the cluster items by similarity to the centroid (prototype) and removes examples with high similarities (prototypical examples).
The authors of SSP-Pruning observed that naive pruning of easy examples across the whole dataset results in strongly increasing class imbalance and degraded performance. As solution, they introduced a class balancing score which enforced a minimum number of images per class. In the absence of class labels, fixed cluster balancing score was used. Instead of fixed score, we here propose to gauge the complexity of a cluster based on simple metrics to decide how many samples to keep from each cluster. To determine the complexity of clusters, we calculate the average intra-cluster distance diutra as the average distance of cluster members to the centroid (Fig. [ middle) and the inter-cluster distance dinter the distance of cluster centroid to its neighboring clusters (Fig. E left). Intuitively, cover the dataset optimally and to equalize the sample density across the embedding space, we need fewer samples from dense clusters and from clusters which have other clusters close nearby. Thus, we define the complexity for the j-th cluster Cj
Cj diuter: dintra.j.
(1)
where dinter is computed for each cluster j as the average cosine distance between cluster centroid
and its I nearest neighbor centroids, and d computed as the average cosine distance between the items of cluster and its centroid. We set the value of / for computing dinter 20 in all experiments (see Section for an ablation over l). Clusters with high dinter and high dintra are considered more complex than clusters with either one of the distances being low. To enable sampling, we turn Eq into a probability distribution by applying a softmax function:
exp(CG/T) Σ exp(Cc/T)*
with the temperature т and the number of clusters k. We set the value of 7 to 0.1 in all experiments (see Section 5.4 for an ablation over T). Multiplying Pj with the target dataset size N, we obtain the number of examples we would like to keep from each cluster. However, it can happen that the original number of samples M, in cluster smaller than the desired Pj N. We wish to sample as close possible to Pj while honoring the dataset constraints and solve this optimization problem using a simple quadratic program solver apsolvers (Caron et al. 2023). We include more details on k-means clustering and the quadratic optimization problem in Appendix and C respectively. and python code for solving the quadratic program and calculating dinter and dintra in Appendix C.I The pruned cluster sizes Pj N are plotted in Fig. 7 in the Appendix.
4

---

Published as a conference paper at ICLR 2024
4 EXPERIMENT DESIGN
Training Datasets.
We report results on three different datasets:
LAION-CAT-440M: (Radenovic et al.
2023) proposed a caption complexity, action, and text spotting filtering (CAT) method and filter the LAION-2B dataset to 440M examples MAION-CAT-440NM
We use SumDeDup (Abbas et al., 2023) to reduce the size of this dataset to 280 million examples, and call it CTOO--DeDup-28MM We refer the reader to (Radenovic et al. 2023) for more details about the
dataset. For safety purposes, we blur all human faces in the LAION-CAT-400M dataset.
2. LAION-50M: random subset from LAION-DeDup-280M. We use this dataset mainly for development and hyperparameter search.
3. DataComp Medium dataset (Gadre et al. 2023): Since the LAION-CAT-44IE dataset has already been pre-filtered in multiple ways, we complement our results on LAION by using a raw dataset with no filtering applied dataset which consists of 128 million raw examples. Because of link failures we were able to download 120 million examples from DataComp.
it. We choose to use the DataComp Medium
Pruning the LAION dataset. For all experiments on LAION, we focus on the training cost we save. Thus, we follow a fixed and simple setting of filtering the dataset 60% of its original size after deduplication. Therefore, we prune LLAIO--DDDupp--
and LAION-5OM to 166M and 30M examples, respectively. For LAION-DeDup 280M, we also experiment with pruning 28% and 40% of its original size. Unless stated otherwise, we train for 32 epochs. For our Density- Based Pruning method, we use image embeddings from a distilled DINOV2-I/14 model (Oquab) et al. 2023). We find that using the distilled HINOV2-L/14 embeddings works better than using multimodal embeddings as discussed in Section 5 We tune the number of clusters for k-means on LAION-DeDup-20 280M and use k=500 (see Section 5.4
Pruning the DataComp Medium dataset. For all experiments on DataComp, we follow the protocol set by the benchmark and train for 128 million examples seen. Keeping the number of examples seen fixed means that if the dataset size decreases, the number of epochs increases. Thus, the goal here is not reduce the training cost but to maximize performance with fixed cost. Similar to LAION, we embed the images using the distilled DINOV2-L/1e image encoder. We tune the number of clusters DataComp and use the value of k=100 as the best value.
Pretrained encoders DBP requires clustering and ranking examples in an embedding space of pretrained model. We experiment with different choices and present overview of the tested encoders in Appendix
Evaluation We use zero-shot accuracy for all evaluations and report the top-1 zero-shot accuracy on ImageNet in addition the DataComp evaluation protocol and evaluate on suite of 38 image classification and retrieval tasks including the VTAB tasks (Zhai et al.. (019b), ImageNet distribution shift tasks, and retrieval tasks. All the evaluation datasets we use are listed in Table
CLIP-score Baselines We use the standard CLIP-score filtering protocol for each dataset. We use the LAION CLIP-score values from the metadata (computed using OpenAl's CLIP-B/32 model) and OpenAl's CLIP-L/14 score for DataComp.
Other Hyperparameters We train the CLIP-ViT-B/33 models using the OpenCLIP [Iharco et al. 2021) default hyperparameters for both LAION and DataComp datsets and fix the training seed. We list the values of different hyperparameters in Table 6 Appendix E
5 RESULTS
Our Results section is organized as follows. We first report best results, obtained on LAION-CAT. 440M (Section and DataComp Medium (Section[5.2) In Sections 5.3 55.3and and [5.4] we analyze our results, explain our hyperparameter and design choices, and conduct ablation studies.
5

---

Published as a conference paper at ICLR 2024
ImageNet
ImageNet dist. shifts
100%
100%
Num. of Examples Seen (9)
(440M Esamplea)
128
Examples Seen
(442M Exampiea)
128
Training Cost %
(P77N)
Training Cost
1221
69.3%
(222
122:2
55.4%
Dice
68
0160N
(16ch)
60
43.0%
42.0%
27.7%
48
27.7%
16400
48
20.8%
20.8%
Top1 Zero-shot Acc.
Top1 Zero gera-shot Acc.
VTAB
Retrieval
100%6
Numn. Examples Seen (8)
100%
Nurn. of Examples Seen (B)
(40H Eamppn)
Training Cost 96
Taininn Paining Cost 9%
69.3%
69.3%
(222M)
88
8B
55.45
1222HI
41.6%
(1
(185H)
-B8
(265H) (16M
-68
27.7%
15490
13:2
48
27.7%
48
20.8%
20.8%
50
51
52
50
58
59
Topl Zero-shot AcC.
Top1 Zero-shot ACC.
LAION-CAT-440M
*
Sup
SemDeDup+CLIP-81 Score
Ours (SemDeDup+
Figure 3: CLIP- VVT- zero-shot evaluation for filtering the LAION-CAT-4400 dataset (Radenovic al. 2023). J. We filter the data by first deduplicating to 277M examples to get LAION-DeDup- 280M (Sum in the Fig.). Then we apply the DBP method filter the LAION-DeDup-200 dataset. We see that we outperform training on the whole LAION-CAT-440X, dataset on ImageNet, VTAB, and IeageNet distribution shifts datasets while using only 27%-41% of the training cost. baseline (green line), we train for 12.7B examples seen during training
For the following the OpenAl CLIP training procedure (Radford et al.) (021a). For all other models, we train for 32 epochs regardless of the dataset size. The y-axis shows the training cost and the number of examples seen for each individual model. See Table performance details individual datasets.
5.1 LAION: OUR METHOD OUTPERFORMS OPENCLIP ON IMAGENET WITH 27% OF THE TRAINING COMPUTE.
We start from LAION-CAT-400M, deduplicate it to LAION-DeDup-277M, and finally, apply the DBP method to obtain four much smaller datasets of sizes 84M, 112M, 166M, and 222M We observe that by training on our smaller curated datasets for fewer number of iterations we can achieve better performance than training on the whole LAION-CAT-440M dataset. We show the zero-shot performance
ImageNet, ImageNet distribution shit, Retrieval, and the VATB tasks in Fig. and Table We observe performance gains despite the massive reduction training compute: training on the I12M subset outperforms OpenCLIP-B/33 on ImageNet (65.44 (65.44% 62.92%) while using only 27% of the training cost. On ImageNet distribution shit tasks and VTAB tasks, we outperform the OpenCLIP baseline using less than 41 % of the training cost. On retrieval tasks, show competitive performance despite using 55.4% of the training cost. We show detailed results for zero-shot evaluation on 38 downstream tasks in Table Appendix.
5.2
OUR METHOD OUTPERFORMS THE CURRENT STATE OF THE ART ON DATACOMP MEDIUM WITH A LOWER DATASET SIZE.
Our approach outperforms the recently proposed and current state of the art on the DataComp leaderboard (T-MARS; Maini et al. 2023) on three (ImageNet, VTAB, and Retrieval) out of four downstream tasks families, as shown in Table I while T-MARS performs better on the ImageNet distribution shifts tasks. Detailed results all shifts
shown in Table IO Appendix. We compare Eimage-bsed
our detailed results to the best baseline released by DataComp (Image- based n CLIP Score (L/14 top 30%) )) and report improved performance in 35 out of 38 distribution shifts. Unfortunately, the

---

Published as a conference paper at ICLR 2024
Table Our approach outperforms the current state of the on DataComp Medium (T-MARS) on most tasks.
Method
Size Imugeeee ImageNet dist. shifts]
VTAB Retrieval Averuge
TMARS (Maini et al. 2023
25M
33.00
27.00
36.30
22.50
36.10
Image-basent ITCP Score (1/14 top 30%) (Gadre et al. 2023
14M
29.70
23,90
34,60
23.10
32.89
CLIP Score (L/14 top 30%)
38M
27.30
23,00
33.80
25.10
32.80
Ours (DeDup, 80% CLIP-L/KA Score, 50% DBP)
19.2M
33.35
24.73
37.26
26.82
34.52
Ours (DeDup, 80% CLIP-L/I4 Score, 40% DBP)
19M
32.02
25.74
37.26
26.80
35.35
Table 2: DBP outperforms CLIP score filtering across different model sizes on LAION-50M. All models are trained for epochs.
Dataset Size
CLIP-S/33
CLIP- B/32 (15IM params.) IN topl acc
CLIP-L/14 (428M params.) IN topl ace
Method/Model
(63M params.) IN topl ace
CLIP score
30M
32.32
38.07
47.61
DB- 18-Pruning
30M
39.04
43.41
53.21
authors of T-MARS have not released their models or the performance on the individual test sets, so we cannot compare our detailed results to theirs. To achieve this result, we deduplicate the 128M examples of the DataComp dataset and retain 80% (96M) of the original dataset size, then we perform CLIP-L/14 score filtering further reduce the dataset size to 40% (38M) 50% (48M) of the deduplicated dataset size, and finally, we perform Density-Based Pruning (DBP) and reduce the dataset size down to around 19M examples, Fig. (Appendix) for ablation the optimal final dataset size as well as on the influence of the number clusters.
5.3 ANALYSIS
A smaller, more balanced dataset can lead to better models (Fig. 3 In this work, we reduce the dataset size while maintaining and/or improving the quality of the data by balancing the data clusters and removing easy examples. This increases the marginal information gain the model gets from every training batch. As a result, we observe better performance on a variety of distribution dataset almost matches or outperforms training on the full LAION-CAT-440M dataset all categories in Fig. despite using only half the compute. This result suggests that, given source dataset, we can find smaller, high-quality dataset through careful filtering. Such dataset not only enhances or maintains performance but also reduces the training cost significantly. Another works (iree/rjovvky al. (2023) also shows theoretically and practically that balancing the data by "removing" examples from the majority groups/classes result in better worst group/class performance and better model even though the dataset size is reduced.
shift tasks with shorter training: The model trained on the BemDDeDup=DDB-2222
The performance on retrieval and ImageNet distribution shifts is relatively lower compared to ImageNet zero-shot accuracy (Fig. and Table []. This trend is consistent across different baselines for retrieval tasks and we hypothesize that retrieval and ImageNet dist. shift tasks need relatively longer training (in Fig. 3 we reduce the number of training iterations/cost seen < 55.4%). To study this behavior, we measure the performance gains obtained with longer training. We increase the number of iterations for training on the 166M dataset from 41.6% (Fig. to 69% and measure the difference in performance on each of the validation tasks. We observe that among the four tasks (ImageNet, ImageNet dist. shift, VTAB, retrieval), the ImageNet dist. shift and retrieval tasks benefit the most from longer training: They each gain 0.9p.p. and 0.8p.p., respectively. In contrast, ImageNet and VTAB gain 0.4p.p. and 0.7p.p., respectively. Therefore, we conclude that the observed performance drops on ImageNet dist. shifts and retrieval tasks can be, partially, attributed shorter training.
Our results hold across different model sizes, Table We test our best approach on LAION-50M using models with different parameter counts: We train CLIP-S/32. CLIP-B/32 and CLIP-L/14 models for five epochs on 30M examples filtered from the LAION-50M dataset using our DBP method. We find that approach outperforms CLIP score filtering for all models tested.
7

---

Published as a conference paper at ICLR 2024
Table Density-based pruning (DBP) helps improve the performance of SSP-Pruning. We dedupli- cate the LAION-CAT-4400 dataset to 277M examples and then apply SSP pruning or DBP to filter the dataset to 112M or 166M examples and train CLIP-B/32 on them for 32 epochs. We report the average zero-shot performance on 38 datasets from Gadre et al. (2023). We set the cluster balancing value SSP-Pruning method to 1.0.
Method/ Dataset size
112M (3.6B examples seen)
166M (5.3B examples seen)
DBP
49.8
51.6
SSP-Pruning
48.6
50.7
100%
ii Rarm Teining
LAD Beasiree
Num: of Example Seen (9)
Thi Zerzzeeeee Zero-shet Imapehet-Acc Acc
43.50
63.25
1.50
Trainina
50%
1.08
40%
20%
C5
Our
L/KONSO to (509)
9.38
42.25
SSP Pruning
Density- Based Pruning
42.00
0.0
0.2
0.6
0.8
1.0
Top1 ImageNet Zero-shot Acc.
Cluster Balancing Ratio
Figure 4: (Ieft) Performance grows consistently with continued training and we close the gap to training the full LAION-50M dataset when training for epochs, despite only using 30M samples. We also outperform the LAION CLIP-B/16 score (CS) filtering. (right) Density-based pruning (DBP) helps improve the performance SSP-Pruning (Sorscher et al. 2022). We prune the LAION-50M dataset to 30M examples and train CLIP-B/32 on it for five epochs.
DBP outperforms SSP-Pruning (LAION). The difference between DBP and SSP-Pruning the choice of how many examples taken from each cluster. In SSP-Pruning, a fixed cluster balancing score is defined while in DBP, we assess the complexity of the different clusters. We show that DBP outperforms SSP- Par-Pin the benefits of DBP over SSP-Pruning on the LAION-50M dataset for different cluster balancing ratios, and find that (a) DBP outperforms SSP-Pruning across all cluster balancing ratios, and (b) cluster balancing is not necessary for DBP since we obtain the best result at ratio of zero.
the LAION-CAT-440M dataset (Table B). We also show in Fig. 4 riight)
Modality of the embeddings and the choice of the encoders are important hyperparame- ters. When performing k-means clustering in
50
Top1 IN Zero-shat Acc.
36.7
the embedding space, we can decide whether we use the image- and/ or caption embeddings. We explore the influence of the encoder in Fig. rigght.. We experiment with image embed-
30
CUP402 MT EN 1574
CUPECO TEAT EM HOM)
BUT NY OW 133PCC
Cawe WAN S9A1 OPNE
20
10
dings extracted from two different pretrained en- coders, CLIP ViT-B/16, and a distilled DINOV2- L/14 model (Oquab et al. 2023). We also ex- plore using caption embeddings from two differ- ent pretrained encoders CLIP-B/16 text encoder and the Sentence BERT model "all-MiniLMM- L6-v2" introduced by (Devlin et al., 2019). In addition, we use multimodal embeddings from the BLIP VIT-B/16 Image-Text Matching (TTM) head (Li et al. (022a) which offers an elegant way to combine both modalities with learned shared embedding. Because tuning the parameters for each model is expensive, we fixed the hyperparameters to the ones tuned on LAION-50M using the DINOV2-L/14 embeddings. The results are displayed in Fig. right) and we achieve the best results with the distilled DINOV2-L/14.
Choice of the Pretrained Encoder and Modality
Figure 5: The choice of the encoder well as the data modality are important hyperparameters.
We obtain consistent improvements with longer training, Fig. 4 left). To show how the per- formance changes with longer training, train the same models for five and forty-five epochs
8

---

Published as a conference paper at ICLR 2024
Tep1 Zero Shot-shet ImageNet Ace
44.0
Top1 Zero-shot ImageNet Acc
44.0
43.
43.5
43.0
42.5
41.5
63.0
43.0
60.5
60.5
<2.0
10
20
30
50
0.0
Nearest Neighbors Computing
Cluster Balancing Ratio
(a)
(b)
Top1 Zaro shet ImageNet Ace
66.0
64.0
Zero-hhi shet Imugee ImageNet
42.5
42.5
62.0
<2.0
43.5
43.5
30006
Influence of the Temperature Parameter
Number of Kimeans Clusters
(c)
(d)
Figure 6: Values of different hyperparameters, for DBP pruning. (a): The number of nearest neighbors N calculate dinter. (b): the cluster balancing ratio, (c): the temperature, and (d): the number of clusters for k-means. We fixed the number of clusters in all experiments 500 except for Fig. (d). We set the temperature parameters to 0.1 all experiments except for Fig. (c). We conduct all experiments pruning LAION-50M dataset to 30M and training it for 5 epochs.
LAION-50M subset, and on 30M examples filtered from it using our pipeline. We consistently outperform CLIP score filtering (CS) throughout training and even close the gap training the full LAION-50M dataset when training for forty-five five epochs.
5.4 HYPERPARAMETER ABLATIONS FOR DBP
DBP has number of hyperparameters such as the number of nearest neighbors calculate dinters the cluster balancing ratio, the temperature 7 in the softmax in Eq 2 and the number of clusters for k-means clusters. The cluster balancing ratio is implemented as another constraint for the quadratic program. We choose all of these hyperparameters on LAION-50M by pruning it to 30M examples and show the results of tuning each them in Fig- 6 Based on these results, we the number of nearest neighbors to compute dinter to 20, the cluster balancing ratio to 0, the temperature to 0. and number of clusters for k-means 500.
CONCLUSION
This research accentuates the potential refining dataset curation techniques to enhance the efficiency of model training. By challenging traditional pruning methods and incorporating the influence of proximate samples into the pruning strategy, we achieved remarkable performance improvements. Notably, the LAION dataset, the approach surpassed the OpenCLIP-ViT-BIA model's ImageNet zero-shot accuracy by 1.1 percentage points using merely 27.7% of the training compute. Furthermore, report state of the art the DataComp Medium benchmark for ImageNet zero-shot accuracy and impressive results across 38 evaluation tasks. This showcases the profound impact optimized dataset pruning the advancement of machine learning models.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
The authors would like to thank Surya Ganguli, Julian Bitterwolf, Anas Mahmoud and Roland Zimmermann for helpful discussions. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen Al Center, FKZ: 011S18039A. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Evgenia Rusak.
REFERENCES
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari Morcos. Semdedup: Data- efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540, 2023.
Martin Arjovsky, Kamalika Chaudhuri, and David Lopez-Paz. Throwing away data improves worst-class error in imbalanced classification. In ICML, 2023.
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE Transactions Medical Imaging, 2018
https
7/pubmed. ncbi..c.i. mlm. nih. gcv/30716025/
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recogmition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Be Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems (NeurIPS), volume 32. Curran Associates, Inc., 2019, ht: /ttttss:/pproceedd ngs neur wiw.rrpp.... ccc wcppappr/ppppppppppp 2019//Lee//7af0/aaaaaccaaaaaaaaa
Paw.:e99--appppppppp pdf
Sara Beery, Elijah Cole, and Arvi Gjoka. The iwildcam 2020 competition dataset, 2020. https
10340
Yonatan Bitton, Nitzan Bitton Guetta, Ron Yosef, Yuval Elovici, Mohit Bansal, Gabriel Stanovsky, and Roy Schwartz. WinoGAViL: Gamified association benchmark to challenge vision-and-langee -anggan models, 2022. htttp::/arxiv.orr
awaaabss/2222 12576
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.077 070.05888. 2021.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. ponents with random forests. In European Conference on Computer Vision (ECCV), 2014. 29
discriminative com-
hittps://1it wwwinger.......... com/ 3tt.:ttee/...000079113-3333331
Stéphane Caron, Daniel Arnström, Suraj Bonagiri, Antoine Dechaume, Nikolai Flowers, Adam Heins, Takuma Ishikawa, Dustin Kenefake, Giacomo Mazzamuto, Donato Meoli, Brendan O'Dongguue, Adam A. Oppenheimer, Abhishek Pandala, Juan José Quiroz Omaña, Nikitas Rontsis, Paarth Shah, Samuel St-Jean, Nicola Vitucci, Soeren Wolfers, @ @bdelhaisse, MeindertHH, @rimaddo, @urob, and @ @shaoanlu. gpsolvers: Quadratic Programming Solvers in Python, April 2023. URL https://github. wttpm//pppoveese/psssoooooo
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server, 2015.
https: wwtg//oo///ao/oo.oooooooo
00325
Gong Cheng. Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the Institute of Electrical and Electronics Engineers (IEEE), 2017. htttps://eeexpppoe
Leee. org/ abstract//bcummnt//
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018, https: //arxiiv brg/ 1tt.ab///1111111111..
10

---

Published as a conference paper at ICLR 2024
Mircea Cimpoi. Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. Conference on Computer Vision and Pattern Recognition (CVPR), 2014. Describi Segrrrbue_n__eextuue________ in iitpiueeeeiii_n______ee
thecvf com/ content, evpr 2014/html/ Cimpoii
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In International Conference on Artificial Intelligence and Statistics (AISTATS),
2011. https::/ppooceedingg /ttpa///poocaeaaaeeo...oese////s wttpss//l///ccateelaa......
Cody Coleman, Christopher Ych, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. arXiv preprint arXiv: /829, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009, doi: 10. 2009 5506888..
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' D'Amour, Dan Moldovan, et al. On robustness and transfcrability of convolutional neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16458-16468, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11999 2020.
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results, 2007. hhtp:: / www. -eetork
http:////////oo//////o//ooaa//a/ hwtpsshhhp/iiiee...mmm nne.. html
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 19358-19369, 2023.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories. Conference on Computer Vision and Pattern Recognition (CVPR) Workshop, 2004. htttps://Leeep:
leee Silee.org//bccmme///S353
Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954-959, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881-2891, 2020.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304. 14108, 2023.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani, Deep bayesian active learning with image data. In International conference on machine learning, pp. 1183-1192. PMLR, 2017.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012. leee. btt./bbbtraac//ccceeee//22000
https://ieeeee

---

Published as a conference paper at ICLR 2024
Muyang He, Shuo Yang. Tiejun Huang, and Bo Zhao. Large-scale dataset pruning with dynamic uncertainty. arXiv preprint arXiv:2306.05/7. 2023.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019. httpss://rrat http//aaxvvv..g...eae...
00029
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: critical analysis of out-of-distribtioon generalization. ICCV, 2021a. https:// /ttp::://axxvvv/o org/ abs/ 2006. 16241
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021b. https
//axxxv.orc/abs//110..
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL pottps::/dol: http:///0oooo/11..22.2.22 If you use this software, please cite it as below.
5143773
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc Le. Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904 4916. PMLR, 2021.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions Big Data, 7(3):335-547, 2019.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. Conference on Computer Vision and Pattern Recognition (CVPR), 2017. https
oraxivvorggabs//// /otb.////11....0 612.068990
Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh Iyer. Retrieve: Coreset selection for efficient and robust semi-supervised learning. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural In- Curran Associates, Inc..
formation Processing Systems, volume 34, pp. neur cwtpppercccpppppppppppppeep files/paper/2221
2021. URL httpps::/poooceetphtttt fil http://cccc2s11111.ccccffffffoo0
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao. Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran Haque, Sara Beery, Jure Leskovee, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of wild distribution shifts, In International Conference on Machine Learning (ICML), 2021. htttss://aaatiiii org abs
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Computer Vision ECCV 2020, 2020.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. object representations for fine-graines categorization. In International Conference on Computer Vision Workshops (/CML), 2013. ht wow.unaaationoooooooo
//www. cv www.:cc-foundere... www.dotoittaioooooooo on. http::oo.ooo.ooeeeeeee////eeooea
icev h httpsshhhpps1111//eeeee 2013/
W19 (tt/html//Kee h119://ee//rtuuuoe02_0000o___eer
2aapaass__2_____c__eeeee BaclVVaapeer.e html
Alex Krizhevsky, Geoffree Hinton, et al. Learning multiple layers of features from tiny images, 2009.
https: http:///wwww. cs www.stt.oooooooooooooooooooooo edu/
2009-
Yann LeCun. The MNIST database of handwritten digits, 1998. exdb/mnist/
12

---

Published as a conference paper at ICLR 2024
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- immae pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp.
PMLR, 2022a.
Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scaling law for clip training. arXiv preprint arXiv:2305.07017, 2023.
Zhiheng Li, Martin Renqiang Min, Kai Li, and Chenliang Xu. Stylet2i: Toward compositional and high-fidelity text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18197-18207, 2022b.
Pratyush Maini, Sachin Goyal, Zachary C Lipton, Zico Kolter, and Aditi Raghunathan. T- mars: Improving visual representations by circumventing text feature learning. arXiv preprint arXiv:2307.03132, 2023.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft, 2013. htttss://arxa org. abs// www.ab///....... 1151111166111111
Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In ACM International Conference Multimedia, 2010.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading
digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems (NeurlPS) Workshops, 2011. htttp:://storaeeeee goog eapis. s com/
pd
Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco. Sewoong Oh, and Ludwig Schmidt. Improving multimodal datasets with image captioning. arXiv preprint arXiv:2307. 10350, 2023.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference Computer Vision, Graphics and Image Processing, 2008.
orge http://dcmmeettt/5 47551411
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.077993, 202.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In Con- ference on Computer Vision and Putlern Recognition (CVPR), 2012. httts:://eeexpll
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34: 20596-20607, 2021.
Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan. Filtering, distillation, and hard negatives for vision-language pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6967-6977, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 202
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021b.
13

---

Published as a conference paper at ICLR 2024
Vikram V. Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron B. Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. Beyond scraping: Crowd-sourcing a geodiverse datase, 2023. http:///axxxx....//bboo
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204. 06125, 1(2):3, 2022.
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? International Conference on Machine Learning (ICML), 2019.
htttp://proceediii
mm prwss/ wtt/ 19a.html
William A Gaviria Rojas, Sudnya Diamos, Keertan Ranjan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. The dollar street dataset: Images representing the geographic and so- cioeconomic diversity of the world. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2022. https::/oppe https::/openrevveeu. http:/ewwweeeerrrrueeeee
id=enfYsave00)
Shibani Santurkar, Yann Dubois, Rohan Taori, Perey Liang, and Tatsunori Hashimoto. Is cap- tion worth a thousand images? a controlled study for representation learning. arXiv preprint arXiv:2207.07635, 2022.
Christoph Schuhmano, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open dataset of clip-filtered 400 million image-text pairs, 2021a. httpss//aaxxv..oz///a..
02114
Christoph
Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprins arXiv:2 1,.02/ 14, 2021b.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Sehramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks n forum i 1dd-3133311111 4
Truck, 2022. URL /tttps://oppnreee
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Be- yond neural scaling laws: beating power law scaling via data pruning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neu- ral Information Processing Systems, volume 35, pp. 19523-19536. Curran Associates, Inc.,
2022. URL |tttps://proceeeiinng http://ppprcceeeeonnneeeeee....e nwurips. www..ipp...ppppppepepeee cc/paper files//appe//022/ File/ /ttp0///00bbbleeeaaaaaaaeeeeeeee Pateddfff---PpP-P--- Caareeer-oonfeeeeeeeee pdf
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian lgel. The german traffic sign recogni- tion benchmark: a multi-class classification competition. In International Joint Conference on Neu-
ral Networks (IJCNN,, 2011.
ieee. org/ dtwpment////////////////oo/tct /e0tmmennt/333395 6633395
O. Tange. Gnu parallel - the command-line power tool. :login: The USENIX Magazine, 36(1):42.47,
2011. URL /wwpw///wwwwwwwooooo gnu. wttp://pa///aaee///pppp/p///a
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt, Measuring robustness natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33:18583- 2020.
Bart Thomee, David Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The new data in multimedia research. Communications of the ACM, 2016.
a150aa/...1 01817
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari S Morcos. D4: Improving Ilm pretraining via document de-duplication and diversification. anXiv preprint arXiv:2308. 12284, 2023.
reghajayan,,and
14

---

Published as a conference paper at ICLR 2024
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint ar arXiv: arXiv:1812.05159, 2018.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Bastiaan S Veeling. Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivari- ant CNNs for digital pathology, 2018. https://arxix/
haa///bbs. abs masss/80556 1806.88800 03962
Pauli Virtanen, Ralf Gommers, Travis Oliphant, Matt Haberland, Tyler Reddy, David Cournaeea,, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake Vand erPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, A. Quintero, Charles R Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1. O Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-277 2020. doi: https://doi.org org/
Alex Jinpeng Wang, Kevin Qinghong Lin, David Junhao Zhang, Stan Weixian Lei, and Mike Zheng Shou. Too large; data reduction for vision-language pre-training. In ICCV, 2023.
Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS), 2019. https: /ttp:://axxvvooo org/ baarg//bb//aa abs / 1905 13549
Ryan Webster, Julien Rabin, Loic Simon, and Frederic Jurie. On the de-duplication of laion-2b, 2023.
Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data- eeft-efficient deep learning. In The Eleventh International Conference on Learning Representations, 2022.
Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision (IJCV), 2016. https::/
Hu Xu, Gargi Ghosh, Po- Yo-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint 1XXi:22009.10084, 2021.
Hu Xu, Saining Xie, Yao Huang, Licheng Yu, Russell Howes, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Cit: Curation in training for effective vision-language data. arXiv preprint N.X::2301.022444 2023a.
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2955-2966, 2023b.
Andy B. Yoo, Morris A. Jette, and Mark Grondona. Slurm: Simple linux utility for resource management. In Dror Feitelson, Larry Rudolph, and Uwe Schwiegelshohn (eds.), Job Scheduling Strategies for Parallel Processing. Springer Berlin Heidelberg, 2003.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2014. https: http:://aelannthooooy..ooo
Q14-10067
15

---

Published as a conference paper at ICLR 2024
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, André Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. The visual task adaptation benchmark, 2019a. http://arxiv/ http://aaiiv....ab...sss 910 04867
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto. Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark, arXiv preprint arXiv: 1xxi::9900004667 2019b.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12104-121 13, June 2022.
16

---

Published as a conference paper at ICLR 2024
A DEDUPLICATION
We follow Sup (Abbas et al. 2023) in order to deduplicate the dataset. SemDeDup dedupli- cates LAION by clustering the image embeddings of a pretrained model, and subsequently removing samples within a certain similarity threshold. We choose the threshold value for SemDeDup manually so that we reach the targeted dataset size. We follow the paper and keep 60%-80% of the data (63% for LAION and 80% for DataComp) as this range of values was shown to perform the best on the LAION dataset. For the k-means clustering step of SemDeDup we use 50,000 clusters for the dataset and 30,000 clusters for the DataComp Medium dataset. We did not tune the number cluster parameters as Abbas et al. (2023) show that it has small effect on SemDeDup. We refer the reader to Abbas et al. (2023) for more details about the SemDeDup method.
B DETAILS ON K-MEANS CLUSTERING
We use the Faiss library (Johnson et al. 2019) for clustering the embeddings single GPU. We normalize the embeddings to have a unit length and run spherical k-means using Faiss. In all experiments, centroids do not change after this number of iterations.
100 clustering iterations. We found that 100 iterations are enough as the
C DETAILS ON THE QUADRATIC PROGRAM FOR DBP
In the main paper, we introduced a complexity criterion how to assess the complexity of individual clusters based on the distances dinter and dintra- We turned the complexity into a probability distribution with a softmax function. Sampling according to this probability distribution requires solving an optimization problem, since the actual cluster sizes impose an upper bound on how many samples we can pick from each cluster. Accounting for this bound while minimizing the squared difference from the desired pruned cluster sizes, we obtain a constrained convex quadratic program:
minimize 21.22 Bk
( - 2- .N.NN xz))
(3)
subject to
Σ x N, 15 I My for all j,
where xy is the sampled number of examples in cluster j and the constraints are given by the pruned dataset size N and the actual cluster sizes My- We solve the program in Eq with the publicly available quadratic program solver qpsolvers (Caron et al. 2023). The pruned cluster sizes vs N are plotted in Fig.
We restate that the difference to SSP-Pruning is the replacement of the class balancing score with a method assess the clusters' complexity decide how many examples keep from each cluster. Following SSP-Pruning, we also keep the least prototypical examples from each cluster.
C.1 PYTHON CODE FOR DENSITY- HANSITY-BASED PRUNING
We include Python-code to solve the quadratic program defined in Eq. 3g.33l in Table The code to calculate dinter and dintra can be found in Table 5
D PRETRAINED MODELS FOR CALCULATING EMBEDDINGS FOR K-MEANS CLUSTERING
Distilled DINOV2-L/14:
We use a distilled DINOV2-L/14 model from Oquab et al. (2023). The model is distilled from DINOV2 and has 300M parameters. We resize the images of the LAION or the DataComp datasets to the size of 224x224 and take the output of the last layer of the model. Each image is embedded into vector of size 1024.
BLIP ViT-B/16: We use the BLIP model to generate a multimodal representation of each image- caption pair in the data. We the BLIP ViT-B/16 model introduced in Li et al. (2022a). The
17

---

Published as a conference paper at ICLR 2024
Num. of examples
Pruned cluster size
Num. of examples
Pruned cluster size
400K
Pj Py-N
250K
Original cluster dize
300K
200K
200K
150K
100K
100K
100
500
Cluster IDs
Cluster IDs
Figure 7: (left) Pruned cluster size vs N,N after solving the quadratic program. (right) Pruned cluster size original cluster size. We observe that the method tends remove more examples from large clusters resulting in more cluster- ballnced dataset. In both plots, the clusters are sorted in the x-axis by the pruned cluster size. The plots are for filtering the LAION-50M dataset down to 30M examples using the distilled DINOV2-L/14 embeddings.
model has 233M parameters and has been pretrained on dataset of 129M examples. To embed an image-caption pair, we first embed the image using the Image Encoder of BLIP into vector of size 768. Then we condition the Image- Grogedde Text Encoder of the model on the image embedding and embed the caption. We take the average of the token embeddings (each of size 768) of the last layer of the model as an embedding.
Sentence BERT: Sentence-BEil BERRT is siamese BERT architecture introduced in Devlin et al. (201)) Our motivation behind using this model is the fact that the model learns to maximize the cosine similarity between embeddings of semantically meaningful sentences using a contrastive learning objective. Namely, we use the "all-MimiiM-Lo-v2" Sentence BERT model from HuggingFace This model has been trained 1B sentence pairs dataset. The model maps each caption onto 384-dimensional vector. This vector is the output of an average pooling layer applied on top of the last layer of the BERT model.
CLIP ViT-B/16 Encoder We embed the images using OpenAl's CLIP-B/16 model (Radford et al. 2021a by mapping each image into 512-dimensional vector using the Vision Transformer Encoder of the CLIP model. This vector is the representation of the CLS token in the output layer of the model.
CLIP B/16 Text Encoder We embed the captions using OpenAl's CLIP-B/16 (Radford et al.) 202 la) model by mapping each caption into 512-dimensional vector using the Text Encoder the CLIP model. This vector is the representation of the last token in the output layer of the model.
E TRAINING HYPERPARAMETERS
We include the training hyperparameters in Table 6
F ADDITIONAL ANALYSIS FOR FILTERING THE DATACOMP DATASET
Deduplication is necessary precursor to DBP, Table 7 Without deduplication, the clusters found by k-means during the first step of DBP are strongly influenced by the duplicates. Then, the crucial assumption of DBP-that the distance the cluster centroid measure the difficulty of a particular sample-does not hold. It is therefore unsurprising that DBP works worse without prior deduplication. We note that this deduplication step has not been necessary on ImageNet where the original SSP-Pruning results have been presented, because ImageNet is highly curated dataset.
meaningful quantity to
CLIP-score filtering leads to better results with prior deduplication, Table 8 Applying CLIP score filtering to reduce the dataset size of DataComp Medium dataset from 120M down to 38M leads to better performance if the dataset is first deduplicated.
18

---

Published as a conference paper at ICLR 2024
Srrp-shot.e Acc. (96)
34.5
34.0
Average Zero-shot
33.0
500 K-means Clusters
100 K-means Clusters
Dataset Size (M) for Datacomp Medium
Figure 8: The performance DataComp Medium is influenced by the dataset size well as by the number of k-means clusters. Starting from a pool size 120M, we first deduplicate it (to 96M), apply CLIP score filtering (to 48M), and finally apply DBP.
G DETAILED RESULTS ON DATACOMP MEDIUM
In addition to the averaged results in Table T in the main paper. We compare sults to the best baseline model released by DataComp and take the results from the
trerrievl_rresults.coo and foithub.com/mmlouuuuutinoioooo wwtpiddaaaioooooooooooooooo /otpo//ooooooccccppoolooloo main/docs/ htt.dddcs//penclpp...ooo aallaasssaiaaaaaa ceteiiattrsusssteeeeesssssssseee csv. We show detailed results for models trained on LAION-CAT-400M and on filtered versions of this dataset in Table
Zero-shot Evaluation We strictly follow the evaluation protocol set up by DataComp on 38 evaluation tasks, including ImageNet, ImageNet distribution shit tasks (ImageNet Sketch, ImageNet v2, ImageNet-A, ImageNet-O,, ImageNet-R, and ObjectNet), retrieval tasks (Flickr and MSCOCO), the VTAB tasks (Caltech-101, CIFAR-100, CLEVR Counts, CLEVR Distance, Describable Textures, EuroSAT, KITTI Vehicle Distance, Oxford Flowers- Oxford-IlIT Pet, PatchCamelyon, RESISC45, SVHN, and SUN397), and other tasks. All evaluation datasets are shown in Table I0] Detailed information on the evaluation tasks can be found in Section N of the DataComp paper (Gadre et al.
2023
H SOFTWARE STACK
We different open-source software packages for experiments, most notably SLURM (Yoo)
et al. 2003), OpenCLIP (Ilharco et al. 2021), scipy and numpy (Virtanen et al. 2020). GNU parallel
(Tagge 2011). Faiss (Johnson al. 2019) PyTorch (Paszke al. (017) and torchvision (Marcel &
2
19

---

Published as a conference paper at ICLR 2024
Table Python code for the quadratic program solver
Import nuмpy np import torch
from apsolvers import solve_qp
Input: inter (List), d_intra (List), temp (float), num cum_centroids (int ), Eiltorod (int), can_ittem__i_cach_cllle (List)
Output: X (list) <- Number of samples cluster
softmax torch. Softmax () 11 probs softmax( (d_inter
d_intra) /temp
12 F np.eye (num_cccuuttlteeee conddoists))
probs filtered dataset_sttt size
np.array (1.0 num centroids)
15 nr.array [fiitere__dtases_siee)
16
17 Define the lower and upper bounds min_ san_samples_ =
bounds ar.array (I mii_sampless num_items__a jinmsii____ _e___in_______ each_clusterii for in range (num_centroids)
20
21
22 X solve_gp (P=P, q=q, A=A, b=b, lb=bounds 1;,01, ub=l:,1), solver='osqp
35
np.rint (X) astype (int)
Table 5: Python code for computing dinter and dintra-
2 import numpy as np import faiss
Input: norm_embs (array), erb_dim (int), num_centroids (int), filtered__atasst__tt (int), niter (int), seed (int), NNs_ (int)
Output: d intra (list), d inter (list)
Cluster the data 10 kmeans faiss. kmoans (dim, num_centroide, niter=niter, seed-seed, spherical-True, gpu-True, verbose-Truc)
kmcans. train (norm_embs)
14 Compute d innt
15 sim_to_centrood, nearest_cent kmeans. index...... search (norm_embs, 1)
16
intra |]
18
for cluster id in range (num centroids):
19
cluster_item_ids np. pp.where( nearest_cent searest_centt-cluster_ddd
20
cluster_d_intra - (1 hmm_oo_ccnrrod@@cccltetttttttte ) mean ()
d_intra.append (clusteerd_iittttttttee
Compute i_inter
24 sim_mtttte to_ Boo_NNccenttrode = kmeans index.idxxi search ( kmeans. centroidss nun_NNa++ 1:]
dist = 1 -
inter mp.mean dist, axis=1
20

---

Published as a conference paper at ICLR 2024
Table 6: Training parameters for CLIP. We follow the standard hyperparameters used for each dataset. We use the OpenCLIP hyperparameters for experiments on the LAION dataset and the DataComp hyperparameters for experiments on the DataComp Medium dataset.
Parameter
Value
Model
CLIP ViT-B-32 2000 training steps 500 training steps 33,792 4,096
Warmup (LAION)
Warmup (DataComp)
Batch size (LAION)
Batch size (DataComp)
Learning rate Optimizer
5.0e-4, cosine scheduler Adam W, wd=0.2, betas=(0.9, 0.98), eps=1.0e-6
Table 7: DBP is more effective on deduplicated vs non-deduplicated DataComp Medium. The result holds for two different dataset sizes. We did not apply CLIP score filtering in this experiment.
Deduplicated?
DBP, Dataset Size (M)
ImageNet top-1 acc.
Average acc.
No
48M
20.70
27.00
Yes, to 80%
48M
21.34
27.71
No
24M
19.10
26.10
Yes, to 80%
24M
20.16
27.30
Table 8: CLIP score filtering is more effective after deduplication DataComp Medium.
DeDup.
Pool Size
Dataset Size
ImageNet 1 acc.
average acc.
No
128M
38M
27.30
32.80
Yes, to 80%
120M
38M
27.93
33.03
21

---

Published as a conference paper at ICLR 2024
Table 9: Evaluation Results 38 datasets for training CLIP-B/32 models on different datasets filtered from the LAION-CAT-44N0 dataset. Datasets are grouped following Gadre et al. (2023). Models are evaluated using the DataComp Gadre al. (2023) evaluation pipeline, and the main metric values defined by DataComp are reported in the table.
Metric
OpenCLIP
LAION-40M SemDeDup
DBP
SemD,+CS
DBP
SSP
DBP
Datset Size
400M
440M
277M
222M
222M
166M
166M
112M
112M
Num. Samples Seen
12.88
12.78
8.88
7.1B
7.18
5.38
5.38
3.6B
3.6B
Training Cost
100%
27.7%
27.7%
ImageNet
Acc
62.93
64.07
64.32
65.17
61.64
65.46
65.09
64.09
62.77
ImageNet Sketch
Acc
49.38
49.78
49.88
49.5
47.2
49.21
49.18
47.36
46.76
ImageNet
ImageNet-A
21.72
25.00
25.65
27.23
23.48
25.83
22.71
ImageNet-O
Acc
53.45
50.6
51.85
52.5
52.7
53.05
55.25
54.6
55.0
IN
ImageNet-R
Acc
73.42
72.25
73.0
72.09
69.9
71.33
71.81
68.77
67.27
ObjectNet
Acc
43.87
47.1
47.5
49.3
45.46
47.55
47.6
46.02
43.67
Caltech-101
Acc
91.18
90.38
89.97
90.28
89.01
90.21
90.81
89.91
89.74
CIFAR-100
Acc
70.29
76.48
75.47
75.33
73.88
75.97
76.03
75.31
73.37
CLEVR Counts
Acc
16.24
23.57
17.21
33.23
22.7
25.44
188.7
21.49
19.97
VTAB
CLEVR Distance
23.91
14.97
17.88
24.51
22.59
20.45
18.12
23.77
24.48
Describable Textures
54,57
54,79
$6.06
46.17
BurSAT
Acc
51.43
52.72
44.37
55.76
47.07
59.56
47.65
44.69
41.15
KITTI Vehicle Distance
Ace
28.97
13.92
9.56
17.02
14.06
25.74
10.97
23.77
23.49
Oxford Flowers. 102
Acc
66. 18
62. 66
63.54
64.59
60.98
65.84
65.71
67.47
68.91
Oxford-IIT Pet
86.71
87.33
88.56
88.18
84.74
88.36
88.9
87.917
87.51
PatchCamelyen
49.53
RESISC45
54.54
59.27
52.49
45.06
SVHN
Acc
30.39
28.28
29.49
22.85
24.36
12.74
11.84
16
7.05
SUN397
Acc
66.99
66.87
66.29
66.55
65.21
67.03
66.86
64.36
63.52
Retrieval
Flickr
Recall
70.21
76.04
76.37
73.86
75.25
73.15
MSCOCO
Recall
43.93
48.06
48.86
48.44
45.76
46.98
45.65
44.24
WineGAViL.
Jaceard Score
40.8
44.91
42.58
42.92
41.72
38.93
37.8
37.26
36.81
CIFAR-10 10
Acc
90.74
93.75
93.79
93.82
92.93
93.28
93.68
92.31
92.41
Country211
FGVC Aircraft
16.58
12.42
13.79
14.01
17.21
11.85
Food-101
Acc
80.86
81.29
81.46
82.41
79.13
82.55
80.2
79.98
75.25
GTSRB
Acc
41.99
35.61
44.98
30.58
38.58
24.11
31.88
20.17
20,86
MNIST
Acc
37.33
37.23
34.03
23.79
29.22
16.63
14.49
9.17
11.09
Pascal
75.82
79.51
78.72
78.59
Rendered SST2
Acc
52.28
49.53
49.37
48.16
52.28
50.25
48.54
49.97
47.17
Stunford Cars
Ace
79.26
74.65
75.41
74.39
73.81
66.35
66.35
$5.11
60.71
ST1-110
Acc
95.6
95.73
96.9
96.3
96.39
96.28
96, 14
95.71
95.36
iWildCam
7.44
8.08
7.28
7.86
7.27
8.82
7.57
7.17
5.0
Camelyon
47.04
62.71
50.0
48.91
50.1
49..6
49.84
FMoW
12.96
12.38
14.75
8.87
0.0
Dollar Street
Acc
54.91
58.41
56.43
57.94
55.84
55.37
$5.96
57.48
55.84
GeoDE
Acc
83.8
86.01
84.76
84.72
84.01
84.22
84.76
84.72
83.76
Average
52.72
52.95
48.65
22

---

Published as a conference paper at ICLR 2024
Table 10: Evaluation results on 38 datasets for training CLIP-B/32 models on filtered DataComp Medium (120M examples). Datasets grouped following Gadre al. (2023). Note that for DataComp all models are trained for the same number of examples seen following the DataComp training settings.
Imagge-hased (1 CLIP-score
DeD
DeDDup.00
(L/14 top 30%)
+CLIP Score,40%
+CLIP-score,
Datasset
Metric
Gadre
2023
DBP
DBP
Num, samples seen
120M
120M
120M
ImageNer
Dong
InageNet Sketch
Wane et
2019
Acc
19.3
20.17
17.73
Dist. Shift
ImageNer
ImageNet-A
Bendrycks
2021b
ImageNet.
Headrecks
ImagoNet-R
Hendryckyy.o
ObjectNet
2019
1altech-101
Fei-Fei
2004
CIFAR-100
Krizhvssk al. 2009
Acc
$4.76
58.54
$9.34
CLEVR Counts
Jebnsoon
2112
2019a
Acc
13.65
14.58
16.38
CLEVR Distance
Johnson al. 2017 Zhai
2019
Acc
22.49
22.37
23.62
Describablle Textures
Cimppii al
2014
Avc
21.33
22.
23.35
VTAB
EuroSAT Holber
2019 Ziei ol
2019g
Avc
33.93
35.98
34,22
KITTI Vehiclee Distance
Geliger er al
2012 Zhal al.
2019a
Acc
21.1
30.8
36.57
Oxford Flowers- 102 Nilsback & Zisseman
2008
Acc
29.65
32.76
33.8
Oxford IIT Pet Parkki et (Vodini et et
2012 Zhai
2019a
Acc
43.11
45.64
47.22
PanchCamelyen
2018 Zhai al
2019a
Asc
$8.62
59.58
48.9
RESISCI5 Cheng et al
2017 Zhai
20
Acc
27.78
29 83
30.92
SVHN Netzer et al. 2011
Zhai er all 2019
Aoc
16.97
14.01
SUN397
Xino et al.
2016
Acc
36.37
43.35
45.08
Retheval
Flickr Youpg et al. 2014
Recall
18.12
27.46
26.85
MSCOCO
Chen
2015
Recall
11.0
16.78
16.45
WinoCiAViL Bittoon et
2022
Iaceand score
43.37
37.17
CIFAR-10
Krizheesky stal
2009
Acc
82.52
$4.56
85.91
Country211
RRdifodd
2021
Thomee
2016
PGVC Aircrafi Maji
Others
Food-101
Stallkamp
MNIST Can
Asc
Pascal VOC
Rendered SST2
Zhai
20196
Acc
$3.16
50.58
$0.14
Stanford
Krunse
2013
Aec
28.03
31.14
28.53
STL-10
Coates al.
2011
Acc
83.65
$4.21
86.21
iWildCamm
Heery et al
2000
Kol et all
2021
Avc
1.42
2.33
1.81
Camelyur17 Bandi et 2015 Koh
2021
Ave
66.69
73.5
47.72
FMoW (Cristie et al
2018 Koh al.
2021
Acc
0.0
0.0
0.0
Dollar Street
Rojas er
202
Aoc
44
46.03
47.08
GooDE (Ramawamy
2123
Acc
65.59
68.94
66.76
Average
32.89
35.35
31,52
23


Reviews:
[{"review_id": "aH8qbuGhvN", "reviewer": "Reviewer_7m8Z", "summary": "This paper seeks to prune large-scale multimodal datasets (e.g., LAION) for training CLIP-style models for training and data efficiency. Building upon SSP-pruning, Density-Based-Pruning picks the number of samples per cluster based on the overall complexity of a particular cluster, and achieves state-of-the-art results on LAION.", "questions": "1.\tIn Fig. 4(right), more sample points are needed for convincing conclusions.\n2.\tIn Table 2, it would be better to include SSP-Pruning as a baseline for comparison.\n\n3.\tThe tasks are relatively simple (ImageNet zero-hot). Could we compare DBP with other baselines on different tasks?\n4.\tIn the methods section, the filtering pipeline includes deduplication, CLIP score filtering, and Density-Based Self-Supervised Prototypes pruning. However, there is no description about CLIP score filtering. The description about deduplication does not provide me with any useful information.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThis paper proposes Density-Based-Pruning to improve data efficiency on large-scale multimodal datasets.\n2.\tThis paper conducts detailed hyperparameter selection experiments, which made the experimental results more convincing.", "weaknesses": "1.\tThe related work section contains excessive content, and there is a considerable amount of duplication. Besides, it is better to introduce “coreset selection[1-3]”, as it is closely related to your work.\n\n2.\tMissing some baselines for comparison, including random selection and other data pruning methods [3-4].\n\n3.\tThe description in the methods section is very confusing.\n[1] Moderate coreset: A universal method of data selection for real-world data-efficient deep learning, ICLR 2023\n[2] Large-scale Dataset Pruning with Dynamic Uncertainty, arxiv23\n[3] RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning, NeurIPS 2021\n[4] Too Large; Data Reduction for Vision-Language Pre-Training, ICCV\n2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M3yUVstU89", "reviewer": "Reviewer_u5qD", "summary": "The paper proposes a dataset pruning method broadly applicable to web scale datasets such as LAION. The authors build upon important prior work at the ImageNet scale (SSP-Pruning, Sorscher et al., 2022), which attempts to rebalance a dataset by clustering, and then sampling datapoints inversely proportional to their cluster centroids. The authors improve upon this procedure by noting that clusters have varying degrees of importance, both as a result of the average distance of points from the centroid as well as the average distance of the centroid from other centroids, and determine the proportion of samples to be pruned proportional to the importance. The authors demonstrate the supremacy of this method, DPB, through an array of experiments that the pruned datasets jointly save on compute while increasing overall performance.", "questions": "If the intuition is that it is density based pruning, why only 500 clusters? The 5 point plot of figure 6-d suggests that clusters between 100 and 10000 should be better explored since there are sharp peaks in that range. \n\nFigure 5: Why not use concatenate CLIP's text + image embedding?\n\nHow is it that SemDeDup does worse than LAION-440M for VTAB, but DBP does better?\n\nA notable decrease in performance on ImageNet dist shifts and Retreival tasks -- does the DBP type of pruning hurt generally hurt robustness? \n\nWhy use CLIP-B16 Score for Figure 3, but CLIP-L/14 Score for Table 1?\n\nUpdate: thank you for taking the time to answer my questions, they are sufficiently addressed.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper's extensive analysis of their new method across multiple datasets and benchmarks makes a compelling case for the high performance of their method. The dimensions of ablations, and the high level takeaways from those ablations in the results section are clear -- DBP outperforms SSP Pruning, outperforms CLIP filtering, DBP outperforms SemDeDup alone, and Deduplicaiton is crucial to the success of the method. These takeaways make the contribution quite compelling.", "weaknesses": "The core weaknesses concern clarity of the results, and the overall difficulty in connecting the many different - at times seemingly disparate - results that are thrown together in the narrative.\n\nFor one, the paper begins by proposing an improvement to SSP, but defers any result about the paper to a small table in the appendix, for a single model and single dataset -- it would help to do a more comprehensive evaluation to make clear the improvement being proposed. The authors attempt to briefly make a broader connection that their work is akin to density based pruning -- this is where the relative lack of optimization over the number of clusters -- which very much controls how the density is approximated -- is puzzling. Indeed, the 5 point plot of Figure 6-d leaves much to want, where we are led to believe 500 is the best, because it is better than 100 (?) and 10000. More ablations here would help clarify things. In general, more time assessing Density based pruning methods (e.g. for a datapoint, if its k nearest neighbors approximate it well, it can be discarded) would significantly improve the narrative of the paper. \n\nOn the other hand, the quadratic program, though interesting and unexpected in a paper of this type, seems like overkill -- assigning the maximum number of points to each cluster whose expected number of samples is above the maximum, and redistributing the remaining samples proportionally among the remaining clusters (and repeating) seems like it would achieve what the authors are seeking to do (if not exactly what the QP does) without the overhead of a QP solver, and all the space in the paper it consumes. \n\nThe discussion section would benefit from a more clear discussion of where the shortcomings of the method are relative to the other methods presented in the paper. For example, DBP does a tiny bit better than TMARS on Imagenet, but more significantly worse on Imagenet Dist and Average but a discussion of the latter point is entirely missing. Whether pruning in a density based manner affects model robustness seems like a subject at least worth touching on. Same for the retrieval plot in Figure 4 -- why is it that SemDeDup does worse than LAION-440B, but DBP catches up? An analysis of what is happening here would help understand the paper considerably. \n\nBeyond that, the many different numbers in the paper -- as a result of different filtering models (CLIP-B16 or CLIP-L/14), differing number of epochs, differing initial dataset size (LAION-50M or 440M) -- make the results very difficult to connect and piece together across tables and figures. If some consistency across variables were afforded -- or at least enough ablations such that there is an extra column in each table connecting the numbers in other tables to that table, it would make the information much easier to process.\n\nUpdate: The rebuttal does well to address most of my concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Eh7f1NejHJ", "reviewer": "Reviewer_mnGt", "summary": "This work scales SSP-Pruning to web-scale datasets, investigate how the complexity of different concepts within a dataset can be used for pruning, and report further improvements over regular SSP-Pruning. They evaluate it with CLIP pre-training and report the results on classification task: it shows data efficiency in terms of pre-training.", "questions": "See weaknesses.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The paper is well structured and the motivation is clear. It is interesting to study the data efficiency of CLIP pre-training to reduce the computational resources.", "weaknesses": "1. The biggest concern I have is about the technical novelty. The main idea is based on SSP-Pruning. Therefore, the technical contribution is very trivial. \n2. This work only reports the results on image classification. How about the zero-shot results of CLIP in retrieval tasks, such as COCO/Flickr30k image-to-text and text-to image retrieval? The zero-shot retrieval performance is also very important to measure the quality of pre-training and the effectiveness of the proposed method. The paper mentions about the retrieval task and shows some results in Figure 3 but I am not sure the implementation details: which dataset do they use? It would be helpful to report the results on standard benchmark (COCO/Flickr30k). \n3. The method involves CLIP-score filtering to curate a new dataset to train CLIP. I am unsure if such involvement is fair, as the filtering itself is similar to distilling some knowledge from a well-trained CLIP. Then, the new CLIP is somehow learning from the well-trained CLIP on large-scale datasets. Therefore, it may not be fair to claim the data efficiency for the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gRDT3dVUvh", "reviewer": "Reviewer_orZi", "summary": "This paper focuses on improving the training and data efficiency of CLIP-style models by pushing the limits of pruning large-scale multimodal datasets. By adapting the pruning rate to the complexity of different concepts within the dataset, the authors are able to reduce training costs to a quarter of regular training. They outperform the LAION-trained OpenCLIP-ViT-B/32 model on ImageNet zero-shot accuracy by 1.1 percentage points while using only 27.7% of the data and training compute. Additionally, they achieve a new state-of-the-art ImageNet zero-shot accuracy and competitive performance on 38 evaluation tasks in the DataComp Medium benchmark. The findings demonstrate the potential of pruning methods for improving the efficiency of training multimodal models.", "questions": "In this paper， the author scales SSP-Pruning to web-scale datasets and demonstrates that the pruning criterion can also transfer to the DataComp benchmark.\nQ1: In Fig. 1, the author shows they reduced the LAION-CAT440M to 166M and improved the zero-shot performance on Imagenet, can you provide more results? such as zero-shot transfer and linear probe performance on different datasets.\nQ2: Does the robustness of model training change when the dataset size is reduced? Please provide robustness evaluation results.\nQ3: The difference between DBP and SSP-Pruning is the choice of how many examples are taken from each cluster. It seems that innovation has some shortcomings\nQ4: In Fig.3, 222M performs worse zero shot accuracy than 166M, please give a reasonable explanation.\nQ5: Why weren't models of the same size, such as CLIP ViT-L-14, OPENCLIP LAION400M ViT-L-14, EVA ViT-L-14, and DINOv2 ViT-L-14, used for clustering comparisons?\nQ6:", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method proposed in this paper exhibits a certain degree of novelty.\n2. The paper extensively validates its findings using a large-scale experimental dataset, although experiments with even larger models were not performed.\n3. The practical applicability of the approach presented in this paper suggests its value in real-world scenarios.", "weaknesses": "1. We cannot determine if the current data is effective for training ViT-B-16 and ViT-L-14 models.\n2. The majority of the experimental evaluation metric is Imagenet zero-shot top1. Is there any bias in the algorithm towards Imagenet data?\n3. Detailed results for each dataset in VTAB are not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)