Demystifying CLIP Data
Authors: Hu Xu
Saining Xie
Xiaoqing Tan
Po-Yao Huang
Russell Howes
Vasu Sharma
Shang-Wen Li
Gargi Ghosh
Luke Zettlemoyer
Christoph Feichtenhofer
Venue: ICLR 2024 spotlight (2024)
Keywords: multi-modal pretraining
CLIP
image
text
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: CLIP data curation and scaling

Abstract: Contrastive Language-Image Pre-training (CLIP) is an approach that has advanced research and applications in computer vision, fueling modern recognition systems and generative models. We believe that the main ingredient to the success of CLIP is its \textit{data} and \textit{not} the \textit{model} architecture or pre-training {objective}. However, CLIP only provides very limited information about its data and how it has been collected, leading to works that aim to reproduce CLIP's data by filtering with its model parameters. In this work, we intend to reveal CLIP's data curation approach and in our pursuit of making it open to the community introduce Metadata-Curated Language-Image Pre-training (MetaCLIP). MetaCLIP takes a raw data pool and metadata (derived from CLIP's concepts) and yields a balanced subset over the metadata distribution. Our experimental study rigorously isolates the model and training settings, concentrating solely on data. MetaCLIP applied to CommonCrawl with 400M image-text data pairs outperforms CLIP's data on multiple standard benchmarks. In zero-shot ImageNet classification, MetaCLIP achieves 70.8\% accuracy, surpassing CLIP's 68.3\% on \mbox{ViT-B} models. Scaling to 1B data, while maintaining the same training budget, attains \textbf{72.4\%}. Our observations hold across various model sizes, exemplified by ViT-H achieving \textbf{80.5\%}, without any bells-and-whistles. Curation code and training data distribution over metadata will be made available.

Published as a conference paper at ICLR 2024
DEMYSTIFYING CLIP DATA
Hu Xu' Saining Xie Xiaoqing Ellen Tan' Po-Yao Huang' Russell Howes' Vasu Sharma' Christoph Feichtenhofer SUniverstty of Washington
Shang-Wen Li
Gargi Ghosh New York University
Luke Zettlemoyer!.'
FAIR, Meta AI
ABSTRACT
Contrastive Language-Iman Image Pre-training (CLIP) is an approach that has advanced research and applications in computer vision, fueling modern recognition systems and generative models. We believe that the main ingredient to the success of CLIP is its data and not the model architecture or pre-training objective. How- ever, CLIP only provides very limited information about its data and how it has been collected, leading to works that aim to reproduce CLIP's data by filtering with its model parameters. In this work, we intend to reveal CLIP's data cura- tion approach and in our pursuit of making it open to the community introduce Saaguage-Imaage Pre-training (MetaCLIP). MetaCLIIP takes raw data pool and metadata (derived from CLIP's concepts) and yields a balanced subset over the metadata distribution. Our experimental study rigorously isolates the model and training settings, concentrating solely on data. MetaCLIP applied CommonCrawl with 400M image-text data pairs outperforms CLIP's data on multiple standard benchmarks. In zero-shot ImageNet classification, MetaCLIP achieves 70.8% accuracy, surpassing CLIP's 68.3% on VYT-B models. Scaling to IB data, while maintaining the same training budget, attains 72.4%. Our ob- servations hold across various model sizes, exemplified by ViT-bigG producing 82.1%. Curation code and training data distribution over metadata is available at https: /ttp://gtttttt wtc.o///cccbooeeeerrccccccccccc
Metadata- Curited
INTRODUCTION
Deep learning has revolutionized the field of artificial intelligence, and pre-trained models have played a pivotal role in democratizing access to cutting-edge Al capabilities. However, the training data used to create these models is often concealed from the public eye, shrouded in secrecy.
The increasing availability pre-trained models for public use contrasts sharply with the lack of transparency regarding their training data. Further, proprietary concerns, such copyright issues, often limit access to the original data sources. Consequently, the need to explore novel approaches for curating high-quality training data that can be shared openly arises.
In the vasion-language domain, the dominant model and learning approach is Contrastive Language- Image Pre-training (CLIP) (Radford et al., 2021), simple technique learn from image-text pairs. We believe that the secret to the dominance of CLIP models is attributed to its high-qualit WIT400M dataset which is curated from the web. Despite its popularity, the specifies of CLIP's curation process have remained mystery, captivating the research community for years.
Follow-up works (Schuhmann et al., 2022: 2021) have attempted replicate CLIP's data, but with a notable difference in their curation method. While CLIP generates data based on its unknown data source and curation methodology, these approaches remove noise by applying the CLIP model as a hard blackbox filter which in turn is a form of distilling WTT400M information captured in CLIP.
The advantages of CLIP's curation are apparent. First, starts from scratch, avoiding the introduc- tion of biases through filters. Second, CLIP's curation process balances the data distribution over metadata, maximizing signal preservation while mitigating, rather than removing, noise in the data!. Such distribution lays the groundwork for task-agnostic data, crucial part of foundation models.
For example, filter on digits can remove noise from date or id strings but remove signal for tasks that involve OCR (e.g., MNIST), filter removing text with less than 5 characters can remove signal "dog".

---

Published as a conference paper at ICLR 2024
In this paper, we attempt to reveal CLIP's method around training data curation. We present empirical study data cura- tion, with frozen model architecture and training schedule. We focus solely the impact of training data, excluding other factors that could confound the results. We make several observations for good data quality and present simple algorithm make CLIP's curation more transparent. Consequently, we shed light on both the curation process and the resulting training data distribution. Our algorithm enables easy adaptation to different data pools, allowing parties to fully own their data pipeline without relying on blackbox fil- ters from external providers.
6.6-
Row DOLE/MAEONI
ImageNet Zero-shot
0.53
RANOL
CUPHEZON)
LAOO-4020
6:35
Our algorithm takes raw data pool D and metadata M (derived from CLIP's queries or visual concepts) and yields a balanced subset D over M: D + f(D; Our approach, named Pre-
5003 100000
190085
200000
250000
10003)
620000
Training Steps
Figure 1: ViT-B/32 on ImageNet zero-shot classifica- tion with fixed training steps (12.8B seen pairs and data has been de-duplicated). Raw:
Metadata- Language-Iman training (MetaCLIP), marks significant step towards making the curation process more transparent and accessible.
raw CommonCrawi (CC) distribution; Raw English: English only CC; MetaCLIP w/o bal.: curated (sub- string matched) data pool from CC; MetaCLIP: cu- rated and balanced metadata distribution. Metadata curation boosts performance significantly and balanc- ing is equally important. Our MetaCLIP data sig- nificantly outperforms CLIP's WTT400M and LAION data(Schuhmann al., 2021).
MetaCLIP applied to CommonCrawl (CC) with 400M data points outperforms CLIP on multiple standard benchmarks. In terms of zero-shot ImageNet classifica- tion, using ViT (Dosovitskiy et al., 2020) models of various sizes. Our MetaCLIP
achieves 70.8% CLIP's 68.3% on ViT-B and 76.2% vs 75.5% ViT-L. Scaling 2.5B data, with the same training budget and similar distribution boosts this to unprecedented accuracy of 79.2% for ViT-L. 80.5% for ViT-H and 82.1% 1% for ViT-bigG in the vanilla training setting (not using any external data, models, or longer training).
In I. we show the impact of metadata curation on ImageNet validation plotted over training steps. First, we are training on Raw English data from the web (400M image-text pairs, 57.4% ac- curacy), after applying Language IDentification (LID) to the random Raw (~1.1B pairs, 54.1 %). Using metadata to curate the training set (MetaCLIP 400M w/o bal, 60.8%) performs significantly better than these baselines, and using balancing significantly increases accuracy further (Meta- CLIP, 65.5%), outperforming similar datasets, WIT400M from CLIP, 63.4% and LAION 400M, (Schuumannnnnn et al., 2021).
RELATED WORK
The training data of CLIP differs significantly from a traditional supervised dataset (Gadre et al., 2023) in various aspects. Firstly, it involves large-scale training with mixed-quality image-text pairs rather than categorized images with human annotated labels, commonly seen in classification datasets. Secondly, CLIP's pre-training is the initial stage of training, assuming no access to previ- ously trained models.
Data Pruning on Established Datasets. Current research on data algorithms primarily revolves around data praming techniques applied to well-established datasets using pre-trained models (Sorscher et al., 2022; Abbas et al., 2023). These approaches, such as coreset selection techniques (Har- & Mazumdar, 2004; Feldman et al., 2011; Bachem et al., 2015; Mirzasoleiman et al.,
2

---

Published as a conference paper at ICLR 2024
2020: Toneva et al., 2018), aim to select subset of data that yields similar performance training on the entire dataset. Post-hoc data pruning with model filters has limited utility, the model is used black-box filter that forbids control biases improve the filter quality.
Handling Noisy Internet Data. Addressing noisy data from the Internet is significant challenge, and existing approaches often heavily rely on human- designdd filter systems. Classical methods involve dataset cleaning and outlier removal (Jiang et al., 2001; Yu et al., 2002) to discard samples that may introduce undesirable biases to models.
Replicating CLIP's Training Data. Recent efforts, such as LAION (Schuhmann et al., 2021; 2022) and concurrent work DataComp (Gadre et al., 2023), attempt replicate CLIP's training data. However, they adopt fundamentally different strategies for several reasons. First, the data used in these approaches post-hoc, filtered, by vanilla CLIPP teacher model. Second, the curation process these methods relies on labor-intensive pipeline of filters, making challenging to comprehend the resulting data distribution from the raw Internet (refer the unknown biases of using CLIP filter in (Schuhmann et al., 2022)). Thirdly, the goal is to match the quantity of CLIP's target data size rather than the data distribution itself, which may lead to an underestimation of the data pool size needed to obtain sufficient quality data. Consequently, the performance on the 400M scale is sub-optimal, with LAION400M only achieving 72.77% (Schuhmann et al., 2021) accuracy on ViT-L/14 on ImageNet, whereas vanilla CLIP obtains 75.5%.
Importance of Understanding CLIP's s Data Curation. The observations made in these studies underscore the critical importance of understanding how OpenAl CLIP curates its data in the first place. A comprehensive understanding of the curation process can shed light on the factors that contribute to success, allowing researchers to devise more effective and efficient algorithms for future vision-language pre- pre-training endeavors.
METACLIP
The original paper (Radford al., 2021) only provides limited details about how CLIP curates its data. Since important design choices for direct reproduction are missing, we will clarify our choices in this section. Our goal uncover CLIP's data curation process, which involves preserv- ing signal in the data while minimizing noise. In this section, will explain the principles we have adopted to achieve this, which may differ from CLIP's s as these are not known publicly.
CLIP's WIT400M is curated with an information retrieval method, quoting (Radford et al., 2021):
" To address this, constructed new dataset of 400 million (image, text) pairs collected from a variety of publicly available sources on the In- ternet. To attempt cover broad set of visual concepts as possible, we search for (image, text) pairs as part of the construction process whose text includes one of set of 500,000 queries We approximately class bal- ance the results by including up to 20,000 (image, text) pairs per query. "
We rigorously adhere this description and provide detailed insights into the construction process of CLIP's metadata (in $3.1)?, sub-string matching (in $3.2), inverted indexing (in $3.3), as well as query and balancing (in $3.4).
3.1 METADATA CONSTRUCTION: M = {entry)
We start by re-building CLIP's 500,000-query metadata, citing Radford al. (2021):
We generalize the term queries (used by CLIPP) entries in metadata hecause metadata describe training data and our algorithm does require search inverted index yet have similar effects.
3

---

Published as a conference paper at ICLR 2024
"
The base query list is all words occurring at least 100 times in the English ver- sion of Wikipedia. This is augmented with bi-grams with high pointwise mutual information as well as the names of all Wikipedia articles above a certain search volume. Finally all WordNet synsels not already in the query list are added.
"
The metadata ('queries' or 'entries") consists of four components: (1) all synsets of WordNet, (2) uni-grams from the English version of Wikipedia occurring least 100 times, (3) bi- bi-grams with high pointwise mutual information, and (4) titles of Wikipedia articles above a certain search volume. We rebuild these components from WordNet and Wikipedia and summarize the statisties in Table 13, We estimate the thresholds for components (3) and (4) as in the 3rd column of Table 1, by first choosing a point-wise mutual information threshold of 30 that meets the budget 100k entries for bi-grams and then fill the rest of the entries with Wikipedia titles.
Source
# of Entries
Desc. of Threshold
Threshold [ALL] (follow CLIP) 100 (follow CLIP) (estimated) 70 (estimated)
WordNet synsets
86,654
N/A
Wiki uni-gram
251,465
Count
Wiki bi-gram
100,646
Pointwise Mutual IPMI)
Wiki titles
61,235
View Frequency
Table 1: Composition of MetaCLIP Metadata.
3.2 SUB-STRING MATCHING: text - entry
After constructing the metadata, CLIP's curation aligns a pool of image-text pairs with metadata entries through sub-string matching. This process identifies texts that contain any of the metadata entries, effectively associating unstructured texts with structured metadata entries. The sub-string matching step retains only high-quality matching texts, automatically filtering out various types of noises that typical filter system would consider on case-by-can basis.
Such alignment referred as sub-string matching in Radford al. (2021):
"
We also restrict this step CLIP text-only querying for sub-string matches while most webly supervised work uses standard image search engines
"
Image-Text Pair Pool We start by estimating the pool size used by CLIP's curation. CLIP's data source is unknown us ("a variety of publicly available sources" in Radford al. (2021)). We adopt CommonCrawl (CCy² as the source to build such a pool and re-apply sub-string matching to this source. We ended with a pool of 1.6B image-text pairs (5.6B counts of sub-string matches). Note that one text can have multiple matches of entries and we have 3.5 matches per text on average.
As a result, sub- sub-string matching builds the mapping txt - entry. This step has two outcomes: (1) low-quality text is dropped; (2) unstructured text now has a structured association with metadata. For all English text, ~500 image-text pairs are kept this stage. Similar to CiT (Xu et al., 2023), this approach looks for quality matches and automatically gets rid of some type of noise (such as date strings) that a typical filter system would remove consider case-by-case (e.g., regular expression on dates, ids etc.).
Note that we cannot find Wikipedia's search volume titles of Wikipedia (4). Instead, we use volumes of Pageviews on Wiki articles. We randomly selected 26 days' Pageviews from Apr. 2018 to Sep. 2022. https:: /cm/mnmmmaaaaaa awl.org
4

---

Published as a conference paper at ICLR 2024
Metadata Subset
# of Entries
# of Counts
Full
500K
5.6B
Counts = 0
114K
Counts > 20000
16K
5.35B
Table 2: Summary of counts for entries.
Entry of the
Counts
Entry
Counts
Entry
Counts
Entry
Counts
120M
107M
and
100M
for 89M
87M
The
67M
with
67M
to
61M
photo
54M
a
50M
image
48M
47M
45M
by
43M
2
43M
Image A
39M
at 38M
Black
33M
30M
29M
Table Top-20 entries with counts.
3.3 INVERTED INDEXING: entry + text
Following sub-string matching, CLIP builds an inverted index of the data pool. All texts associ- ated with each metadata entry aggregated into lists, creating mapping from each entry the corresponding texts, entry + text.
As an analysis, we count the number of matches for each entry and summarize that in Table 2. The counts exhibit a long-tailed distribution. Out of the 500k entries, 114k entries have no matches. This signifies the importance of knowing the training data distribution since it is very likely the training data does not have certain visual concepts. We observed that only 16k entries had counts higher than 20k, accounting for only 3.2% (16k/500k) of the entries, but their counts made up 94.5% (5.35B/5.6B) of the total counts of all entries.
Top Entries. We show the top entries of the matching in Table 3. Interestingly, many of these are stopwords, which don't carry specific meaning but can enhance the overall text quality (e.g.. by generating grammatically correct sentences rather than just keyword lists). It's important note that although sub-string matching aims to select only high-quality texts, there are instances where common entries may still include irrelevant texts. For instance, the entry "photo" could match with the popular but unhelpful term "untitled photo". These relse-related issues can be addressed in the subsequent stage processing.
3.4 QUERY AND BALANCING WITH & <20K
The key secret behind OpenAl CLIP's curation is balance the counts of matched entries. For each metadata entry, the associated list of texts (or image-text pairs) is sub-sampled, ensuring that the resulting data distribution more balanced. This step aims to mitigate noise and diversify the distribution of data points, making the data more task-agnostic as foundation data for pre-training.
The magic number t 20k is a threshold used to limit the number of texts/pairs for each entry. Entries with fewer than pairs (tail entries) retain all associated pairs, while entries with more than t pairs (head entries) are sub-sampled to pairs. The selection based on the density of information in texts; texts with more matched entries have higher chance of being curated (recall that the average 3.5 matches per text).
To study the effect of the magic number t 20k, plot the cumulative sum of counts for entries sorted by counts from tail to head Fig. Interestingly, the value of = 20k seemingly represents the transition from tail to head entries, when the head entries start exhibiting exponential growth rate. By applying a max count of f, the growth rate of total counts (i.e., the scale resulting data points) is reduced to linear. This significantly flattens (and balances) the training data distribution. We further study the optimality of 20k for the 400M data scale in our experiments.
In summary, balaneing yields three interesting outcomes:
5

---

Published as a conference paper at ICLR 2024
Cumulative Entry Counts
Pool (1.68)
1=20k (400M)
Metadata Entries Sorted by Counts
Figure 2: Cumulative sum of counts entries from tail to head data pool with 1.6B image-text pairs (5.6B match counts). (1) raw/unbalanced cumulative counts, = oo; (2) balanced cumulative counts after applying t 20k. The limit t defines the transition of tail/head entries.
(i) reduces dominance and noise from head entries, like common web terms. E.g., out 400M pairs, only 20k texts containing "photo" are kept (while there are 54M "photo" instances in the pool).
(ii) It diversifies the data distribution and balances tail/head entries, leading more agno-tgi foundation.
(iii) Sampling for each entry ensures that data points with more matched entries denser informa- tion are prioritized for curation.
Discussion. CLIP employs pure NLP-based approach, requiring no access to ML models and minimizing explicit/implicit priors from humans. The metadata plays a central role in mitigating noise and preserving signal in the data distribution. The balancing step effectively flattens the data distribution, diversifying the data and making more suitable as foundation data for pre-training tasks. We analyze the effects of balancing in Appendix A.3.
3.5 A SIMPLE ALGORITHM FOR CURATION
This section presents algorithm that formalizes the curation process described earlier. The al- gorithm aims improve scalability and reduce space complexity for operations across data points, such inverted indexing and sub-sampling Instead of building inverted indexes, the algorithm only maintains total counts for each entry.
We assume that CLIP curation constructs an inverted index that maps entries documents (image- text pairs) to enable efficient search for each entry ("we search for (image-text) pairs" Radford et al. (2021)). In contrast, sampling. This avoids the need to build an inverted index that could potentially store hundreds of millions of conerete pairs for popular entries, thereby improving efficiency and scalability.
algorithm approaches the balancing process through independent
Our algorithm takes three inputs: metadata M, data pool D, and a hyper-paramett parameter t. It aims to find a subset D* with balanced distribution over M, denoted as D* < f(D;M,t) The algorithm consists of two parts, each corresponding to specific stage of the curation process.
We provide the Python pseudo-code in Algorithm

---

Published as a conference paper at ICLR 2024
Algorithm 1: Pseudo-code of Curation Algorithm in Python style (see Sec. A.10 for samples).
D: raw image-tex: paira; metadata;
t: max matches par entry in metadata;
D_atar: ourated image-text, pairs;
D_star UI
Part 1: sub-string matching: store entry indexes in sext mast.matched_eettp and output ccunts por entry in entry_count.
entry_count
substs_matching M)
Part 2: balancing via indepenent sampling
entry county [entry_ entry_prob =t entry_oourt
<]
for image, text in D:
for entry_id
randon. random() pnryy_probbentry_.lo
apptaz (inage, text)
break
Part 1: Entry Counts from Sub-string Matching. This corresponds to Sec. 3.2. The substr_matchiie function outputs the total counts matches per entry, entry_count, represented as a NumPy array indexed by entry_id. Each text is associated with Beetched_entr____ ids that contains a list matched entries.
Part 2: Balancing via Independent Sampling. This part corresponds to Sec.3.3 and Sec.3.4 and focuses balancing counts on entries. Instead of building expensive inverted index with associated lists of texts for each entry, we sample each data point independently.
We first compute the probability of sampling each entry, entry_prob, where tail entries (entry_count < t) have a probability equal to I, and head entries have a probability less than 1. We iterate through all image-text pairs and sample/curate each pair. When image-text pair has a matched entry sampled/selected, we include that pair in D
more This procedure is equivalent to CLIP's curation, because if one image-text pair has matched entries, the chance of that pair being selected is determined by the probability of sampling for each individual entry: f/entry__st count [entry ry contry_idi _d]. As long one entry selects that pair, will be kept in DD". Our independent sampling approach allows us to scale balancing for each data point independently and reduces the global operation to counting the total matches for each entry- We demonstrate case studies in experiments on (1) scaling curation in data pipeline and (2) online balancing in data loader.
EXPERIMENTS
Data Pools.
We collect two pools of data:
Pool 1 contains 1.6 billion image-text pairs with total of 5.6 billion counts of matches. This pool was used to estimate a target of 400M image-text pairs, collected from 15 snapshots of Common- Crawl (CC) from January 2021 January 2023.
Pool 2 aims to scale curation in our data pipeline. We parsed all 90 CC snapshots from 2013 to April 2023, using our algorithm (see SA.2 for details on the curation pipeline) curate from a pool of 10.7B matched image-text pairs that are originally from large set of URL-text pairs, which have undergone de-duplication, English Language IDentification (LII) and sub-string matching- However, only perform (expensive) image downloading, storing, and transferring for data points that are catiibrrto and selected by our algorithm.
For balancing we consider 2 scenarios on this data: (i) t 170k, which is resulting in 2.5B image- text pairs. This t = 170k configuration has tail counts amounting to 6% of the total counts, the same tail/head ratio that the 400M Pool data has, produced by applying t 20% on the 1.6B Pool data. (ii) The = 20k threshold applied to Pool which results in 1B image- text pairs and compared to the 400M set from Pool I only increases tail metadata matches (head counts capped 20k).

---

Published as a conference paper at ICLR 2024
CUP eul
56.6
63.4
1937
89
66.
53.7
630
59.7
1966
64.0
67.
87.4
66.9
18.
86.6
19
46.9
610
12A
26.7
17.
10.5
63.9
180
236
564
586
OpeaC ou end.
$7.6
02.9
91.7
19.7
10.
6122
85.4
19922
16.7
154.5
86.5
50.7
56.1
31.4
45.3
95.6
52.1
42A1
148
65.0
42.
21.3
53.5
52.5
MeteCLiP
58.3
655
90.6
91.
10.
63.
6300
70.7
25.8
52.1
66.7
91.9
18.5
415
35.9
95.4
15.6
64.2
35.8
30.7
17.2
53.5
66.
45A
300
56A
53.4
CUR oer evll
Ope
01.
59.6
68.3
88.8
91.7
90.5
11.4
66.3
55.6
86
640
11.6
64.6
24.0
17.4
45.1
56.9
59.1
50.8
89.1
66.1
69.4
16.3
51.
53.0
06.1
1963
56.8
53.3
65.3
65.5
13.3
21.7
22.8
16.3
51.1
06.5
07.0
533
15.5
15.7
916
MIP
71.3
52.9
OpEiCI
MeGaCLIP
00.1
WIT
955
TTA
17.M
705
517
(010
62.0
91.
76.4
617
46.5
59.7
718
47.5
755
15.1
05.
Table 4: data(Schuhmann et al., 2021)). We use 3 different model scales (ViT-B/32 and -B/16 and -L/14) and identical training setup CLIP.
MctaCLIP-400M. vs. CLIP (WIT400M data) and OpenCLIP (LAION-400M
McaKLIP--0MM) 58.2
05.5
90.6
91.
702
63.4
63D
NUL
26.5
52.6
RK.7
91.9
68.5
41.5
3599
154
51A
14.2
35.8
36.7
17.2
55.5
05.1
43.4
30.
56.4
53.4
67.5
81.9
95.3
76.
71.
659
75D
N6.9
3M4
V2.1
72.
35A1
15.3
S6.7
MA
1877
33
17
11.2
75.3
(7.8)
146
519
56.8
1226
982
12.3
86
1713
SAL9
02.9
627
46-3
16.3
25.3
19.2
1944
515
11:
70.8
16.
W11
M6
71.
66.6
76.1
77.
45.9
904
V1L
72.3
1T.
41.6
97.7
15.4
18.8
41.8
11.4
77.6
S7.
680
88.5
77.
518
61.
McsCLIP 03.2
17.44
88.1
94.5
78.2
17.5
66.4
79.3
198.0
57.1
9233
98.5
75
36.4
41:8
98.0
50.5
70.1
49:3
36.8
21.6
13.7
[241
12.71
21.6
56.8
61.6
M53 00.55
19577
79.0
(2.9
30.3
(2.1
75.9
66.
22.7
30.8
57.44
Mearr IPOMOMM
671
40.7
05.5
77.4
75.9
62.0
96.4
76.1
61.7
99.1
59-7
11.8
76.9
13.9
76.1
57.1
96.6
65.
MesCLIPNLR
1002
84-7
7100
31.2
MITHOT
McXALINS258)
724 9955 142 96.0
96.4
83.4
74.1
10L0
2012
95.4
15.5
15.1
12.7
55.2
19.4
6533
14.5
625
11.2 3122 05.
822
64.1
30.1
50.5 01
McUCLIN225B)
73.2 82.1 949 98.5 866 SAD 743 MLD 523
726
96.1
033
593
18.1
560
59.5
13.7
155
617
31.0
413
656
856 638 24.5 58.5 63.
Table 5: Scaling MetaCLIP from 400M (t=20k) to 1B (t=20k) and 2.5B (t=170k) training data.
Training Setup We strictly follow the CLIP training setup, using V100 32GB GPUs and an equiv- alent global batch size of 32,768. For ViT-B/32 and ViT-B/16, we use 64 GPUs with a per GPU batch size of 512 and for ViT-L/14 we use 128 GPUs with a 256 per GPU batch size. It takes 4 days to train ViT-B/32 and a month to train ViT-L/14, We use 256 A 100 80GB GPUs to train ViT-H/14 and ViT-bigG/14 model for I week and 2 months, respectively. We train in all experiments for the same number of iterations that correspond to 12.8B seen image-text pairs during training (32 epochs for 400M). We pre-process with face-blurring.
4.1 RESULTS
Zero-shot Image Classification. We follow the standard evaluation benchmark and made sure all prompts and class names were the same as those used by CLIP Radford et al. (2021). We also re- evaluated OpennnIOpsn checkpoints to avoid differences caused by benchmark data copies. The results are shown in Tab 4. The standard deviation of training multiple seeds is relatively small with + 0.1 1% for ImageNet ViT-B/32.
In Table 4, we observe that MetaCLIP outperforms OpenAl CLIP on ImageNet and average accuracy across 26 tasks, for 3 model scales. With 400 million training data points on ViT-B/32, MetaCLIP outperforms CLIP by +2.1% 1% on ImageNet and by +1.6% on average. On ViT-B/16, MetaCLIP outperforms CLIP by +2.5% on ImageNet and by +1.5% on average. On ViT-L/14, MetaCLIP outperforms CLIP by +0.7% on ImageNet and by +1.4% on average across the 26 tasks.
We next turn Pool 2 which is larger set of image-text text pairs and study the effect of scaling data. In Table 5, we scale data to IB and 2.5B and observe a large gain over 400M, with similar performance for both 1B and 2.5B scales. Note that the number of training iterations (and therefore compute) is the same for all rows. The main difference between 1B and 2.5B is the threshold t, where 1B is a more balanced set by adding more data points (compared to the 400M set) to tail entries (up to t = 20k), instead the 2.5B set adds (up to 170k) data points to all, head and tail, entries. The extra data in the tail entries (IB set), seems to benefit downstream accuracy for tasks on specific data such as CUB fine-graince bird classification, Flowers. KITTI, PCAM, while the larger 2.5B data that has more head entries increases broadly over more datasets, but each at a smaller amount. The overall average accuracies are similar for 1B and 2.5B (e.g., 70.2% 69.8% for ViT-L model
8

---

Published as a conference paper at ICLR 2024
Cumulative Entry Counts
2.0
2.5-
Pool 2 (10.78)
1.0-
Pool 2, t=170k (2.56)
0.5-
Pool (1.68)
Pool 2, 1=20k (16)
0.0
Pool 2.22 (400M)
Metadata Entries Sorted by Counts
Figure 3: Cumulative sum of counts on entries from tail to head on a data Pool 2. We again show (1) raw/unbalanced cumulative counts), = oc; (2) balanced cumulative counts after applying = 20k
and = 170k. t defines maximum number of counts per entry and the transition of tail/head entries. We show the Pool 1 configuration from Fig. as dashed lines for reference.
size). On ImageNet, the 2.5B training data achieves 67.6% on ViT-B/32 that breaks the previous believed saturated B/32 models (Cherti et al., 2022), 79.2% on ViT-L/14, 80.5% on ViT-H/14 and 82.1% on ViT-bigG/14.
We plot the cumulative sum of counts for entries sorted by counts from tail to head in Fig. 3 for all these cases, similar Fig. for Pool (and the Pool configuration as dashed lines). The plot shows that the 2.5B data is still relatively long-tail, while the IB data is more balanced, explaining it's better performance on specific data such as bird and flower types observed above.
4.2 ABLATION STUDY
We show ablations for MetaCLIP for the 400M scale and ViT-B/32 in Table 6. We first ablate different balancing thresholds f. We observe that the choice of 1 = 20k by CLIP yields the best performance for ImageNet and averaged accuracy and 15k = 35 are slightly worse.
To understand the key effect of balancing, we use the whole matched pool (1.6B image-text pairs) to train CLIP. Surprisingly, training on 4x more data (on head entries) significantly hurts the accuracy on ImageNet (61.9 vs 65.5) and averaged accuracy across 26 tasks (56.6 vs 58.2).
Balancing can also be applied online in the data loader with head entries down-sampled leading to slightly better performance (58.5 vs 58.2); see appendix for details. This is useful if head data has already been collected and one wants to train on different distribution. The better accuracy for online balancing is explained by the larger diversity in head data.
MestLiP
$7.5
66.5
25
32.1
09.
35.8
64.1
52.3
33.5
57.
654
91.2
M6.0
63.0
72.11
263
52.7
06.
42.0
3011
618
402
12:7
16.1
64.2
41.3
512
snbutaneed (1 RB3
30.6
61.9
76.9
MLD
67.5
368
7900
18M
31.0
KL1
V1.5
66.3
36.2
31.0
95.1
55.2
562
41.4
321
15.1
51.6
39.2
41A0
11.2
55.6
32.6
[61
901
999
63
654
71A
779
85.1
2
M
68.6
16.1
13.1
86.5
51.5
W6
78
176
86.0
193
33.0
55.0
$7.4
Table 6: Ablation studies on balancing in MetaCLIP. Default: /=20k, 400M. Model: ViT-B/32.
CONCLUSION
In this paper, we attempt to reveal CLIP's data curation. Our MetaCLIP builds upon metadata for curation and balancing of raw data sourced from the web. Curating with metadata and balancing are essential for good data quality, significantly outperforming the use of raw data. Our experiments show that MetaCLIP performs well for different scales sourced from CommonCrawl data and out- performs CLIP's s proprietary data source, without reliance on any external model. We make our pipeline for generating the data publicly available.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank Zeyuan Allen-Zhn -Zuu, and Chunting Zhou for the insightful discussion and Brighid Meredith for suggestions on scaling the pipeline.
A
APPENDIX
A.1 ADDITIONAL RESULTS
Curation from DataComp-12.88.
The concurrent work Gadre et al. (2023) released a collection of 12.8B image-text pairs from CommonCrawl from 2014-2022. We further inves- tigate whether we can apply the al- gorithm on its 12.8 unfiltered pool. Although the unfiltered pool seem- ingly offers an opportunity to apply our algorithm source, our initial studies show that,
0.5
publicly available
ImageNet Zero-shot
0.50
implicit biases may still be present
in this pool. For example, we no-
tice that all image URLs are collected
0.5
as a string starting with http. This excludes relative URLs that could be frequently used by quality web-
sites (with potentially good image- text pairs). We curate from Data- Comp's 12.8B unfiltered pool with 1=60k, which has 6% of tail counts that the same as 1=20k for 400M from our 1.6B pool.
CLB740056
5:000 100030
159000
220000
250003 336000
35:000 400000
Training Steps
When using IB image-text pairs cu- rated from DataComp's pool we no- tice a quality drop during training, compared to data curated from pools, see Fig. 4. Our smaller 400M set is slightly better than ing DataComp- IB and our larger sets (IB, 2.5B) are significantly better.
Figure 4: VIT-B/32 unfiltered 12.8B pool. We show ImageNet zero-shot classi- fication with fixed 12.8B seen pair training budget. Meta- CLIP's curation is effective for all pools. However, with the same curation method, the unfiltered DataComp-122 -2.8B pool lacks quality (we suspect it is caused by implicit filters in the parser DataComp).
Pool 1, Pool 2 and DataComp's
In Table 7, we show our 400M data vs our curated DataComp- IB data at various model scales, where the same observation holds, suggesting our raw data pool more effective.
Meat HOIM:
58.2
0655
91.
57%
55.7
91.
11.5
15.4
454
65.1
154
306
56.1
$1.4
Mext (iiinneeer
$7.
654
81
57%
19-1
107
16.3
193
17.00
51.
NTERIS
MIKCIP
718
513
90.1
72.5
41.
15.4
5K
15.5
22.8
543
686
MICLIP DannChepp
83.
7177
56.1
68.37
70.7
99.6
396
46.2
96.
559
MACLIP HRODS 67.1
76.2
91.1
55.5
17.
25.9
10.5
$4.7
2044
62D
99.7
94.4
16:4
61.7
46.5
1933
59.2
42.5
23.9
389
755
57.1
35.1
55.6
65.6
MACCIP DeDeer
65.3
16.7
92.
89
189
22.1
69.5
05.3
76.0
65.1
42.4
99.1
$1.0
15.6
59.5
203
59.1
808
Table 7: MetaCLIP curating our 400M data vs curating IB data from DataComp-12.82 12.8B: The pool of DataComp leads a quality drop with performance closer our 400M set.
DataComp Benchmark We also evaluate MetaCLIP on the benchmark used by (Gadre et al., 2023) that contains 38 tasks including variants of ImageNet, retrieval, VTAB, etc. For simplicity, average the scores over each category.
10

---

Published as a conference paper at ICLR 2024
Avg.
IN
Dist. Shift
VTAB Avg. Retrieval
VTT-B/22
CLIP (400M)
51.5
63.4
48.2
50.5
48.0
OpenCLIP (407M)
52.7
62.9
48.5
53.0
50.7
MetaCLIP (400M)
53.5
65.5
50.4
54.1
50.6
MetaCLIP (1B)
54.2
67.3
51.9
53.6
51.1
MctaCIII (2.5B)
55.4
67.6
52.3
55.3
52.6
VTT-B/16
CLIP (400M)
55.5
68.3
54.1
54.4
50.2
OpenCLIP (407M)
56.1
67.0
52.6
54.9
53.9
MctaCLIP (400M)
57.5
70.8
55.5
56.7
53.9
MetaCLIP (IB)
58.4
72.4
57.8
56.3
54.3
MetaCLIP (2.5B)
60.0
72.1
57.7
59.0
54.0
VIT-L/14
CLIP (400M)
61.4
75.5
61.6
59.5
53.6
OpenCLIP (407M)
59.7
72.7
57.3
58.6
55.9
MctaCLIP (400M)
62.2
76.2
61.3
59.8
57.3
MctaCLIP (1B)
65.0
79.0
64.5
62.5
58.3
MetaCLIP (2.5B)
65.6
79.2
64.1
60.1
VTT-1/44
MetaCLIP (2.5B)
66.5
80.5
66.1
64.6
60.4
ViT-bigG/14
MetaCLIP (2.5B)
68.3
82.1
66.5
62.6
Table 8: Zero-shot classification and retrieval tasks from (Gadre et al., 2023).
Note that prompts and class names used by (Gadre et al., 2023) could be different from prompts and classnames used by OpenAl CLIP.
From Table 8, we can see that MetaCIIIP outperforms CLIP and OpenCIII across various model sizes. First, for the same data scale (400M), MetaCLIIP outperforms OpenCI.IP, which is better than CLIP this benchmark, by +1.4% for ViT-B/16 and +2.5% for ViT-L/14, when comparing average accuracy across the 38 tasks. Second, for increasing our MetaCLIP data size to IB we see significant gain, especially for the larger model, from 62.2% 65.0% average accuracy. Using larger dataset with 2.5B and more head entries leads to further gain to 65.5%.
We further detail the breakdown of each dataset in Table 9 (please view in landscape orentation).

---

Published as a conference paper at ICLR 2024
638
165.0
88.1
13.3
8800
87.
88.
89.3
91.1
91.4
50.2
702 925
53.0
51.9
S5.0
19.5
5588
20.3
60.6
660
030
61.3
67.5
69-4
67.4
W
13.0
159
16.7
13.5
19.9
19.4
15.2
17.0
18
28.1
15.6
63
2.4
X-4
11.1
10.5
11.2
12.5
12.3
12.5
14.1
16.2
15.0
17.3
183
95.6
96
96.3
97.0
97.2
98-1
99.4
99.3
99.3
994
9945
79 S
730
77
64.7
33.7
74
$300
77.5
$4.8
36.5
89.5
52.3
S7.4
532
54.4
60.
591
65.6
66.0
66.3
(2. 60L
631
6223
55
S010
56
30.7
596
619
5912
52.
703
754
73
70
78.5
78.8
72.2
78.2
7M.
80.9
75.0
741
44.3
42.9
407
52
55.3
59
614
60
739
74.6
76.5
7722
S7--
356
426
513
602
478
68
76.3
613
539
64.4
72.
90.5 41.5 52.7 948 61.7 7N1
CISIM
A20
43.3
43.4
43.7
46.
50.0
40.9
47.5
59.6
56.3
61.
800
S4.7
86,1
67.
58.
SE.1
90.
98.1
S8.1 51.2 510 9422
Table 9: Zero-shot evaluation of Table 8 broken down by datasets.
27
24.3
17.7
2554
305
31.7
393
196
21.
D8.1
22.6
22.7
31.9
30,9
312
33.0
9007
952
90.8
90.1
99.7
55.0
95.5
96
38 0
S5.6
36
35.3
40.2
36.9
33.5
39.2
40.2
38.6
57.5 38.8
44.4
Ramioval
40.3
43.0
451
42.8
46.9
48.2
50.4
46.4
51.9
54.2
55.7
56.1
70
7266
72.9
72.2
74.5
76-7
77.1
78.
75.1
29
922
85.3
85.0
#544
47.1
49.4
56.6
60.2
50.9
67.7
89
00.6
00
75.5
66.2
76.1
20,4
20-3
21.7
34
40.2
24.3
252
29.1
52.8
52.4
46.8
62.5
604
670
633
66.3
660
644
668
668
676
226
208
729
330
244
251
$4.5
6.6
912
5K2
583
666
67.
033
67.4
085
680
T3A
70.3
7401
38.9
M02
904
913
5922
93.
952
64.5 213 845 55.
96.2
662
732
6944
69.
609
723
735
792
765
352
3911
SITTE FERLL BEEEE
VTAR
274
28
250
2005
187
263
1KL
242
184
21
242
232
27.9
26.3
S
560
202
557
52.7
626
604
69.5
29.0
72.8
44.9
557
622
55.3
624
65.5
15M 72.7
72.1
24
22.5
23
22.3
22-4
226
2012
25.
22.5
22.6
2033
23
7
21.2
28.7
30.1
29.0
19.
22.7
13.
21.1
17.1
70.3
76-S
66.9
71.2
66.5
78.9
75.8
72.3
348
30.4
S8.5
012
91.3
934
933
923
9
38.1
953
05.7
60.4
73
76.1
78
77.7
77.9
8LK
842
$2.9
58.9
91.0
301 934
94.1
Phariittion Shit
535
40.8
48.2
483
423
506
392
415
3233
283
3166
289
21-7
2M,6
20-7
28
28.2
47.0
48.
70.7
06.4
70.4
72.3
75.4
784
$5.1
$7.4
59.2
6
61.9
59.7
62.6
65.0
65.
00.8
72.5
7U. 74.1
76.0
S533
560
4K-3
22.4
57-9
6022
59.5
65.0
08.9
6699
728
0
6556
67.3
67-
6K3
6700
7008
72.1
750
728
7X2
790
792
MLLS
X21
OpenCLIP PEINCLIPPOPMM H07M0
MoucI.ie (40MM)
MasCIIP PP (1B)
McnCLIP (2)58)
CLI CLIP (00M0)
OpenkLiP (UTM))
McMCLIP (4AP)))
MenCLIP
MeuCLIP (2.50) (2550)
CLIP CLIPAADON (ACMM))
OpenCTJIP (4TTN)
MeDCLLIP HP (OOMI
McoCLIP (ID)
MeuCLI IP (2.58)
MAUCLIP (2.58)
(2.58)
MITER In
MEDR
MetaCLiP
IP
12

---

Published as a conference paper at ICLR 2024
A.2 DETAILS ON EFFICIENT CURATION
Curation in Data Pipeline. Our data collection pipeline contains 5 major components: HTML Baaiing/Laagggg-ldeeniffiee
URLs & Text deduplication, Image Download, NSFW image filter & deduplication and packaging, which are described next.
Our HTML Parser is applied all WAT files of CommonCrawl by keeping all <img> tags with both relative and absolute URLs and one or multiple text keys.
internal language identification system that can detect 191 languages with different dialects and keep all texts that are tagged as English (or its dialects).
kanguage-Identifcattoon
URLs / Text deduplication: We use 24bit sha224 hashing encode URL and into tables to deduplicate columns with the same hash, avoiding downloading the same image URL multiple times. URLs with illegal domains are removed: if there are multiple texts associated with the same image, these are further deduplicated; texts with NSFW keywords are removed;
NSFW filter:
We use an internal system that can classify inappropriate content in images into 96 types of dangerous content and discard such image/text pairs. Images are further deduplicated by 64-bit PCA hash, derived from a similarity search model's feature embeddings with PCA reduction to 64 dimensions and sign quantization.
Our curation algorithm does not require access to images, making suitable for integration into pipeline to reduce the scale of data points after parsing and before image downloading. We designed the algorithm to be modular, allowing different parts to be placed at different stages in the pipeline, shown in Figure 5.
Specifically, sub-string matching can be placed immediately after HTML parsing to reduce data points for English-only pairs (c.g. by ~50%). Balancing can be applied earlier, before image down- loading, to further reduce data points by ~77%. This approach led to a total reduction of ~990, allowing for curation of the entire CommonCrawl data without the need to store and transfer all data points, which allows us to curate the whole CommonCraw/ since 2013 with 300B+ scale URL-text pairs without spending the storage/transfer on all ~ 10x data points, where the rate of keeping data point with MetaCLIP curation is ~011 (0.5 x 0.23).
HTML Parsing/LID
URLs Text Dedup
Image Download
NSFW Image Dedup
Packaging
Partl: Sub-string Matching
Part2: Balancing
50% reduction
77% reduction
Figure 5: Case study:
Curation implementation in our data pipeline.
Curation in Data Loading We applied the balancing/sampling part of the algorithm to the data loader adjust data distribution on-the-fly. Although data points for tail entries always sampled in Algorithm 1, the diverse pairs from the head entries are sub-sampled from a larger pool while maintaining similar distribution offline curation. This diversification of pairs that match the head entries improved the performance, as shown in Table 6.
A.3 HUMAN STUDY ON THE EFFECTS OF CURATION
In this section, we study the impact of MetaCLIP curation on data distribution by using human evaluation. We approach this exploration from three distinct angles: noise reduction, alignment of visual content, and task-agnostic attributes. In the pursuit of comprehending the first two aspects, we undertake a human study aimed at comprehending the data quality implications of implementing the balancing technique (outlined in Part of Algorithm 1). This evaluation encompasses three dimensions: image-only, text-only, and image-text alignment. We collect evaluation set of 100 random image-text pairs for balanced and unbalanced data, respectively, and ask annotators to score on the image, text, and pair quality metrics, separately, scale of to 5.
13

---

Published as a conference paper at ICLR 2024
Annotation Guidelines. Annotators follow guidelines to assess both images and texts, evaluating informativeness (how well information is conveyed) and aesthetics. For images, aesthetics considers visual elements like composition, color, lighting, balance, contrast, texture, and subject matter. For texts, aesthetics gauges factors like delimiters, sentence structure, capitalization, prefixes/suffxes, recognized words, generic words, and overall text quality. The alignment metric for image-text pairs measures the relevance between the two modalities, assessing how well the text describes the image content. Ratings are averaged across annotators for each dimension.
We show the study results in Table 10 and discuss the different criteria next.
Evaluation Dimension
Rating for Balanced Data
Rating for Unbalanced Data
P-value
Image
4.60, [4.50, 4.70]
4.36, [4.23, 4.48)
0.001
Text
4.67, [4.56, 4.78)
4.06,
[3.82, 4.30]
0.001
Alignment
4.41, [4.23, 4.59]
3.72, [3.46, 3.99]
< 0.001
Table 10: Average human rating on the effect of balancing on data quality, with confidence intervals shown in parentheses. Higher rating is better. Balanced data is rated of higher quality.
Noise Mitigation in Image and Text. As shown in Table 10, significant quality improvement for all the three evaluation dimensions observed after applying balancing. MetaCLIP curation has no specific hard filters such as removing shorter text, removing dates, etc. However, curation by sub-string matching and balancing has a different filtering effect. For example, a sub-string itself can never curate date-only text. Further, balancing allows signal and noise to co-exist when they are difficult to be separated by human designed filters. For example, if one entry such as "image" or 20k, it can only contribute 0.005% of 400M data.
"photo" is capped
Visual Content Alignment. Although MetaCLIP curation does not directly involve images, it has a positive effect on aligning visual content by controlling the quality and distribution of text. First, sub-string matching increases the chance of having (visual) entities mentioned in the text, thereby improving the likelihood of finding corresponding visual content. Second, balancing favors long- tailed entries that could have diverse visual content than head entry (such as the text "1"). In Table 10, we observe significant improvement on pair quality from unbalanced data to balanced data.
A.4 MEASURING TASK- -LIIGAELN
With metadata, one can measure the alignment of pre-training data distribution with the data distri- bution in downstream tasks. First substring matches that are directly corresponding to class names in the downstream tasks. In the first row Table 11, we show the accuracy of MetaCLIP (400M), ViT-L (cf. Table4) for reference. The second row shows the number of classes that are present in the metadata, for each downstream dataset. For example, for ImageNet 703 of the 998 unique class names are present in the metadata. Interestingly, there seems to be correlation with the accuracy and the number of classes matched in the metadata.
do simple measure of counting the number of metadata
ME (OMMM)
W0.7
T7.
703
SAT
230
BLT
463
993
293
509 71 153
351 356
15TL TOUD
120700 ME
calal. 51
6.5
9.
75
6.1
11.4
6.7
50
12.0
03
11.7
733
186
3.4
89
7.1
84
76
9.3
0.0
57
14.5
8.9
8.9
3.8
02 88
hol
7.4
61
1044
161
80
MT
74
86
K1
81
81
93
32
184
00
533
12.5
7.1
8:3
1044
97
Table 11: Measuring task-alignment. First row: MetaCLIP (400M) ViT-L/14 accuracy, second row:
number of classes matched in metadata, Third and fourth row: We use KL-divergence (lower is better) to measure the similarity between pre- pre-training distribution and benchmark task distribution. Note that, for ImageNet, two classes have duplicated class names and therefore there are only 998 unique classes (out of 1000).
14

---

Published as a conference paper at ICLR 2024
Hyperparameter
OpenAl CLIP/ MetaCLIP
OpenCLIP
DataComp
Activation Function
QuickGELU
GELU 13B (407M epochs)
GELU
Seen Pairs
12.88(400Mx33 epochs)
12.8B
Batch
32768 (B0e4((B32../16),, 4.0c-4(L144)
32768 (B/32), 33792 (B/16), 38400 (L/14)
90112 (L/14)
Learning Rate
5.0c-4(B/32)
1e-3(L/14)
Warm-up
2k
2k (B/32)
10k (1/114
Table 12: Hyperparameters of OpenAl CLIP vs OpenCLIP on LAION-40M(SShuhmann et al., 2021) and DataComp IB.
Next, use KI Kl--eiveeeence measure alignment:
P(m) (2L(T)))) = E T(m) log T(m)
where T(m) represents the task distribution over M and P(m) represents the pre-training data distribution. We compute T(m) using the benchmark data distribution over class labels of task. example from Table 11, T(m) ImageNet has 998 entries uniformly dis-
Taking ImageNet tributed (each has evenly examples) and P(m) has normalized counts over all 500k entries but only 703 entries used to compute KL-divergence. In the third and fourth row of Table 11, we can see improvements for balanced data points over unbalanced ones for each task, showing balancing improves similarity with most tasks.
A.5 TRAINING SETUP OPENAI CLIP VS OPENCLIP
Our work strictly follows CLIP's setup for a controlled comparison focusing on data curation and quality. We notice differences in the training setup of OpenCLIP' and list the difference (known to us). OpenCLIP varies the setup from CLIP (c.g., global batch size, learning schedule, etc.). Here only list the difference for LATON-400M(Schumaam et al., 2021), which is closer the CLIP setup. We note that DataC differs even more, e.g., by curating images close to ImageNet training data, a large batch size of 90k that is almost 3 larger than CLIP, and using the CLIP model filter data.
A.6 BENCHMARK DEDUPLICATION
Our pools are deduplicated from the benchnarkk/mageeNtt data using a 64-bit PCA hash, derived from similarity search model's feature embeddings with PCA reduction to 64 dimensions and sign quantization. DataComp- 12.8B is already deduplicated.
A.7 NEGATIVE RESULTS LEARNED FROM ABLATING CLIP CURATION
We briefly describe few ideas close CLIP curation that did not look promising in initial attempts and were abandoned:
1. Self-curated Metadata. We initially attempted build metadata directly from the text in raw image-text pairs (i.e., using terms appearing in text above a certain threshold of counts). We rank entries by count and keep the top 500,000. Metadata built this way appeared worse. We notice that although the top frequent entries are similar to CLIP's metadata, the long-tailed part very different. For example, the minimal count be in the 500,000 budget needs at least 130 counts. In contrast, our metadata has 114K entries that have no matches. This approach results in worse quality metadata including low-quality spelling/writing (instead of high-quality entries from WordNet or Wikipedia). Further, the effect of balancing saturates earlier for such data (in larger f, verified by CLIP training) since low-quality entries are also heavily in long-tail.
2. Cased WordNet. We also notice many cased words are missing from metadata (e.g., Word- Net in lowercase). After adding cased WordNet into metadata, we notice performance drop on ImageNet. The reason could be class names are more likely in lower case and upper case entry matching may reduce the written quality of texts.
httpp::// //gi thub. com/m1 Foundat i cans/open_cc11
15

---

Published as a conference paper at ICLR 2024
3. Stopwords/Useless Entries Removal We further study the effect of whether removing stopwords and useless words such as "photo" and "image" is beneficial. This led to al- most difference since balancing entries reduced the effects of useless entries (each entry contributes to 0.0002% (1/500k) level the total data points). To encourage a simplified solution, we do not intend add more artificial filters.
A.8 MORE DETAILS ON DISTRIBUTION ON METADATA
Extending the top-20 matches in Table 3, further group counts of metadata entries and show examples per group as in Table 13.
Group
5 Examples (Entry:Count)
O-10k
ivailo: 12, Kunta Kinte:201, vikernes:33, peria:50, ankoku:20
10k-20k
queer: 19k, barry: 10k, bandages: 12k, The Police:15k, sigma:14k 14k
20k-50k
polygonal:2 Ik, widely:28k, however:35k, toppers:25k, executives:2
50k-100k
planted:52k, olive oil:58k, yours:63k, packages:82k, Spokane:53k
100k-500k
compact:1 vertical:222k, underwear: 11k,, powder:323k, weekly:
500k-1M
Tokyo:713k, Lead:620k, Diagram:809k, Dye:858k, unnamed:512K
1M-50M
see: .MM, Door:3.2M, News:2.3M, sea:1. 1M, street:
50M-130M
with:67M, and: 100M, to:61M, in: 107M, of: 1f:1
Table 13: Distribution of metadata entries with counts, similar head shown in Table 3.
A.9 RANDOMNESS OF ALGORITIM 1
We further study the randomness of algorithm by running 3 times for Pool 1(400M) and Pool 2(2.5B). This ended with a standard deviation of 4035 examples for Pool 1 and 2104 examples for Pool 2, respectively.
A.10 QUALITATIVE DATA EXAMPLES
In Table 14, we illustrate data before/after sub-string matching and balancing. We also highlight class labels from ImageNet the table. mark matched entry with a bigger font size indicating higher probability of sampling that entry. Intuitively, sub-string matching removes low quality text and balancing favors longer text with long-tail entities to improve data quality. Table 15, we show more examples of matched text that include ImageNet tail entries.
16

---

Published as a conference paper at ICLR 2024
Text
Substr. Bal. IN Head IN Tail
control 14ct
X
cudd2008
x
x
product-img
X
x
X
X
A4omote
X
X
X
X
hanukkah-pann Baiiiiaah-prtt-----e part--accbooo-- patty-hhccoolleeett--ooeetttehet
X
X
johnny_ sl___aash_h_hhe beassshhhli_dee _do (2)
X
X
8533 GOLDEN RIDGE COURT
X
x
X
X
How to build a stone patio on your own
x
battery plate
barn wedding
Picture sand machine, Concept jeep the concept, the
/
x
front, Slim, Wrangler,
X
X
desk
Adult T-shirt
x
Imix m00 stage moniter
x
X
google wallet md3
Distressed beach bar sign Pearly's Oyster Bar
x
Why the Kilimanjaro Trek should be top of your bucket list
/
x
J70 desk model
/
x
/
Whitby castle
/
x
/
Inside the castle
x
/
Hama Hama Oyster Saloon restaurant 338446 US-101 Lilliwaup, WA 98555, USA 3608775811 OR +1 360-877-5811
x
x
beach
Caramelized onions. sauteed red bell peppers and
zucchini combined create winning egg frit-
tata breakfast dish.
Vector layered paper cut craft style music composition of sasaophonne guitar trumpet violin music in- struments notes abstract color background. Jazz concert festival party poster banner card template
Nautilus hot tub
night binoculars for hunting
2017 cat eyes women's sunglasses for women vintage sun glasses round women sun glasses oculos oculos de sol feminino
Table 14: Examples categorized by whether passing sub-string matching and balancing: Words in violet color are in metadata and their font size indicates probability of being sampled, ranging from
13pt that has probability close to 0, to 22pt that has probability 1. ImageNet labels in head entries are cyan.
17

---

Published as a conference paper at ICLR 2024
Text
Substr. Bal. IN Head IN Tail
Antique German sterling silver soo cup julep goblet.
A journey to the East Sea by high-speed train
New KTX line makes Gangneung's wonders all the more accessible
little arrow design co watercolor rainbow blush shower curtain and mat
photo of antique silver top break revolver
trombone model
X
Staffordshire Bull Terrier
KI Plantation Timbers fire truck on the night of Jan- uary 3, 2020. Photography Tim Wilson
/
X
water buffalo bath
Basset Hound pup with Lionhead X Lop rabbit
Single serving of peach trifle
x
(hairy arachnid belonging to the Theraphosidae family of spiders) in box, standing still
A tarantula
Close-up shot.
Modern Staffordshire Bull TerrierLED
Night Light Animal Pet Dog Puppy 3D Optical illusion
Lamp Home Decor Table Lamp Desk Light
Amazon rocking chair, CANVA. camping chairs
kukri machete fitted inside scab- bard
dispalying
jacksons chameleon
X
Wall Mural Insects pollinating blooming
rapeseed crops in field
Scottish Terrier
Phone Pocket
Table 15: Examples passing both sub-string matching and balancing wit ImageNet tail classes: Words in violet color are in metadata and their font size indicates probability of being sampled,
ranging from 13pt that has probability close to 0, to 22pt that has probability 1. ImageNet labels in tail entries are in cyan.
18

---

Published as a conference paper at ICLR 2024
REFERENCES
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data- efficient learning web-scale through semantic deduplication. arXiv preprint arXiv:2303.0955 2023.
Olivier Bachem, Mario Lucic, and Andreas Krause. Coresets for nonparametric estimation-the case of dp-means. In International Conference on Machine Learning, pp. 209-217. PMLR, 2015.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-iman image learning. arXiv preprint arXiv:2212.07145. 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- ence on Learning Reprexentations, 2020.
Dan Feldman, Matthew Faulkner, and Andreas Krause. Scalable training of mixture models via coresets. Advances in neural information processing systems, 24, 2011.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim En- tezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexan- der Ratner, Shuran Song. Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets, 2023.
Sariel Har-Peled and Soham Mazumdar. On coresets for k-means and k-median clustering. In Proceedings of the thirty-sixth anmual ACM symposium Theory of computing, pp. 291-300, 2004.
Mon-Fong Jiang, Shian-Shyong Tseng, and Chih-Ming Su. Two-phase clustering process for out- liers detection. Pattern recognition letters, 2266-7):6117700, 2001.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950-6960.
PMLR, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- $748-8763. PMIR, 2021.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02111 2021.
Christoph Schuhmann, Romain Beaumont, Cade W Gordon, Ross Wightman, Theo Coombes, Aarush Katta, Clayton Mullis, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. 2022.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neu- ral scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523-19536, 2022.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler. Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint /8/2: 2018.
Hu Xu, Saining Xie, Ya-Yo Huang, Licheng Yu, Russell Howes, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Cit: Curation in training for effective iision-langusa data. arXiv preprint ar XrXiv:2301 022441 2023.
19

---

Published as a conference paper at ICLR 2024
Dantong Yu, Gholamhosein Sheikholeslami, and Aidong Zhang. Findout: Finding outliers in very large datasets. Knowledge and information Systems, 4:387-412 2002.
20


Reviews:
[{"review_id": "hg9i0Xhdwi", "reviewer": "Reviewer_7LK5", "summary": "The paper proposes a data curation pipeline for CLIP training. The paper evaluates the model performance resulting from different data curation pipelines, and shows that the proposed pipeline outperforms closed-source CLIP, as well as Open CLIP. Moreover, an interesting finding from the paper is that balancing the metadata distribution is the key to improving performance.", "questions": "1. Page 3\"this post-hoc data pruning approach has limited utility, as the computational resources saved have already been expended during the initial training of the model.\" The reviewer has some concern regarding this comment. While posthoc data pruning still requires training an initial model, the pruned dataset can also allow efficient model improvement, which is usually an iterative process involving experimentation with different model hyperparameters.\n\n2. Is there a specificreason to omit some other data curation baselines proposed in the DataComp paper? \n\n3. What is the meta data distribution for OpenCLIP?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles an important problem in the community regarding the opacity of data curation processes of foundation models. Moreover, it promises to open-source part of the efforts, including data curation code and training data distribution.\n\n- The paper is also strong in terms of empirical evaluation, including various data sizes, and different implementations of the balancing steps. Huge resources are devoted to the evaluation part.", "weaknesses": "- The main weakness lies in the technical novelty. The paper is a commendable effort to reproduce the data curation pipeline that has already been described in the original CLIP paper (Radford et al, 2021), and report the findings for reproducing an existing data curation technique. The paper's novelty can be greatly enhanced by exploring some new technical components beyond what's already described in Radford et al 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "khwjF7k48c", "reviewer": "Reviewer_33mj", "summary": "This paper aims at reproducing the data collection and construction process of the original CLIP, which is still unknown to the reserach community, despite of the brief introduction in the CLIP paper, and some followup public datasets like LAION or DataComp. Specifically, it first collects some high-quality metadata based on Wikipedia and WordNet, uses them to retrive text entry from snapshots of CommonCrawl, and employs a balancing strategy to build a more diverse and balanced data distribution. The CLIP model trained on the resulting dataset shows comparable or superior performance compared to CLIP or OpenCLIP.", "questions": "1. How does different choices of metadata affect the final dataset? I know the authors follow the procedure listed in the original CLIP paper, but I feel like the decision to use Wikipedia and WordNet can be futher explained by 1) either a comprehensive distribution probing or visualization of the metadata used in the paper (right now only top-20 entried are showed, and they seem very general and vague wordings); 2) remove some metadata sources (like no wordnet synset), or tune the threshold hyper-parameters like PMI or view frequency.\n\n2. Could in data construction process inspire data creating efforts in other areas? The authors are encouraged to add some related discussion, possibly in a *broader impact* section.\n\n3. In section 4, it is mentioned multiple times that using a smaller *t* can *increases tail metadata matches*, like the last sentence in page 7. But my understanding is that a smaller *t* only leads less matches of head metadata, while all the matches of tail metadata are kept. Am I understanding it wrong?\n\n4. Since randomness is introduced in the balancing stage, I wonder how much impact does it have with different random seeds?\n\n5. Will the authors released detailed code (not just some demo) to reproduce their whole pipeline of data curation from scratch? For this work, I think it is not the resulting dataset MetaCLIP that is most valuable. How the authors arrive at the final dataset from some wiki pages and wornet synets to the final collection of diverse and high-quality image-text pairs really matters, and could make a big difference and push the community forward once open sourced.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The effort to reproduce the exact data construction procedure of the original CLIP paper is well motivated and appreciated. As the authors pointed out, the later datasets like LAION or DataComp, all adopt trained CLIP model during data collection. How to build a high-quality and diverse image-text dataset from scratch, like WIT400M, is still a mystery to the community.\n2. Given that CLIP is such a important foundation model that connects image and text, the data crafting pipeline and the resulting dataset are of clear importantce to the community. Also, the lessons learnt in this work could potentially benefit future efforts to construct high-quality open datasets in other areas as well.\n3. The writing is good. The overall logistics follow a reasonable thread of investigation.\n4. The performance of the model trained on MetaCLIP often show superior performance compared to LAION or WIT.", "weaknesses": "1. I find that the authors use the average accuracy across multiple datasets as a major performance metric throughout the paper. This is examplified by table 4/5 in the main texts, and also some tables in the appendix. This does not make sense to me. Those datasets come with different number of classes and number of samples. For instance, averaging the accuracy of a dataset of 10 classes (e.g. EuroSAT), and a dataset of 102 classes (e.g. Flowers), is unreasonble, because misclassifying all samples in one class corresponds to 10% accuracy in the former dataset, but corresponds to ~1% accuracy in the former dataset. I know that some other works, like DataComp, also keep this practice. But I feel a more reasonable approach is to compare the performance on individual datasets, and how many datasets on which the MetaCLIP model prevails. As such, the authors should list results on individual datasets, not report some average accuracies.\n\n2. Following previous one, I find results in table 8 in appendix tend to be more comprehensive, as the model is also evaluated on ImageNet distribution shift datasets and retrieval datasets. The organization of this paper can be enhanced if the table 4/5 in the main text also report results on those dataset.\n\n3. Some data filtering details could be further detailed. For example, the last sentence in page 4 (sorry but this submission does not have a line number) talks a bit about some *noise reduction... (e.g., regular expression on dates, ids etc.)*. Also, figure 5 in the appendix shows some *URL & Text Dedup* and *NSFW Image Dedup*, but does not describe them in details. The authors are encouraged to list more detailed descriptions on those noise reduction and deduplication processes, and also explain their effects on the resulting dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fNgcANHHkV", "reviewer": "Reviewer_f2FV", "summary": "This paper attempts to uncover the data curation process of the CLIP paper by Radford etc. The curation process is composed of several steps: 1) 500k entry construction 2) text2entry matching 3) entry2text indexing 4) entry balancing. Applying this pipeline to the CommonCrawl dataset, the resulting dataset achieves even better performance CLIP.", "questions": "1. How many runs are conducted to get the quantitative experimental results in the paper (e.g., Table 4 and 5)? Are the standard deviations sufficiently smaller than the differences between different settings?\n2. Any analysis on where the improvement over CLIP dataset is from in Table 4? Different raw data sources?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. After almost three years the CLIP paper came out, it is great to see efforts following and investigating the data curation pipeline in the CLIP paper, whose proposed data diversification (balancing) was ignored by the other works, such as LAION. The paper would be interesting to the researchers working on data curation, and contrastive pre-training, too.\n2. The experimental results are impressive and set the new state of the art.", "weaknesses": "1. The contribution of the paper is limited. The main contribution is the entry balancing used by CLIP. This balancing operation actually plays a similar role to the deduplication used in [1] and [2], where its effectiveness has been proven.\n2. In Sec 3.4, when sub-sampling image-text pairs for each entry, in addition to the information density based rule, it is worth trying some model-based rules, e.g., image-text matching based rules. Although the paper mainly aims to reproduce the CLIP paper's results, it would lead to more contributions to go beyond CLIP reproduction and curate higher-quality datasets.\n\n[1] Abbas, Amro, et al. \"SemDeDup: Data-efficient learning at web-scale through semantic deduplication.\" arXiv preprint arXiv:2303.09540 (2023).\n[2] Yu, Haichao, et al. \"The Devil is in the Details: A Deep Dive into the Rabbit Hole of Data Filtering.\" arXiv preprint arXiv:2309.15954 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F24UKzgPU3", "reviewer": "Reviewer_Smdn", "summary": "The authors of the paper try to replicate the curation process employed by OpenAI to train their CLIP models. To that end, they highlight important excerpts from the original CLIP paper, and they try to replicate them as close as possible. The models trained on the resulting dataset show improved performance in imagenet zero-shot accuracy.", "questions": "- how does the curation step proposed in this paper compare with the one used for LAION-400M\n- can you please explain why your curation is not overfit towards ImageNet?\n- can you please compare against approaches in datacomp?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- the paper highlights several important excerpts form OpenAI's curation technique and tries to replicate them\n- the authors do an ablation study over their choices to validate whether their choices are helpful\n- the authors plan to release the curation data to the community", "weaknesses": "- the evaluation shows improvement in some downstream tasks, but not others. it would be nice to investigate this mixed performance, as to make sure that the curation technique is extremely tailored towards imagenet\n- there are other proposed approaches [1] for recreating WIT-400M, however, the authors do not compare their proposed approach with other approaches\n\n[1] DataComp: In search of the next generation of multimodal datasets, Gadre at al., 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)