CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning
Authors: Yiping Wang
Yifang Chen
Wendan Yan
Alex Fang
Wenjing Zhou
Kevin Jamieson
Simon Shaolei Du
Venue: NeurIPS 2024 spotlight (2024)
Keywords: contrastive learning
visual-language pretraining
data selection
CLIP
Primary area: active_learning
TL;DR: we design universal data selection methods for CLIP pretraining and achieve near SOTA results with less than 10% of preprocessing resources. Combining our methods with the current best one, we can achieve a new state-of-the-art.

Abstract: Data selection has emerged as a core issue for large-scale visual-language model pretaining (e.g., CLIP), particularly with noisy web-curated datasets. Three main data selection approaches are: (1) leveraging external non-CLIP models to aid data selection, (2) training new CLIP-style embedding models that are more effective at selecting high-quality data than the original OpenAI CLIP model, and (3) designing better metrics or strategies universally applicable to any CLIP embedding without requiring specific model properties (e.g., CLIPScore is one popular metric).  While the first two approaches have been extensively studied, the third remains under-explored. In this paper, we advance the third approach by proposing two new methods. Firstly, instead of classical CLIP scores that only consider the alignment between two modalities from a single sample, we introduce $\textbf{negCLIPLoss}$,  a method inspired by CLIP training loss that adds the alignment between one sample and its contrastive pairs as an extra normalization term to CLIPScore for better quality measurement. Secondly, when downstream tasks are known, we propose a new norm-based metric, $\textbf{NormSim}$, to measure the similarity between pretraining data and target data. We test our methods on the data selection benchmark, DataComp [Gadre et al., 2023]. Compared to the best baseline using only OpenAI's CLIP-L/14, our methods achieve a 5.3\% improvement on ImageNet-1k and a 2.8\% improvement on 38 downstream evaluation tasks. Moreover, both $\textbf{negCLIPLoss}$ and $\textbf{NormSim}$ are compatible with existing techniques. By combining our methods with the current best methods DFN [Fang et al., 2023] and HYPE [Kim et al., 2024], we can boost average performance on downstream tasks by 0.9\%, achieving a new state-of-the-art on the DataComp-medium benchmark.

CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning
Yiping Wang* University of Washington
Yifang Chen" University of Washington
Wendan Yan University of Washington
Alex Fang University of Washington
Wenjing Zhou University of Michigan
Kevin Jamieson University of Washington
Simon Shaolei Du University of Washington
Abstract
issue for large-scale visual-language model pretraining (e.g., CLIP), particularly with noisy web-curated datasets. Three main data selection approaches are: (1) leveraging external non-CLIP models aid data selection, (2) training new CLIP-style embedding models that are more effective selecting high-quality data than the original OpenAl CLIP model, and (3) designing better metrics or strategies universally applicable to any CLIP embedding without requiring specific model properties (e.g., CLIPScore is popular metric). While the first two approaches have been extensively studied, the third remains under-explored. In this paper. we advance the third approach by proposing two new methods. Firstly, instead of classical CLIP scores that only consider the alignment between two modalities from a single sample, we introduce negCLIPLoss method inspired by CLIP training loss that adds the alignment between one sample and its contrastive pairs as an extra normalization term CLIPScore for better quality measurement. Secondly, when downstream tasks are known, we propose a new norm-based metric, NormSim, to measure the similarity between pretraining data and target data. We test our methods the data selection benchmark, DataComp [I] Compared to the best baseline using only OpenAl's CLIP-L/14, methods achieve a 5.3% improvement on ImageNet- 1k and a 2.8% improvement on 38 downstream evaluation tasks. Morcover, both negCLIPLoss and NormSim are compatible with existing techniques. By combining with the current best methods DFN 2 and HYPE [I,, we can boost average performance downstream tasks by 0.9%, achieving new state-of- offtthe-rrt the SitaCoom--meiimm benchmark"
Data selection has emerged
methods
1 Introduction
Curating large-scale visual-languad datasets from web-sourced data has become common for pretraining multi-modal models. However, the quality of these web-curated data pairs remains a critical bottleneck. Research has shown that the choice of dataset significantly impacts model performance, irrespective of the models and training techniques employed [T, and this motivates
'Equal contribution. Correspondence to ypwang61@cs washington. edu. Codes are available
DDttaComp benchmark: https::/www.datacom wttps/aaaacooaaaaacccaa/ae al//ccllplledeeroorr..ccc
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the development of various data selection strategies. This paper focuses on optimizing subset selection from fixed data pool to train CLIP model 4 that achieves superior performance on zero-shot downstream tasks.
Classical methods rely solely on OpenAl's (OAI) pretrained CLIP model (i.e., a teacher model) and focus on better utilizing the embeddings. The most commonly used one is calculating CLIPScore, which measures the cosine similarity between the visual and language embeddings of the CLIP model for the same sample, to eliminate low-quality data with mismatches between text and image. Other works also leverage heuristic distribution alignment techniques to select samples relevant to downstream tasks, such as aaaage-bsee filtering M. These approaches are generally viewed as providing only limited enhancements. However, we argue that the potential of those embeddings has been heavily under- explerexppored. This work seeks a universal method to better employ any given embeddings, not only from OAT CLIP, but also from other CLIl--ttle models.
On the other hand, recent leading data filtering methods, instead of focusing on improving embedding utilization stategy itself, mainly follow the other two directions, both employing external resources. They either (1) use external non-CLIP models that aid in data selection, (2) or use external high- quality multi-modal data to train a better CLIP-style embedding model than the original OAI CLIP to filter out low-quality data. Specifically, in the first line of works, HYPE 3 leverages embeddings from hyperbolic models instead of the classical Euclidean- based CLIP to measure how each data point has semantically overlaps with other data points and filters out data with low specificity. T-MARS [12] removes images where the text is the only feature correlated with the caption using FAST [3], an off-the-shelf OCR text detection model. Devil [14] applies fasttext [15] to remove non-English texts and use BI.IP-2 [66 model for digit recognition to keep useful images with digits. The second direction, represented by Data Filtering Network (DFN) (2). involves training a new CLIP-style teacher model that uses high-quality datasets like HQITP-350M. Although the embeddings extracted from this model perform worse than the OAI CLIP in downstream tasks, it is particularly good at filtering out low-quality data. Notably, some of these methods can be combined and indeed, merging the selected data from DFN and HYPE achieves current state-of-art shown in HYPE 3
Previous works mainly focus on improving the CLIP embedding quality or utilizing an external model to do filtering but employ the CLIP embedding in a suboptimal way by only using classical methods like CLIPScore. In contrast, in this work, we focus on improving the filtering methods themselves for any given CLIP embedding. We show that there are universal and more effective strategies for utilizing any CLIP teacher model, regardless of its architecture (e.g., B/32 or L/14) or the dataset was trained on (e.g.. OpenAI-WIT-400M or DFN's high-qualit dataset). These strategies should always be orthogonal to the use of any newly trained CLIP-style models like DFN and might also be compatible with methods using external models like FAST and BLIP-2.
Our Contributions. We propose an alternative to CLIPScores that we call negCLIPLoss that more accurately characterizes data quality. We also introduce a new distribution metric we call the p-Norm Similarity Score (NormSim) when knowledge about downstream tasks available. Two major observations directly inform our proposals:
Firstly, we observe that classical methods measure the quality of a multi-modal sample by com- puting the cosine similarity between its visual and language embeddings. believing that lower similarity indicates that the text does not match its image part well. However, we find that some less informative samples may have a systematic bias, which leads to higher CLIPScores. For example, the language part containing the word "image" can result in higher similarity with any visual part, even when the text does not accurately describe its image content. Our proposed method Ceg inspired by the standard CL.IPLoss, normalizes the original CL.IPScore by the similarity between a sample and its contrastive pairs. For example, the high score caused by the word "imagee" is typically consistent across its contrastive pairs, our adjustment reduces this bias. As we have highlighted, such replacement can be universally applied across different embedding models. See Fig. 2 for illustrations.
Secondly, if one has access to examples drawn from the same distribution as the target task, is natural to assume that this extra knowledge could be leveraged to inform the data filtering process. We propose the NormSim metric to measure the vision similarity between a training sample and the target task dataset X E x defined (Xiige)) fo
where
fo RDD + IRd is the vision encoder of teacher model so that (Xirnne)))) E Raxd fa())) E Rd. and fo (x((ixi) ffii))))(((" E R", and ||. I|p is the p norm; effective choices are p = or po. Notably, unlike previous ImagetNet-batt based filtering [1]. which tries to keep the training set as diverse as
2

---

downstream tasks by clustering the training set and finding the nearest neighbor group for every target sample, our method does not explicitly consider the diversity but select examples as long as it is close to any target sample (i.e. select high NormSim score). Notably, negCIIPLLoss and NormSim enjoy complementary effect data selection. See Fig. 3
To illustrate the effectiveness of our methods, we use a widely used benchmark DataComp [ as our primary method of evaluating the datasets created by our data filtering methods. We show that, by simply replacing the CLIPScores with CeLCLIPLoss and utilizing NormSim we are able to exceed the best AAA-CCLP(L//4)-bbaee
baseline by 5.3% ImageNet-Il and 2.8% average across 38 downstream tasks, which is similar or even better than the performance achieved by many -esermaa--essurcss----e based methods. Notably, even if the target downstream tasks not available, using NormSim on a proxy downstream task constructed from the training set, called NormSimz--, combined with negCLIPLoss, can also gain a 1.9% improvement on 38 downstream evaluation.
Moreover, the improvements achieved by our methods are not limited to OAI CLIP-based methods but can also be obtained by combining resources. By merging the subset selected by negCLIPLoss and NormSim with the subset selected by current state-of the-art method "HYPE U DFN", we can further improve it by 0.9% on both ImageNet- Ik and on average 38 downstream tasks. Besides, we can also achieve a 0.8% improvement on average 38 tasks over "HYPE DFN" using only the data selected by DFN and strategies. More importantly, we demonstrate that negCLIPLosss as a replacement for CLIPScoree can be applied to any other embedding models like OAI-L/14, OAI-B/32, and DFN-B/32, universally boosting performance from 0.4% to 3.0% on an average of 38 tasks. This result is not only technically insightful for understanding the information available in embeddings but also practically significant. Compared to existing methods, our approach saves a significant amount of computational time on both reprocessing and new embedding retraining as shown in Table 3
methods with advanced models that require external
2 Problem Setup
Data Filtering on Multimodal Dataset. We are given a training dataset Duaio = fx", , ,'}, where (x", .²) E T is the image-text (vision-language) training pair. For convenience, we will let superscript vl denote either modality so that, for example, x E xx" x.. Our goal is identify a subset S C Duain that maximizes the zero-shot accuracy of the CLIP model on some downstream tasks when S used train the CLIP model.
CLIP score and embedding. Recent efforts, such as LAION 5 and DataComp [. use Ope- nAI's CLIP ViT-L/14 model [4] as teacher model to obtain quality score. Here denote this vanilla CLIP model as ful- For any pair æ² the model outputs a normalized unit-vector ( ). If X = {att 1.... .....a .m denotes a dataset containing m samples, then we define Rmxd as the embedding matrix. The popular filtering metric "CLIPScore" is defined as (((=)))(())) E [-
(X" = [f' ...., fa(("))]
Dataset and model. Here follow the pipeline of Datacomp ① to standardize the training and evaluation process. This is testbed for dataset experiments aiming to open-source and further improve the vanilla CLIP model and is widely adopted in previous data selection papers [17 18 12 1 7
We will give more details in Sec.
Data Filtering Strategy
3.1 negCLIPLoss: A Better Metric than CLIPScore
In this section, we introduce better and statistically interpretable quality metric called negCLIPLoss, which directly replaces the common metric CLIPŠcore. Fig. iillustrrae how negCLIPLoss works. This new metric only requires negligible extra computational costs and no additional external data collection costs. As the name suggested, this metric is inspired by the standard CLIP loss used in the actual training process of the teacher CLIP model, which is defined as
log exp(J.(e)) F((DDD/ ( log EjER- exp(f.(x;)) f((c))/T) ΣSER exp(f((x;)) f(()))))
(
(1) Here B" is the random batch where i-th sample belongs during a particular training step, and is the learnable temperate parameter. Notably, the teacher loss differs from CLIPScore primarily by
3

---

Quality Metric for data i
CLIPScore = fla
negCLIPLoss
fige 9t oc negative CLIP loss for data
where the normalization term
& [log efa ())) log (I eff aut))
fig Image/text embedding
ri temperature
8: batch
CLIPScore underestimate the quality
CLIPScore overestimate the quality
SBn Juan lalends Fridy Harbor
"Amesionn foorbal
"CIMG917G Woolly sheep Dunk's Green"
"Canbo Sap-down verbooring 67-46"
11758 Grein Willow Place Photo 25"
"I "Lisiin Image 7
Soun
negULIPLOSS Loss: Top 34%
Score Top 78% CLIPS.CO Tap B61o
neyCLIPLoss
Top 37%
negCUILLON Loss: Too 42%
cors Too 85%
CLIPScor:: 27% ses: Tos 19%
negil
negClIPL Tep
Top
I: Top 100%
RR Top 99%
9t: Top 100%
00: Top 17%
Figure 1: Illustration of negCLIPLoss. CLIPScore may underestimate (bottom left, where the data quality high but CLIPScore is low) or overestimate (bottom right, where the data quality is low but CLIPScore is high) the quality of image-text pairs. However, this issue be mitigated by simply subtracting a normalization term R. negCLIPLoss employs the teacher model to calculate the negative CLIP loss training data and serves more accurate metric. Here, "Top X%% denotes that the score represents the top X% high values within the entire dataset (i.e., the (100-X)% percentile among all the values). For example, "R Top 100%" means this data has almost the smallest R among the whole dataset, which represents that it contains highly specific elements in both images and texts.
normalization term R* as follows:
exp( fo fa(a)) fi(a")) + log Σ exp( fa(xf) f((o)) JE
-T. (I.
Folzt) ((a))
log
JER"
CLIPScore
Dornalization term R"
In practice, since the training dataset of teacher CLIP models, like OAL-WIT400M and the actual batch divisions B" is inaccessible, randomly select K batches from the student model's 's training data and use the averaged results from {
1K to estimate the normalization term R" on B":
(ACCLIPLsss("")) = K
T Σ Em (a:') 2 CLIPScore(x.") - R
(2)
Here [K are some batches randomly selected from the student model's training data and J2 E BR, vk. We choose K == in our experiments, but any sample size larger than 5 sufficiently stable for estimating the original CLIPLoss (Details in Appendix D.I). Besides, in Sec. 4.3.3 also show that the computational cost introduced by remains negligible compared to other basclines. The temperature and batch size |B*| can be directly obtained from the parameters of the pretrained teacher model. More details of negCLiPLoss are in Appendix, including the concentration analysis of R (Appendix A.I], pseudocode (Algorithm Db. and the ablation study of r and BB (Appendix C.2
Motivation behind negCLIPLoss. Other existing works also use loss-guided data selection, such as LESS 20 in NLP, CoDis (2 CV, and RHO [22]
ImageNet-1k
Average
Sad meeriig formanca
28
in general data scheduling scenarios. However, it is
still unclear whether selecting based on teacher loss
is suitable for multi-modal contrastive learning. Here we give an affirmative answer as shown in Fig. 2 where we or on par with CLIPScore consistently.
CUPPScor LiggCi
CLIPScore LPgosP
negCLIPLoss performs better than
Dowesamping
Downsampling
Figure Comparison of negCLIPLoss and CLIPScore across different downsampling ra- tios DataComp-mediun.
To illustrate how teacher loss helps our selection, we demonstrate that normalization term provided by negCLIPLOss Losss is crucial for correcting the overestima- tion or underestimation inherent in CLIPScore. A high normalization term implies that either the image embedding, text embedding, or both can easily match multiple contrastive pairs beyond their

---

NormSim Krangee x) =Il fo (Xaaggtt of 10p
Typeli: target quality
Typez: terget-ralated, high quality
Top
( (IN-1k)
20%
No Filtering
Zeijcle
120121
"Comna Sulads"
Hores Fiier Sear Unia
Num(Ta) Numi122 21 Num)ra) NumiT4)
NormSimy
Too
50%
Quality Filtering
TypeS:
Numi72) Num ae Num/T1) Num,T3)
CV FOCUS
80%
Norm Similarity Filtaring
NumiT11 Numi
NumiT4)
wealh
95%
Quality Norm-Sim Filturing
Bair Touer llat: Fawishae"
"Ortwey Enalee
flos kadge"
Num(F2) ze NuncT 1) NumiT3 3] 4
Top 95%
Top negCLIPLoss
20%
Top 5%
(b) Data Selection Methods Comparison
Data Type for Different negCLIPLoss and NormSimg
Figure 3: Illustration of NormSim on DataComp. Xurget is the target prior data. "Top X% denotes that the represents the top X% high values within the entire dataset. (a) Visualization of data with different NormSim and negCLIPLoss. Here we use VormSiny(ImageNNe 1k) as an example. Although both Type 2 and Type 4 data have high negCLIPLoss and thus high quality, data with low NormSima (Type 4) are more irrelevant to downstream tasks like ImageNet, VTAB, and MSCOCO. For example, they contain many images dominated by OCR content and make little contribution to improving downstream performance. (b) Illustration of a rough comparison of sampling data for different filtering methods. Using "negCLIPLoss NormSim" filtering can balance the quality and relevance to downstream tasks, thus increasing the proportion of Type 2 data. (Refer to Appendix B for more visualization.)
corresponding counterparts. For example, the bottom right of Fig. the text containing "Image" or "Photo" can be easily matched with any visual content. Similarly, the image of "verloopring" only contains very simple features and can be matched with many words like "white", "empty" or "circle", etc. Consequently, despite high absolute CLIPScore, the relative negCLIPLoss within its batch can be lower. contrast, the bottom left features highly specific elements in both text and images, such "Islands Harbor," "American foothall", and "sheep green". These elements are specific and less likely to match with contrastive pairs, resulting in a higher relative negCLIPLoss.
3.2 NormSim: New Training- Target Similarity Metric
Our proposed negCLIPLoss a universal approach to improve filtering performance by estimating quality better, and it does not rely any downstream task. Now, if we access some knowledge of the downstream tasks, we could further improve the performance by using a vision-only p-norm similarity targer data metric measure the relationship between each training sample and the downstream target data. We will discuss the reason to use vision-only embedding later in this section.
Specifically, we assume access to the target set of downstream tasks and denote them as Xurget = (Turget, (1). ..... Trarget, (m)) ), where each Ftargtt E TRd is i.i.d--ampled from the target downstream distribution P Purged Fl but without overlapping with the test set. Then, for each training sample x! and the corresponding target set Xurgess the NormSim is defined as:
1/D
NormSim, 2) = (X(inge) ( (=(""))
Σ (0.60).)(+))))
(3)
We select the subset S by choosing the samples with top-N highest NormSim scores. The choice of the norm type p can be based on the data distribution and training process. In this paper, we consider two instantiations of p:
When p 2, our data selection method can be regarded as the following equation. It's equivalent to selecting a subset that aligns with the principal components of the target set variance (Appendix I.6.I).
Althoogh ou distribbutii tasks like "WILDS" have distribution shift between training data and test data, they still provides useful information of the test data.
5

---

1/2
S arg max (ornSima(er,..)) 2:),, (or =
Σ
[o(=))) [T
(4)
[S]
Nunci
When p= x., the distance metric can be regarded as an even more optimistic measure, such that training sample will be selected if it has high similarity to any target sample. Note that this is different from nearest-neighbor-baee based method used in image-basee filtering [1, where they are trying find the nearest training sample of every larget sample. In this case, can be regarded as:
S arg max Σ [9] RES
(1e,
NormSim (1),
max Xurn
fo(x)) fo(x))
(5)
In Appendix D.3 we also show that our NormSim can outperform the nearest neighbor selection on the downstream target tasks. Here, we show an example selected via the NormSimy (ImageNet-I] 1 in Fig. [ showing that this lision-target-wwar method is complementary to the quality- -based one.
Choice of Target Data. In the experiment parts, we try two kinds of target data: training data from ImageNet-Ik (1.3M) or training data from all 24 accessible downstream tasks (2. We denote respectively.
them as (aatnSii,,NNNTTA and NormSim,,Taage
Necessity of using vislon-only information We use only the visual information x" instead of multi- modal information n for measuring similarity. This is because common crawled text often has brief captions, making the OAI CLIP language embedding weaker than its visual embedding model 25 Consequently, the language part cannot characterize the pre-training and downstream task distribution well the visual part. This phenomenon is also observed in Gadre et al. [. where image-based filtering (select data whose image embeddings are similar to that from ImageNet- 1k) outperforms text-based filtering (select data whose captions contain words from ImageNet-211k). More ablation studies are provided in Appendix D.4
Generality of NormSim in choosing teacher model. Notably, since just image embeddings in the NormSim metric, we believe it unnecessary to use CLIP model to obtain NormSim. Norm- Sim can be general metric for selecting target-related image/image-tex data if any good image representations are given, like the representations obtained from pretrained ResNet-50.
Theoretical justification. Unlike many existing methods that force diversity by selecting training samples around each Furgets our strategy maximizes similarity without directly considering data diversity. For the p = case, we demonstrate that maximizing NormSima is optimal under linear model J., as shown Appendix A.2 Our theorem also provides error guarantees for noisy embeddings and explains when vision-only embeddings outperform combined vision and language embeddings. Recent work by Joshi et al. [26) provides a similar analysis but focuses on high- data and methods for filtering noisy datasets, as discussed above.
between images and texts. This approach is less effective than image-only
Using proxy when downstream Xtarget is inaccessible. also be used when only the pre-training set available. this case, we construct a proxy "target" set from the pre- pre-training set itself. Specifically, let S; be the selected subset at step i, then we treat the current S, as the proxy "target" set. To construct the next smaller set, we select the next data batch Si+1 satisfying arg m Exes NormSimz(S;; 2),, until reaching an N size subset. We call this approach NormSim;-D (Dynamic) and will specify the algorithm details in Appendix C.3
Surprisingly, we show that the 2-norm can
4 Experimental Results
In this section, we evaluate the performance of negCLIPLoss and NormSim, aiming to address the following questions: Q1: Given fixed CLIP teacher model, can our methods more effectively utilize CLIP embeddings for data filtering? Q2: Are our methods applicable to diverse CLIP teacher models with varying architectures or different pretrained datasets? Q3: How does our method compare to other leading approaches that utilize external models or multimodal datasets? Additionally, could our method be compatible with these methods and enhance their effectiveness?
'Here we only use the target data for data selection, instead of training on them. The target dataset is significantly smaller than pretraining set like DataComp mediuu (128M) or external datasets like HQRTP-350M utilized by DFN 2

---

4.1 Setup
We adhere the standardized training and evaluation protocols of the DataComp benchmark D. Training configuration. We employ the medium-scale training configuration of DataComp (attaoop-medium) It provides a substantial dataset comprising 128 million low-quality, web- curated image-text pairs to be filtered. Once the data subset is obtained by some data filtering strategy, it will be used to train fixed CLIP-B/32 model in fixed training budget that allows the model to pass 128 million data points an epoch. Therefore, smaller subsets will be repeated more frequently, ensuring a fair comparison. We note that the size of the DataComp dataset becomes smaller over time since some URLs of images become invalid and we only successfully downloaded about 10M data. Therefore, the results of baselines on the leaderboard do not apply to our datasets, and we reproduce all the top baselines on the leaderboard with their public UIDs of the selected data.
Evaluation. We measured the model performance on 38 downstream datasets including image classification and retrieval tasks followed by DataComp. The image classification tasks contain ImageNet-Ik (27]. ImageNet distribution shifts [28) 11 datasets from the Visual Task Adap- tation Benchmark (VTAB) [32] and 3 datasets from WILDS B3 34 Retrieval datasets contain
Flickr30k [35), MSCOCO 36 and WinoGAViL [37
Teacher model architecture. Our experiments utilize two architectures for OpenAl's CLIP teacher models: ViT-L/14 and ViT-B/32. Additionally,
the public version of DFN (DFN-P) proposed by Fang et al. [2] as a teacher model, and its architecture is also ViT-B/32.
4.2 Baselines
We restate the three current research directions mentioned before based on how much external resources are employed: (D1) using OAI CLIP alone while optimizing embedding employment strategies, (D2) training and using a more advanced CLIP embedding model based on external data, and (D3) utilizing non-CLIP external models to aid data selection. It is important to note that D2 and D3 may also incorporate strategies from DI. For example, CLIPScore (D1) has been used in almost all the top methods. Therefore, we categorize baselines by the largest possible category they encompass. According to the above categorization, we summarize the baselines we used in our experiments as follows. Please refer to Fig. and Appendix C.4 for more details.
DI: OAT CLIP embedding only. The learner can only access the pretraining dataset (like DataComp- medium), the original OAI CLIP teacher model that is used to extract embeddings, and some target data of the downstream tasks which is much smaller than the pretraining dataset (like ImageNet- 1k).
In this category, we don't use any existing external non-CLIP models any newly trained CLIP model based on external multi-modal dataset. In detail, This category includes (1) CLIPScore (38 which only uses CLIPScore for filtering as we mentioned before. (2) Image-based filtering M. which uses ImageNet- IK training data as the downstream target data for data filtering. It applies k-means clustering to the image embeddings of training data and selects clusters closest to the ImageNet- IK embeddings. Gadre et al. [ also try to combine image-baser filtering and CLIPScore together. (3) 2 Pruning [18], which represents the dataset as an undirected graph and selects the data by combining difficulty and diversity. They use the CLIP score to initialize their graph.
D2, D3: Accessible external model and multi-modal data. All the current top baselines enable the learner to utilize external resources, either to train a better CLIP teacher model or to help filtering using existing models' properies. In detail, (1) DFN (2 trains another CLIP data filtering network via external high-quality data. Their currently public model (DFN-P) is trained on CC1 MM [39] +
CC3M [40] SS15M (1), while the best DFN is trained on nonpublic HQrTP-350M 2 which
is even larger than DataComp-medium (2) HYPE (3) leverages hyperbolic embeddings (different from CLIP embedding) and the concept of entailment cones filter out samples with meaningless or underspecified semantics, enhancing the specificity of each sample.
(3) HYPE U DFN proposed by 3 samples subset separately for each method and then merge them. This is the state-of-the-an method on the DataComp benchmark for medium size. (4) Other methods including T-MARS [2] 13
Devils 14] MLM [2), which leverage external models such as text detection model FAST
BLIP-2 16 and 1.a 43 44 to heuristically select data. See details in Appendix C.4
Cross-setting comparison. We make these separations for fair comparison. Intuitively, performance should be ranked as D2, D3 > D1. However, our results show that cross-setting comparisons are possible and our DI methods can perform similar or even better than most of D3 methods.
SSee http://ihhlcommttooooddoooosssos Similar issues are proposed by D22 pruning [18
7

---

Table 2: Results DataComp-medium from methods that only OpenAl's CLIP-L/14 model, i.e., all methods are from the DI category. The "dataset size" represents the size of the subset obtained from different approaches. NormSim(IN-Ik denotes using the training data of ImageNet-Ik as the target while NormSim(Target) represents using that of all 24 available downstream tasks. NormSim-D refers to the methods that use iteratively selected subset from the training set as the target proxy.
Dataset Size
IN-IK task)
IN Dist. Shift
VTAB
Retrieval Avg. (3)
Filtering Strategy
(5)
(11)
(38)
No filtering
110M
17.3
15.0
25.2
21.3
25.6
CLIPScore (20%)
22M
25.4
22.7
31.8
22.0
31.0
CLIPScore (30%)
33M
26.4
23.6
32.6
24.5
32.2
Image-based
[
24M
25.5
21.9
30.4
24.6
29.9
CLIPScore (30%) Image-based
11M
27.4
23,9
31.9
21.4
30.8
D Pruning 18
22M
23.2
20.4
31.4
18.7
29.5
negCLIPLoss (20%)
22M
27.4
23.8
23.7
32.5
negCLIPLosss (30%)
33M
27.9
24.6
33.2
25.1
32.9
CLIPScore (30%) NormSima-D
22M
28.3
25.0
34.5
22.7
32.9
negCI.PPI.oss (30%) n NormSimg-D
22M
29.8
26.1
34.8
24.6
34.1
CI.IPScore (30%) 0 NormSima (IN-1k)
22M
29.1
25.4
35.8
24.1
33.4
CLIPScore (30%) NormSim, (Target))
22M
28.9
25.1
32.7
23.6
32.5
CLIPScore (30%) 0 NormSim...
22M
29.7
25.9
33.7
24.1
33.7
CLIPScore (30%) NormSim.. (Target)
22M
30.2
26.2
35.0
23.4
33.9
CegCLIPLsoss (30%) (1 NormSima(IN-1k)
22M
30.4
26.4
35.4
25.6
34.3
negCLIPLoss (30%) ( (Trggimaa(Trget)
22M
30.6
26.2
35.2
25.5
33.9
negCLIPLoss (30%) 0 NormSim...N--Iae
22M
31.9
27.3
34.8
25.0
34.4
negCLIPLoss (30%) n (Tarriee.TTTreeee
22M
31.7
27.2
36.0
26.0
35.0
4.3 Main Results and Discussions
Table Results on DataComp-medium from the top methods that use only OpenAl's CLIP. B/32 model or public version of DFN (DFN-P). "NormSim B/32., represents using OAI CLIP-B/32 to calculate NormSim
4.3.1
Comparision on D1 Category (Q1)
In Table 2, we compare the methods where only the OAI CLIP model is allowed to be used.
Strategy
Size
IN-Ik
VTAB
Avg.
Our Methods leverage OAI CLIP-L/14 better. First, negCLIPLoss outperforms CLIPScore on all metrics, regardless of whether is used alone or combined with other methods. These results support our claim that negCLIPLOos can more accurately estimate the data quality.
OAI CLIP-B/32
CLIPScore (30%)
33M
27.6
33.6
33.2
CLIPScore (20%)
22M
27.0
33.0
32.2
negCI.PPLoss (30%)
28.8
33.7
33.6
negCI.IPL.os (20%)
22M
28.9
34.3
33.0
negCLIPLose (30%) NormSim..(Targee
Second, even when target knowledge unavail- able, use NormSim,-D together with negCLI- PLoss can still improve the filtering performance by 1.9% on average 38 downstream tasks. Third, when target knowledge is available, NormSim, and NormSimgo can improve filtering more sig- nificantly compared with NormSim;-DD and in general, NormSim is the best choice. Espe- cially, compared with the best baseline 'CLIP- Score (30%)', our best combination 'negCLI- PLoss n NormSim (ormSii..(Tarrett)) improves 5.3% on ImageNet-1k and 2.8 % average 38 down- stream tasks, respectively. Later in Table we will that this result outperform all the D3 baselines except DFN U HYPE. On the other hand, when using ImageNet- 1k the target data, the choice of norm has very little influ- ence.
32.4
35.9
35.2
DEN-P
CLIPScore (30%)
33M
28.4
33.2
32.7
CLIPScore (20%)
22M
29.7
33.0
33.1
CLIPScore (17.5%)
19M
30.2
34.1
33.8
CLIPScore (15%)
16M
25.9
32.9
31.6
negCLIPLOss (30%)
33M
28.9
33.4
33.2
negCLIPLoss (20%)
22M
30.7
33.6
33.8
negCLIPLoss (17,5%)
19M
31.2
35.7
34.7
negCLIPLoss (15%)
16M
31.3
35.8
34.6
negCI.IPI Loss (30%)
22M
29.4
33.5
32.5
MormSii...TTargeee
negCLIPLoss (17.5%
16M
31.5
34.6
34.4
NormSimau ((Trge))
negCLIPLoss (17.5%)
16M
31.6
37.2
35.7
NormSimm"

---

Table 3: Results all DI&D2&D3 top methods on DataComp-medium. The results of MLM from their paper, while all other baselines reproduced on downloaded dataset using their official UIDs. "Ours (20%)" refers to use "negCLIPLoss (30%) n NormSim, (Target) to get 20% of original data, while "Ours (10%)" denotes applying "negCLIPLoss (20%) n NormSim.o (Target)" to get 10%. And we use "*" to indicate the case where we choose the intersection of the data selected by using OAI CLIP-B/32 and OAI CLIP-L/14 separately, which results in about 15M data for "Ours (20%)** and 7.4M data for "Ours (10%)*".
Dataset Size
IN-1k
IN Dist. Shift
VTAB (11)
Retrieval (3)
Avg. (38)
Type
Filtering Strategy
(1)
(5)
D3
T-MARS 12
22M
30.8
26.3
34.8
25.4
34.1
D3
Devil
20M
31.0
26.7
35.9
24.7
34.5
D3
MLM
42
38M
30.3
25.6
36,0
29.0
34.5
HYPE
10M
30.3
25.8
22.2
31.9
D2
DFN
2
16M
36.0
30.1
36.2
27.0
35.4
D3
DFN
U HYYP 3
20M
36.4
30.8
38,5
28.0
36.8
DI
Ours (20% %)
22M
32.4
27.4
35.9
26.3
35.2
D3
DFN U Ours (20 %)
23M
36.4
30.9
38.6
28.1
37.6
D3
DFN U HYPE Ours (10%)*
22M
37.3
31.4
38.5
27.6
37.7
4.3.2 Try Other Teacher Models (Q2)
To evaluate whether our method applies to other CLIP teacher models, we replaced OAI CLIP-L/14 with OAI CLIP-B/32 and DFN-P as embedding models. We compare the best baseline "CLIPScore" with our "negCLIPI and Appendix D.2 Note that the original DFN paper selects subset comprising 19.2M data points, which accounts for approximately 17.5% of our dataset and 15% of their dataset, we incorporate these sampling ratios into comparison.
and best strategy "negCLIPLoss NormSim,..
as shown Table
negCLIPLoss can be applied different CLIP embedding models. Our proposed negCLIPLoss, as a replacement of CLIPScore, not only leads to better performance compared to all the other baselines using OAI CLIP-L/14 as shown Table 2 but also achieves universal improvement on the other two CLIP embedding models, OAI CLIP-B/31 and DFN-P as shown in Table [ Our methods can consistently outperform all downstream tasks for different filtering ratios and models, like 0.5%-5.49% increase on ImageNet-1k. k.
Embedding required by NormSim should have good downstream performance. When combining negCLIPLoss with NormSim.o. OAI CLIP-B/32 and DFN-P exhibit completely different behaviors. The former obtains results even better than those in Table 2 which uses OAI CLIP-L/14 as the teacher model, while DFN-P achieves results even worse than using negCLIPLoss alone The reason that, unlike OAI CLIP-B/32. DFN-P is specially designed for data filtering the expense of downstream task performance, as claimed by its authors. For example, the ImageNet- k accuracy for DFN-P, OAI CLIP-B/32, and OAI CLIP-L/14 are 45%, 63%, and 75%, respectively. This indicates that the embeddings obtained from DFN on target data might be highly unreliable, leading to inaccurate similarity calculations between training and target data. To support this, if we use DFN-P to evaluate negCLIPLoss but utilize OAI CLIP-B/32 for calculating NormSim, shown in "negCLIPLoss (17.5%) n No (Targee))" we can further improve the results compared to using negCLIPLoss alone. Its average performance on tasks is even higher than utilizing the best DFN (trained on PrQTTP-350M with CLIPScore, shown in Table
4.3.3 Comparison with D2& & D3 Categories (Q3)
In this part, we compare all the D2 & D3 baselines mentioned Sec. 4.2 together with our best strategy Table 3 Here reproduce all the baselines if their official UIDs are available. For "A U B" mentioned Table 3 we follow the way of "HYPE U DFN" Kim et al. B to merge the data, which generates the sampling subset separately for each method and then merge them. This will result in oversampling the shared data, which is intuitively more important We also show the best result
"see "negCLiPLoss (30%) NormSim. (Tareet)" versus "negCLIPLoss (20% ((3%)))))) "negCLIPLoss (17.5% NormSim... (Target) versus "negCLIPLoss (17.5%)(1555) For the dataset size of "AUB", """"", count the number of the unique data in the dataset followed HYPE 3

---

we obtain by combining our method with DFN [2] and HYPE [3] on the full DataComp-medium dataset in Table 4 where the baselines are from DataComp benchmark.
Our methods can outperform most of the D3
methods. In Table 3 we show that without us- ing any external models data, our best combina- tion, i.e. using OAI CLIP-B/S3 B/32 for "negCLIPLoss (30%) n NormSim,o (Target)" (Ours (20%) )), still outperforms all methods except DFN and "DFN HYPE". This answers the first part of Q3 and further indicates that some external models may be redundant since CLIP embeddings already contain necessary information.
Table 4: Results of the top methods on the full
DataComp-medda mediium dataset (128M data).
Strategy
IN-1k
Avg.
No filtering
17.6
25.8
CLIPScore [8
27.3
32.8
T-MARS 12
33.0
36.1
Devils [14
32.0
37.1
DFN
37.1
37.3
DFN HYPE
38.2
37.9
We can further improve the SOTA method. In Table 3 we show that our model can further boost the performance of the current SOTA method "HYPE U DFN" by 0.9% both ImageNet- and on average 38 downstream tasks, and close results can be achieved even without combining HYPE which utilizes the external embedding model MERU [45). And update the SOTA performance of the PataComp-medium (full dataset) benchmark as shown in Table Here we use the data selected by both OAI CLIP-B/31 and L/14, which we found is more robust than using of them alone. Our better results answer the second part of Q3, that is, our methods can be compatible with other D2&D3 methods.
DFN Ours (20%)
37.5
38.6
DFN U HYPE U Ours (10%)
38.2
38.8
5 Conclusion and Limitation
In this paper, introduce metries, negCLIPLoss and NormSim, enhance data selection in multimodal contrastive learning without relying on external resources. negCLIPLoss provides a more accurate quality metric compared to the commonly used CLIPScore, while NormSim measures the similarity between pretraining data and target data for known downstream tasks. Experiments show that our methods achieve results that competitive with or even better to approaches using external models or datasets. Additionally, negCLiPLoss and NormSim are compatible with existing top techniques, allowing us achieve a new state-of the-art by combining them.
notable limitation of our study is the exclusion of larger pretraining datasets, such as the large and xlarge scales of DataComp. However, DataComp-medie medium is the most commonly used benchmark for data selection in CLIP pretraining, and our method has demonstrated both effectiveness [Table[[] and efficiency (5 on it. Future directions include exploring better ways merge data selected by different methods and incorporating our methods into data scheduling scenarios.
6 Acknowledgement
We thank Tong Chen, Pang Wei Koh, Xiaochuang Han, Rui Xin, Luyao Ma. Lei Chen, and other members in the UW MI. Group for many insightful discussions and helpful feedback. The research
of Kevin Jamieson and Yifang Chen are partially supported by the NSF through the University of
Washington Materials Research Science and Engineering Center, DMR-2308977 and awards CCF 2007036. SSD acknowledges the support NSF IIS 10170, NSF DMS 2134106, NSF CCF 2212261, NSF IIS 2143493, NSF CCF 2019844, and NSF IIS 2229881.
10

---

References
[1] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304. 14108, 2023.
[2] Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309. / 17425, 2023.
[3] Wonjae Kim, Sanghyuk Chun, Tackyung Kim, Dongyoon Han, and Sangdoo Yun. Hype: Hyper- bolic entailment filtering for underspecified images and texts. arXiv preprint arXiv:2404.17507 2024.
[4] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. -766. PMLR, 2021.
[5] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 25294,, 2022.
[6] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818 21882222,,42 2023.
[7] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vasion-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
[8] Huy V Vo, Vasil Khalidov, Timothée Darcet, Théo Moutakanni, Nikita Smetanin, Mare Szafraniec, Hugo Touvron, Camille Couprie, Maxime Oquab, Armand Joulin, et al. Au- tomatic data curation for self-supervised learning: A clustering-based approach. arXiv preprint arXiv:2405.15613. 2024.
[9] Tzu-Heng Huang, Changho Shin, Sui Jiet Tay, Dyah Adila, and Frederic Sala. Multimodal data curation via object detection and filter ensembles. arXiv preprint arXiv:2401.1222 2024.
[10] Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36:34201-34227,, 2023.
[11] Amro Abbas, Evgenia Rusak, Kushal Tirumala, Wieland Brendel, Kamalika Chaudhuri, and Ari S Morcos. Effective pruning of web-scale datasets based on complexity of concept clusters. arXiv preprint arXiv:2401.04575 2024.
[12] Pratyush Maini, Sachin Goyal, Zachary C Lipton, J Zico Kolter, and Aditi Raghunathan. T- mars: Improving visual representations by cireumventing text feature learning. arXiv preprint arXiv:2307.031.2 2023.
[13] Zhe Chen, Jiahao Wang, Wenhai Wang, Guo Chen, Enze Xie, Ping Luo, and Tong Lu. Fast: Faster arbitrarily-shape shaped text detector with minimalist kernel representation, 2021.
[14] Haichao Yu, Yu Tian, Sateesh Kumar, Linjie Yang, and Heng Wang. The devil is in the details: deep dive into the rabbit hole of data filtering. arXiv preprint arXiv:2309. 15954, 2023.
[15] Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint arXiv: /607. 007000/759.9. 2016.
[16] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19745 PMLR, 2023.
11

---

[17] Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt. Improving multimodal datasets with image captioning, arXiv preprint arXiv:2307.10350, 2023.
[18] Adyasha Maharana, Prateek Yadav, and Mohit Bansal. D2 pruning: Message passing for balancing diversity and difficulty in data pruning. arXiv preprint arXiv:2310.079:3 2023.
[19] Anas Mahmoud, Mostafa Elhoushi, Amro Abbas, Yu Yang, Newsha Ardalani, Hugh Leather, and Ari Morcos. Sieve: Multimodal dataset pruning using image captioning models, arXiv preprint arXiv:2310.02110, 2023.
[20] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333, 2024.
[21] Xiaobo Xia, Bo Han, Yibing Zhan, Jun Yu, Mingming Gong, Chen Gong, and Tongliang Liu. Combating noisy labels with sample selection by mining high-discrepancy examples. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1833-1843, October 2023.
[22] Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized training on points that are learnable, worth learning, and not yet learnt. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 15630-15649, PMLR, 17-23 Jul 2022.
[23] Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai- Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit iision-and-language tasks? arXiv preprint arXiv:2 2021.
[24] Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. In International Conference Machine Learning, pages 25994 26009. PMLR, 2022.
[25] Yutaro Yamada, Yingtian Tang, and Ilker Yildirim. When are lemons purple? the concept association bias of clip. arXiv preprint arXiv:22 2022.
[26] Siddharth Joshi, Arnav Jain, Ali Payani, and Baharan Mirzasoleiman. Data-efficient contrastive language-inen iaatgge--magepeee pretraining: Prioritizing data quality over quantity. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 1000-1008. PMLR, 02-04 May 2024.
[27] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-20 228.255.22 Icce, 2009.
[28] Haohan Wang, songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global repre- sentations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
[29] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In Kamalika Chaudhuri and Ruslan Salakhutdinov, edi- tors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5389-5400. PMLR, 09-15 Jun 2019.
[30] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardr, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271, 2021.
[31] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: critical analysis of out-of out-of-distribution generalization. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 8340-8349 2021.
12

---

[32] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 19 1910.04867, 2019.
[33] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasumaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.
[34] Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, et al. Extending the wilds benchmark for unsupervised adaptation. arXiv preprint arXiv:2 12.05000, 2021.
[35] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[36] Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015.
[37] Yonatan Bitton, Nitzan Bitton Guetta, Ron Yosef, Yuval Elovici, Mohit Bansal, Gabriel Stanovsky, and Roy Schwartz. Winogavil: Gamified association benchmark to challenge vision-and-language models. Advances in Neural Information Processing Systems, 35:26549- 26564, 2022.
[38] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint 1rXiv:2104.08711 2021.
[39] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Sorieut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tal tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558-3568, 2021.
[40] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2565, Melbourne, Australia, July 2018. Association for Computational Linguistics.
[41] Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. Advances in Neural Information Processing Systems, 35:21455-22 1115552--1999 21469, 2022.
[42] Weizhi Wang. Khalil Mrini, Linjie Yang, Sateesh Kumar, Yu Tian, Xifeng Yan, and Heng Wang. Finetuned multimodal language models are high-quality image-text data filters. arXiv preprint arXiv:2403.02677, 2024.
[43] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 2023.
[44] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna, Imsys. org (accessed 14 April 2023), 263) 2023.
[45] Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ra- makrishna Vedantam. Hyperbolic image-text representations. International Conference on Machine Learning, pages 7694-7731. PMLR, 2023.
[46] Ryumei Nakada, Halil Ibrahim Gulluk, Zhun Deng, Wenlong Ji, James Zou, and Linjun Zhang Understanding multimodal contrastive learning and incorporating unpaired data. In International Conference Artificial Intelligence and Statistics, pages 4348-4380 PMLR, 2023.
13

---

[47] Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013.
[48] Martin J Wainwright. Hligh-dimensionl statistics; A non-asymptotic viewpoint, volume 48. Cambridge university press, 2019.
[49] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions Big Data, 7(3):535-555 547, 2019.
[50] Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt, Im- proving multimodal datasets with image captioning. Advances in Neural Information Processing Systems, 36, 2024.
[51] Sachin Goyal, Pratyush Maini, Zachary Lipton, Aditi Raghunathan, and J Zico Kolter. Scaling laws for data filtering- data curation cannot be compute agnostic. arXiv preprint arXiv:2404.0711 077777 2024.
14

---

A Theoretical Interpretation
A.1 Concentration of Normalization Term in negCLIPLoss
In this section, we construct a theorem using the concentration inequality to show that when the batch size is sufficiently large, the normalization term Rtt obtained from actual batch Bg can approximate R calculated using ground truth batch B* quite well. The details are as follows:
We assume that the pretraining dataset is ndependent and identically distributed (i.i.d.) sampled from some distribution P. Besides, to use pretraining data batch to approximate the ground truth batch, one necessary condition is that their distribution similar. Here for simplicity, we assume that they are also i.i.d.
Assumption A.1. We assume that the ground-truth batch of data B used by the teacher model is i..dd to the pretraining dataset DD which is required to be filtered.
For simplicity, we denote $1, = Fu (x)) f((x))... E be the cross coass-image-tex similarities in the batch B. Then the normalization term can be written as
T R₂¹
log exp($1//T)) + log( Σ exp($58/7)
2
jEB
jEB
Here 8ij E [.] We will show that R (1 + o(1) R.P for all i when B is sufficiently large, which means that we can use the random batch to approximate the ground-truth batch.
Theorem A.1. If Assumption A.7 holds and the batch size satisfies |B] = [B*,, then we have ) for any i B
R. O[log( while [ R B" = =0 \im
Proof. Since 8.j [],,1], It's obvious that R. E, [xxp(ss:)/T]]] then aij is zero-mean. Note that since the data is i.idd. so does oigs and denote 7 := := [. Note that Jayyl S e//T M, from Bernstein inequality we have
([log(BB)), Let ox == exp(s1/TT)
1e2
P(I
Σ asyl 2 2) S 2exp(
|B + M
jEB
A similar conclusion holds for B"
These result that with probability at least - we have
1 JEB
any S max / Bly In 2 4 M In 7
2) =: =
Thus we have Ejen exp( (o1 ) it's easy prove that log(z1) -oog(g)()))))))))
Σ jEBB exp
2t B1, y, 7. Furthermore, for any 21, 1, b - ==
Therefore, we have log(E exp( (h) ). Similar claims hold for IR RR.11.
log( E exp( +2)) S (
V
A.2 Optimality of NormSimg Under Linear Assumption
In this section, we give a theoretical justification on the NormSim metric when p 2 under the linear model assumptions when low quality image and mismatched text has already been removed. In other words, we mainly focus on the following strategy.
S = arg max
E(G;))) N
Σ
fe(of)
(6)
X Vargel
1E8
aee Xorga
Euupet pony
15

---

A.2.1 Theoretical Setup
Training data. For any x", ,", E Rd observable image and text training paits, we define z", 2' to be the corresponding latent vectors which contain all semantically pertinent information about our tasks of interest. Similar to previous theoretical work [46], we assume each i.idd pair z"d follows zero-mean sub-gaussian distribution whose cross-covariance satisfies
Cov(=", "ov==,z²') z²)
= Mimiin = diag(o), .2,...)
01 =
and each x" is generated based on a linear model such that
x² = 12 +
Here Gi E Odxr is the othonormal ground truth representation mapping from the latent vector space to the input space, and e" (O, Ia) are i.idd random noise.
Also we denote the cross covariance of any finite dataset S' (e.g. the given train set Dwain) as Egr.
Test data. For any zero-shot downstream task, assume shares almost same data generation process as the training set, except its the cross-covariance Enaget does not necessarily equal Emaain which necessitate the choice of prexy
CLIP embedding model as teacher. Under the linear model assumption, we have a teacher model fel Got, whose generated clip embedding can partially recover the ground truth hidden vector x with error.
Formally, we say teacher has ch error for all possible n budget subsets C Duains
1 JS|
G E5
Σ
xx'x(x") (x (a
Σ =²(x") x Es
where the same notation applies for the language modal. By the orthonormal assumption on the ground truth matrix G we see that G is aiming to inverting the map. In addition, we say the teacher has c cross modal eтror
1 S
(T) Σ ES
> c.u.
When all E" E + 0 as co, then we say the teacher is strong for both modalities. But might also be possible that only one modal, for example, visual is strong. That is e - 0, 67, e > Ey.
Model and training. According to Lemma 4.1 1 in 446. using the CLIP loss to optimize the linear same training dynamics using the regularized linear loss. Therefore,
model has approximately the here we assume that we are learning Gv, G2 by maximizing the clip score gap between the contrastive pairs, plus a regularizer.
Eies Ejes(Si- 88) [S((SS|
|S|
min GG Gi) = min Go.Gi
2 2|S| - |G.G 12
G..Gr
where Sty = (G] ; GT 25) and > 0 is some regularizer-related constant. Note that this objective maximizes self- similariyy and minimizes similarity between disparate pairs. Note that this "loss" can be negative, avoiding the trivial null solution of all zeros. We denote this training process from any given G. (S)).
Goal and metric. Under the same principle as our training loss function, we measure the per- formance of any learnt Go, Gj on some downstream task with distribution Dunget test loss Luagal (G., Gt)
E End (G.). x".G. GT(z) 2/)) (G x", G.T x!))
mi DDage
16

---

This is inspired by the following classification accuracy. Assume that the test data including C' class, and the class distribution is C. For every class c, the training data a (x", x²) satisfies distribution Pe- We further assume the corresponding classification templates are (...)==1- Thus we define classification accuracy
AC Gi) Cxx [E [ sic']]
Therefore our goal is to minimize its gap between the best hind-side subset, for any p. without budget constraints,
(S) Caares (G.t) (Gnt))(G..))
min S'EDiin
(A = A
A.2.2 Generalization Guarantees
We now provide theoretical guarantees and postpone proof into Appendix A.2.3 Firstly, are going to prove the intuition behind NormSim,score.
Lemma A.1 (Intuition behind NormSimz). With high probability at least hind-side best subset has at least n number of samples, then we have
si suppose the
(V dllogd]SS))
dlog(d (d] S|)
(S
max (Tr (Enazee EEg Es))) S'e D'an
O n
NormSimyrelateed
noise
Proof sketch. 1 Under the assumption that both nd Eot is zero-mean, maximizing the clip score gap equivalent maximizing the clip score of the sample.
LurgelG Gi) == -Eg (G x", GTx²)
@ By minimizing the regularized training loss (.S(Go,, Gi) using Eckart Young-Mirsky Theorem, we get a closed form solution of G as
G. = G.Es (Gi) + noise depend on S
Combining the result in and O, we have
Graga((Gol) ~e - Tr Es) noise depend on S
P
The same analysis can be applied on minsre Lunge (A(S')) as well. Rearranging these two equations gives the final result.
This lemma shows the the ASS) is depend on the NormSimz-related term and the noise term which comes from g. When n and [S] is large enough, then the NormSima-relant term will become dominant. This aligns with our practice experience that the final performance is less sensitive to the small variation in the number of select data as long as that is sufficient. Moreover, some special cases where test distribution has identity cross-var vorsinari, then sampling by choosing CLIP score might be enough.
Now we are ready to give a proof on the choice of Earret and visual-only information. Specifically, the strategy error mainly comes from (1). The unknown test distribution shift from training. (2). The unobservable ground truth Σg. To tackle error (1), we assume some prior knowledge on test by using the proxy test variance Exurgen To tackle the error (2), there are two possible solutions as shown below. Based on the theoretical interpretation, we should choose different strategy based on the property of teacher embedding model.
Svision+language arg max Tr Emanger Σ x.
G. ( (Tt)
s
Syision only = arg max TI Enaneer
Σ xx'
17

---

Theorem A.2 (Main). Under the assumption of Lemma A.1
1
S noise +
Euargr
Enargel ||
Ebeall
p
1 + p
vul
(visiontlanguage
(z", 21))
(vision only)
18
Firstly, it evident that the greater the difference between Etarger and Etrrett the less improvement we can expect. Moreover, in scenarios where is large (indicating lower accuracy in the language part) while e, is small (indicating higher accuracy in the vision part), it may be advisable to opt for vision- only embeddings. However, the learner should also consider the term
[ST (ies(z", =*), which represents the alignment between the ground truth visual and language latent vectors, essentially reflecting the intrinsic quality the data. If this term is already significant, relying solely on vision information as proxy for language information could lead to suboptimal results.
A.2.3 Detailed proofs
Lemma A.2. Let
G3 G = arg min Ge,GgERdxr
L(G,,
Gt)
Then we have
G - +
+ P P3 P3+ PA
(8)
where noise terms P. are defined in 12
and 15
Proof. Note that 8iz = (2/) G.G.T oj = TrG o (x+) T Gi), like the proof Corollary B.1. in 46). we have
T
L(G,, Gi) = Eies E (Bij - 88)
+ p |S|
|G, l1
S 1)
2 2 -
Eies Ejes 8ij
|S| Σ Eies* 8ü
|S|
||G GT l|
(
- 1)
2 1
|S|
|||
Tr G
aa(a
xt' (a')
Gi
+ 2 |s|
|. G 1|
|S|
|$] -
iES
Tr(G. rG)) +
0
|
G...G
1B
||
where oo = E i x 2.4.8 in Golub et we know that
//S| Then by the Eekart- Young-Mirsky Theorem (For example, Theorem
arg
min L(Go, Gi) G.ERdxr dR
|S|
Tr.G..IGi)
2 ||G,GT 12
arg max GRE
Rdxr dxr G.G.T = 1 S| SVD,(T))
= {(G,, Gi)
(Eckart-Young-ae Young- Theorem)
[S]
where the notation SV), means choosing the first components of the matrix r. Further note that
S
r
(
(9)
|S|
1
iES
Po P P
(10)
18

---

Here note that Es
1
Eies 2 (x we have Ps as follows:
[5]
S
Po
G
Es (Gi)
(II)
|S|
P
2 (E))
(12)
|S|
iES
P2
Σ (x)) (Gt)
(13)
S||
P3
(14)
|S|
SS
P4 =
(
(15)
|
It's clear that the rank of the matrix Po is no more than so SVD, = Po- i E (1,2, 4}.
P; are noise terms with P O.
Lemma A.3. For any fixed S. w.h.p 5 the noise term can be upper bounded by
[3]
Proof. To upper bound the P1 and P2, have
(E 11 Ir
Σ
(2( (T T 22
) (E;")
i
E1
zi ml (c T 112 E
(x
23 ol (c: end
= 1S|d
Si weakly dependent variable, then by using Bernstein inequality, we have, with high probability 6,
Regarding each (z!"²) ) 2, (gx1)
T
Σ
2 al (cr")
T < ISId+ dS log(1/6) < (SIdd/log(11//
So 19 Σ
2
(S T 1| <
d (1/4) Note that |
S
log S d)
(like Proposition 2.5 in
Wainwright et al. 48 ) it is easy to see that P3 ad P4 are the low order terms if S T
L
Lemma A.4 (Intuition behind VAS). With high probability 5, suppose the hind-side best subset has at least number of samples, then we have
1 max (Tr (Enngert Eg Es))) + P S' D
dlg((188) +
(108(188)
(SS)
[S]
Proof. For any learnt Go, Gj based on dataset S, we have
Lrest (Gv, Gi)
Tr(G., Eaa (x'(x) GG;)
-Tr(Emwr x0" (x') "
1
Tr (Bam xx' (GFEs(;)) Tr (Ear xx (x') ( [nossss)
P
Tr (
Eew ( - .Ir (Ler 100 (x!) nnisess)
P
Tr (Eunge Es) Tr DEam
1x0 (xx) (=n)Tooos))s)
19

---

Here the first equation comes from Theorem Consequently, have
A.4 and the third equation comes from Lemma A.2
( Tr (Earge Tr (E Dungi |x (x"())) |noises))
min Duan
Lusa (A(S))
max Duain
(Tr (ar(Enn (Etrges Esr) + [BBaa Pugee (xr (x') IIll noises ||
max
p Duni
1 max
(Tr (Eurges Egs +0 (v
(Valos1/0)
/dlo(110))
p Sinin
n
Therefore, have the final result
A(S) = Leea(Guu
min S'
Liea (A(S'))
(v) ddog(110)
1 max (Tr (Erage (Es: Es))) + +
d log((1) (1))
p S'E Ezam
[s]
Theorem A.3 (Main). Under the assumption of Lemma
A. we have
(SS) S noise [Snareet
Smerger
||||
E> E ll
S +
(vision+ + llaguagge)
(2E 33
(vision only)
Proof. Based on Lemma A. we will focus on the error cause from selecting subset S, that is, Tr Eg. Since the exact Lurget is unknown, assume the access to some proxy Euager instead.
Recall that, for any S, we have ground-truth Σg- = Ez,es== (z')T Unfortunately. this is not directly observable by the learner. Instead, the learner is able to observe some proxy Egg based on the teacher model Gel and therefore solving
arg max Tr (Enarger Es)
S
and therefore, denote Dest = arg max E Dem
Tr (Enarget
Tr (2ieee 2ss) 'r (ianger Whest
Es)) 'T (Erangor Es) Tr ((2rarget Etarger ( (2 Es))
< Tr (Surger Es Σss) + ((Enarge Emanged ( (Eeest Es))
Eurget ES
Esll. + 1|5 Enarger
EE Exest
where the first inequality is by the definition of Σg and the second inequality comes from holder's inequality. Now the key is to upper bound | h/ based on our chosen strategy.
In option I, we use the clip embedding from both visual and language modal. That is, choose Σg = (G.)s(G.)) xx" (xx") (ax))TG, GG. Then have
[É's Esll- S
1 S
(GG) xF 2.ES
Σ 2z 11- es.
In option 2, we use the clip embedding from language model only. That is choose Es
Exves G.] x (xx)) G.. Then, by definition of eg, we have
[Es -Esls <
1
||
Σ F.IES
Σ 2"(2")Tl1 l1.
|
Σ 22(2") Esll.
|S]
|s]
aNES
x"
Es|l.
20

---

Now to further bound the second term, we have
1
1
Σ
2²(=")
Esll.
Zz
11.11
Zi/1=
S
[S]
1
TT Zo V /TrZzo (Zx 7i)
[S]
1
(T((nxx) /2'rr (Iuxa - -Z,2i)
[S|
2)S((SS)
Σ (2",
|
SE[S]
(2", 211)
S
VE(S]
Therefore, we finish the proof.
Theorem AA (A simplified version of test loss). Under the assumption that both zet, Eet is zero-mean, maximizing the clip score gap is equivalent to maximize the clip score of the same sample.
Lirrge (Gus Gi) -Eaer-Dane (G Dus GTx))) x01)
Proof. For any Det, we have
Ex Dane ((G) wv. G. x01) (GT Bos GT mi))
[G Bv., Gj Exg, DDemm (a0/ 21))
-G Bvs GJ xi)
21

---

Target
Image-Texe Dataset
(27M)
OAI IT400M
HQITP-35OM
downstream tasks (2:1M)
All available
IN-21K,
IN-1K
captions
(1.3M)
CLIP Teacher Model
External Model
(CLIP embedding)
FAST
fasitext
BLIP-2
LLaVA-155 (GPT-4V)
MERU
NormSim
negCLiPLoss
S core
Ours
Image- based
pruning
Text-based
HYPE
Figure 4: Illustration of different directions for data selection methods for multimodal contrastive learning. Here four colors to denote the four main resources model, downstream target data (which is much smaller than the external multimodal dataset or pretraining dataset), the external image-text dataset, and the external non-CLIP model. Direction I denotes the methods that only use the original OAI CLIP teacher model and the downstream target data. Direction represents the methods that use external datasets train new CLIP teacher model for improving filtering, like DFN (2). Direction 3 denotes the methods that use external non-CLIP model select the data that may heuristically helpful for downstream tasks, like image without too much text or be more special. In general, D) method using only CLIP embedding. like neg negCLIPLoss, orthogonal D2. And both DI and D2 can be combined with D) to explore better filtering results. In the experiments part of the main paper (Sec. 4) further show that proposed D1 methods: NormSim and negCLIPLoss, can outperform all the D3 baselines except the best method "HYPE DFN", And we can achieve the new state-of-the-arr by combining our methods with that method.
obtain: CLIP teacher
B Illstration of Different Directions for Data Selection in Multimodal Contrastive Learning
We summarize our main idea of categorizing the current top data selection methods in Figure
C Details of Experiments
C.1 Computation Cost
Our algorithm can significantly reduce the computational cost compared to many existing works as shown in Ta For example, when the CLIP embeddings are obtained (cost about 50 hours for CLIP-B/32), both T-MARS [12] and MLM 42 still require more than 900 hours data propcccessing time to extract the required information from 11MM size dataset of DataComp- medeum,, while only need about hours. On the other hand, DFN, although has similar forward speed (i.e. preprocessing time), requires retraining new CLIP teacher model on the HQRTP-350M. which is larger than DataComp-medium.
We give some details in estimating the preprocessing time of other methods:
For T-MARS and DD2 pruning, we run their official code on DataComp-small (11M) data, and simply scale the preprocessing time by 10 for DataComp-nt DataComp-medium, -emommm given that the preprocessing time for T-MARS proportional to the size of the pretraining dataset, while ID2 pruning no faster than linear.
22

---

Table 5: Comparison of preprocessing time and external resources needed between method and other D3 category methods. We skip DFN since it's orthogonal to our negCLIPLos method and we directly improve it mentioned in Table [ Here since all the baselines below except MLM use pretrained CLIP model, we only count the time that doesn't contain that for inferring CLIP image/text embeddings (about 50 L40 hours for OAI CLIP-B/32), which is also adopted in DataComp The external dataset corresponds to the external multimodal dataset used for training or finetuning the external model. Notably, the preprocessing time for the following methods are all
benchmark (.
approximately linearly proportional to the amount of unfiltered pretrained dataset.
Filtering
Ext. Model Used
Size of Ext. Dataset
Preprocess
Training Time
Type
Avg.
Strategy
Time
D2 Pruning 18
NA FAST L3
NA
>70 L40 h
65 L.40 h
29.5
D3
T-MARS [2]
NA
950 L40 h
65 L40 h
34.1
D3
MLM [2
LLaVA-1.5 1.5 43
50k
1120 A100 h
65 L40 h
34.5
D3
Devil [4
fasttext [S] BLIP-2 16
NA
510 A100 h
65 L40 h
34.5
D3
HYPE 3
MERU 45
27M
120 L40
65 L40 h
31.9
D1
Ours (20%)
NA
NA
5 TT40 h
65 1.40 h 35.2
For MLM, we get the estimated time from their paper. They mention that they need 6. minutes to process 10k samples A100, which results in 1120 A100 hours for dataset (110M). We need to mention that their estimation time of calculating CLIP embedding is inaccurate and can do it much faster than their claim using the DataComp pipeline.
For Devil, needs to the k-means clustering algorithm from the faiss library on the embedding space, which is estimated to cost 120 L40 hours on DataComp- PataComp-medium. medium. Using BLIP-2 [16] scan the whole dataset will need about 470 A100 hours from the experimental details in [77. From https://ammdallass. httpo/aaaaaaaaaaapuuuoooooooo.
we roughly assume that 120 L40 hours are at least comparable 40 A100 hours for K-means clustering.
For HYPE, they claim that MERU is as efficient as CLIP, but they still need least 120 L40 hours for processing 1 1OM data for their final score, since it uses the image embedding clusters on DataComp-medium obtained from running k-means clustering algorithm.
C.2 Details of negCLIPLoss
We give the pseudocode of calculating negCLIPLoss in Algorithm I which is specially designed for pytorch-style parallel matrix calculation. It can be fully accelerated and the computation cost introduced by the normalization term is negligible compared with the training time preprocessing time of other top baselines as detailed in T.ble
In negCLIPLoss, we need to get the batch size [B] and the value of the learnable temperature parameter the final step of the teacher model pretraining stage. For OAI CLIP-I/14 and OAT CLIP-B/32, these values are 7 = 0.01 and [B] = 32768.
We also have an ablation study about the temperature parameter and batch size chosen for CLIP teacher models as shown in Table 6 We will see that in general, larger batch size will result in better performance, and The reason for such a batch size is that larger batch can contain more contrastive data pairs, which is also supported by the concentration result of the normalization term proved in Appendix A and thus it can check the image-text matching between more different data. Therefore, we always consider the largest batch size 32768 which can fit into single 24G GPU in the CLIP forward pass, which is also the OAI CLIP training batch size.
0.01, 32768 is the best choice for both OAI CLIP-B/32 and DFN-P.
C.3 Details of -ormSim,-D
In this section, we illustrate the details of our NormSima-D algorithm. The top-N selection method is aiming to achieve the object:
$ arg max
Σ (+)))(((+))7
fo(2!)
(16)
S
Xianget
iE5
Ie Ximen
23

---

Table 6: Ablation study about the temperature parameters and batch size for CLIP teacher model. The values obtained from the last training step of the teacher models are 7 = 0.01, 32768 for OAT CLIP-B/32, OAI CLIP-L/14, and b 16384, 0.07 for DFN-P. P. In the main paper, b = 32768, T 0.01 for all three kinds of teacher models.
OAT CLIP-B/32
Size
IN-1k
IN Dist. Shift
VTAB
Retr.
Avg.
CLIPScore (30%)
38
33M
27.6
24.2
33.6
25.1
33.2
CeCPLLsss (30%)
b 16384, 0.01
33M
28.8
25.0
32.5
26.2
33.0
b 16384, 0.02
33M
28.6
24.8
33.3
25.3
33.1
5 16381, T 0.07
33M
28.0
24.2
33.5
25.1
32.6
6 32768, T = 0.001
33M
16.0
13.9
25.1
19.4
24.4
6 32768, 0.005
33M
28.5
25.0
33,6
27.0
33.0
32768,
33M
28.8
25.1
33.7
26.6
33,6
32768, F 0.02
33M
28.5
24.8
33.6
26.2
32.9
h = 32768, T = 0.07
33M
28.2
24.5
32.8
25.2
32.7
CeLPLILsss (30%) NormSim,.. (Target)
16384, T = 0.01
22M
32.4
27.4
34.5
26.1
34.7
b = 16384, T = 0.02
22M
31.8
26.7
35.0
24.9
34.2
5 16384, 0.07
22M
31.0
26.3
35.0
25.5
33.9
5 32768, 0.005
22M
32.2
27.2
35.3
26.5
34.8
32768, T
22M
32.4
27.4
35.9
26.3
35.2
DFN-P
Size
IN-1k
IN Dist. Shift
VTAB
Retr.
Avg.
15%, b 16384, r 0.07
16M
31.0
27.0
35.2
26.8
34.2
15%, 32768, т
16M
31.3
27.3
35.8
26.4
34.6
17.5%, 16384, т = 0.07
19M
31.3
27.2
33.5
27.6
33.5
17.5%, b 32768, T = 0.01
19M
31.2
27.5
35.7
27.0
34.7
negCIIPLLoss (17.5%) n NormSim (aaiSiimmmMTTTraaaa (Target)
6 b= 16384, r = 0.07
16M
31.1
27.4
34.8
26.1
34.2
b 32768, 0.01
16M
31.6
27.3
37.2
25.5
35.7
when the actual Xaarett is unknown. In practice, removing one data time slow. Therefore, remove batch of data for every step. In detail, if the number of steps т, and let E [S. Ejes, [u(=:f) (m))T where S; is the selected subset step then will remove the data satisfies the following equation step-by-ster until reaching the final subset size:
5; S = min 21ES:
(x
1
Σ (((a))o(4))))) FLES:
Fu(lot)
i 10-
S.
Then we can detail the algorithm process of NormSimz-D in Algorithm 2 In general, the smaller the step size, the better the results. But in experiments, we find that it's already enough to get good results when 7 500.
C.4
Details of Related Works
We add some details about the baselines used in our paper follows.
Text-based filtering. ① proposes a text-based filtering that tries to select the data that contains caption overlapping with the class name from ImageNet-21K ImageNet-IK. I.
Image-based filtering. [ also proposes heuristic way to sample the visual content overlaps with ImageNet- 1K classes. They first apply filtering by language (only choose English caption by fasttext [15) and caption length (over two words and 5 characters). Then they cluster the image embeddings from training data to 100K groups using Faiss [49], and keep the groups whose cluster center is the nearest neighbor to at least one image embedding of ImageNet- IK image.

---

Algorithm negCLIPLoss
image/text embeddings of the pretraining data Fo [(fr((=)))) .,..., (f(( RNxd batch size b, temperature parameter the number of times negCLIPLoss is random {-= 10).
Inputs:
Initialize neg array ...... ,0 E RN
for k == to K do
Get a random batch division Sy: [Bi....,BB. B.} such that [N/b]. Every B, E Sy. the index of a batch of data.
for j 1 to do
Get batch of embeddings in batch j: F2 F E faoxd
Get the similarity matrix: Ej F7 (F)) E Rbxb
Get the CLIPScores: Cj = diag E IR
Define G; exp(EE/T))
Define 9j E IR be the vector containing the sum of each row vector in Gj (i.e., over image).
Define 9) R be the vector containing the sum of each column vector in G, (i.e., over text). Get the negCLIPLoss: r r[B]] Cj 0.5r (log(g)) + log(g))), here we use element-wise operation.
end for end for
Take the mean each random division as output: negCLIPLoss r//
Algorithm 2 NormSim-D strategy
Inputs: image embeddings of the data after CLIP score filtering (ff (x)))ssss target size N, number of steps T Initialize S S, No |S|
fo to do
Size step :: N₂ = No- - 4No N).
Prior matrix: E = - (o))))
Updated NormSimz-D for each sample in S
(ommSim-D(zz) = fo (T Eieat11- fx(x))
Construct St such that it contains the data with highest NormSing-D in St-1 and satisfies |S,] = N,. end for
2 Pruning. 188 tries to represent the dataset undirected graph for coreset selection. They assign the difficulty for each example and use message passing to update the difficulty score incorporating the difficulty of its neighboring examples, and finally try to keep both diverse and difficult subsets. For our experiments, we adhere to the default hyperparameters TD2 on DataComp specified in their official codebase.
T-MARS 12 uses text detection model like FAST 13 filter out the data that only contain the texts of caption in the image and don't have other useful image features.
Devils 14 combines many ways for data filtering. At the very first it filter data based on heuristic rules like text length, frequency of texts, and image size, and it also CLIPScore for cross-mi modolity matchment. Then it adopts target distribution alignment methods similar to image-based filtering, but instead of using ImageNet- Ik only, uses 22 downstream tasks as the target set. Further, it adopts external models fasttext [15] to remove non-Ean oon-Esh captions and image-captioning model BIIP-2 50 to select images with MNIST-style digits.
MILM 42 prompts GPT-4V to construct instruction data including the image-text data, and use it to fine-tune smaller vasion-language model like LLaVA-1.5 43 into filtering network. Nevertheless, the number of parameters of L.L still much larger than CL.PP, and thus LLaVA-15 has much longer preprocessing time mentioned in Table C.I
25

---

C.5 How to Choose Hyperparameters
The main hyper-parameters of our negCLIPLoss and NormSim are the target numbers for filtering (refer Appendix C.2] for the setting of temperature and batch size), which also the main concerns for all the top baselines like DFN, MLM, and T-MARS, In the case of DataComp settings, noting that all the top baselines in CeedCoomp-miium benchmark keep the downsampling ratios ranging from 15% 30% to achieve the best results, we can set the sampling ratio as some previous baselines. Our method with OAI CLIP teacher model first selects the data with the top 30% negCL.PLLoss and then selects the top 66. 66.7% NormSim scores to keep 20% of the original pool. We don't 't tune the target size carefully here for fair comparison.
In more general cases, we can recommend some training-datased tlaatseet-nddeeedeett thresholds for Norm- Sim, since the scores only depends on the norm p and target data rather than other data in the pool. We recommend to set the threshold as 0.7 for NormSim.e (Target) and 0.15 for NornSimz(IN- (N) in general. On the other hand for negCLIPLoss, note that like NormSim, CLIPScore also training- dataset-inddeendden, we recommend to first find the percentile of the data with CLIPScore=0.21. and then downsample the dataset using CLIPLoss until that particular percentile.
Overall, finding optimal filtering ratio for data selection algorithm is always difficult and out of the scope of this paper. From the paper about the scaling law for data filtering (51), downsampling size even depends on the computation budget. When you have more budget, you should sample more data for learning. And thus another possible solution is use their fitting formula to get some recommended downsampling ratios.
At last, also note that in data selection problem, visualization is simple but effective way for tuning parameters or finding downsampling ratios. People can first randomly select a small subset (like 1000 data) some pretraining data subset, and then calculate the target scores (CLIPScore, negCLIPLoss, NormSim or any other metrics) on them, and fianlly visualize the data corresponding to scores different percentiles, like top 10%, 30%, 50% and 70% of the negCLIPLoss, In this way, we can determine the threshold of filtering directly by observating the data. We also give some visualization examples of our methods in Appendix [E] We believe this is effective way to give some guidance on how to roughly select the initial downsampling ratios.
C.6 Discussion of NormSim
C. How NormSim; Connects Selecting the Data in Principal Components.
For convenience, we let f(;) denote the image embedding of the target data x, E Xy, and fxx) denotes the image embeddings of training data x, Xs. Then the definition of NormSim data s, is
1/p
NormSim, (XT, x3)= )=
Σ 1z ExY
[f(x)) [f f(xx))))
(17)
Then when 2, we have
1/2
CamSiim(Xxy.aa
Σ If(x)) f(x)) (f(x.) f(x.))
(18)
NEXT
1/2
[(x.)
Σ (axe)((xa)) (xxs)
(19)
w.EXr
1/2
[(xa)T
Σ {as))())) f(xx)
x
(20)
[xT]
26

---

Note that A= [X Ex Ex,x, f(x)) is the variance matrix of the target image embeddings.
Then using NormSim, for filtering, we have
arg max 18|=N
Σ
NormSimg (x))
(21)
2, EXEX
NormSima (Xr, v. f(xx) T. A f(x))
(22)
= fxx)) TU S. U f f(x,)
(23)
Σ 8) [f(z)) uuy1
(24)
Here A = USU is the eigen decoposition of A, where S diag(s), ,.., are the matrix of cigenvalues, and U [u....., Up] E gdxr are the corresponding cigenvectors (i.c., the principal component directions). Note that the column vectors of U and f(z,) are all unit vectors, (24) shows that NormSim, select the data that match with the principal components, i.e., eigen directions U, with large eigen values 8j.
87) with 81 >...> 8r
C.6.2 Why NormSim works well without explictly considering data diversity.
We answer this question by the following reasons:
Many top baselines, such DFN and T-MARS, also don't explicitly consider diversity, yet they still provide good performance. Devil even shows that valuable data is worth sampling multiple times, which they call "quality duplication". Therefore, why NormSim works well without explicitly considering diversity may be that when the computing budget is limited, in the DataComp benchmark, the model first needs to learn the most useful and representative data, which should be similar to some target data.
important reason
Moreover, we chose validation data from 24 downstream tasks ranging from ImageNet to EuroSet, which may have covered a sufficiently diverse range of target examples for NormSim to calculate similarity. The diversity of the target data will consequently result in the diversity of the selected subset. And this also implies the importance of selecting good target dataset.
An additional reason may be that our proposed negCLIPLoss already implicitly selects more diverse data, as shown in Figure of the main paper. If some training data are diverse, they
will match less with other data and thus have lower normalization term. This results in larger negCLIPLoss and a higher probability of being sampled.
D Additional Results
D.1 Stability Analysis of Batch Sampling Numbers in negCLIPLooss
We show that negCLIPLoss is not sensitive to the number of random select batches K in Figure 5
D.2
Universality of negCLIPLoss over Different Teacher Models
We show the complete results of applying our methods to different teacher models like OAI CLIP-B/32 and DFN-P in Table 7 Detail descriptions are in Sec.
D.3 NormSim is Better than Nearest Neighbor Selection
We also try to use near-neighbor selection for aligning downstream distribution. Here, we calculate the ranks of pretraining data for each target (the higher the rank, the higher the similarity), and then for each pre-train data, we keep its highest rank. Finally, we select the data with the highest ranks as the nearest neighbor selected subset.
In Table we show that given the training data of 22 downstream tasks, our NormSim can outperform near neighbor selection under the same downsampling ratio. The reason may that the distribution between the target and pretraining set is not well aligned, if you force the algorithm to
27

---

Comparison of Methods Across Different Metrics
Filtering Strategy 20%
Filtering Strategy 30%
30
CegCoss
30
negCUPLos
25
25
R-15
Recioval
Festival
10
50
100
10
K
K
Figure 5: Results of negCLIPLoos with a different number of batch samples (denoted as K) on DataComp-mediun Solid lines denote negCLIPLoss, while dashed lines denote CLIPScore. Here, we use OAI CLIP-L/14 as the pretrained model. We can see that once 2 5, negCLIPLoss consistently outperforms CLIPScore across all subtask metrics. In the main paper, we K 10.
Table 7: Results on DataComp-mediun from the top methods that use only OpenAl's CLIP-B/32 model or public version of DFN (DFN-P).
Dataset Size
IN-1k
IN Dist. Shift
VTAB
Retrieval Avg.
OAI CLIP-B/32
(1 sub-task)
(5)
(11)
(3)
(38)
CLIPScore (20%)
22M
27.0
33.0
22.9
32,2
CLIPScore (30%)
33M
27.6
24.2
33.6
25.1
33.2
megCI.IPI.oo (20%)
22M
28.9
34.3
33.0
negCLIPLLoss (30%)
33M
28.8
25.1
33.7
26.6
33.6
neg (30%) ( NormSim. (Tareet)
22M
32.4
27.4
35.9
26.3
35.2
DFN-P
CLIPScore (15%)
16M
25.9
32.9
21.9
CL.PS (17.5%)
19M
30.2
26.8
34.1
26.5
33.8
CLIPScore (20%)
22M
29.7
26.8
33.0
27.0
33.1
CLIPScore (30%)
33M
28.4
24.7
33.2
26.8
32.7
negCI.PPI.oss (15%)
16M
31.3
27.3
35.8
26.4
34.6
negCL.PPI.oss (17.5%)
19M
31.2
27.5
35.7
27.0
34.7
negCLIPLoss (20%)
22M
30.7
negCLIPLoss (30%)
33M
28.9
25.5
33.4
27.3
33.2
negCLIPLoss (30%) O JormSim..(TTrgeer
22M
29.4
23.6
33.5
24.2
32.5
negCLIPLoss (17.5%) n NormSim,o. (Target)
16M
31.5
26.4
34,6
25.4
34.4
negCLIPLosss (17.5%) NormSim (Target) (Target)
16M
31.6
27.3
37.2
25.5
35.7
find the nearest train data for each target, that train data may be sometimes random and not helpful. On the other hand, NormSimæe will not select this kind data. will select the data whose best similarity score exceeds general threshold, rather than just consider ranks.
D.4 Vision-Only NormSim is Better than Using Both Vision and Language
In DataComp [1. they show that image-based filtering is better than text-based filtering. In our paper, we also an ablation study to support this. Due to the restriction of computation resources, we run NormSim, (IN-1k) and NormSimg-D DataComp-small
example. Since ImageNet- lk
28

---

Table Comparison between NormSim. and nearest neighbor selection. We use OAI CLIP-L/14 as the teacher model and assume both methods have been intersected with negCLIPLoss (30%). The size of the selected subset 22M.
Filtering Strategy
IN-1k
VTAB
Avg.
negCLIPLoss (30%)
27.9
33.2
32.9
Nearest Neibor Selection No (Target)
34.9
34.0
31.7
36.0
35.0
only has labels rather than long texts for describing images, we need to generate the caption before calculating NormSim;(IN-1k), We select 80 templates as the original CLIP paper 44, generate prompts for each class, and take the mean of their embeddings as the representative text embedding for images within that class.
The results are in Table We can see that both metrics, we have "image only" > "image x text" > "text only". We believe the reason for NormSim,(IN-Ik) is that the images themselves can convey significantly more features than the text prompts generated by labels. For NormSimg-D. D, should be related to the large amounts of low-quality captions in the web-curated dataset. And "image text" will also be influenced by the informativeness and the quality captions. In short, for NormSim, using vision-only embeddings is best choice.
Table 9: Ablation Study the NormSim and its variants DataComp-small (11M). All experiments first select 45% data based on the CLIP score, then use corresponding approaches to obtain 3.3M data. "image" or "text" means using the variance of image text embeddings represent Emaget, and "image text" means representing Earget with the cross-covariance of image and text embeddings.
Filtering Strategy n CLIP score (45%)
IN-1k
IN Dist. Shift
VTAB
Retrieval
Average
Random Sampling
4.2
4.9
17.2
11.6
15.6
NormSim (IN- Ik, image)
5.2
5.5
19.0
12.2
17.4
NormSim IN-Ik, text)
3.9
4.2
16.3
11.3
14.9
NormSim (IN- lk, image text)
4.3
4.9
17.5
11.8
15.9
NormSim-D (image)
4.7
5.4
19.7
11.7
17.3
NormSim-D (text)
3.5
4.1
16.7
11.1
15.4
NormSim-D (image X text)
3.6
4.2
18.4
11.1
15.8
E
Additional Visualization
We further visualize more data with different negCLIPLoss in Figure 6
and 8
And similar for
NormSim (Target) in Figure and and 11
*We use http:///wwaaaaa//.oo.ooo/oo//a/a ageCaption Visualiieo for visualizing the dataset. We also recommend visualizing basic dataset statistics
(Im-

---

mat laurns
Guacutessyy' e ustovego ase 512 mis 912 pr S.A. 8646one, bybounes Lumino
top facl presents araaz
u dicingue suf petter Black Becroom
Stock photo
hone
Deno)
Eacaga Ply-iin Stula uk Broomss guris ford
1011021-goo
21-9-77 S4-97 ane
IONBRAAD
e beth NOZ 575 JAK
AR
377695
792
laarrucz! gameotie.rett
shisport
Habep the 4559 sourry welch man Binee trm clocs ma sport Big
5i jaluts Ble Usa St 5357 ACE
Bo43
PNG R63-A
C Ditur
the eiler en emor ou
CarTucT SHNOODE TOXTO HEUBRAAIL USO INc.
103 Ducoee porque as
Chiktrnn rear water are
Mirme arR crnyyoa at
FRRS 211 1M
usraatian, der mi Pute
WELCAME e
Hartcore ezare aditcioo gaión masechcaasine
icusi Blngg star wather som
[vaull (Sud d.pp-l
Tehu do BroddRocTOOS
Loading.
atp SANGS
to 183
Stattsch ver begelinn or de Veri not na van he
tin Vaeee lde, Kids Silde, trampodine, Pio
JEN TORTUGA
Sae rless steed vah pen
Genniaa BERO, 301
O
GAME CRITICS
Figure 6: Visualization of small subset whose negCLIPLoss rank top 100% high in DataComp- medium.
30

---

07204000 3410 4881 5F17 Squat doguoo by
Snapcran, RORAAm 2014 10629682) 2042 92C2
meirdde's
Gecoponta von t5pazaa
Plucc argenica noc
Xystec Hub de burnay ab mm
Mizt Boxi Acceosorie Stop
puserirc soMfunecii lso
Seal Res
Esl chadel 000037
Yee be Ka ton.ttnng
mn image
B
Holiday tott fantll langerho's
Handy, Intel lggenies
agam 21xE8.c0m Bltt
Rebot Soreer Task dalle Faging
cuilk acoed yeu wody
426-01-156
BOST, SPECIAL YHUS
I8 RULAWA IEAA ROUB
what presertittio
sentors- wwienn-2dd0011..pp
Radd Delate Pheto
spricam praettpau
cs
LAicY
Consiteraples Sobre do inflacio
porta BRCTE Enga
Thactions ys Video Gui
Lie paquet
ipal
AGU1D
stidy, de Mayo Martin
ean Bage (103.11
PDF
The Vingin in and Child lo wi Strendrd Bhrthees
Card Sew (flwwc
Perivk meittneRRR pro
Work esion
Figure 7: Visualization of small subset whose negCLIPLoss rank top 50% high in DataComp- medium.
31

---

getiraa on The tived
Lakowood rnch iroust Grade
Shimang Stella 13000 sw s 5-4228 Head Visee
miless 225 eng
gathers likig
Ruby
countiess bhr
250 KoNier Harine
DDose
KOHLER. Marine
Dree
nouber Whiler 347 866 441-3401
Visl pasee Wellswogen
Street 655 Goyy
Taps lyte city
rgen armmnn choin hittena hai 2013
Roake 160 T6l, Chinaa tallie Colored backgrsu
trazent piletalence idler
monochrome snalng oym
41
Mindliness the Go
egration
in
GO
Keuzil,, Gildur, bunde 100 Coube
Purtab Wetter. Sewet-Seeeeeeeee-rooom
emeraid diamon dtater engagement
Selloats Bocied Mari Carrica
mww.brntneccspeeeeee mil--27082
White StenSemg:e30s itammc 1ohitt
Panascricc
1g
treechy leadhanck for sed Seil ct4 (funcy -V
Man and sex on artiat from hapedaale,
containne on May 20
Ewer Early 20th century
2003-4 2130 Litre carreo
ga Panging off tree bw!
carre, some 50cmm morth
Inv023 essel inttin walue tamplatte
Novern Llebromadel Klometar wot
Papler Alasota 3Dcm
aby BMAZIAA
Humger
Mision Duotes Memss
Shower Curtan
ARMY
Part
Ht soN Bur ChA Souial
beryy Pamigg analte Organ
mall
Pg
cro0600 Logs
Pg
ICML 2019
Figure 8: Visualization of a small subset whose negCLIPLoss rank top 10% high in DataComp- medium.
32

---

Civmetter
GATE 20247
One Gool
VOWWO
Golls
PDF
BUGOUT
REAL MADRIO
GETAFE
JS] digistore24
TUTORIAL UN
MEMORIES
$450
LOGO
ON PRES
Pork?
CENIAMLINE
Lorth
IN 15 MINS
Srgalaati
FAMILY FUN NIGHT
AARE
Painer Npprentee
LOBB UESTAISINIIIIEEE..
LADIA
PLANSS
PENGIONES
LLP baL
SKYRIM
Hada
3,000
PARTER
CHEEAP TH E3
ENG SOERRR
ROC/USOT
PRACUEE
KUNNING
D
F
FREDO
"SCAYY
EMPIRE
SISTO DISTRROOK
SITE.
MANUEL CREANOLINI
THEIS DNSSENTATION PRRJECT
ECO
DENASTADO FUERTE
Ravotta 2apaling
BEACH
STARLING
RSS
Mårbacka
WNTIL 1 FOUND YOU
Welling les Tireesszion
with Leall raigey
Selu Logglid
KA
SYSTRABUND
PELIUU NAUDU
ARRSS
RC.0 ISA Bo at DD
propel
BROS
CaumpiI Beeting
ZND AUGUST
July
ac
Usuñ Hemoroidy
FREE LASIK CONSULTATION
MILK DAY
LOWONGAN KERJA
SumFiI OtD Eirence
HUN CNB
2020/2021 Feal
NVIIKON
Figure 9: Visualization of the images from small subset whose NormSim,c (Target) rank top 100% high in DatCComp-medium.
33

---

PDF
ON ORDER
14
Thankful Tolii
Figure 10: Visualization of the images from a small subset whose NormSim (Target) rank top 50% high in DataComp-medium.
34

---

hubbiz
hubbiz
Figure 11: Visualization of the images from a small subset whose (Targe)..(Target) rank top 10% high in DataComp-medium.
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes we clearly define 1. the benchmark are using; 2.the methods with its key insights 3.the empirical improvement.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss this briefly in the last section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
36

---

The full version of theory of NormSim results are in Appendix. and we provide all the assumptions and proofs. We briefly mentioned this in Sec. 3.2
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
We also provide experiment details in Justification: The main results are in the Sec.
Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
37

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code will be provided according to Neurips code submission guidance. After got accepted, we will open source that.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https ctspps.cc/pullic//ge
Cotm//umeeeeeeeeeeeeeee ssi onsliicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The main results are in the Sec 4 We also provide experiment details in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Almost all existing works, like DFN, HYPE, and MLM, only run the training once on DataComp-mediun. Training on a 128M size dataset is very costly and relatively stable, commonly believed that there no need rerun experiments with different training seeds. In the experiments, we fix all the training seeds to be 0 for fair comparison. For our algorithm, most of them are deterministic. The only one involving randomness is negCLIPLoss, which requires resampling K=10 times. For it we provide sensitivity analysis in Fig. 55)
Guidelines:
The answer NA means that the paper does not include experiments.
38

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We discuss the computing cost estimation and comparison in Appendix C.I We didn't 't explicitly calculate the memories since is quite standard under the DataComp benchmark.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
39

---

This research focuses on the methodology part of data selection. All experi- ments are performed under the existing standard dataset. So as long as those datasets itself maybe harmless, our research will not make any negative impact.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
This paper will only provide UID's of selected data from existing datasets (DataComp-medium (1). This paper will not release any model or new dataset.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the DataComp 1 which introduces the URL for the dataset and the code/models used to implement the benchmark.
Guidelines:
The answer NA means that the paper does not existing assets.
40

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaeed, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects. All metrics are fixed evaluations.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing research with human subjects
41

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
42


Reviews:
[{"review_id": "o9hB4VN14G", "reviewer": "Reviewer_3xCf", "summary": "Data selection is crucial in the pretraining stage to clean the web-crawled, large, and noisy pretraining dataset. Typically, existing methods use embeddings to compute CLIPscore in order to assess the data sample alignment quality. This paper introduces two new methods to enhance this measurement:\n1. negCLIPLoss: a better adjustment to reduce bias within a given batch.\n2. NormSim: provides additional information when downstream tasks are known, allowing the selection of samples that are close to the target downstream tasks.\nEmpirical results demonstrate that these proposed methods can be easily combined with existing filtering approaches. The authors also illustrate that their approach yields state-of-the-art results on the DataComp leaderboard.", "questions": "1. In Figure 1, R scores on the left side are in the top 100%, while on the right side they are in the top 10%. How should these be interpreted and categorized as underestimates or overestimates of quality?\n2. When the downstream targets are not accessible, we may use the current filtered dataset as a reference, but how do we find the first-round reference dataset as a proxy to compute NormSim? \n3. I would like to list several papers that I found and read for data selection.\n\nhttps://arxiv.org/abs/2405.15613, https://arxiv.org/abs/2401.12225, https://arxiv.org/abs/2302.03169, https://arxiv.org/abs/2401.04578, https://arxiv.org/abs/2404.07177", "limitations": "I didn't see any potential negative societal impact of their work.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* Originality: Most of the work in data curation relies heavily on the original CLIP score. It's a new idea to adapt the CLIP score and elevate this measurement for better use.\n* Quality: The resulting performance is solid and achieves the top position on the leaderboard (medium-scale).\n* Clarity: The motivation behind the two approaches is clear, but some areas need further clarification. Questions are listed below.\n* Significance: Data selection in the pretraining dataset is important to the field, and they have demonstrated that their approaches are effective in achieving state-of-the-art results.", "weaknesses": "1. I think we need more clarification on how to interpret the Top X% in three different metrics in Figure 1. Can the authors provide a more detailed description? Also, how is the R score derived from the batched data? How to find the proper batched data to use?\n2. It seems that the negCLIPLoss is not incorporated into the training loss. We use it as a measurement when CLIP embeddings are provided. In this scenario, how do we determine the batch data, B, for subtracting the regularization term? Would the size of the batched data affect the measurement? The sampling method to find batched data is unclear to me.\n3. I am unclear about the process for greedily selecting samples using NormSim, especially when the raw data pool is massive, and how to define the size of S. \n4. I would suggest moving algorithm steps from the Appendix into the main body, or showing some steps in the main body. They are good at understanding filtering steps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback to help us improve our paper. We will revise our paper based on your feedback. We detail our response below and please kindly let us know if our response addresses your concerns.\n\n> **Q1**: I think we need more clarification on how to interpret the Top X% in three different metrics in Figure 1. Can the authors provide a more detailed description? (How should these be interpreted and categorized as underestimates or overestimates of quality?)\n\n**A1**: Thanks for mentioning this. We show the modified Figure 1 in the one-page supplementary based on your advice, and we illustrate it in detail as follows. \nWe use the ‘Top X%’ of a metric to denote the score which is top X% high among all the scores of this metric in the data pool. For example, in Figure 1, R scores on the left side are top 100%, indicating that these examples have the smallest R in the dataset. \n\nBesides, in this case, we note them as ‘CLIPScore can underestimate the quality’, mainly just because their CLIPScore is relatively small (like Top 78%) while negCLIPLoss are high (like Top 34%). As we can see from both visualization and the experimental result, those data have high quality which is underestimated by CLIPScore. Similar claims hold overestimation cases. In Lines 154-165 in the main paper, we further show the intuition behind the normalization term $R$. \n\n> **Q2**:  How is the R score derived from the batched data? How to find the proper batched data to use? \n\n**A2**: We summarize how we choose random batch and obtain R score and negCLIPLoss from different batched data as follows:\n\n(1) We split the whole data into batches randomly, from which we obtain batches $\\{B_1,\\ldots, B_k\\}$.\n\n(2) For each batch $B_s$, we calculate the cross-image-text similarity between the data in the batch, i.e., $f_l(x^l_i)^\\top f_v(x^v_j)$ for any $i, j \\in B_s$. \n\n(3) Using these scores, we can calculate the metrics of all the data in this batch from Eqn.1-2, and we record them for each data.\n\n(4) Repeat (1) - (3) for K times (Note each data will have multiple $R$ calculated from K different batches which all contain the data itself), we then calculate the mean value for these K different R scores and negCLIPLoss, and use them to approximate the ground-truth values.\n\nDetails can be found in Algorithm 1 in Appendix C.1. We mention that this process isn’t the only choice to get the random batch. We choose this method mainly to avoid double calculation of the cross-image-text similarities. \n\n> **Q3**:  negCLIPLoss is not incorporated into the training loss. We use it as a measurement when CLIP embeddings are provided...Would the size of the batched data affect the measurement? \n\n**A3**: Yes, we use negCLIPLoss only for data filtering rather than training. We want to emphasize that the main focus of our paper is on data selection with fixed training pipelines. In A2 in the 'reply to all reviewers' parts, we show how the batch size affects the measurement. In A1 in the ‘reply to all reviewers’ parts, we also theoretically show that using a larger batch size, negCLIPLoss will have a smaller approximation error.\n\n> **Q4**: the process for greedily selecting samples using NormSim, especially when the raw data pool is massive\n\n**A4**: We note that NormSim is only determined by each data itself like CLIPScore, so the ‘greedily selecting samples using NormSim’ just means simply selecting the data with top NormSim scores. We use the words ‘greedily’ because for this particular NormSim-D algorithm (Details in Algorithm 2 in Appendix C.3), theoretically, we should solve harder optimization problems, but here we use a greedy way (select the top scores) to do approximation. In the revised paper we would change the word to prevent confusion.\n\n> **Q5**: how to define the size of S\n\n**A5**: In general, for all the top filtering methods, like CLIPScore, HYPE, and T-MARS, we always need to set the target size of the filtered dataset manually. Like in DataComp, all these top baselines keep the downsampling ratios ranging from 15%~30%. Our method with OAI CLIP first selects the data with the top 30% negCLIPLoss and then selects the top 66.7% NormSim scores to keep 20% of the original pool. We don’t tune the target size carefully here for fair comparison.\n\nIn practice, this remains an open problem for all leading baselines when dealing with a large raw data pool. Here we found that a simple but very useful way to define $S$, is random sampling a small subset (like 1000 data) from the large pool and visualizing these data based on their scores, as Figure 6-11 in the main paper. From this we can determine the filtering threshold of the metric scores and thus the target size. (like we find 0.7~0.75 is a good threshold for NormSim). Details are shown in Appendix C.5.\n\nBut overall, deciding a proper S is beyond the scope of this paper. We agree that this can be a meaningful direction for future research. We are also aware of some recent works [1] that suggest there are scaling laws for data filtering, indicating that the target size for filtering is strongly dependent on the computing budget.\n\n[1] Goyal, Sachin, et al. \"Scaling Laws for Data Filtering--Data Curation cannot be Compute Agnostic.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n> **Q6**: I would suggest moving algorithm steps from the Appendix into the main body\n\n**A6**: Thanks for your advice! We will add them in the revised version.\n\n> **Q7**: how do we find the first-round reference dataset as a proxy to compute NormSim in NormSim-D?\n\n**A7**: For the first run, we just use the whole original dataset as the proxy for calculating $\\text{NormSim}_2$. For effectiveness, we only randomly downsample 10% of data for calculating $\\text{NormSim}_2$, and the results are similar to using all the data. \n\n> **Q8**: list several related papers \n\n**A8**: Thanks for your advice! We would cite all these papers in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TIFC5ITUSg", "reviewer": "Reviewer_XD1b", "summary": "This work proposes two new approaches for data selection for vision-language pre-training. The first approach, negCLIPLoss, adds the contrastive loss as a normalization term on top of the existing CLIPScore metric. The second approach, NormSim, further improves the performance if examples from target task distribution are available. Together, the methods achieve state-of-the-art results on ImageNet-1K and 38 downstream tasks with DataComp-medium without any external data or model. Several important theoretical justifications and interpretations are provided for the methods.", "questions": "1. In Lines 135 - 136, the inaccessible batch division $B^*$ from teacher CLIP models is different from the actual batch $B_k$ in this work, in terms of both the actual image-text pairs and the batch size. Are there any potential theoretical guarantees or approximations to show that such a difference is reasonably negligible?\n\n2. Could the authors further show the derivations of the discussions on the two important NormSim instances? 1) Lines 179 - 180 ($p=2$, equivalent to selecting a subset that aligns with the principal components), and 2) Lines 181-182 ($p=\\infty$, a sample will be selected if it has high similarity to any target)? These may help other readers to understand NormSim better.", "limitations": "The authors discussed the limitations of this work.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Quality: the quality is overall high. Without external resources (on which previous methods rely), the proposed approaches improve evaluation performances by 5.3% on ImageNet and 2.8% on average of 38 downstream tasks. Further, there are many theoretical results that focus on the guarantees of NormSim (though with strong assumptions).\n\nClarity: this paper is very well written. It is well-motivated, the distinctions of previous approaches are succinctly laid out, the methods are well presented, and the results have a clear structure.\n\nSignificance: this paper will bring significant impacts. The data selection problem has been increasingly vital for training higher-quality vision-language models. The proposed approaches, which focus on metrics instead of models or data, are compatible with different techniques that can be combined with advanced models in the future. The approaches also provide significant efficiency improvements (e.g., from 70 L40 hours to 5 L40 hours). The theoretical analyses can provide useful tools for future research as well.", "weaknesses": "Quality: this is a minor complaint, but in Lines 229 - 230 the authors state that \"the results of baselines on the leaderboard do not apply to our datasets, and we reproduce all the top baselines on the leaderboard with their public UIDs of the selected data\" because some URLs of images become invalid. The leaderboard scores of baselines seem higher than the reproduced results in the submission. Could the authors also include the DataComp leaderboard results in the Appendix for fair comparison?\n\nThere are also some minor questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of our paper together with your valuable comments and suggestions. We will revise our paper according to your comments. We respond to your questions below and would appreciate it if you could let us know if our response addresses your concerns.\n\n\n> **Q1**: this is a minor complaint, but in Lines 229 - 230 the authors state that \"the results of baselines on the leaderboard do not apply to our datasets, and we reproduce all the top baselines on the leaderboard with their public UIDs of the selected data\" because some URLs of images become invalid. The leaderboard scores of baselines seem higher than the reproduced results in the submission. Could the authors also include the DataComp leaderboard results in the Appendix for fair comparison?\n\n**A1**: Thanks for your advice! We would include the DataComp leaderboard results in appendix in the revised version.\n\n> **Q2**: In Lines 135 - 136, the inaccessible batch division B∗ from teacher CLIP models is different from the actual batch Bk in this work, in terms of both the actual image-text pairs and the batch size. Are there any potential theoretical guarantees or approximations to show that such a difference is reasonably negligible?\n\n**A2**: Thanks for mentioning this. We construct a theorem using the concentration inequality to show that when the batch size is sufficiently large, the normalization term $R^{B_k}$ obtained from actual batch $B_k$ can approximate $R^{B^*}$ calculated using ground truth batch $B^*$ quite well, i.e., $R^{B_k} = (1+o(1))R^{B^*}$. The details have been shown in A1 in the ‘reply to all reviewers for the major concern’ parts. Here we assume that $B^*$ and $B_k$ are i.i.d. for simplicity since the claim cannot hold if the teacher batch is very different from the actual batch. We also assume that $|B|=|B^*|$. In practice, we claim that a larger batch size is better since it can contain more contrastive pairs in a batch, and we do some ablation studies as shown in A2 in the ‘reply to all reviewers for the major concern’ parts (Table R1) to support our claim.\n\n\n> **Q3**: Could the authors further show the derivations of the discussions on the two important NormSim instances? 1) Lines 179 - 180 (p=2, equivalent to selecting a subset that aligns with the principal components), and 2) Lines 181-182 (p=∞, a sample will be selected if it has high similarity to any target)? These may help other readers to understand NormSim better.\n\n**A3**: Thanks for your advice, we show the derivations as follows and we add them in the revised paper. For convenience, we let $f(x_t)$ denote the image embedding of the target data $x_t \\in X_T$, and $f(x_s)$ denotes the image embeddings of training data $x_s \\in X_S$. Then the definition of NormSim on a data $x_s$ is \n\n$$\nNormSim_p(X_{T}, x_s) =  \\left(\\sum_{x_t \\in X_T}  [f(x_t)^\\top f(x_s)]^p\\right)^{1/p} \\qquad (R1)\n$$\nThen when $p=2$, we have \n$$\nNormSim_2(X_{T}, x_s) = \\left(\\sum_{x_t \\in X_T} [f(x_s)^\\top f(x_t)]\\cdot [f(x_t)^\\top f(x_s)] \\right)^{1/2} = \\left(f(x_s)^\\top \\cdot\\sum_{x_t \\in X_T} [f(x_t) f(x_t)^\\top ]\\cdot f(x_s) \\right)^{1/2}\n$$\nNote that $\\Lambda=\\frac{1}{|X_T|}\\sum_{x_t \\in X_T} [f(x_t) f(x_t)^\\top]$ is the variance matrix of the target image embeddings. Then using $NormSim_2$ for filtering, we have\n\n$$\nS = \\arg \\max_{|S|=N}\\sum_{x_s \\in X_S} NormSim_2(X_{T}, x_s) = \\arg \\max_{|S|=N}\\sum_{x_s \\in X_S}  f(x_s)^\\top \\cdot \\Lambda \\cdot f(x_s) \\qquad (R2)\n$$\nTake $\\Lambda=USU^\\top$ as the eigen decoposition of $\\Lambda$, $S = \\text{diag}(s_1,\\ldots,s_r)$ where $s_1>\\ldots > s_r$ is the matrix of eigenvalues, and $U=[u_1,\\ldots,u_r] \\in R^{d\\times r}$ are the corresponding eigenvectors, i.e., the principal component directions. \nNote that the column vectors of $U$ and $f(x_s)$ are all unit vectors, so we get that Eqn. R2 means $\\text{NormSim}_2$ select the data that best match with the principal components of the target variance.\n\nBesides, when $p=\\infty$, from Eqn. R1 and the definition of infinity norm, we know that $NormSim_{\\infty}(X_{T}, x_s) = \\max_{x_t \\in X_T} f(x_t)^\\top f(x_s)$, thus it measures the max similarity between the data $x_s $ with any target data $x_t \\in X_T$. Therefore, a sample will be selected if it has high similarity to any target data.\n\nWe will add these discussions in the revised papers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "38ltMnXEXQ", "reviewer": "Reviewer_NSn3", "summary": "This paper introduces a novel metric, negCLIPLoss, for selecting high-quality data. Additionally, the paper proposes a norm-based metric, Normsim, which offers an improved measure of data quality and is compatible with existing methods. Both negCLIPLoss and NormSim demonstrate significant performance improvements, outperforming state-of-the-art methods, while maintain low preprocessing time. Theoretical interpretations are provided for NormSim within the framework of a linear model.", "questions": "1. The algorithm 1 requires the knowledge of the batch size B and the parameter $\\tau$ from the teacher model. If the teacher model is private and both B and $\\tau$ are not accessible (for example, only api is provided), are the proposed methods still workable? How critical is the batch size B to model performance? Can the parameter $\\tau$ be estimated?", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is intuitive, well-motivated and well-written.\n2. The proposed methods are simple and effective.\n3. The experiments are sufficient.", "weaknesses": "1. In Line 86, the authors assert that NormSim does not explicitly consider diversity but provide no further explanation. Since diversity is often linked to the generalization performance of models, it is unclear how the proposed methods implicitly connect with diversity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of our paper and your constructive feedback. We have responded to your concerns and will revise our paper based on the discussions. We would also appreciate it if you could let us know if our response addresses your concerns.\n\n\n> **Q1**: In Line 86, the authors assert that NormSim does not explicitly consider diversity but provide no further explanation. Since diversity is often linked to the generalization performance of models, it is unclear how the proposed methods implicitly connect with diversity.\n\n\n**A1**: We reply to this concern as the following points:\n1) Many top baselines, such as DFN and T-MARS, also don't explicitly consider diversity, yet they still provide good performance. Devil [1] even shows that valuable data is worth sampling multiple times, which they call 'quality duplication'. Therefore, one important reason why NormSim works well without explicitly considering diversity may be that when the computing budget is limited, as in the DataComp benchmark, the model first needs to learn the most useful and representative data, which should be similar to some target data.\n2) Moreover, we chose validation data from 24 downstream tasks ranging from ImageNet to EuroSet, which may have covered a sufficiently diverse range of target examples for NormSim to calculate similarity. The diversity of the target data will consequently result in the diversity of the selected subset.\n3) An additional reason may be that our proposed negCLIPLoss already implicitly selects more diverse data, as shown in Figure 1 of the main paper. If some training data are diverse, they will match less with other data and thus have a lower normalization term $R$. This results in a larger negCLIPLoss and a higher probability of being sampled.\n\nThanks for your concern and we would like to add these discussions into the NormSim section in the revised paper.\n\n[1] Yu, Haichao, et al. \"The devil is in the details: A deep dive into the rabbit hole of data filtering.\" arXiv preprint arXiv:2309.15954 (2023).\n\n> **Q2**: Algorithm 1 requires the knowledge of the batch size B and the parameter $\\tau$ from the teacher model. If the teacher model is private and both B and $\\tau$ are not accessible (for example, only api is provided), are the proposed methods still workable? How critical is the batch size B to model performance? Can the parameter $\\tau$ be estimated?\n\n**A2**: This is a good concern about the limitation of our method. First we claim that most of the CLIP model is either close-sourced (no API, like the SOTA filtering model of DFN) or fully open-sourced (providing model weights, like OAI CLIP, openclip, LAION, etc), so our method should be workable for most of the current CLIP models. Besides, when only the API is provided, the recommended values for $B$ and $\\tau$ are 32768 and 0.01, respectively. The reason is that\n\n1) In general, similar to the training stage, a larger batch size can result in better performance in negCLIPLoss filtering since it contains more contrastive data pairs in a batch. 32768 is the training batch size of the OAI CLIP model, and these data can be fitted into a single 24G GPU in the CLIP forward pass. In A1 in the ‘reply to all reviewers’ parts, we also theoretically show that using a larger batch size, negCLIPLoss will have a smaller approximation error.\n\n2) For $\\tau$, when the model is accessible, we can directly get it by the model parameters since it’s learnable (And note that their temperature is the reciprocal of our definition). However, their values of $\\tau$ are always as regular as 0.01. The reason is that there is a manually set lower bound in the CLIP training setup (for the original definition it’s upper bound 100) for the trainable $\\tau$, and after training they always reach this bound. Therefore, when the model parameter is unavailable, we recommend first trying 0.01 for $\\tau$, and then sampling a small subset and tuning $\\tau$ around it. For tuning the parameters, except for training a small-scale model, we also recommend sampling a small subset and calculating the negCLIPLoss on it with different hype-parameters settings, and then visualizing them (like Figure 6-11 in the main paper) for choosing. Details are shown in Appendix C.5.\n\nWhat’s more, to show how batch size B influences the model performance, we do some ablation study on $B$ and $\\tau$. Due to the limited time and resources, we mainly focus on the OAI CLIP-B/32 model. The results are shown in A2 in the ‘reply to all reviewers’ parts (Table R1). In R1 we can see that in general, |B| = 32768 is better than |B|=16384, and $\\tau=0.01$ performs the best for both batch sizes. These results support our claims above. We will add these ablation studies and the discussion in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Reply to all reviewers for the major concern

We sincerely appreciate all reviewers for their insightful and constructive feedback to make our paper better. We will revise our paper according to these comments. Here we will address the most common concerns of the reviewers and will put other responses in separate rebuttals.

Most of the reviewers have some concerns related to whether there is any (theoretical) guarantee that we can use the random batch from the pretraining dataset to approximate the inaccessible ground-truth batch in calculating $\mathcal{R}$ and negCLIPLoss, and how the batch size and temperature will affect our method negCLIPLoss. We answer these questions as follows

> **A1**: Concentration of Normalization Term $\mathcal{R}$

We construct a theorem using the concentration inequality to show that when the batch size is sufficiently large, the normalization term $R^{B_k}$ obtained from actual batch $B_k$ can approximate $R^{B^*}$ calculated using ground truth batch $B^*$ quite well. The details are as follows:

We assume that the pretraining dataset $\mathcal{D}$ is *i.i.d.* sampled from distribution $\mathcal{P}$. Besides, to use pretraining data batch to approximate the ground truth batch, one necessary condition is that their distribution is similar. Here for simplicity, we assume that they are also *i.i.d.*.

**Assumption R1**: We assume that the ground-truth batch of data $B^*$ used by the teacher model is *i.i.d.* to the pretraining dataset $\mathcal{D}$ which is required to be filtered.

For simplicity, we denote $s_{ij} = \bar f_{v}(x^v_i)^\top \bar f_{l}(x^l_j), i, j \in B$ to be the cross-image-text similarities in the batch $B$. Then the normalization term can be written as $\mathcal{R}^B_i = \frac{\tau}{2}\left[\log(\sum_{j \in B} \exp(s_{ij}/\tau)) + \log(\sum_{j\in B}\exp(s_{ji}/\tau))\right]$. 
Here note that $s_{ij} \in [-1,1]$. We show that $\mathcal{R}_i^B = (1+o(1))\mathcal{R}_i^{B^*}$ for all $i$ when $|B|$ is sufficiently large, which means that we can use the random batch to approximate the ground-truth batch.

**Theorem R1**: If Assumption R1 holds and the batch size satisfies $|B|=|B^*|$, then we have $\mathcal{R}_{i}^B=\Theta(\log(|B|))$ while $|\mathcal{R}_i^B - \mathcal{R}_i^{B^*}| = O(\frac{1}{\sqrt{|B|}})$ for any $i \in B \cap B^*$.

*Proof*:
Since $s_{ij} \in [-1,1]$, It's obvious that $\mathcal{R}_i^B=\Theta(\log(|B|))$.  

Let $\alpha_{ij} := e^{(s_{ij}/\tau)} - E_j[e^{(s_{ij}/\tau)}]$, then $\alpha_{ij}$ is zero-mean. 
Note that since the data is *i.i.d.*, so does $\alpha_{ij}$.
Therefore, we denote $\gamma := E_{j}[\alpha_{ij}^2]$.
Note that $|\alpha_{ij}|\leq e^{1/\tau} =: M$, from Bernstein inequality we have
$$
    \mathbb{P}(|\sum_{j \in B}\alpha_{ij}| \geq t) \leq 2\exp(-\frac{\frac{1}{2}t^2}{|B|\gamma + \frac{1}{3}Mt})
$$
A similar conclusion holds for $B^*$. These result that with probability at least $1-\eta$, we have 
$$
|\sum_{j \in B}\alpha_{ij}| \leq \max \left( 2\sqrt{|B|\gamma\ln(\frac{2}{\eta})}, \frac{4}{3}M\ln(\frac{2}{\eta}) \right) =: t(|B|,\gamma, \eta, M)
$$  
Thus we have $|\sum_{j\in B}\exp(\frac{s_{ij}}{\tau})-\sum_{j\in B^*}\exp(\frac{s_{ij}}{\tau})| \leq 2 t(|B|,\gamma, \eta)$. Furthermore, for any $x_1, x_2 > 1$, it's easy to prove that $|\log(x_1)-\log(x_2)| \leq \frac{|x_1 - x_2|}{\min(x_1, x_2)}$. Therefore, we have $|\log(\sum_{j\in B}\exp(\frac{s_{ij}}{\tau}))-\log(\sum_{j\in B^*}\exp(\frac{s_{ij}}{\tau}))| \lesssim O(\frac{1}{\sqrt{|B|}})$, and thus similar claims hold for $|\mathcal{R}_i^B - \mathcal{R}_i^{B^*}|$.

> **A2**: Ablation study on batch size and the temperature.

All the reviewers are concerned about the choice of batch size. We claim that in general, similar to the training stage, **a larger batch size always results in better performance in negCLIPLoss filtering** since it can contain more contrastive data pairs in a batch, and thus it can check the image-text matching between more different data. Therefore, we consider the largest batch size 32768 which can fit into a single 24G GPU in the CLIP forward pass, and we note that this is also the training batch size that OpenAI used for training CLIP.

To support our claim, we do some ablation studies on $B$ and $\tau$. Due to the limited time and resources, we mainly focus on the OAI CLIP-B/32 model. Results are as in Table R1:

**Table R1**: Ablation study of $B$ and $\tau$ using OpenAI CLIP-B/32 model on DataComp-medium.

|  negCLIPLoss           | Dataset Size | ImageNet (1)   | ImageNet Dist. Shift (6) | VTAB (11) | Retrieval (3) | Avg. (38) |
|---------------|---------------|-------|------------|----|----|-----|
| $\|B\|=16384, \tau=0.01$| 33M | **28.8**| 25.0 | 32.5 | 26.2 | 33.0 |
| $\|B\|=16384, \tau=0.02$| 33M | 28.6 | 24.8 | 33.3 | 25.3 | 33.1 |
| $\|B\|=16384, \tau=0.07$| 33M | 28.0 | 24.2 | 33.5 | 25.1 | 32.6 |
| $\|B\|=32768, \tau=0.005$| 33M | 28.5 | 25.0 | 33.6 | **27.0** | 33.0|
| $\|B\|=32768, \tau=0.01$| 33M | **28.8** | **25.1** | **33.7** | 26.6 | **33.6**|
| $\|B\|=32768, \tau=0.02$| 33M | 28.5 | 24.8 | 33.6 | 26.2 | 32.9|
| $\|B\|=32768, \tau=0.07$| 33M | 28.2 | 24.5 | 32.8 | 25.2 | 32.7|
|  **negCLIPLoss $\cap$ NormSim**           |  |    |  |  |  | |
| $\|B\|=16384, \tau=0.01$| 22M | **32.4** | **27.4** | 34.5 | 26.1 | 34.7|
| $\|B\|=16384, \tau=0.02$| 22M | 31.8 | 26.7 | 35.0 | 24.9 | 34.2|
| $\|B\|=16384, \tau=0.07$| 22M | 31.0 | 26.3 | 35.0 | 25.5 | 33.9|
| $\|B\|=32768, \tau=0.005$| 22M | 32.2 | 27.2 | 35.3 | **26.5** | 34.8|
| $\|B\|=32768, \tau=0.01$| 22M | **32.4** | **27.4** | **35.9** | 26.3 | **35.2**|

We can see that in general, negCLIPLoss with a larger batch size ($|B|=32768$) indeed has better or comparable downstream performance. Nevertheless, $|B|=16384, \tau=0.01$ still has good performance when being combined with NormSim ($\tau=0.01$ performs well for both batch sizes). These match our theoretical findings in A1: using a larger batch size, negCLIPLoss will have a smaller approximation error.

Meta-review: The submission proposes two data selection methods for large-scale visual-language model pretraining (e.g., CLIP). All three reviewers agree that this submission is well-written and intuitive with extensive experiments. The AC therefore recommends accepting the paper and asks the authors to include their discussions with the reviewers in the final manuscript.
Decision: Accept (spotlight)