CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning

Authors: yiping  wang
yifang  chen
wendan  yan
alex  fang
wenjing  zhou
kevin  jamieson
simon shaolei du
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 00:11:22

Abstract: Data selection has emerged as a core issue for large-scale visual-language model pretaining (e.g., CLIP), particularly with noisy web-curated datasets. Three main data selection approaches are: (1) leveraging external non-CLIP models to aid data selection, (2) training new CLIP-style embedding models that are more effective at selecting high-quality data than the original OpenAI CLIP model, and (3) designing better metrics or strategies universally applicable to any CLIP embedding without requiring specific model properties (e.g., CLIPScore is one popular metric).  While the first two approaches have been extensively studied, the third remains under-explored. In this paper, we advance the third approach by proposing two new methods. Firstly, instead of classical CLIP scores that only consider the alignment between two modalities from a single sample, we introduce $\textbf{negCLIPLoss}$,  a method inspired by CLIP training loss that adds the alignment between one sample and its contrastive pairs as an extra normalization term to CLIPScore for better quality measurement. Secondly, when downstream tasks are known, we propose a new norm-based metric, $\textbf{NormSim}$, to measure the similarity between pretraining data and target data. We test our methods on the data selection benchmark, DataComp [Gadre et al., 2023]. Compared to the best baseline using only OpenAI's CLIP-L/14, our methods achieve a 5.3\% improvement on ImageNet-1k and a 2.8\% improvement on 38 downstream evaluation tasks. Moreover, both $\textbf{negCLIPLoss}$ and $\textbf{NormSim}$ are compatible with existing techniques. By combining our methods with the current best methods DFN [Fang et al., 2023] and HYPE [Kim et al., 2024], we can boost average performance on downstream tasks by 0.9\%, achieving a new state-of-the-art on the DataComp-medium benchmark.

Main paper: 
 introduction: 
 Curating large-scale visual-language datasets from web-sourced data has become common for pretraining multi-modal models. However, the quality of these web-curated data pairs remains a critical bottleneck. Research has shown that the choice of dataset significantly impacts model performance, irrespective of the models and training techniques employed [4] [5] [6] [7] [8] [9] [10] [11] , and this motivates the development of various data selection strategies. This paper focuses on optimizing subset selection from a fixed data pool to train a CLIP model [4] that achieves superior performance on zero-shot downstream tasks.Classical methods rely solely on OpenAI's (OAI) pretrained CLIP model (i.e., a teacher model) and focus on better utilizing the embeddings. The most commonly used one is calculating CLIPScore, which measures the cosine similarity between the visual and language embeddings of the CLIP model for the same sample, to eliminate low-quality data with mismatches between text and image. Other works also leverage heuristic distribution alignment techniques to select samples relevant to downstream tasks, such as image-based filtering [1] . These approaches are generally viewed as providing only limited enhancements. However, we argue that the potential of those embeddings has been heavily under-explored. This work seeks a universal method to better employ any given embeddings, not only from OAI CLIP, but also from other CLIP-style models.On the other hand, recent leading data filtering methods, instead of focusing on improving embedding utilization stategy itself, mainly follow the other two directions, both employing external resources. They either (1) use external non-CLIP models that aid in data selection, (2) or use external highquality multi-modal data to train a better CLIP-style embedding model than the original OAI CLIP to filter out low-quality data. Specifically, in the first line of works, HYPE [3] leverages embeddings from hyperbolic models instead of the classical Euclidean-based CLIP to measure how each data point has semantically overlaps with other data points and filters out data with low specificity. T-MARS [12] removes images where the text is the only feature correlated with the caption using FAST [13] , an off-the-shelf OCR text detection model. Devil [14] applies fasttext [15] to remove non-English texts and use BLIP-2 [16] model for digit recognition to keep useful images with digits. The second direction, represented by Data Filtering Network (DFN) [2] , involves training a new CLIP-style teacher model that uses high-quality datasets like HQITP-350M. Although the embeddings extracted from this model perform worse than the OAI CLIP in downstream tasks, it is particularly good at filtering out low-quality data. Notably, some of these methods can be combined and indeed, merging the selected data from DFN and HYPE achieves current state-of-art as shown in HYPE [3] .Previous works mainly focus on improving the CLIP embedding quality or utilizing an external model to do filtering but employ the CLIP embedding in a suboptimal way by only using classical methods like CLIPScore. In contrast, in this work, we focus on improving the filtering methods themselves for any given CLIP embedding. We show that there are universal and more effective strategies for utilizing any CLIP teacher model, regardless of its architecture (e.g., B/32 or L/14) or the dataset it was trained on (e.g., OpenAI-WIT-400M or DFN's high-quality dataset). These strategies should always be orthogonal to the use of any newly trained CLIP-style models like DFN and might also be compatible with methods using external models like FAST and BLIP-2.Our Contributions. We propose an alternative to CLIPScores that we call negCLIPLoss that more accurately characterizes data quality. We also introduce a new distribution metric we call the p-Norm Similarity Score (NormSim) when knowledge about downstream tasks is available. Two major observations directly inform our proposals: • Firstly, we observe that classical methods measure the quality of a multi-modal sample by computing the cosine similarity between its visual and language embeddings, believing that lower similarity indicates that the text does not match its image part well. However, we find that some less informative samples may have a systematic bias, which leads to higher CLIPScores. For example, the language part containing the word "image" can result in higher similarity with any visual part, even when the text does not accurately describe its image content. Our proposed method negCLIPLoss, inspired by the standard CLIPLoss, normalizes the original CLIPScore by the similarity between a sample and its contrastive pairs. For example, the high score caused by the word "image" is typically consistent across its contrastive pairs, so our adjustment reduces this bias.As we have highlighted, such replacement can be universally applied across different embedding models. See Fig. 2 for illustrations. • Secondly, if one has access to examples drawn from the same distribution as the target task, it is natural to assume that this extra knowledge could be leveraged to inform the data filtering process. We propose the NormSim metric to measure the vision similarity between a training sample x and the target task dataset X v target ∈ R n×D defined as ∥f v (X v target )f v (x v )∥ p , where f v : R D → R d is the vision encoder of teacher model so that f v (X v target ) ∈ R n×d , f v (x v ) ∈ R d , and f v (X v target )f v (x v ) ∈ R n , and ∥ • ∥ p is the p norm; effective choices are p = 2 or ∞. Notably, unlike previous ImagetNet-based filtering [1] , which tries to keep the training set as diverse as downstream tasks by clustering the training set and finding the nearest neighbor group for every target sample, our method does not explicitly consider the diversity but select examples as long as it is close to any target sample (i.e. select high NormSim score). Notably, negCLIPLoss and NormSim enjoy complementary effect in data selection. See Fig. 3 . To illustrate the effectiveness of our methods, we use a widely used benchmark DataComp [1] as our primary method of evaluating the datasets created by our data filtering methods. We show that, by simply replacing the CLIPScores with negCLIPLoss and utilizing NormSim we are able to exceed the best OAI-CLIP(L/14)-based baseline by 5.3% on ImageNet-1k and 2.8% on average across 38 downstream tasks, which is similar or even better than the performance achieved by many external-resources-based methods. Notably, even if the target downstream tasks are not available, using NormSim on a proxy downstream task constructed from the training set, called NormSim 2 -D, combined with negCLIPLoss, can also gain a 1.9% improvement on 38 downstream evaluation.Moreover, the improvements achieved by our methods are not limited to OAI CLIP-based methods but can also be obtained by combining our methods with advanced models that require external resources. By merging the subset selected by negCLIPLoss and NormSim with the subset selected by current state-of-the-art method "HYPE ∪ DFN", we can further improve it by 0.9% on both ImageNet-1k and on average 38 downstream tasks. Besides, we can also achieve a 0.8% improvement on average 38 tasks over "HYPE ∪ DFN" using only the data selected by DFN and our strategies. More importantly, we demonstrate that negCLIPLoss, as a replacement for CLIPScore, can be applied to any other embedding models like OAI-L/14, OAI-B/32, and DFN-B/32, universally boosting performance from 0.4% to 3.0% on an average of 38 tasks. This result is not only technically insightful for understanding the information available in embeddings but also practically significant. Compared to existing methods, our approach saves a significant amount of computational time on both reprocessing and new embedding retraining as shown in Table 5 . 
problem setup: 
 Data Filtering on Multimodal Dataset. We are given a training dataset D train = {x v , x l }, where (x v , x l ) ∈ R D is the image-text (vision-language) training pair. For convenience, we will let superscript vl denote either modality so that, for example, x vl ∈ x v , x l . Our goal is to identify a subset S ⊂ D train that maximizes the zero-shot accuracy of the CLIP model on some downstream tasks when S is used to train the CLIP model.CLIP score and embedding. Recent efforts, such as LAION [5] and DataComp [1] , use Ope-nAI's CLIP ViT-L/14 model [4] as a teacher model to obtain quality score. Here we denote this vanilla CLIP model as fvl . For any pair x vl , the model outputs a normalized unit-vector fvl (x vl ). If X vl := {x vl 1 , . . . , x vl m } denotes a dataset containing m samples, then we define fvl (X vl ) = [ fvl (x vl 1 ), . . . , fvl (x vl m )] ⊤ ∈ R m×d as the embedding matrix. The popular filtering metric "CLIPScore" is defined as⟨ fv (x v ), fl (x l )⟩ ∈ [-1, 1].Dataset and model. Here we follow the pipeline of Datacomp [1] to standardize the training and evaluation process. This is a testbed for dataset experiments aiming to open-source and further improve the vanilla CLIP model and is widely adopted in previous data selection papers [17, 18, 12, 2, 19, 7] . We will give more details in Sec. 4.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss this briefly in the last section. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
negcliploss: a better metric than clipscore: 
 In this section, we introduce a better and statistically interpretable quality metric called negCLIPLoss, which directly replaces the common metric CLIPScore. Fig. 1 illustrates how negCLIPLoss works. This new metric only requires negligible extra computational costs and no additional external data collection costs. As the name suggested, this metric is inspired by the standard CLIP loss used in the actual training process of the teacher CLIP model, which is defined asEQUATIONHere B * is the random batch where i-th sample belongs during a particular training step, and τ is the learnable temperate parameter. Notably, the teacher loss differs from CLIPScore primarily by a denotes that the score represents the top X% high values within the entire dataset (i.e., the (100-X)% percentile among all the values). For example, "R : Top 100%" means this data has almost the smallest R among the whole dataset, which represents that it contains highly specific elements in both images and texts. normalization term R * as follows:-τ • ℓ B * (x vl i ) = fv(x v i ) ⊤ fl (x l i ) CLIPScore - τ 2 log j∈B * exp( fv(x v i ) ⊤ fl (x l j ) τ ) + log j∈B * exp( fv(x v j ) ⊤ fl (x l i ) τ ) normalization term R *In practice, since the training dataset of teacher CLIP models, like OAI-WIT400M [4] , and the actual batch divisions B * is inaccessible, we randomly select K batches from the student model's training data and use the averaged results from {B k } K i=1 to estimate the normalization term R * on B * :EQUATIONHere {B k } K i=1 are some batches randomly selected from the student model's training data and x i ∈ B k , ∀k. We choose K = 10 in our experiments, but any sample size larger than 5 is sufficiently stable for estimating the original CLIPLoss (Details in Appendix D.1). Besides, in Sec. 4.3.3 we also show that the computational cost introduced by R remains negligible compared to other baselines. The temperature τ and batch size |B * | can be directly obtained from the parameters of the pretrained teacher model. More details of negCLIPLoss are in Appendix, including the concentration analysis of R (Appendix A.1), pseudocode (Algorithm 1), and the ablation study of τ and |B| (Appendix C.2). Motivation behind negCLIPLoss. Other existing works also use loss-guided data selection, such as LESS [20] in NLP, CoDis [21] in CV, and RHO [22] in general data scheduling scenarios. However, it is still unclear whether selecting based on teacher loss is suitable for multi-modal contrastive learning. Here we give an affirmative answer as shown in Fig. 2 , where we can see negCLIPLoss performs better than or on par with CLIPScore consistently.To illustrate how teacher loss helps our selection, we demonstrate that the normalization term provided by negCLIPLoss is crucial for correcting the overestimation or underestimation inherent in CLIPScore. A high normalization term implies that either the image embedding, text embedding, or both can easily match multiple contrastive pairs beyond their ) corresponding counterparts. For example, in the bottom right of Fig. 1 , the text containing "Image" or "Photo" can be easily matched with any visual content. Similarly, the image of "verloopring" only contains very simple features and can be matched with many words like "white", "empty" or "circle", etc. Consequently, despite a high absolute CLIPScore, the relative negCLIPLoss within its batch can be lower. In contrast, the bottom left features highly specific elements in both text and images, such as "Islands Harbor," "American football", and "sheep at green". These elements are specific and less likely to match with contrastive pairs, resulting in a higher relative negCLIPLoss.Our proposed negCLIPLoss is a universal approach to improve filtering performance by estimating quality better, and it does not rely on any downstream task. Now, if we can access some knowledge of the downstream tasks, we could further improve the performance by using a vision-only p-norm similarity to target data metric to measure the relationship between each training sample and the downstream target data. We will discuss the reason to use vision-only embedding later in this section.Specifically, we assume access to the target set of downstream tasks and denote them as X target = {x target,(1) , . . . , x target,(m) }, where each x target,(i) ∈ R d is i.i.d.-sampled from the target downstream distribution P target3 , but without overlapping with the test set. Then, for each training sample x vl and the corresponding target set X target , the NormSim is defined as:EQUATIONWe select the subset S by choosing the samples with top-N highest NormSim scores. The choice of the norm type p can be based on the data distribution and training process. In this paper, we consider two instantiations of p:When p = 2, our data selection method can be regarded as the following equation. It's equivalent to selecting a subset that aligns with the principal components of the target set variance (Appendix C.6.1).EQUATIONWhen p = ∞, the distance metric can be regarded as an even more optimistic measure, such that a training sample will be selected if it has high similarity to any target sample. Note that this is different from nearest-neighbor-based method used in image-based filtering [1] , where they are trying to find the nearest training sample of every target sample. In this case, it can be regarded as:EQUATIONIn Appendix D.3, we also show that our NormSim ∞ can outperform the nearest neighbor selection on the downstream target tasks. Here, we show an example selected via the NormSim 2 (ImageNet-1k) in Fig. 3 , showing that this vision-target-aware method is complementary to the quality-based one.Choice of Target Data. In the experiment parts, we try two kinds of target data: training data from ImageNet-1k (1.3M) or training data from all 24 accessible downstream tasks (2.1M) 4 . We denote them as NormSim p (IN-1k) and NormSim p (Target), respectively.We use only the visual information x v instead of multimodal information x vl for measuring similarity. This is because common crawled text often has brief captions, making the OAI CLIP language embedding weaker than its visual embedding model [1, [23] [24] [25] . Consequently, the language part cannot characterize the pre-training and downstream task distribution as well as the visual part. This phenomenon is also observed in Gadre et al. [1] , where image-based filtering (select data whose image embeddings are similar to that from ImageNet-1k) outperforms text-based filtering (select data whose captions contain words from ImageNet-21k).More ablation studies are provided in Appendix D.4. Theoretical justification. Unlike many existing methods that force diversity by selecting training samples around each x target , our strategy maximizes similarity without directly considering data diversity. For the p = 2 case, we demonstrate that maximizing NormSim 2 is optimal under a linear model fv , as shown in Appendix A.2. Our theorem also provides error guarantees for noisy embeddings and explains when vision-only embeddings outperform combined vision and language embeddings. Recent work by Joshi et al. [26] provides a similar analysis but focuses on high-quality data and cross-variance between images and texts. This approach is less effective than image-only methods for filtering noisy datasets, as discussed above.Using proxy when downstream X target is inaccessible. Surprisingly, we show that the 2-norm can also be used when only the pre-training set is available. In this case, we construct a proxy "target" set from the pre-training set itself. Specifically, let S i be the selected subset at step i, then we treat the current S i as the proxy "target" set. To construct the next smaller set, we select the next data batch S i+1 satisfying arg max Si+1⊂Si x∈S NormSim 2 (S i , x), until reaching an N size subset. We call this approach NormSim 2 -D (Dynamic) and will specify the algorithm details in Appendix C.3.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experimental results: 
 In this section, we evaluate the performance of negCLIPLoss and NormSim, aiming to address the following questions: Q1: Given a fixed CLIP teacher model, can our methods more effectively utilize CLIP embeddings for data filtering? Q2: Are our methods applicable to diverse CLIP teacher models with varying architectures or different pretrained datasets? Q3: How does our method compare to other leading approaches that utilize external models or multimodal datasets? Additionally, could our method be compatible with these methods and enhance their effectiveness?We adhere to the standardized training and evaluation protocols of the DataComp benchmark [1] .Training configuration. We employ the medium-scale training configuration of DataComp (DataComp-medium). It provides a substantial dataset comprising 128 million low-quality, webcurated image-text pairs to be filtered. Once the data subset is obtained by some data filtering strategy, it will be used to train a fixed CLIP-B/32 model in a fixed training budget that allows the model to pass 128 million data points an epoch. Therefore, smaller subsets will be repeated more frequently, ensuring a fair comparison. We note that the size of the DataComp dataset becomes smaller over time since some URLs of images become invalid 5 , and we only successfully downloaded about 110M data. Therefore, the results of baselines on the leaderboard do not apply to our datasets, and we reproduce all the top baselines on the leaderboard with their public UIDs of the selected data.Evaluation. We measured the model performance on 38 downstream datasets including image classification and retrieval tasks followed by DataComp. The image classification tasks contain ImageNet-1k [27] , ImageNet distribution shifts [28] [29] [30] [31] , 11 datasets from the Visual Task Adaptation Benchmark (VTAB) [32] and 3 datasets from WILDS [33, 34] . Retrieval datasets contain Flickr30k [35] , MSCOCO [36] and WinoGAViL [37] .Teacher model architecture. Our experiments utilize two architectures for OpenAI's CLIP teacher models: ViT-L/14 and ViT-B/32. Additionally, we use the public version of DFN (DFN-P) proposed by Fang et al. [2] as a teacher model, and its architecture is also ViT-B/32.We restate the three current research directions mentioned before based on how much external resources are employed: (D1) using OAI CLIP alone while optimizing embedding employment strategies, (D2) training and using a more advanced CLIP embedding model based on external data, and (D3) utilizing non-CLIP external models to aid data selection. It is important to note that D2 and D3 may also incorporate strategies from D1. For example, CLIPScore (D1) has been used in almost all the top methods. Therefore, we categorize baselines by the largest possible category they encompass. According to the above categorization, we summarize the baselines we used in our experiments as follows. Please refer to Fig. 4 and Appendix C.4 for more details.D1: OAI CLIP embedding only. The learner can only access the pretraining dataset (like DataCompmedium), the original OAI CLIP teacher model that is used to extract embeddings, and some target data of the downstream tasks which is much smaller than the pretraining dataset (like ImageNet-1k).In this category, we don't use any existing external non-CLIP models or any newly trained CLIP model based on external multi-modal dataset. In detail, This category includes (1) CLIPScore [38] , which only uses CLIPScore for filtering as we mentioned before. ( 2 (3) HYPE ∪ DFN proposed by [3] samples subset separately for each method and then merge them. This is the state-of-the-art method on the DataComp benchmark for medium size. (4) Other methods including T-MARS [12] , Devils [14] , MLM [42] , which leverage external models such as text detection model FAST [13] , BLIP-2 [16] and LLaVA-1.5 [43, 44] to heuristically select data. See details in Appendix C.4.Cross-setting comparison. We make these separations for fair comparison. Intuitively, performance should be ranked as D2, D3 > D1. However, our results show that cross-setting comparisons are possible and our D1 methods can perform similar or even better than most of D3 methods. 3 we will see that this result outperform all the D3 baselines except DFN ∪ HYPE. On the other hand, when using ImageNet-1k as the target data, the choice of norm has very little influence.Table 3 : Results of all D1&D2&D3 top methods on DataComp-medium. The results of MLM [42] are from their paper, while all other baselines are reproduced on our downloaded dataset using their official UIDs. "Ours (20%)" refers to use "negCLIPLoss (30%) ∩ NormSim ∞ (Target)" to get 20% of original data, while "Ours (10%)" denotes applying "negCLIPLoss (20%) ∩ NormSim ∞ (Target)" to get 10%. And we use "*" to indicate the case where we choose the intersection of the data selected by using OAI CLIP-B/32 and OAI CLIP-L/14 separately, which results in about 15M data for "Ours (20%)*" and 7.4M data for "Ours (10%)*". To evaluate whether our method applies to other CLIP teacher models, we replaced OAI CLIP-L/14 with OAI CLIP-B/32 and DFN-P as embedding models. We compare the best baseline "CLIPScore" with our "negCLIPLoss" and best strategy "negCLIPLoss ∩ NormSim ∞ (Target)" as shown in Table 1 and Appendix D.2. Note that the original DFN paper selects a subset comprising 19.2M data points, which accounts for approximately 17.5% of our dataset and 15% of their dataset, we incorporate these sampling ratios into our comparison.negCLIPLoss can be applied to different CLIP embedding models. Our proposed negCLIPLoss, as a replacement of CLIPScore, not only leads to better performance compared to all the other baselines using OAI CLIP-L/14 as shown in Table 2 , but also achieves universal improvement on the other two CLIP embedding models, OAI CLIP-B/32 and DFN-P as shown in Table 1 . Our methods can consistently outperform all downstream tasks for different filtering ratios and models, like a 0.5%-5.4% increase on ImageNet-1k.Embedding required by NormSim should have good downstream performance. When combining negCLIPLoss with NormSim ∞ , OAI CLIP-B/32 and DFN-P exhibit completely different behaviors. The former obtains results even better than those in Table 2 , which uses OAI CLIP-L/14 as the teacher model, while DFN-P achieves results even worse than using negCLIPLoss alone 6 . The reason is that, unlike OAI CLIP-B/32, DFN-P is specially designed for data filtering at the expense of downstream task performance, as claimed by its authors. For example, the ImageNet-1k accuracy for DFN-P, OAI CLIP-B/32, and OAI CLIP-L/14 are 45%, 63%, and 75%, respectively. This indicates that the embeddings obtained from DFN on target data might be highly unreliable, leading to inaccurate similarity calculations between training and target data. To support this, if we use DFN-P to evaluate negCLIPLoss but utilize OAI CLIP-B/32 for calculating NormSim, as shown in "negCLIPLoss (17.5%) ∩ NormSim B/32 ∞ (Target)", we can further improve the results compared to using negCLIPLoss alone. Its average performance on 38 tasks is even higher than utilizing the best DFN (trained on HQITP-350M) with CLIPScore, as shown in Table 3 .In this part, we compare all the D2 & D3 baselines mentioned in Sec. 4.2 together with our best strategy in Table 3 . Here we reproduce all the baselines if their official UIDs are available. For "A ∪ B" mentioned in Table 3 , we follow the way of "HYPE ∪ DFN" in Kim et al. [3] to merge the data, which generates the sampling subset separately for each method and then merge them. This will result in oversampling the shared data, which is intuitively more important. 7 We also show the best result we obtain by combining our method with DFN [2] and HYPE [3] on the full DataComp-medium dataset in Table 4 , where the baselines are from DataComp benchmark. Our methods can outperform most of the D3 methods. In Table 3 , we show that without using any external models or data, our best combination, i.e., using OAI CLIP-B/32 for "negCLIPLoss (30%) ∩ NormSim ∞ (Target)" (Ours (20%)), still outperforms all methods except DFN and "DFN ∪ HYPE". This answers the first part of Q3 and further indicates that some external models may be redundant since CLIP embeddings already contain necessary information.We can further improve the SOTA method.In Table 3 , we show that our model can further boost the performance of the current SOTA method "HYPE ∪ DFN" by 0.9% on both ImageNet-1k and on average 38 downstream tasks, and close results can be achieved even without combining HYPE which utilizes the external embedding model MERU [45] . And we update the SOTA performance of the DataComp-medium (full dataset) benchmark as shown in Table 4 . Here we use the data selected by both OAI CLIP-B/32 and L/14, which we found is more robust than using one of them alone.Our better results answer the second part of Q3, that is, our methods can be compatible with other D2&D3 methods. 
conclusion and limitation: 
 In this paper, we introduce two metrics, negCLIPLoss and NormSim, to enhance data selection in multimodal contrastive learning without relying on external resources. negCLIPLoss provides a more accurate quality metric compared to the commonly used CLIPScore, while NormSim measures the similarity between pretraining data and target data for known downstream tasks. Experiments show that our methods achieve results that are competitive with or even better to approaches using external models or datasets. Additionally, negCLIPLoss and NormSim are compatible with existing top techniques, allowing us to achieve a new state-of-the-art by combining them.A notable limitation of our study is the exclusion of larger pretraining datasets, such as the large and xlarge scales of DataComp. However, DataComp-medium is the most commonly used benchmark for data selection in CLIP pretraining, and our method has demonstrated both effectiveness (Table 2 -3) and efficiency (Table 5 ) on it. Future directions include exploring better ways to merge data selected by different methods and incorporating our methods into data scheduling scenarios. 
acknowledgement: 
 A Theoretical Interpretation 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We discuss the computing cost estimation and comparison in Appendix C.1. We didn't explicitly calculate the memories since it is quite standard under the DataComp benchmark.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This research focuses on the methodology part of data selection. All experiments are performed under the existing standard dataset. So as long as those datasets itself maybe harmless, our research will not make any negative impact. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper will only provide UID's of selected data from existing datasets (DataComp-medium [1] ). This paper will not release any model or new dataset. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite the DataComp [1] which introduces the URL for the dataset and the code/models used to implement the benchmark. Guidelines:• The answer NA means that the paper does not use existing assets. 
 
 Appendix: In this section, we construct a theorem using the concentration inequality to show that when the batch size is sufficiently large, the normalization term R B k obtained from actual batch B k can approximate R B * calculated using ground truth batch B * quite well. The details are as follows:We assume that the pretraining dataset D is ndependent and identically distributed (i.i.d.) sampled from some distribution P. Besides, to use pretraining data batch to approximate the ground truth batch, one necessary condition is that their distribution is similar. Here for simplicity, we assume that they are also i.i.d..Assumption A.1. We assume that the ground-truth batch of data B * used by the teacher model is i.i.d. to the pretraining dataset D which is required to be filtered.For simplicity, we denote s ij = fv (x v i ) ⊤ fl (x l j ), i, j ∈ B to be the cross-image-text similarities in the batch B. Then the normalization term can be written asR B i = τ 2   log( j∈B exp(s ij /τ )) + log( j∈B exp(s ji /τ ))   Here s ij ∈ [-1, 1]. We will show that R B i = (1 + o(1)) • R B * ifor all i when |B| is sufficiently large, which means that we can use the random batch to approximate the ground-truth batch. R B i = Θ(log(|B|)) while |R B i -R B * i | = O( 1 √ |B| ) for any i ∈ B ∩ B * . Proof. Since s ij ∈ [-1, 1], It's obvious that R B i = Θ(log(|B|)). Let α ij := exp(s ij /τ ) - E j [exp(s ij /τ )], then α ij is zero-mean. Note that since the data is i.i.d., so does α ij , and we denote γ := E j [α 2 ij ]. Note that |α ij | ≤ e 1/τ =: M , from Bernstein inequality we haveP(| j∈B α ij | ≥ t) ≤ 2 exp(- 1 2 t 2 |B|γ + 1 3 M t )A similar conclusion holds for B * . These result that with probability at least 1 -η, we have| j∈B α ij | ≤ max{2 |B|γ ln( 2 η ), 4 3 M ln( 2 η )} =: t(|B|, γ, η, M ) Thus we have | j∈B exp( sij τ )-j∈B * exp( sij τ )| ≤ 2t(|B|, γ, η). Furthermore, for any x 1 , x 2 > 1, it's easy to prove that | log(x 1 )-log(x 2 )| ≤ |x1-x2| min(x1,x2) . Therefore, we have | log( j∈B exp( sij τ ))- log( j∈B * exp( sij τ ))| ≲ O( 1 √ |B| ). Similar claims hold for |R B i -R B * i |.A.2 Optimality of NormSim 2 Under Linear AssumptionIn this section, we give a theoretical justification on the NormSim metric when p = 2 under the linear model assumptions when low quality image and mismatched text has already been removed. In other words, we mainly focus on the following strategy.EQUATIONTraining data. For any x v , x l ∈ R d observable image and text training pairs, we define z v , z l to be the corresponding latent vectors which contain all semantically pertinent information about our tasks of interest. Similar to previous theoretical work [46] , we assume each i.i.d pair z vl follows zero-mean sub-gaussian distribution whose cross-covariance satisfiesCov(z v , z l ) = Σ train = diag(σ 1 , σ 2 , . . .), ∥z vl ∥ = 1and each x vl is generated based on a linear model such thatx vl = G * vl z vl + ξ vl . Here G *vl ∈ O d×r is the othonormal ground truth representation mapping from the latent vector space to the input space, and ξ vl ∼ N (0, I d ) are i.i.d. random noise.Also we denote the cross covariance of any finite dataset S ′ (e.g. the given train set D train ) as Σ S ′ .Test data. For any zero-shot downstream task, we assume it shares almost same data generation process as the training set, except its the cross-covariance Σ target does not necessarily equal Σ train , which necessitate the choice of Σtarget_proxy .CLIP embedding model as teacher. Under the linear model assumption, we have a teacher model fvl = Ḡvl , whose generated clip embedding can partially recover the ground truth hidden vector z vl with error.Formally, we say teacher has ϵ n v error if for all possible n budget subsets S ⊂ D train ,1 |S| x vl ∈S Ḡ⊤ v x v (x v ) ⊤ Ḡv - x vl ∈S z v (z v ) ⊤ * ≤ ϵ n vwhere the same notation applies for the language modal. By the orthonormal assumption on the ground truth matrix G * vl , we see that Ḡ⊤ v is aiming to inverting the map. In addition, we say the teacher has ϵ n v * l cross modal error1 |S| ∥ x vl ∈S Ḡ⊤ v x v (x l ) ⊤ Ḡl - x vl ∈S z v (z l ) ⊤ ∥ * ≤ ϵ n v * lWhen all ϵ n v , ϵ n l , ϵ n v * l → 0 as n → ∞, then we say the teacher is strong for both modalities. But it might also be possible that only one modal, for example, visual is strong. That isϵ n v → 0, ϵ n l , ϵ n v * l ≫ ϵ n v .Model and training. According to Lemma 4.1 in [46] , using the CLIP loss to optimize the linear model has approximately the same training dynamics as using the regularized linear loss. Therefore, here we assume that we are learning G v , G l by maximizing the clip score gap between the contrastive pairs, plus a regularizer,min Gv,G l L ρ S (G v , G l ) := min Gv,G l i∈S j∈S (s ij -s ii ) |S|(|S| -1) + ρ 2 |S| |S| -1 ∥G v G ⊤ l ∥ 2 F where s ij := ⟨G ⊤ v x v i , G ⊤ l x l j ⟩and ρ > 0 is some regularizer-related constant. Note that this objective maximizes self-similarity and minimizes similarity between disparate pairs. Note that this "loss" can be negative, avoiding the trivial null solution of all zeros. We denote this training process from any given S as G vl = A ρ (S).Goal and metric. Under the same principle as our training loss function, we measure the performance of any learnt G v , G l on some downstream task with distribution D target as test lossL target (G v , G l ) := E x vl ∼Dtarget x vl 2 ∼Dtarget (⟨G ⊤ v x v , G ⊤ l x l 2 ⟩ -⟨G ⊤ v x v , G ⊤ l x l ⟩)This is inspired by the following classification accuracy. Assume that the test data including C class, and the class distribution is C. For every class c, the training data x = (x v , x l ) satisfies distribution P c . We further assume the corresponding classification templates are {x c } C c=1 . Thus we define classification accuracy asAC(G v , G l ) = E c,c ′ ∼C×C [E xi∼Pc 1[s ic > s ic ′ ]]Therefore our goal is to minimize its gap between the best hind-side subset, for any ρ, without budget constraints,∆ ρ (S) = L target ( Ĝvl ) -min S ′ ∈Dtrain L target (A ρ (S ′ )), Ĝvl = A ρ (S)We now provide theoretical guarantees and postpone our proof into Appendix A.2.3. Firstly, we are going to prove the intuition behind NormSim 2 score. Lemma A.1 (Intuition behind NormSim 2 ). With high probability at least 1 -1 |S|d , suppose the hind-side best subset has at least n number of samples, then we have∆ ρ (S) = 1 ρ max S ′ ∈Dtrain (Tr (Σ target (Σ S ′ -Σ S ))) NormSim2related term + O d log(d|S|) n + d log(d|S|) |S| noiseProof sketch. ❶ Under the assumption that both z vl , ξ vl is zero-mean, maximizing the clip score gap is equivalent to maximizing the clip score of the same sample.L target ( Ĝv , Ĝl ) := -E x vl ∼Dtarget ⟨ Ĝ⊤ v x v , Ĝ⊤ l x l ⟩❷ By minimizing the regularized training loss L ρ S (G v , G l ) using Eckart-Young-Mirsky Theorem, we get a closed form solution of Ĝ asĜ⊤ l ≈ 1 ρ G * v Σ S • (G * l ) ⊤ + noise depend on S❸ Combining the result in ❷ and ❶, we haveL target ( Ĝvl ) ≈ - 1 ρ Tr (Σ target Σ S ) -noise depend on SThe same analysis can be applied on min S ′ ∈Dtrain L target (A(S ′ )) as well. Rearranging these two equations gives us the final result.This lemma shows the the ∆(S) is depend on the NormSim 2 -related term and the noise term which comes from ξ. When n and |S| is large enough, then the NormSim 2 -related term will become dominant. This aligns with our practice experience that the final performance is less sensitive to the small variation in the number of select data as long as that is sufficient. Moreover, in some special cases where test distribution has identity cross-variance, then sampling by choosing CLIP score might be enough.Now we are ready to give a proof on the choice of Σtarget and visual-only information. Specifically, the strategy error mainly comes from (1). The unknown test distribution shift from training. (2). The unobservable ground truth Σ S . To tackle error (1), we assume some prior knowledge on test by using the proxy test variance Σtarget . To tackle the error (2), there are two possible solutions as shown below.Based on the theoretical interpretation, we should choose different strategy based on the property of the teacher embedding model.S vision+language = arg max S Tr   Σtarget ( x vl ∈S Ḡ⊤ v x v (x l ) ⊤ Ḡl )   S vision only = arg max S Tr   Σtarget ( x vl ∈S Ḡ⊤ v x v (x v ) ⊤ Ḡv )   Theorem A.2 (Main).Under the assumption of Lemma A.1,∆ ρ (S) ≤ noise + 1 ρ ∥ Σtarget -Σ target ∥∥Σ S -Σ best ∥ * + 1 ρ ϵ S v * l (vision+language) ϵ S v + 1 -1 |S| i∈[S] ⟨z v , z l ⟩) (vision only)Firstly, it is evident that the greater the difference between Σtarget and Σ target , the less improvement we can expect. Moreover, in scenarios where ϵ l is large (indicating lower accuracy in the language part) while ϵ v is small (indicating higher accuracy in the vision part), it may be advisable to opt for visiononly embeddings. However, the learner should also consider the term 1 -1 |S| i∈S ⟨z v , z l ⟩, which represents the alignment between the ground truth visual and language latent vectors, essentially reflecting the intrinsic quality of the data. If this term is already significant, relying solely on vision information as a proxy for language information could lead to suboptimal results.Ĝv , Ĝl = arg minEQUATIONThen we have ĜvEQUATIONwhere noise terms P i are defined in ( 12) , ( 13), ( 14) and (15) .Proof. Note thats ij = (x l j ) ⊤ G l G ⊤ v x v i = Tr(G ⊤ v x v i (x l j ) ⊤ G l ), like the proof of Corollary B.1. in [46] , we haveL(G v , G l ) = i∈S j∈S (s ij -s ii ) |S|(|S| -1) + ρ 2 |S| |S| -1 ∥G v G ⊤ l ∥ 2 F = i∈S j∈S s ij -|S| i∈S s ii |S|(|S| -1) + ρ 2 |S| |S| -1 ∥G v G ⊤ l ∥ 2 F = -Tr G ⊤ v 1 |S| -1 i∈S x v i (x l i ) ⊤ - |S| |S| -1 xv ( xl ) ⊤ G l + ρ 2 |S| |S| -1 ∥G v G ⊤ l ∥ 2 F =: -Tr(G ⊤ v ΓG l ) + ρ 2 |S| |S| -1 ∥G v G ⊤ l ∥ 2 Fwhere xvl := ( i∈S x vl i )/|S|. Then by the Eckart-Young-Mirsky Theorem (For example, Theorem 2.4.8 in Golub et al. [47] ), we know that arg minGv∈R d×r ,G l ∈R d×r L(G v , G l ) = arg max Gv∈R d×r ,G l ∈R d×r Tr(G ⊤ v ΓG l ) - ρ 2 |S| |S| -1 ∥G v G ⊤ l ∥ 2 F = {(G v , G l ) ∈ R d×r × R d×r : G v G ⊤ l = 1 ρ |S| -1 |S| SVD r (Γ)} (Eckart-Young-Mirsky Theorem)where the notation SVD r (Γ) means choosing the first r components of the matrix Γ. Further note thatEQUATION=: P 0 + P 1 + P 2 + P 3 + P 4 (10) Here note that Σ S = 1 |S| i∈S z v i (z l i ) ⊤ , we have P i as follows:EQUATIONEQUATIONEQUATIONEQUATIONEQUATIONIt's clear that the rank of the matrix P 0 is no more than r, so SVD r (P 0 ) = P 0 . And for i ∈ {1, 2, 3, 4}, P i are noise terms with E[P i ] = O.Lemma A.3. For any fixed S, w.h.p 1 -δ the noise term can be upper bounded by d log(1/δ)Proof. To upper bound the P1 and P2, we have∥ i z vl i (ξ vl i ) ⊤ ∥ 2 * = Tr   i,j ξ vl i (z vl i ) ⊤ z vl j ξ vl j   = i,j (z vl i ) ⊤ z vl j (ξ vl j ) ⊤ ξ vl i E∥ i z vl i (ξ vl i ) ⊤ ∥ 2 * = E i (z vl i ) ⊤ z vl i (ξ vl i ) ⊤ ξ vl i = |S|dRegarding each (z vl i ) ⊤ z vl j (ξ vl j ) ⊤ ξ vl i as weakly dependent variable, then by using Bernstein inequality, we have, with high probability 1 -δ,∥ i z vl i (ξ vl i ) ⊤ ∥ 2 * ≤ |S|d + d|S| 2 σ 2 ξ log(1/δ) ≤ |S|d log(1/δ) So 1 |S| ∥ i z vl i (ξ vl i ) ⊤ ∥ * ≤ d log(1/δ) |S| . Note that ∥ xvl ∥ ≲ log(|S|d) |S|(like Proposition 2.5 in Wainwright et al. [48] ), it is easy to see that P3 ad P4 are the low order terms if δ ≲ 1 |S|d .Lemma A.4 (Intuition behind VAS). With high probability 1 -δ, suppose the hind-side best subset has at least n number of samples, then we have∆(S) = 1 ρ max S ′ ∈Dtrain (Tr (Σ target (Σ S ′ -Σ S ))) + d log(1/δ) n + d log(1/δ) |S| Proof.For any learnt G v , G l based on dataset S, we haveL test (G v , G l ) = Tr(G ⊤ v E x vl ∼Dtarget [x v (x l ) ⊤ ]G l ) = Tr(E x vl ∼Dtarget [x v (x l ) ⊤ ]G l G ⊤ v ) = 1 ρ Tr E x vl ∼Dtarget [x v (x l ) ⊤ ]G * l Σ S (G * v ) ⊤ -Tr E x vl ∼Dtarget [x v (x l ) ⊤ ]noise S = 1 ρ Tr (G * v ) ⊤ E x vl ∼Dtarget [x v (x l ) ⊤ ]G * l Σ S -Tr E x vl ∼Dtarget [x v (x l ) ⊤ ]noise S = - 1 ρ Tr (Σ target Σ S ) -Tr E x vl ∼Dtarget [x v (x l ) ⊤ ]noise SHere the first equation comes from Theorem A.4 and the third equation comes from Lemma A.2. Consequently, we have -minS ′ ∈Dtrain L test (A(S ′ )) = max S ′ ∈Dtrain 1 ρ Tr (Σ target Σ S ′ ) + Tr E x vl ∼Dtarget [x v (x l ) ⊤ ]noise S ′ ≤ 1 ρ max S ′ ∈Dtrain (Tr (Σ target Σ S ′ )) + ∥E x vl ∼Dtarget [x v (x l ) ⊤ ]∥∥noise S ′ ∥ * ≤ 1 ρ max S ′ ∈Dtrain (Tr (Σ target Σ S ′ )) + O d log(1/δ) nTherefore, we have the final result as∆(S) = L test ( Ĝvl ) -min S ′ ∈Dtrain L test (A(S ′ )) = 1 ρ max S ′ ∈Dtrain (Tr (Σ target (Σ S ′ -Σ S ))) + O d log(1/δ) n + d log(1/δ) |S| Theorem A.3 (Main).Under the assumption of Lemma A.1, we have∆(S) ≤ noise + ∥ Σtarget -Σ target ∥∥Σ S -Σ best ∥ * + ϵ S v * l (vision+language) ϵ S v + 1 -1 |S| i∈[S] ⟨z v , z l ⟩) (vision only)Proof. Based on Lemma A.1, we will focus on the error cause from selecting subset S, that is, Tr Σ target Σ S . Since the exact Σ target is unknown, we assume the access to some proxy Σtarget instead.Recall that, for any S, we have ground-truth Σ S = E z vl ∈S z v (z l ) ⊤ . Unfortunately, this is not directly observable by the learner. Instead, the learner is able to observe some proxy ΣS based on the teacher model Ḡvl and therefore solving arg maxTr Σtarget ΣS and therefore, denoteΣ best = arg max S ′ ∈Dtrain Tr (Σ target Σ S ′ ) Tr (Σ target (Σ best -Σ S )) = Tr Σtarget (Σ best -ΣS ) + Tr Σtarget ( ΣS -Σ S ) + Tr (Σ target -Σtarget )(Σ best -Σ S ) ≤ Tr Σtarget ( ΣS -Σ S ) + Tr (Σ target -Σtarget )(Σ best -Σ S ) ≤ ∥Σ target ∥∥ ΣS -Σ S ∥ * + ∥ Σtarget -Σ target ∥∥Σ S -Σ best ∥ *where the first inequality is by the definition of ΣS and the second inequality comes from holder's inequality. Now the key is to upper bound ∥ ΣS -Σ S ∥ * based on our chosen strategy.In option 1, we use the clip embedding from both visual and language modal. That is, chooseΣS = x vl ∈S ( Ḡv ) ⊤ x v (x l ) ⊤ Ḡl . Then we have ∥ ΣS -Σ S ∥ * ≤ 1 |S| ∥ x vl ∈S ( Ḡv ) ⊤ x v (x l ) ⊤ Ḡl - x vl ∈S z v (z l ) ⊤ ∥ * ≤ ϵ S v * lIn option 2, we use the clip embedding from language model only. That is choose ΣS =x vl ∈S Ḡ⊤ v x v (x v ) ⊤ Ḡv .Then, by definition of ϵ S , we have∥ ΣS -Σ S ∥ * ≤ 1 |S| ∥ x vl ∈S Ḡ⊤ v x v (x v ) ⊤ Ḡv - x vl ∈S z v (z v ) ⊤ ∥ * + 1 |S| ∥ x vl ∈S z v (z v ) ⊤ -Σ S ∥ * ≤ ϵ S v + 1 |S| ∥ x vl ∈S z v (z v ) ⊤ -Σ S ∥ *Now to further bound the second term, we have1 |S| ∥ x vl ∈S z v (z v ) ⊤ -Σ S ∥ * ≤ 1 |S| ∥Z ⊤ v ∥ * ∥Z v -Z l ∥ * = 1 |S| Tr Z v Z ⊤ v Tr(Z v -Z l ) ⊤ (Z v -Z l ) = 1 |S| Tr(I n×n ) 2 Tr I n×n -Z v Z ⊤ l = 1 |S| 2|S|(|S| - i∈[S] ⟨z v , z l ⟩) = 1 - 1 |S| i∈[S] ⟨z v , z l ⟩)Therefore, we finish the proof.Theorem A.4 (A simplified version of test loss). Under the assumption that both z vl , ξ vl is zero-mean, maximizing the clip score gap is equivalent to maximize the clip score of the same sample.L target (G v , G l ) := -E x vl ∼Dtarget ⟨G ⊤ v x v , G ⊤ l x l ⟩Proof. For any x vl , we have [2] . Direction 3 denotes the methods that use external non-CLIP model to select the data that may be heuristically helpful for downstream tasks, like image without too much text or be more special. In general, D1 method using only CLIP embedding, like negCLIPLoss, is orthogonal to D2. And both D1 and D2 can be combined with D3 to explore better filtering results.E x ′ vl ∼Dtarget (⟨G ⊤ v x v , G ⊤ l x ′ l ⟩ -⟨G ⊤ v x v , G ⊤ l x l ⟩) = ⟨G ⊤ v x v , G ⊤ l E x ′ vl ∼Dtarget (x ′ l -x l )⟩ = -⟨G ⊤ v x v , G ⊤ l x l ⟩ CLIP TeacherIn the experiments part of the main paper (Sec. 4), we further show that our proposed D1 methods: NormSim and negCLIPLoss, can outperform all the D3 baselines except the best method "HYPE ∪ DFN". And we can achieve the new state-of-the-art by combining our methods with that method.We summarize our main idea of categorizing the current top data selection methods in Figure 4 .Our algorithm can significantly reduce the computational cost compared to many existing works as shown in Table 5 . For example, when the CLIP embeddings are obtained (cost about 50 hours for CLIP-B/32), both T-MARS [12] and MLM [42] still require more than 900 hours data pre-processing time to extract the required information from 110M size dataset of DataComp-medium, while we only need about 5 hours. On the other hand, DFN, although has a similar forward speed (i.e. preprocessing time), requires retraining a new CLIP teacher model on the HQITP-350M, which is larger than DataComp-medium.We give some details in estimating the preprocessing time of other methods:• For T-MARS and D 2 pruning, we run their official code on DataComp-small (11M) data, and simply scale the preprocessing time by 10 for DataComp-medium, given that the preprocessing time for T-MARS is proportional to the size of the pretraining dataset, while D 2 pruning is no faster than linear.Table 5 : Comparison of preprocessing time and external resources needed between our method and other D3 category methods. We skip DFN since it's orthogonal to our negCLIPLoss method and we can directly improve it as mentioned in Table 1 . Here since all the baselines below except MLM use a pretrained CLIP model, we only count the time that doesn't contain that for inferring CLIP image/text embeddings (about 50 L40 hours for OAI CLIP-B/32), which is also adopted in DataComp benchmark [1] . The external dataset corresponds to the external multimodal dataset used for training or finetuning the external model. Notably, the preprocessing time for the following methods are all approximately linearly proportional to the amount of unfiltered pretrained dataset. • For MLM, we get the estimated time from their paper. They mention that they need 6.1 minutes to process 10k samples on A100, which results in 1120 A100 hours for our dataset (110M). We need to mention that their estimation time of calculating CLIP embedding is inaccurate and we can do it much faster than their claim using the DataComp pipeline.• For Devil, it needs to run the k-means clustering algorithm from the faiss library on the embedding space, which is estimated to cost 120 L40 hours on DataComp-medium. Using BLIP-2 [16] to scan the whole dataset will need about 470 A100 hours from the experimental details in [17] . From https://lambdalabs.com/gpu-benchmarks, we roughly assume that 120 L40 hours are at least comparable to 40 A100 hours for K-means clustering.• For HYPE, they claim that MERU is as efficient as CLIP, but they still need at least 120 L40 hours for processing 110M data for their final score, since it uses the image embedding clusters on DataComp-medium obtained from running k-means clustering algorithm.We give the pseudocode of calculating negCLIPLoss in Algorithm 1, which is specially designed for pytorch-style parallel matrix calculation. It can be fully accelerated and the computation cost introduced by the normalization term is negligible compared with the training time or preprocessing time of other top baselines as detailed in Table C .1.In negCLIPLoss, we need to get the batch size |B| and the value of the learnable temperature parameter τ at the final step of the teacher model pretraining stage. For OAI CLIP-L/14 and OAI CLIP-B/32, these values are τ = 0.01 and |B| = 32768.We also have an ablation study about the temperature parameter and batch size chosen for CLIP teacher models as shown in Table 6 . We will see that in general, a larger batch size will result in better performance, and τ = 0.01, b = 32768 is the best choice for both OAI CLIP-B/32 and DFN-P.The reason for such a batch size is that a larger batch can contain more contrastive data pairs, which is also supported by the concentration result of the normalization term proved in Appendix A.1, and thus it can check the image-text matching between more different data. Therefore, we always consider the largest batch size 32768 which can fit into a single 24G GPU in the CLIP forward pass, which is also the OAI CLIP training batch size.In this section, we illustrate the details of our NormSim 2 -D algorithm. The top-N selection method is aiming to achieve the object: when the actual X target is unknown. In practice, removing one data at a time is too slow. Therefore, we remove a batch of data for every step. In detail, if the number of steps is τ , and let Σtest,i =EQUATION1 |Si| j∈Si fv (x v j ) fv (x v j )⊤ where S i is the selected subset at step i, then we will remove the data satisfies the following equation step-by-step until reaching the final subset size:S i \ S i+1 = arg min x l ∈Si fv (x v l ) T • 1 |S i | xt∈Si fv (x v t ) fv (x v t ) ⊤ • fv (x v l ) , i ∈ {0, . . . , τ -1}Then we can detail the algorithm process of NormSim 2 -D in Algorithm 2. In general, the smaller the step size, the better the results. But in experiments, we find that it's already enough to get good results when τ = 500.We add some details about the baselines used in our paper as follows.• Text-based filtering. [1] proposes a text-based filtering that tries to select the data that contains caption overlapping with the class name from ImageNet-21K or ImageNet-1K. • Image-based filtering. [1] also proposes a heuristic way to sample the visual content overlaps with ImageNet-1K classes. They first apply filtering by language (only choose English caption by fasttext [15] ) and caption length (over two words and 5 characters). Then they cluster the image embeddings from training data to 100K groups using Faiss [49] , and keep the groups whose cluster center is the nearest neighbor to at least one image embedding of ImageNet-1K image.The main hyper-parameters of our negCLIPLoss and NormSim are the target numbers for filtering (refer to Appendix C.2 for the setting of temperature and batch size), which is also the main concerns for all the top baselines like DFN, MLM, and T-MARS. In the case of DataComp settings, noting that all the top baselines in DataComp-medium benchmark keep the downsampling ratios ranging from 15% 30% to achieve the best results, we can set the sampling ratio as some previous baselines. Our method with OAI CLIP teacher model first selects the data with the top 30% negCLIPLoss, and then selects the top 66.7% NormSim scores to keep 20% of the original pool. We don't tune the target size carefully here for fair comparison.In more general cases, we can recommend some training-dataset-independent thresholds for Norm-Sim, since the scores only depends on the norm p and target data rather than other data in the pool. We recommend to set the threshold as 0.7 for NormSim ∞ (Target) and 0.15 for NormSim 2 (IN-1k) in general. On the other hand for negCLIPLoss, note that like NormSim, CLIPScore is also trainingdataset-independent, we recommend to first find the percentile of the data with CLIPScore=0.21, and then downsample the dataset using CLIPLoss until that particular percentile.Overall, finding optimal filtering ratio for data selection algorithm is always difficult and out of the scope of this paper. From the paper about the scaling law for data filtering [51] , downsampling size even depends on the computation budget. When you have more budget, you should sample more data for learning. And thus another possible solution is to use their fitting formula to get some recommended downsampling ratios.At last, we also note that in data selection problem, visualization is a simple but effective way for tuning parameters or finding downsampling ratios. People can first randomly select a small subset (like 1000 data) on some pretraining data subset, and then calculate the target scores (CLIPScore, negCLIPLoss, NormSim or any other metrics) on them, and fianlly visualize the data corresponding to scores at different percentiles, like top 10%, 30%, 50% and 70% of the negCLIPLoss. In this way, we can determine the threshold of filtering directly by observating the data. We also give some visualization examples of our methods in Appendix E, We believe this is an effective way to give some guidance on how to roughly select the initial downsampling ratios. For convenience, we let f (x t ) denote the image embedding of the target data x t ∈ X T , and f (x s ) denotes the image embeddings of training data x s ∈ X S . Then the definition of NormSim on a dataEQUATIONThen when p = 2, we haveEQUATION∝ f (x s ) ⊤ 1 |X T | xt∈X T f (x t )f (x t ) ⊤ f (x s ) 1/2 (20) Note that Λ = 1 |X T | xt∈X T f (x t )f t ) ⊤ isthe variance matrix of the target image embeddings. Then using NormSim 2 for filtering, we haveEQUATIONHere Λ = U SU ⊤ is the eigen decoposition of Λ, where S = diag(s 1 , . . . , s r ) with s 1 > . . . > s r are the matrix of eigenvalues, and U = [u 1 , . . . , u r ] ∈ R d×r are the corresponding eigenvectors (i.e., the principal component directions). Note that the column vectors of U and f (x s ) are all unit vectors, (24) shows that NormSim 2 select the data that match with the principal components, i.e., eigen directions u j with large eigen values s j .C.6.2 Why NormSim works well without explictly considering data diversity.We answer this question by the following reasons:• Many top baselines, such as DFN and T-MARS, also don't explicitly consider diversity, yet they still provide good performance. Devil even shows that valuable data is worth sampling multiple times, which they call "quality duplication". Therefore, one important reason why NormSim works well without explicitly considering diversity may be that when the computing budget is limited, as in the DataComp benchmark, the model first needs to learn the most useful and representative data, which should be similar to some target data. • Moreover, we chose validation data from 24 downstream tasks ranging from ImageNet to EuroSet, which may have covered a sufficiently diverse range of target examples for NormSim to calculate similarity. The diversity of the target data will consequently result in the diversity of the selected subset. And this also implies the importance of selecting a good target dataset. • An additional reason may be that our proposed negCLIPLoss already implicitly selects more diverse data, as shown in Figure 1 of the main paper. If some training data are diverse, they will match less with other data and thus have a lower normalization term. This results in a larger negCLIPLoss and a higher probability of being sampled.We show that negCLIPLoss is not sensitive to the number of random select batches K in Figure 5 .We show the complete results of applying our methods to different teacher models like OAI CLIP-B/32 and DFN-P in Table 7 . Detail descriptions are in Sec. 4.We also try to use near-neighbor selection for aligning downstream distribution. Here, we calculate the ranks of pretraining data for each target (the higher the rank, the higher the similarity), and then for each pre-train data, we keep its highest rank. Finally, we select the data with the highest ranks as the nearest neighbor selected subset.In Table 8 , we show that given the training data of 22 downstream tasks, our NormSim ∞ can outperform near neighbor selection under the same downsampling ratio. The reason may be that the distribution between the target and pretraining set is not well aligned, so if you force the algorithm to Figure 5 : Results of negCLIPLoss with a different number of batch samples (denoted as K) on DataComp-medium. Solid lines denote negCLIPLoss, while dashed lines denote CLIPScore. Here, we use OAI CLIP-L/14 as the pretrained model. We can see that once K ≥ 5, negCLIPLoss consistently outperforms CLIPScore across all subtask metrics. In the main paper, we set K = 10. find the nearest train data for each target, that train data may be sometimes random and not helpful. On the other hand, NormSim ∞ will not select this kind of data. It will select the data whose best similarity score exceeds some general threshold, rather than just consider ranks.In DataComp [1] , they show that image-based filtering is better than text-based filtering. In our paper, we also do an ablation study to support this. Due to the restriction of computation resources, we run our NormSim 2 (IN-1k) and NormSim 2 -D on DataComp-small as an example. Since ImageNet-1k only has labels rather than long texts for describing images, we need to generate the caption before calculating NormSim 2 (IN-1k). We select 80 templates as the original CLIP paper [4] , generate prompts for each class, and take the mean of their embeddings as the representative text embedding for images within that class.The results are in Table 9 . We can see that for both metrics, we have "image only" > "image × text" > "text only". We believe the reason for NormSim 2 (IN-1k) is that the images themselves can convey significantly more features than the text prompts generated by labels. For NormSim 2 -D, it should be related to the large amounts of low-quality captions in the web-curated dataset. And "image × text" will also be influenced by the informativeness and the quality of captions. In short, for NormSim, using vision-only embeddings is a best choice. We further visualize8 more data with different negCLIPLoss in Figure 6 , 7 and 8. And similar for NormSim ∞ (Target) in Figure 9 , 10 and 11. Justification: Yes we clearly define 1. the benchmark we are using; 2.the methods with its key insights 3.the empirical improvement. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: Yes Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Although out-of-distribution tasks like "WILDS" have distribution shift between training data and test data, they still provides useful information of the test data.Here we only use the target data for data selection, instead of training on them. The target dataset is significantly smaller than pretraining set like DataComp-medium (128M) or external datasets like HQITP-350M utilized by DFN[2].See https://github.com/mlfoundations/datacomp/issues/3. Similar issues are proposed by D 2 pruning[18].see "negCLIPLoss (30%) ∩ NormSim∞(Target)" versus "negCLIPLoss (20%)/(30%)" and "negCLIPLoss (17.5%) ∩ NormSim∞(Target)" versus "negCLIPLoss (17.5%)/(15%)"For the dataset size of "A∪B", we count the number of the unique data in the dataset followed HYPE[3].We use https://github.com/ypwang61/research_tools/blob/main/visualization2.py (Im-ageCaptionVisualizer) for visualizing the dataset. We also recommend visualizing basic dataset statistics by https://lst627.github.io/visdatacomp.github.io/. 


Decision: The submission proposes two data selection methods for large-scale visual-language model pretraining (e.g., CLIP). All three reviewers agree that this submission is well-written and intuitive with extensive experiments. The AC therefore recommends accepting the paper and asks the authors to include their discussions with the reviewers in the final manuscript.

Reviews: review: paper_summary: paper_summary: Data selection is crucial in the pretraining stage to clean the web-crawled, large, and noisy pretraining dataset. Typically, existing methods use embeddings to compute CLIPscore in order to assess the data sample alignment quality. This paper introduces two new methods to enhance this measurement:
1. negCLIPLoss: a better adjustment to reduce bias within a given batch.
2. NormSim: provides additional information when downstream tasks are known, allowing the selection of samples that are close to the target downstream tasks.
Empirical results demonstrate that these proposed methods can be easily combined with existing filtering approaches. The authors also illustrate that their approach yields state-of-the-art results on the DataComp leaderboard.
strength_weakness: strength_weakness: 1. I think we need more clarification on how to interpret the Top X% in three different metrics in Figure 1. Can the authors provide a more detailed description? Also, how is the R score derived from the batched data? How to find the proper batched data to use?
2. It seems that the negCLIPLoss is not incorporated into the training loss. We use it as a measurement when CLIP embeddings are provided. In this scenario, how do we determine the batch data, B, for subtracting the regularization term? Would the size of the batched data affect the measurement? The sampling method to find batched data is unclear to me.
3. I am unclear about the process for greedily selecting samples using NormSim, especially when the raw data pool is massive, and how to define the size of S. 
4. I would suggest moving algorithm steps from the Appendix into the main body, or showing some steps in the main body. They are good at understanding filtering steps.
questions: questions: 1. In Figure 1, R scores on the left side are in the top 100%, while on the right side they are in the top 10%. How should these be interpreted and categorized as underestimates or overestimates of quality?
2. When the downstream targets are not accessible, we may use the current filtered dataset as a reference, but how do we find the first-round reference dataset as a proxy to compute NormSim? 
3. I would like to list several papers that I found and read for data selection.

https://arxiv.org/abs/2405.15613, https://arxiv.org/abs/2401.12225, https://arxiv.org/abs/2302.03169, https://arxiv.org/abs/2401.04578, https://arxiv.org/abs/2404.07177
limitations: limitations: I didn't see any potential negative societal impact of their work.
paper_summary: paper_summary: This work proposes two new approaches for data selection for vision-language pre-training. The first approach, negCLIPLoss, adds the contrastive loss as a normalization term on top of the existing CLIPScore metric. The second approach, NormSim, further improves the performance if examples from target task distribution are available. Together, the methods achieve state-of-the-art results on ImageNet-1K and 38 downstream tasks with DataComp-medium without any external data or model. Several important theoretical justifications and interpretations are provided for the methods.
strength_weakness: strength_weakness: Quality: this is a minor complaint, but in Lines 229 - 230 the authors state that "the results of baselines on the leaderboard do not apply to our datasets, and we reproduce all the top baselines on the leaderboard with their public UIDs of the selected data" because some URLs of images become invalid. The leaderboard scores of baselines seem higher than the reproduced results in the submission. Could the authors also include the DataComp leaderboard results in the Appendix for fair comparison?

There are also some minor questions below.
questions: questions: 1. In Lines 135 - 136, the inaccessible batch division $B^*$ from teacher CLIP models is different from the actual batch $B_k$ in this work, in terms of both the actual image-text pairs and the batch size. Are there any potential theoretical guarantees or approximations to show that such a difference is reasonably negligible?

2. Could the authors further show the derivations of the discussions on the two important NormSim instances? 1) Lines 179 - 180 ($p=2$, equivalent to selecting a subset that aligns with the principal components), and 2) Lines 181-182 ($p=\infty$, a sample will be selected if it has high similarity to any target)? These may help other readers to understand NormSim better.
limitations: limitations: The authors discussed the limitations of this work.
paper_summary: paper_summary: This paper introduces a novel metric, negCLIPLoss, for selecting high-quality data. Additionally, the paper proposes a norm-based metric, Normsim, which offers an improved measure of data quality and is compatible with existing methods. Both negCLIPLoss and NormSim demonstrate significant performance improvements, outperforming state-of-the-art methods, while maintain low preprocessing time. Theoretical interpretations are provided for NormSim within the framework of a linear model.
strength_weakness: strength_weakness: 1. In Line 86, the authors assert that NormSim does not explicitly consider diversity but provide no further explanation. Since diversity is often linked to the generalization performance of models, it is unclear how the proposed methods implicitly connect with diversity.
questions: questions: 1. The algorithm 1 requires the knowledge of the batch size B and the parameter $\tau$ from the teacher model. If the teacher model is private and both B and $\tau$ are not accessible (for example, only api is provided), are the proposed methods still workable? How critical is the batch size B to model performance? Can the parameter $\tau$ be estimated?
limitations: limitations: NA

Comments: title: Thank you for reviewing
Response
comment: Thanks for your reply and support, and we will add the suggested details mentioned in the rebuttal in the next version. Thanks for taking the time to make our paper better!
Dear Authors,

I have read your general response and individual comments. Thanks for your reply. Thanks for addressing studies on batch size and clarifying some details in the paper. In general, this paper gives a new idea and a good adjustment to replace CLIPScore, but some places lack detailed descriptions. I support this paper and keep my original score here. Thanks.
We sincerely thank you for your time and constructive advice on improving our work!
The reviewer thanks the authors for the global and the specific responses. The reviewer is satisfied with the response and will maintain the score.