Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining
Authors: Mikey Shechter
Yair Carmon
Venue: NeurIPS 2025 poster ()
Keywords: dataset filtering
dataset curation
vision-language pretraining
CLIP
representation learning
Primary area: deep_learning
TL;DR: Introducing a learned algorithm for curating datasets for CLIP pretraining that achieves state-of-the-art ImageNet accuracy on the DataComp benchmark.

Abstract: We introduce Filter Like You Test (FLYT), an algorithm for curating large-scale vision-language datasets that *learns* the usefulness of each data point as a pretraining example. FLYT trains a scoring model that learns to weigh each example's features using gradient signals from downstream tasks training sets. Based on FLYT, we implement Mixing-FLYT (M-FLYT), which takes the per-example scores generated by different scoring methods as features, and learns to unify them into a single score. FLYT naturally produces a distribution over the training examples, which we leverage through Soft Cap Sampling (SCS), a strategy for obtaining a filtered pretraining dataset from per-example probabilities that samples examples while preventing over-representation through a repetition penalty. Using these methods, we achieve 40.1\% ImageNet zero-shot accuracy on the DataComp medium scale filtering benchmark, a 2\% absolute accuracy increase over all previous results and a 5.5\% increase over results that---like us---use only public resources. Our approach also yields 37.7\% on the average of 38 DataComp evaluation tasks, outperforming previous public-resource approaches by 0.4\%.

Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining
Mikey Shechter"
Yair Carmon"
Abstract
We introduce Filter Like You Test (FLYT), an algorithm for curating large-scale lisionlanguage datasets that learns the usefulness each data point as a pretrain- ing example. FLYT trains a scoring model that learns to weigh each example's 's features using gradient signals from downstream tasks training sets. Based on FLY',, we implement Mixing-FLYT (M-FLYT), which takes the per-example scores gener- ated by different scoring methods as features, and learns to unify them into a single score. FLYT naturally produces distribution over the training examples, which we leverage through Soft Cap Sampling (SCS), a strategy for obtaining a filtered pretraining dataset from per-example probabilities that samples examples while preventing over-representation through a repetition penalty. Using these methods, achieve 40.1 % ImageNet zero-shot accuracy the DataComp medium scale filtering benchmark, a 2% absolute accuracy increase over all previous results and a 5.5% increase results that-like us only public resources. Our approach also yields 37.7% on the average of 38 DataComp evaluation tasks, outperforming previous public-resoure, approaches by 0.4%.
Introduction
Filtering the pretraining data of CLIP models [35] has a dramatic effect on the resulting model's quality [39, 40, 17, 51]. Despite intense research and considerable progress, the best existing methods select data based heuristic proxies for usefulness [16, 22, 49, 29, 27, 21]. In this work, we take different approach: adhering to the core principle of machine learning, we optimize a loss function that measures the usefulness of data for pretraining.
We consider the setting of the DataComp filtering benchmark [17]. Our goal is to find a filtering method that selects subset from a candidate pool of web-sourced image-text training data. To evaluate the quality of the selected subset, it is fed to a fixed training pipeline and the resulting model evaluated a fixed set of zero-shot image comprehension tasks. The DataComp rules do not constrain the computational cost of the filtering method. Morcover, DataComp allows using external data (except evaluation test sets) for making the filtering decisions, long the selected training set a subset of the candidate pool.
In this setting, we introduce methods that together achieve state-of-the-art performance the DataComp medium scale. At the core is Filter Like You Test (FLYT), a method for training a scoring model that predicts the weight an image-text pair should have in CLIP pretraining. Mixing- FLYT (M-FLYT) is an implementation of FLYT that uses scores generated by existing scoring methods as input features the scoring model. Complementing FLYT, Soft Cap Sampling (SCS) is a simple sampling strategy that leverages the probabilistic structure of FLYT's scores to construct the final pretraining dataset.
Filter Like You Test (FLYT) (Section 3.2). FLYT recasts data selection as data re-weighting [37], and learns to upweight (and hence select) the pretraining data that contributes the most to
Tel Aviv University, sachter@mail. tau.ac. and ycarmon@tauex. tau.ac. ac.il. il.
2The name is inspired by FLYP [18].
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

update V Poddow!!!
Upstreom
Feature Extroctor
Scoring Model
Scores
softmax
Weights W
V
Weighted Clip
Lup
Reference
Upstream Embaddings fo (z)
Va
update( V Lap
Valaowwa
Updated Reference Model 0+
Embeddings
fo+ (2)
Loss
L
In each FLYT training loop, the scoring model takes features extracted from a batch of upstream data and generates score for each example. These scores are converted to weights via softmax. A reference model processes the upstream batch, and the resulting embeddings, together with the weights, are used to compute weighted CLIP loss. The reference model is then updated using gradients from this loss. Next, this updated reference model processes downstream data to compute a downstream loss, which produces gradient signal that passes through the updated reference model the way to the scoring model parameters.
Figure 1:
the model'ss downstream performance. We jointly train two models: a scoring model that evaluates the usefulness of training examples using extracted features, and reference model that provides feedback on these evaluations. During training, the scoring model assigns weights to each example in the batch, which are then used in our weighted CLIP loss function to update the reference model. The updated reference model then evaluated on downstream data, allowing gradient signal from the downstream loss to flow through it, to the weights, and finally back to the scoring model. This allows the scoring model to give higher weights to examples that reduce the downstream loss reference model. We then the trained scoring model to generate per-example scores. Figure 1 illustrates single training step of FLYT and Algorithm provides pseudocode.
Mixing-FLYT (M-FLYT) (Section 3.4). Previous work aggregates different filtering methods to obtain the best results. They either sum together different scores [22]. take the intersection//uion of the filtered datasets [17, 29, 22, 49] or pipeline the filtering methods one after the other [53, 49].
Departing from heuristic approaches, we leverage FLYT to directly optimize over space of mixing methods defined by a simple linear combination scores. Replacing the aggregation method of [22] with M-FLYT improves IetgeNet accuracy by 1.3%. More importantly, M-FLYT can aggregate more scores. When used with 12 different scores, outperforms the best single score by 1.7% on ImageNet accuracy, and by least 0.8% average accuracy. M-FLYT outperforms all baseline aggregation methods we evaluated, including summation and ImageNet accuracy-weighter sums by 0.5% on ImageNet accuracy and 0.6% on average accuracy.
different natural choice for FLYT's feature extractor component a multimodal neural network, such CLIP; refer to this Embeddings-FLL. (E-FLYTT), While E-FLYT does not improve the -fate-of-the-an on its own, incorporating its scores as an input to M-FLYT leads to improved overall performance.
Soft Cap Sampling (SCS) (Section 3.6). FLYT produces probability distribution over the training examples, allowing us to obtain filtered subset by simply sampling from the distribution. However, sampling with replacement produces too much repetition of the top-scoring examples, and sampling without replacement produces too few. To interpolate between the two sampling methods we propose SCS, a strategy that samples with replacement but reduces repetition using penalty term (see Algorithm 2). Using only publicly available resources, M-FLYT with SCS has 40.1 % ImageNet zero-shot accuracy, which is 2% absolute increase compared to the best prior work, and 5.5%

---

Algorithm 1: Filter Like You Test (FLYT)
Algorithm 2: Soft Cap Sampling (SCS)
Scoring model Göo with initial parameters Φp. reference model fe, with initial parameters 8, feature extractor V, upstream dataset D, downstream dataset C Parameters: Number of steps T, upstream data batch size B, downstream data batch size B', weighted loss EWCL, and downstream loss EDS for = 0 to T 1 do
Input:
Input: Training examples 21:M and their
scores 81:M
Parameters: Repetition penalty target dataset size N and sample count per
iteration G
D [] while |D| < N do
W1:ad = softmax(s1:a4)
f::G = Sample G indices according to
21:8 ~ D. 21:B'
distribution WI:M without repetitions.
(:B (=1.B)
δ concatenate (D, zin.o)
$1:B qd. (C1.B)
$1.0 = Si.. a
W1.B softmax($ ($1:B)
return D
Lup EWCL (fe, (21.B), W1:B)
0.+1 update( (V, lup))
Ldown ( (8).)))
Φ++1 = updato( V Ldoon)
> Appendix A
return ΦT
improvement compared to the best prior work that uses only publicly available resources. Our filtered dataset also achieves 37.7% average accuracy across all evaluation tasks, which is within 0.1% of our reproduction the best previously published filtering method, and a 0.4% improvement over the best prior work using only publicly available resources. On the DataComp small scale benchmark (which we did not evaluate until the end of the project) our method outperforms previous work on both ImageNet and average accuracy (see Table 4).
Data and code release. We share all the code required to reproduce our work, including E- FLYT, M-FLYT training, and SCS sampling at https: / github. ally, publish best-performing scoring and mixing models along with the resulting filtered datasets, and the M-FLYT input scores at flyt- 0tt.-66666677666660ff0ddd66666
Addition-
Related work
Early work on curating large-scale image-text datasets established methods to extract, filter, and organize pairs of images and captions from the web [41, 5, 45, 13]. Beyond assembling large collections, several lines of research address dataset pruning and deduplication [44, 2], example characterization [28, 47], and quality assessment [34].
CLIP [35] leveraged large image-text datasets via contrastive learning, producing the first generation of lasion-language foundation models and revolutionizing computer vision. While the original CLIP was trained on proprietary set of 400M image-text pairs, subsequent efforts introduced open-source alternatives. LAJON-400M [39] and LAION-55 -B [40] filtered large web-crawled corpora primarily using the cosine similarity between image and text embeddings from a pretrained CLIP model, along with basic image and text quality metrics. Xu et al. [51] attempted to reproduce CLIP's original data pipeline and managed to outperform the original model.
Recognizing the importance of data quality for CLIP training. Gadre et al. [17] introduces the DataComp benchmark. DataComp fixes the data pool as well the training and evaluation procedures, and allows researchers to focus on comparing filtering methods while keeping all other parameters fixed. Below, we survey the literature on data filtering for DataComp.
Filtering methods. Gadre et al. [17] consider several baseline filtering methods, and achieved best results by combining CLIP score with clustering- filtering method using similarity to the ImageNet training set. T-MARS [29] observe that images containing texts that overlap with the

---

caption received high CLIP scores. while not contributing much to the model training. To address this issue, they mask out these texts before calculating CLIP score. For better results, they take the intersection of this approach and their variation of Maini et al. [28]. Yu et al. [53] developed pipeline merging various techniques categorized as: single modality filtering, cross modality filtering, and distribution alignment. DFN [16] use CLIP score from a model fine-tuned on large high-quality private data. HYPE [22] argues CLIP's image-text alignment alone is insufficient, and adds individual information of the images and texts. They use hyperbolic embeddings to generate hyperbolic similarity score, and image and text specificity scores, which they sum with the traditional Euclidean CLIP and the image-bare based scores. s- s-CLIPLoss [49] introduce variation to the standard CLIP score that normalizes it by the example's 's similarity score with other examples in a batch, and after filtering using this score, filters remaining data using downstream example similarity metric. While these heuristic approaches are successful, our approach directly learn how to estimate an example's 's quality.
Data sampling. Given a scoring of the data pool, most filtering approaches select the top 15-20% of examples, resulting in each example appearing 5-7 times during training in the DataComp benchmark. Goyal et al. [19] analyzed the quality-quantity trade-off of dataset filtering, and found that for the compute budget of the medium scale DataComp benchmark selecting the top 20% of examples is optimal, with more aggressive filtering leading to diminishing returns. However, their analysis only considers uniform sampling while top performing methods created non-uniformly sampled datasets through concatenation of multiple subsets of data [22, 49]. Our proposed SCS method extends this approach of non-uniform sampling using a simple probabilistic procedure; in our top performing dataset, individual examples appear up to 56 times.
Learned data weighting. FLYT shares conceptual and technical similarities with meta-learning paradigms. Particularly similar meta-learning works address the problem of learned data weighting. Ren et al. [37] introduced algorithm for online example weighting which utilizes gradient signals from a validation dataset for producing per-example weights. Shu et al. [43] employs a similar idea to Ren et al. [37] with the addition of a weighting model that is optimized to weigh the examples. FLYT is similar to these methods in that it too learns weights and backpropogates through a "target loss" after gradient update on the "training loss." However, unlike FLYT, these methods consider online data weighting for standard supervised learning, with the intention of improving robustness to noise and data imbalance, mostly in small-scale settings.
Concurrent work. Concurrent with our research, two related works also developed data filtering algorithms evaluated on the DataComp benchmark. Xu et al. [52] introduced EcoDatum which transforms unimodal and multimodal scores into discrete labels and uses weak supervision ensemble model to aggregate them based on agreements and disagreements [36]. Their approach is similar to M- FLYT in that it combines multiple filtering methods, but unlike us. it doesn't 't optimize downstream objective for aggregation. Separately, Engstrom et al. [14] proposed metagradient descent (MGD), a model-free dataset curation method that, similar to our approach, leverages gradient signals from downstream tasks. While MGD learns to score each example separately, we train scoring model that applies to unseen examples, which allows us to extend our method to larger dataset scales at a low computational cost. Additionally, whereas MGD optimizes for all available downstream tasks in the DataComp benchmark, we use only the ImageNet training set as the downstream data. MGD sets new state-of-the-art average accuracy of 40.2% on the DataComp medium benchmark, though its ImageNet accuracy is only 27%
3 Method
In this section, we describe FLYT in detail. We introduce our notation by going over standard CLIP training (Section 3.1), then move to describing FLYT (Section 3.2). We then go over the weighted CLIP loss we use to train FLYT (Section 3.3), followed by two different feature extractors for FLYT's implementation (Section 3.4), and our choice of downstream data (Section 3.5). Finally, we present SCS (Section 3.6).
3.1 Notation and problem setting
CLIP training. We denote a CLIP training example = = (= E D where our pretraining dataset and 2² are the training image and text, and denote a batch of B training examples by 21.B. We train a CLIP model with parameters that maps each example z to features fo(=) = (fo(z)), fo(z )). Letting fe(21.) denote the features of all the examples in the batch, the standard

---

CLIP loss is
lc (fo(21:B) = (Pinuge + 12ex))22
(I)
where
image
contrast (fo(z (o Conntrast ( (1I.B,
lex = Leontrast Leontra((F(("") foo
B
U1:B)
=
log
j=1 e² eu(unies)
and r the learnable temperature parameter.
3.2 Filter Like You Test (FLYT)
In this section describe FLYT. Our goal is learn scoring model de parameterized by ó. The scoring model takes feature vector Ç = (2) produced by applying feature extractor to example 2, and maps it to score s, such that the higher is, the more useful is pretraining example. We overload notation to let
$1.B (SIBB)
denote the scores computed element wise over the batch. We obtain weights over the batch W1.B using a softmax transformation:
exp(s:)
W1BB = softmax(#1.8)) where softmax(s:)
Ej=1 exp(s;)
Now consider a reference CLIP model fg. We define a weighted CLIP loss
Lup (fo W1.B)
which is function of both the reference model embeddings, and the weights produced by the scoring model. This loss serves as an approximation to the loss we would have gotten had we used the scoring model filter the batch. We diseuss the weighted CLIP loss in detail in section Section 3.3.
We can use this loss and its gradient to update let
04 (W1:B) = update (VatteeVVlll VgLup)
be the result of gradeent-based update to 0. For example, using SGD with learning 7/, have 0 (Wi:s) = V Tup- Using a more sophisticated optimizer like Adam 04 (W1:B) takes a more complicated form, but
still think about it function of W1.B-
Next, we introduce the key part our approach, batch downstream data (e.g., the ImageNet training set). Let a1.8r denote this batch (note that B' does not necessarily have to be equal to B). We use this batch to obtain gradient signal for updating W1.B and hence the scoring model parameters 6. To that end, define:
Lawon EDS (fo (E..B
Where ls is some loss over the downstream data: we experiment with standard CLIP loss lc, as well as cross-entropy (CE), with or without temperature; see Appendix C.8 for details. We can then update & using V VoLLown-
Finally, we score each example with our trained scoring model, then filter based on these scores.
3.3 Weighted CLIP loss
Our definition of weighted CLIP loss follows these guiding principles:
1. Crucially, larger weight means the corresponding example is more represented in the loss.
2. Setting an example's weight to 0 should be equivalent to excluding it from the batch.
3. Uniform weights should result in the standard CLIP loss.
4. The weight should reflect the example's importance both positive and negative example.
To realize these principle modify the CLIP loss Equation (1), showing modifications in red:
E +
wer, (fo(21.n) W1.R)
text
(3)
2
5

---

where
image
contrast
(o(("") (..B)
= (fa(="), fo(z') (.BB)) B), and
contrntt
(U1:B, 01:B;
W1.B)
Σ
wi log
Eラ=1 my uyyer(un))))
B
3.4 Feature extractors
Mixing-FLYT (M-FLYT) Inspired by past success in mixing multiple filtering heuristics [17, 22, 49, 53], we implement FLYT using existing scoring methods as our feature extractor. Specifically, the feature extractor ! produces a vector Ç E I& of k different scores, generated by k different scoring methods. Our scoring model qo I + R then takes as input the different scores to output a single unified score 8. We refer to this implementation of a scoring model as mixing model.
Using M-FLYT gives us the advantage of not needing to learn example importance from scratch, and instead use the information gained by other scoring methods. In fact, we notice that a simple linear mixing model is enough for obtaining strong results.
Embedding-FLY's (E-FLYT) With E-FLYT, we use a pretrained CLIP model as the feature extractor // to generate image and text embeddings. Our scoring model gø then maps the concatenated image and text embeddings to a score s.
Despite CLIP's ability produce semantically rich representations, our current E-FLYT implementa- tion does not yet match M-FLYT's performance. Appendix C.7 documents our experiments, which show improvements but suggest further work is needed fully leverage these embeddings.
3.5 Choice of downstream data
FLYT requires high-quality downstream data to effectively learn which examples are more valuable for pretraining. For our implementation, we use the ImageNet training set as proxy for downstream task data, which aligns with the DataComp competition rules permitting "use of other images associated with these tasks (e.g., supervised training sets)" [17]. This approach follows standard practice, as all other top-performing methods [17, 49, 53, 16) similarly utilize the ImageNet training set for their filtering algorithms. The DataComp benchmark includes 38 diverse tasks, some of which differ significantly from ImageNet. This range of evaluation tasks means that the average downstream score provides practical measure of model performance unfamiliar tasks.
3.6 Soft Cap Sampling (SCS)
We now consider the task of obtaining filtered pretraining dataset from per-example scores. A principled data filtering approach should use these scores for deciding the number of repetitions of each example in the filtered dataset. It clear how do that when using similarity based score as in most prior work. In contrast, sampling strategy emerges directly from the definition of FLYT, since the scores naturally correspond to the (log) probabilities of including examples in the batch.
A straightforward approach would be to sample from this distribution 128M times to create our final dataset. In practice, doing that results over-represented examples. To correct this, we introduce SCS (Soft Cap Sampling): every time we add an example to our filtered data, we subtract "repetition penalty" a from its score, reducing probability perform the sampling procedure iteratively. In each iteration, sample batch of G 100K examples without repetitions, then subtract a from the score of each batch member. Algorithm presents SCS, and Figure illustrates the distribution generated via SCS.
sampled again. For implementation efficiency,
Experiments
In this section, we present the steps we took to create our top performing dataset which achieves 40.1% zero-shot ImageNet accuracy on the DataComp medium scale benchmark and 37.7% average accuracy across all DataComp evaluation tasks. We first review key details about our training and evaluation setup (Section 4.1,, then discuss the challenges, design choices and results for M-FLYT (Section 4.2) and SCS (Section 4.3). Table I summarizes our findings and compares them to the state-of-the-art. Section 4.4 we apply our algorithm to the DataComp small scale benchmark, demonstrating that approach works effectively across different data scales.

---

10'
Method
Samaliss
Repediiors
Imageket
10F
SCS aw0
25.74
56
0.401
0.377
SCS
37.2M
0.300
10
Number of Repetitions
Figure 2: Histogram of example repetitions using SCS on the probabilities produced by M-FLYT. See Section 4.3 for more details.
Table 1: Comparison our method to top performing methods on the DataComp medium scale filtering benchmark. DFN-FT best reproduction of DEN [16]. denotes results from the original paper, and + indicates evaluations using our downloaded 19M candidate pool using the authors' provided UIDs. Table 15 compares evaluations to the results reported in the corresponding papers.
Public
ImageNet dist. shifts
Filtering Method
ImageNet
VTAB
Retrieval
Average
Resources
Image-basedr ICLIP score' [17]
Yes
0.305
0.243
0.342
0.250
0.328
DFN-FT [our reproduction 16]
Yes
0.342
0.274
0.357
0.290
0.348
HYPE 20%
[22]
Yes
0.338
0.269
0.357
0.286
0.343
CLIPLoss [49]
Yes
0.333
0.273
0.361
0.251
0.352
DEN+ [16]
No
0.367
0,303
0.371
0.280
0.364
HYPE+DEN [22]
No
0.372
0,304
0.374
0.287
0.368
DEN+s-CLIPL.sS [49]
No
0.371
0.304
0.390
0.285
0.378
IYPE-DDEN+-CLIPLOS [49]
No
0.381
0.310
0.392
0.284
0.378
M-FLYT
Yes
0.359
0.294
0.383
0.310
0.371
M-FLYT&SCS
Yes
0.401
0.311
0.396
0.292
0.377
4.1 General setup
We experiment mainly with the medium scale DataComp filtering track benchmark [17]. In this benchmark, the goal is to filter a 128M example candidate pool sourced from Common Crawl [1]. The filtered data is then used to train a ViT-B/32 CLIP using fixed procedure, and the resulting model undergoes a fixed evaluation on 38 downstream tasks, with performance typically reported in four main categories: ImageNet, ImageNet distribution shifts, VTAB, and retrieval, along with the overall average across all tasks.
For our experiments we were only able to download 119 out of the 128M DataComp candidate pool. To compare our filtered datasets to those published in prior work, we reapply the DataComp training and evaluation procedure on the intersection of the published datasets and our downloaded pools. Table 15 shows that the accuracy degradation due to the missing data is between 0.1% and 1 1%.
The fixed DataComp medium training procedure goes through 128M input examples (hence using fixed amount of compute) regardless of the size of the filtered data; if the filtered dataset has less than 128M examples, reshuffled and repeated until reaching 128M examples. except those specifically evaluating sampling strategies, we follow the standard strategy of selecting top 20% scoring examples, which means each example is seen by the model or times during training. Using SCS, control how many times each example will be seen, forming dataset with repeating data and using a shuffle buffer to spread data repetitions across training.
experiments
For both M-FLYT (Section 4.2) and E-FLYT (Appendix C.7), we use CLIP VIT-B/32 model as the reference model. Unless otherwise mentioned, we use the unfiltered DataComp medium pool as the upstream dataset, and the ImageNet training set the downstream dataset.
To enable large-scale training of FLYT, we develope data parallelism approach, which detailed in Appendix A. The computational costs are outlined in Appendix C.2.

---

Table 2: Left: Performance and linear weight of each input to M-FLYT. For HYPE [22] and
CLIPLoss [49], we replicate the scores and mixing methodology used in the original papers. Right: Baseline aggregation experiments as described Section 4.2.
Input
Weight
ImageNet
Average
VIT-B/32
0.08
0.282
0.327
ViT-L/14
0.21
0.267
0.317
Baseline
ImageNet
Average
DFN-Base
0.60
0.297
0.333
Sum
0.327
0.331
DFN-FT
0.80
0.342
0.348
Standardized Sum
0.348
0.360
HYPE -d
0.45
IN-weighted 2
0.350
0.364
HYPE
-0.05
0.309
0.341
IN- IN-weighted = 4
0.354
0.359
HYPE
0.02
IN IN-weighted = 8
0,353
0.365
s-CLIPLoss
0.51
0.331
0.363
WN-weighted 16
0.352
0,362
NormSim
0,55
INIK-Classine
0.36
0.268
0.294
M-FLYT
0.359
0.371
CC2M-Classfier
0.17
0.287
0.308
E-FLYT
0.65
0.316
0.323
4.2 M-FLYT results
We train a linear mixing model using input scores that are standardized to zero mean and unit standard deviation. The standardization both improves performance and provides interpretable weights for each input method. We empirically found that initializing the reference model with a CLIP model trained using DataComp's medium scale configuration their unfiltered dataset and using standard cross entropy (CE) as the downstream loss produced better results; see Appendix C.3
We create and reproduce 12 input scores. We use the CLIP score of OpenAl's ViT-B/32 and ViT-L/14 models [35], as well as reproduced DFNs [16], before and after fine-tuning, which refer to as DFN-Base and DFN-FT (Appendix B. 1). We reproduce the image and text specificity negative Lorentzian distance yy introduced by HYPE [22] using their codebase, and the S-CLIPLoss and NormSim (INIK) scores introduced by Wang et al. [49] using theirs. We also add three new inputs. Two binary classifiers that classify between some "high quality" dataset and the DataComp medium scale dataset, and an E-FLYT scoring model described in Appendix C.7. See Appendix B for more details about the inputs.
and
We compare M-FLYT to individual input components and to several baseline approaches. These baselines include simple sum of all input scores and sum of scores after standardization to zero mean and unit standard deviation. Finding that standardization significantly improves performance, we further explore weighted standardized sums where inputs are weighted by their normalized Ima- geNet scores adjusted to achieve a specific max-to-min weight ratio r, for details see Appendix C.5.
Table 2 demonstrate that M-FLYT outperforms all individual scores as well as all baseline aggrega- tions. Table 9 demonstrates that excluding any individual input score from M-FLYT's training leads to performance degradation, suggesting that the method effectively leverages multiple signals, and could potentially achieve stronger performance given more (or better) input scores.
In Appendix E we complement methods. particular, we plot the top-scoring training example according to different filtering methods. Figures 5 to 7 show that M-FLYT prioritizes examples containing multiple ImageNet classes, while sorting by OpenAl's ViT-B/32 CLIP scores favors images with visible text that matches the caption text.
quantitative results with qualitative comparison of filtering
4.3 SCS results
For SCS, we experiment with repetition penalty between 0.1 and 0.6, and sample in batches of G = 100K. We choose G to balance efficient sampling while ensuring the resulting dataset closely approximates what we would get using G 1. Appendix C. 10 shows that the exact value of G has minimal impact on performance. When preparing the filtered dataset, we use a 1.5M shuffle buffer in the DataComp resharder to prevent batches from containing many copies of the same example.

---

ImageNet Accuracy
Average Accuracy
0.38
Imagelet Accuure Accuracy
Average Accuracy
0.37
0.36
0.36
0.35
0.34
0.34
0.33
20
30
40
10
20
30
40
Unique Samples
Unique Samples
Soft Cap
Hard Cap
Threshold
Cap
Figure 3: Comparison of sampling strategies (SCS, HCS, threshold filtering, and No Cap) showing their effect on ImageNet accuracy (left) and average accuracy (right), SCS was tested with a values from 0.1 to 0.6, HCS with values from 5 25 and standard threshold with values from 5% 25%
Table Using SCS with M-FLYT compared IN-weighted.
Mixing method
Sampling method
ImageNet
Average
M-FLYT
Threshold top 20%
0.359
0.371
M-FLYT
SCS (a = 0.15)
0.401
0.377
IN-weighted
Threshold top 20%
0.353
0.365
IN-weighted
SCS (a 0.15)
0.379
0.362
When using the scores produced by M-FLYT, SCS outperforms every other filtering method in the DataComp ImageNet Leaderboard (Table 1). Figure 3 shows the influence of the repetition penalty parameter a, and compares SCS to standard threshold filtering, regular sampling (No Cap), as well as to a sampling method we call Hard Cap Sampling (HCS). HCS allows example to be sampled until it reaches a "frequency limit" 3, and then its sampling probability is set to We note that while SCS outperforms HCS, both methods outperform the standard threshold filtering.
Figure 3 also shows a pattern where for threshold filtering, example diversity improves average accuracy but reduces ImageNet accuracy: selecting the top 10% of examples (1 1. 11.93M) performs better on ImageNet compared to selecting the top 20% (23. 23).6M), but results in lower average accuracy across tasks. This trade-off is not as apparent with SCS.
To test SCS independently of M-FLYT, we apply it with repetition penalty 0.15 scores from our strongest baseline, IN-weighted with r = 8. We convert these scores to probabilities via softmax, following the same approach with M-FLYT. Table 3 shows that SCS improves IN-weighted ImageNet accuracy by 2.6% while decreasing average score by 0.3%. In comparison, applying SCS M-FLYT improves ImageNet accuracy by 4.2% and average score by 0.6%. This demonstrates that a key advantage of FLYT is that it naturally produces distribution over training examples, which can be leveraged through sampling approaches like SCS.
4.4 Small scale results
We evaluate our method on the small scale DataComp benchmark. We use the mixing model trained by M-FLYT described Section 4.2, and sample using SCS with a repetition penalty 0.5 and a batch size G = 10K. Table 4 shows that our method achieves 8% zero-shot ImageNet zero-shot accuracy and 18.5% average accuracy, outperforming previous work both categories. These results suggest that M-FLYT and SCS can be effectively used across different computational scales. Figure shows the effect of different values.
5 Limitations and future work
Computational scale a clear limitation of our work: we only had the resources to experiment the DataComp medium scale, and could not test our methods on the larger scales. However, in prior work

---

Table Comparing our methods to top performing methods on the DataComp small scale filtering benchmark, * denotes results from the DataComp leaderboard that did not provide a paper to cite.
Filtering method
ImageNet
Average
EcoDatum [52]
0.053
0.182
WS [21]
0.056
0.180
HYPE [22]
0.063
0.180
CAM n Flipped CLIP
0.064
0.178
M-FLYT
0.072
0.185
M-FLYT SCS 0.5)
0.080
0.185
progress on medium scale usually translated to progress on larger scales, which makes us optimistic that our proposed methods will also scale well.
The computational overhead of FLYT is potential limitation, but may also be negligible in some scaling regimes. In our experiments, M-FLYT achieves high performance using approximately 3 fewer FLOPs than training DataComp medium-scale model, due training relatively few examples (20M). Scaling up the number of training steps would proportionally increase computational costs, as detailed in Appendix C.2. Whether the scoring model needs increase in computational cost as the data and final model size increases and hence the relative overhead of FLYT in large scale settings-
topic for future work.
Another notable limitation is that E-FLYT underperforms methods such as DFN, though its scores did contribute M-FLYT. To surpass state-of-the-arr methods we had leverage existing filtering methods through M-FLYT rather than relying on a general feature extractor. We believe this is likely deficiency in optimization stack, solvable by more careful tuning of learning rate (schedules) for individual components and regularization techniques such as dropout. In similar vein, we believe that more careful parameter tuning should allow us leverage downstream data that is more diverse than ImageNet, though these hypotheses require empirical validation.
A fourth limitation-ant direction for future research- is that we use data weighting as proxy for data selection when defining the FLYT objective function. It would be interesting to try to model data selection more directly via reinforcement learning: each reference model training step, the scoring model picks a batch training data and observes a reward signal for the chosen data.
A fifth direction for future work is closing the data selection / pretraining loop by iteratively re- re-training the scoring model to select the next data to pretrain our model on. FLYT is particularly well-suited to this form of active learning, as it allows us to use the pretraining model as the reference model at each scoring iteration, ensuring each selected batch optimal for the model's current state.
Finally, FLYT should be useful past DataComp and vision-language data filtering. In particular, will be exciting extend it data filtering and mixing for language models.
Acknowledgments
We thank Sonia Lichtenzveig Sela and Alexander Tsverin from the TAU CS systems for crucial technical support. We thank our anonymous reviewers for experiment and writing suggestions, Alex Fang and Vaishaal Shankar for helpful discussion of the DFN implementation details, and Sanghyuk Chun for sharing the HYPE codebase.
This research was partially supported by the computing resources of the Tel Aviv University Center for Al and Data Science, the Adelis Foundation, and the Israeli Science Foundation (ISF) grant no. 2486/21.
References
[1] URL WtL.ttts///ccmoncraat
[2] A. K. M. Abbas, K. Tirumala, D. Simig, S. Ganguli, and A. S. Morcos. SemDeDup: Data- efficient learning at web-scale through semantic deduplication, In International Conference on Learning Representations (ICLR), 2023.
10

---

[3] S. Beery, A. Agarwal, E. Cole, and V. Birodkar. The iWildCam 2021 competition dataset. arXiv:2105.03494, 2021.
[4] L. Bossard, M. Guillaumin, and L. Van Gool. Food-101- discriminative components with random forests. European Conference on Computer Vision (ECCV), 2014.
[5] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021.
[6] G. Cheng, J. Han, and X. Lu, Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the Institute of Electrical and Electronics Engineers (IEEE), 2017.
[7] Cherti, Mehdi, Beaumont, Romain, Wightman, Ross, Wortsman, Mitchell, Ilharco, Gabriel, Gordon, Cade, Schuhmann, Christoph, Schmidt, Ludwig, and Jitsev, Jenia. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[8] G. Christie, N. Fendley, J. Wilson, and R. Mukherjee. Functional map of the world. In Conference Computer Vision and Pattern Recognition (CVPR), 2018.
[9] M. Cimpoi, S. Maji, I. Kokkinos, Mohamed, and A. Vedaldi. Describing textures in the wild. In Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
[10] A. Coates, A. Ng, and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2011.
[11] L. Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 2012.
[12] Deng, Jia, Dong, Wei, Socher, Richard, Li, Li-Jia, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE conference on computer vision and pattern recognition, 2009.
[13] K. Desai, G. Kaul, Z. T. Aysola, J. Johnssn. RedCaps: Web-curated image-text data created by the people, for the people. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
[14] L. Engstrom, A. Ilyas, B. Chen, A. Feldmann, W. Moses, and A. Madry. Optimizing ML training with metagradient descent. arXiv:2503. 2025.
[15] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal html.
[16] A. Fang, A. M. Jose, A. Jain, L. Schmidt, A. T. Toshev, and V. Shankar. Data filtering networks. In International Conference on Learning Representations (ICLR), 2024.
[17] S. Y. Gadre, G. Ilharco, A. Fang, Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang, E. Orgad, R. Entezari, G. Daras, S. Pratt, V. Ramanujan, Y. Bitton, K. Marathe, S. Mussmann, R. Vencu, M. Cherti, R. Krishna, P. W. Koh, O. Saukh, A. Ratner, S. Song, H. Hajishirzi. A. Farhadi, R. Beaumont, S. Oh, A. Dimakis, J. Jitsev, Y. Carmon, Shankar, and L. Schmidt. DataComp: search of the next generation of multimodal datasets. 2r04v:2304. /4108, 2023.
[18] S. Goyal, A. Kumar, Garg, Z. Kolter, and A. Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[19] S. Goyal, P. Maini, Z. C. Lipton, A. Raghunathan, and Z. Kolter. Scaling laws for data filtering- data curation cannot be compute agnostic. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, 2024.

---

[20] P. Helber, B. Bischke. A. Dengel, and D. Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
[21] T.-H. Huang, C. Shin, S. J. Tay, D. Adila, and F. Sala. Multimodal data curation via object detection and filter ensembles. arXiv:2401.12222 2024.
[22] W. Kim, S. Chun, Kim, D. Han, and S. Yun. HYPE: Hyperbolic entailment filtering for underspecified images and texts. In European Conference on Computer Vision (ECCV), 2024.
[23] J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine- fineggrained categorization. In International Conference on Computer Vision Workshops (ICMI.), 2013.
[24] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images, 2009.
[25] I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv: 1600:6008.., 2016.
[26] I. Loshehilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2017.
[27] A. Mahmoud, M. Elhoushi, A. Abbas, Y. Yang, N. Ardalani, H. Leather, and A. S. Morcos. Sieve: Multimodal dataset pruning using image captioning models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
[28] Maini, S. Garg. Z. C. Lipton, and J. Z. Kolter. Characterizing datapoints via second-split forgetting. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[29] P. Maini, S. Goyal, Z. C. Lipton, J. Z. Kolter, and A. Raghunathan. T-MARS: Improving visual representations by circumventing text feature learning. In International Conference on Learning Representations (ICLR), 2024.
[30] S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft, arXiv:1306.514 5151, 2013.
[31] Y. Netzer, Wang. A. Coates, A. Bissacco, B. Wu, and Y. Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems (NeurlPS) Workshops, 2011.
[32] M.-E. Nilsback and A. Zisserman. Automated flower classification over large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, 2008.
[33] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. Cats and dogs. In Conference on Computer Vision and Puttern Recognition (CVPR), 2012.
[34] Radenovie, A. Dubey, A. Kadian, T. Mihaylov, S. Vandenhende, Patel, Y. Wen, V. Ra- manathan, and D. Mahajan. Filtering, distillation, and hard negatives for vision-language pre-training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023.
[35] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, G. Krueger, and Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
[36] A. Ratner, S. H. Bach, H. Ehrenberg, J. Fries, S. Wu, and C. Ré. Snorkel: Rapid training data creation with weak supervision. In Proceedings of the VLDB endowment. International conference on very large data bases, 2017.
[37] M. Ren, W. Zeng, B. Yang, and R. Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning (ICML), 2018.
[38] W. A. G. Rojas, S. Diamos, K. R. Kini, D. Kanter, V. Reddi, and C. Coleman. The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2022.
12

---

[39] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki. LAION-400M: Open dataset of CLIP-filterd 400 million image-text pairs. arXiv:2111.02114 2021.
[40] C. Schuhmann, R. Beaumont, R. Vencu, C. W. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. R. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev. LAION-5b: An open large-scale dataset for training next gener- ation image-text models. In Advances Neural Information Processing Systems (NeurIPS), 2022.
[41] Sharma, Piyush, Ding, Nan, Goodman, Sebastian, and Soricut, Radu. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the Association for Compulational Linguistics, 2018.
[42] N. Shazeer. GLU variants improve transformer. arXiv:2002.05202 2020.
[43] Shu, Q. Xie, L. Yi, Q. Zhao, Zhou, Xu, and Meng. Meta-weightt-et. Learning an explicit mapping for sample weighting. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[44] B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and A. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[45] K. Srinivasan, K. Raman, J. Chen, M. Bendersky, and M. Najork. WIT: Wikipedia-hasee image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, 2021.
[46] Stallkamp, M. Schlipsing, J. Salmen, and Igel. The german traffic sign recognition benchmark: multi-class classification competition. In International Joint Conference on Neural Networks (JJNN)) 2011.
[47] S. Swayamdipta, R. Schwartz, N. Lourie, Y. Wang, H. Hajishirzi, N. A. Smith, and Y. Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. arXiv:2009.10795 2020.
[48] B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li. YFCC100M: The data in multimedia research. Communications the ACM, 2016.
[49] Y. Wang, Y. Chen, W. Yan, A. Fang. W. Zhou, K. Jamieson, and S. S. Du. CLIPloss and norm-based data selection methods for multimodal contrastive learning, arX1v:2405. 2024.
[50] Xiao, K. A. Ehinger, Hays, A. Torralba, and A. Oliva. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision (IJCV), 2016.
[51] H. Xu, S. Xie, X. Tan, P.-Y. Huang, R. Howes, V. Sharma, S.-W. Li, G. Ghosh, L. Zettlemoyer, and C. Feichtenhofer. Demystifying CLIP data. In International Conference on Learning Representations (ICLR), 2024.
[52] Xu, Y. Song, D. Wang, W. Zhao, M. Chen, K. Chen, and Q. Li. Quality over quantity: Boosting data efficiency through ensembled multimodal data curation. arXiv:2502.082/1 2025.
[53] H. Yu, Y. Tian, S. Kumar, L. Yang, and H. Wang. The devil is in the details: A deep dive into rabbit hole of data filtering. arXiv:2309. 15954, 2023.
[54] X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, Djolonga, A. Pinto, M. Neumann, A. Dosovitskiy, L. Beyer, O. Bachem, M. Tschannen, M. Michalski, O. Bousquet, S. Gelly, and N. Houlsby. The visual task adaptation benchmark, 2019.
13

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
Our claims are supported by experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations and future work are discussed Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
14

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide the codebase needed to reproduce all of our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce
15

---

the model (e.g., with an spen-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the codebase needed to reproduce all of our results, along with instructions on how to execute it.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ public/ httec//ggieeec/eeeeeeeee SoooeeSumisssoonPP olicy) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //n /nippscc/pppii http:cccppppccccceeeeeee.le..cct httpp//wwwpguuuueeeeeeeee/ee/e/e wewggguddd/eeeeee
htteggguddeee/dduubiiieeeeeeee hat////oooSuissseeeeee wwabbbmmsssoooooooio ssssoniiicyy)) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and test details are discussed in Section 4, and provided in more detail in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
16

---

7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
The experiments are too computationally expensive to repeat using multiple seeds, but we see the metrics vary smoothly across different hyperparameters (e.g., in Figure 3) indicating that the results are statistically significant.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Computational cost is discussed in Appendix C.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
: /////nnuippp http://pp.....ppppe.//ecccccccaa
17

---

Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics and believe that the research conducted in the paper conforms to it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification:
There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring
18

---

that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The resources used in the paper are properly credited and used exactly follow- ing the DataComp benchmark, which respects their license.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi theode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: carefully documented all the assets we release
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
19

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.LM for what should or should not be described.
(https htt.eerrrpp..cccooooeeeeee//////
20

---

A Implementing data-parallel distribution
CLIP training benefits from a large batch size [35, 7J. FLYT trains a CLIP model while weighting each example compared others in the batch, which also requires sufficiently large batch size. However, training FLYT requires more memory than training a standard CLIP model since we have to keep the computational graph from ø to Ldown in order to calculate VoLdown. This means that in order to train FLYT on large scale, we need to implement data-parah pataaaplalllll distributed training.
Our goal is to compute V VoLdown using data parallelism across multiple GPUs, which is not trivial considering FLYT introduces different dependencies between examples in the batch. We start by using the chain rule write:
18wi:B 0Φ
V Lownn
V VWI:B V Lrdown:
and focus the second factor
whe8 down
= V V
where
0+ (W1:B) = update(((wr.z))
with
((W1B) = (flwcI(f(EE1.B) W1.B)
and update(-) is an optimizer update function. We then use the chain rule again write:
Ög
wy:n Ldown
V
@wi:B
We may further apply the chain rule (and the fact the partial derivatives commute) to write
Ölwer (FI. w1.B) Vofo(z))
Of:
where f; is shorthand for the embedding fo(2;) that implicitly says we only need to store the value of the embedding and not the computational graph behind it. This way each GPU calculates g for its own sub-batch, and we can then gather and sum their result to get for the full batch using the CLIP gradient accumulation technique from Cherti, Mehdi et al. [7]. Using this, define
V LD (fupdate(g) (21. (21.B)))
which we can again compute using gradient accumulation. Treating as a constant vector independent of WI.B. the gradient we would like to calculate is
o(g.v) Dw1.B
ØlweL (f1.B. W1.B)
WL.B Iidown
(VSJ(zz),
OWI.B
Of.
EE[B]
If we define the function
dewor W1.B)
ha(n)
fo+mo
05.
Then our gradient is simply
Litoon
1((0).
Own:
(E(
Which is amenable to gradient accumulation.
The first factor
sosotmax
OΦ
d6
can be calculated separately using gradient accumulation, and we get
(@w1.B
Looonn
V Laoon-
21

---

Table 5: DFN-Base reproduction experiments.
Initialization
Num. samples
Notes
Training datasct
ImageNet
Average
Random
10M
CC12M
0.230
0,300
Random
50M
CC12M
0.274
0.324
Random
128M
CC12M
0.281
0.328
OpenAl
128M
DFN-Basc
CC12M
0.297
0.333
OpenAl
128M
CCI2M+CCM+SS115M
0.286
0.334
OpenAl
512M
2x Batch Size (8192)
CC12M
0.296
0.338
Table 6: DFN-FT reproduction experiments
Initialization
Num. samples
Notes
Training dataset
ImageNet
Average
DFN-Base
400K
DEN-FT
INIK+MS COCO+Flickr30k
0.342
0.348
DFN-Base
1.6M
INIK+MS COCO+Flickr30k
0,340
0.348
DFN-Base
3M
INIK COCO+Flickr30k
0.335
0.338
DFN-Bat Base
400K
1/10 LR (5e-5)
INIK+MS COCO+Flickr30K
0.321
0.344
DFN-Base
IM
1/10 LR (5e-5)
INIK COCO+Flickr30k
0.324
0.342
DFN-Base
12M
1/10 LR (Se-5)
IN COCO+Flickr30
0.319
0.340
DFN-Bec -ase
800K
INIK
0.333
0.343
B M-FLYT input details
In order experiment with M-FLYT, reproduce some existing scoring methods and create some new ones:
1. CLIP scores. We use the CLIP scores calculated using OpenAI ViT-B/32 and ViT-L/14 CLIP models [35].
2. DFN. The results of Fang et [16] are proprietary and we do not have access to their best DFN model or the scores created by it. The authors provided version that uses only publicly available resources, but it was not trained using all the improved techniques they discovered. Thus, we reproduced their work using only publicly available resources. We use the CLIP score DFN networks: one before fine-tuning on ImageNet, and one after. Appendix B. provides the details of our reproduction.
3. HYPE. Kim et al. [22] provided a PyTorch implementation which includes their hyperbolic CLIP weights, and the functionality required to reproduce the image specificity c, text specificity 6,, and negative Lorentzian distance dpp- We used these three values as inputs.
4. s-CLIPLoss. Wang al. [49] also provided PyTorch implementation. We used OpenAl ViT-B/32 embeddings to calculate s-CL.PPLos. and NormSim,
5. Binary classitiers. We trained two línear binary models that elassify between "high quality" dataset and the DataComp medium scale dataset. We describe them in Appendix B.2 below.
6. E-FLYT. We use our E-FLYT-Baad scoring model, detailed in Appendix C.7.
B.1 DEN reproduction
Since M-FLYT requires good inputs perform well, we made effort train high performing DFN. In the paper, Fang et al. [16] trained ViT-B/32 CLIP model initialized with OpenAl's checkpoint on the high-quality private dataset HQITP-350M. Since we do not have access this dataset, we trained our DFN on CC12M [41]. Similar to Fang et al. [16], we then fine-tune on the combined training sets INIK with sampled OpenAl templates as texts, MS COCO, and Flickr30k. Unless otherwise specified, we use the same perrat-prreestrr as DataComp medium scale training. Tables 5 and 6 show results we obtained with different hyperparameter choices when our reproductions DFN-Base and DFN-FT, respectively.
22

---

Table Detailed hyper parameters of the different configurations used for training and ablations of M-FLYT and E-FLYT. * Initialization "DataComp medium" means we train model from scratch on the unfiltered DataComp medium scale dataset using the DataComp medium scale training configuration.
Parameter
E-FLYT-Basc
E-FLYT-22
E-FLYT-LR
M-FLYT-Base
Optimizer
AdamW
AdamW
AdamW
AdamW
Weight decay
0.2
0.2
0.2
(81,84)
(0.9, 0.98)
0.98)
(0.9, 0.98)
(0.9, 0.98)
Scheduler
cosine
cosine
cosine
cosine
Reference model LR
Se-5
5e-5
Variable
5e-5
Scoring model
le-3
Variable
le-3
Feature extructor init
OpenAl
OpenAl
OpenAl
None
Reference model inil
OpenAl
OpenAl
OpenAl
DataComp medium"
Downstream
CE temperature
Downstream batch size B'
3072
768
1536
3072
Upstream batch size
4096
4096
4096
4096
Num, downstream samples
7.7M
Num. upstream samples
20.5M
20.5M
20.5M
20.5M
Num, steps
5000
5000
5000
5000
Num. warmup steps
100
100
100
100
Downstream dataset
ImageNet
22 tasks
ImageNet
ImageNet
Upstream dataset
DataComp medium
DataComp medium
DaraComp medium
DataComp medium
Trinable reference parameters
all
all
Trainable scoring parameters
all
all
all
all
B.2 Binary classifiers input
When training our top performing M-FLYT, we use two binary linear classifiers. These models take input embeddings generated by OpenAl's ViT-B/32 model and predict which two datasets these embeddings come from. I-Classsieee classifies between the ImageNet training set and the Data- Comp medium scale dataset image embeddings. image and text embeddings of the DataComp medium scale dataset and CC2M. CC2M is dataset of 2M examples we created by running our INIK -Classiiff on the CC12M dataset [41] and taking the top 2M examples that the model classified as ImageNet examples. As shown in Table 2, while INIK-Classifer underperforms compared to CLIP score filtering, still demonstrates improvement over no filtering and receives substantial weighting from M-FLYT. In contrast, achieves performance similar to CLIP score filtering but receives a lower M-FLYT weighting. We trained them for 40K steps with batch size of 256, cosine learning rate with maximal value of 0.01 and 500 warm-up steps.
classifies between concatenated
C Experimental setup
C.1 Experiment configurations
When training M-FLYT and E-FLYT, we optimize the mixing model, the embeddings scoring model, and the reference models using AdamW [26] with a weight decay of 0.2. (81, (2) (0.9, 0.98), and a cosine scheduler [25]. Due to computational constraints, some experiments compare against different model configurations. We list their full hyper parameters in Table 7 and describe their main differences here. We refer to the configuration from Section 4.2 as M-FLYT-BBaR Base, and to the E-FLYT configuration used in the experiments described in Appendix C.7 as E-FLYT-Bar -ase. E-FLYT-22 maintains the same parameters as E-FLYT-Base. with three key differences: it employs the standard CLIP loss as the downstream loss, uses downstream batch size of 768, and trains with 22 downstream tasks. E-FLYT-LR uses the same configuration as E-FLYT-Base. except it employs the CE loss without temperature and uses a downstream batch size B' of 1536; this configuration was used exclusively for learning rate experiments.
C.2 Computational cost
In terms of FL.OPs, M-FLYT requires slightly more than the FLOPs of regular CLIP model training per example, while E-FLYT requires slightly more than 3x. Since only train our models
23

---

Table Experiments using the M-FLYT-Bass configuration (Table
Experiment
Bascline
Variation
ImageNet
Average
M-FLYT
0.359
0.371
Downstream loss
CE
+ temperature
0.352
0.366
Random initialization
0.341
0,354
Reference model initialization
Trained on DC medium
OpenAl
0.342
0.355
Input standardization
Yes
No
0.323
0.337
Table 9: Comparison of training M-FLYT with all input methods vs leaving one method out. Numbers in parentheses show performance degradation compared using all inputs. We highlight values with the largest degradation, indicating which methods provide the most benefit when included.
ImageNett shifts
Inputs
ImageNet
VTAB
Retrieval
Averago
Inputs
0.359
0.294
0.383
0.310
0.371
No VIT-B/22. VIT-L/14
0.355 (0.004)
0.288 (0.006)
0.367 (0.016)
0.307 (0.003)
0.360 (0.011)
No DFN-Base, DFN-FT
0.341 (0.018)
0,280 (0.014)
0.375 (0.008)
0.281 (0.029)
0.358 (0.01
No -d,,, fi.
0.351 (0.008)
0.291 (0.003)
0.366 (0.017)
0,308 (0.002)
0,364 (0.007)
s CLIPLoss, NormSim,,, (INIK)
0.354 (0.005)
0.292 (0.002)
0.377 (0.006)
0.307 (0.003)
0.364 (0.007)
INKKCClasifeer CC2M-Clasiieer
0.352 (0.007)
0.288 (0.006)
0.368 (0.015)
0.301 (0.009)
0.355 (0.016)
E.FLNT
(0.007)
0.292 (0.002)
0.376 (0.007)
0.309 (0.001)
0.366 (0.005)
for 20M examples (compared to DataComp's 128M), M-FLYT uses approximately 3x fewer FLOPs than training DataComp medium scale model, and E-FLYT approximately 2x fewer.
With our implementation, M-FLYT takes 40 A000 hours to train, and E-FLYT takes A100 hours, roughly equivalent to running DataComp medium scale experiment. For SCS with batch size G 100K, takes about 100 minutes to sample 128M examples using naive Python implementation single CPU. For larger scales (DataComp large Xlarge), the filtering computation becomes negligible relative to the overall training cost.
Comparing to other filtering methods:
Fang al. [16] reported (in Table 2) training their ViT-B/32 DFN for 5.12B samples seen, which requires approximately 80-120x more FLOPs than our method
Kim et al. [22] trained ViT-L/14 CLIP model on V100 GPUs for 256M samples, taking 488 GPU hours (61 hours x 8 GPUs)
Wang et al. [49], which doesn't require additional model training beyond the pretrained OpenAl CLIP model, reported 5 hours of preprocessing on an I.40 machine
C.3 M-FLYT ablations
Table shows experiments using M-FLYT. The standard CE loss slightly outperforms CE with temperature (Appendix C.8), and reference model initialized with "DataComp medium" (explained in Appendix C.7) outperforms OpenAl initialization. We also find that input standardization greatly increases performance.
Table shows the result of training M-FLYT while excluding groups of related inputs. Removing any input group degrades performance, with several notable results: DFN reproduction scores have the strongest impact on ImageNet, ImageNet distribution shifts, and retrieval scores (degrading performance by 1.8%, 1.4%, and 2.9% respectively). For VTAB tasks, excluding HYPE scores causes the largest drop (1.7%), while unexpectedly, removing the linear classifiers- being the weakest standalone filtering methods -eads to the largest decline in average performance (1.6%).

---

Comparing M-FLYT the original mixing method of HYPE [22] and s-CLIPLoss [49]. HYPE combines scores by summing their three components (-d,, + 6, + 6,), while s-CLIPLoss applies a two-stage filtering process: selecting the top 30% based on s-CLIPLoss scores, then selecting the top 20% according to their NormSim scores from this filtered subset.
Table 10:
Input method
Mixing method
ImageNet
Average
Original (sum scores)
0,309
0.341
HYPE
M-FLYT
0.322
0.347
Original (two stage)
0,331
0,363
S CLosL and NormSim
M-FLYT
0.334
0.357
C.4 Comparing M-FLYT to aggregation methods in prior work
Table 10 compares M-FLYT against the original mixing strategies of HYPE and s-CLIPLoss. For HYPE's components, M-FLYT outperforms the simple summation approach both ImageNet (+1.3%) and average (+0.6%) metrics. When mixing S-CI.PLLoSS scores, M-FLYT performs similarly to the original two-stage filtering method, showing marginal improvement ImageNet (0.3%) but slightly lower average performance (0.6%).
C.5 M-FLYT baseline ImageNet-weighted standardized sum
Here we explain the SaageNet-weightee standardized sum baseline we compare against in Section 4.2. For K' ImageNet accuracies IN corresponding to K scoring methods, we first standardize the per-example scores produced by these methods individually across all training examples to have zero mean and unit standard deviation. We denote by s/ the standardized score that method I,.., gave example 2j.
We then normalized the ImageNet accuracies IN l:K between zero and one, and add scalar to achieve our desired ratio
IN, min min(IN
0; max max 1:k) min(INI.K)
w; w;
where min (IN1:K) is the lowest ImageNet score, and max max(IN1:k) (IN1:) the highest. This ensures that the ratio between the maximum and minimum weights equals
max(wi:k)
min(w1:k)
The final score for example zj is then Σ w
In Table 2 we experiment with max-to-min ratios r E 42,4,8,16
C.6 E-FLYT learning rate tuning
Table 11 compares different scoring model and reference model learning rates using the E-FLYT-LR configuration. We notice that E-FLYT is not very sensitive to the scoring model LR (with reference fixed at le-4) and select le-3 based on slightly better results. E-FLYT is somewhat sensitive to reference model LR, and with scoring model LR fixed at le-3, 5e-5 gives the best results. These values are used for subsequent experiments.
C.7 E-FLYT results
For E-FLYT, model. We initialize both the feature extractor and the reference model with OpenAl's checkpoint.
CLIP VIT-B/32 [35] as the feature extractor and FFNGLU [42] scoring
In Table 12 we show experiments using E-FLYT-Base We evaluate three alternatives for the downstream loss function: The CLIP loss, as defined in Equation (1) lc(Je (wI.B) ($I.F)) standard cross entropy (CE) loss, and CE with temperature parameter (see definitions in Appendix C.8). Our experiments show that CLIP loss and CE with temperature demonstrate similar performance, both outperforming the standard CE loss. For subsequent experiments, we CE with temperature
25

---

Table 11: Learning rate tuning using the E-FLYT-LR configuration (Table 7). We first optimize the scoring model LR with reference model LR fixed at le-4, then optimize reference model LR with scoring LR fixed at le-3.
Scoring model LR
Reference model LR
ImageNet
Average
5e-4
le-4
0.286
0.300
le-3
le-4
0.286
0.306
5e-3
le-4
0,280
0.310
le-2
le-4
0.277
0.300
le-3
le-5
0.158
0.238
le-3
5c-5
0.292
0.319
le-3
le-4
0.286
0.306
le-3
5e-4
0.267
0.296
because it better resembles the accuracy metric used to evaluate most DataComp tasks. We found it essential to separate learned temperature parameter for the upstream and downstream loss functions when using either the CLIP or CE with temperature loss. We initialize the downstream temperature parameter TDS 1/0.07, as the default when training CLIP model from scratch, while the initial value for the upstream temperature is set to 100 as part of the pretrained reference model weights. The downstream temperature is updated during training using Vns Ldown-
Table 12 also compares reference model and feature extractor initializations, where initializing with stronger models outperform weaker ones. We evaluate four initializations in ascending order of performance: Random initialization, "DataComp medium", where train model from scratch on the unfiltered DataComp medium scale dataset using the DataComp medium scale training configuration, OpenAl checkpoint, and "DataComp-133" which is model trained by Gadre et al. [17] on the DC-IB dataset for 13B samples seen. Further results in the table show that training the reference model much better than freezing and that fine-tuning the feature extractor prevents the training from succeeding. Our ablation studies show that performance remains stable with downstream batch sizes ranging from 1024 4096. Similarly, performance is consistent when training for 1250 to 12500 steps, suggesting that E-FLYT could be optimized using less compute.
As discussed in Section 5, believe that E-FLYT training stack has significant room for improvement.
C.8 Downstream loss
We now explain how construct the downstream loss from a downstream batch 21. containing labeled data of the form 2; = (I;, 4)), where I; is an image and c, E {1,..., ....,,K K} is a class label. First, we uniformly sample templates for each of the classes following DataComp' 's class-specific template definitions t1:k sample(1 K). Then, the CE loss is given by
B'
(DS(21:B))
Σ log
> Σ (0os(
1=1
where 7DS is learnable temperature parameter only use in the CE+temperature variation for the downstream data. Using the standard CE loss, TDs is set to 1. Using this notation, we define the downstream CLIP loss mentioned in Appendix C.7 with separate temperature parameter rDS as
B'
e mmsttaaa
B'
gos((altt,),ff(an)) log
fc(21.B)) =
log
1=1
=1
1=1
==
C.9 E-FLYT with diverse downstream data
Instead of using the ImageNet training set alone for the downstream data, we experiment with using 22 training sets from the DataComp benchmark. We now detail the training configuration, and then discuss the results of experiments using E-FLYT-22. First, list the datasets used:
26

---

Table 12: Experiments using the E-FLYT-Baa configuration (Table 7). Initialization "Trained on DC-IB" means using the ViT-B-32 model trained by Gadre et al. [17] on DC-1B for 13B samples seen (Hugging Face model card).
Experiment
Baseline
Variation
ImageNet
Average
E-FLYT-Bas
0.316
0.323
CE (no temperature)
0.287
0.305
Downstream loss
CE temperature
CLIP
0.313
0.330
Random initialization
0,270
0.312
Reference model initialization
OpenAl
Trained on DC medium
0.290
0,308
Trained on DC-1B"
0.314
0.320
Random initialization
0.157
0.234
Feature extractor initialization
OpenAI
Trained on DC medium
0.287
0.302
Trained on DC-1B"
0.325
0,329
Reference model + feature extractor initialization
OpenAl
Trained on DC-1B"
0.326
0.331
Only scoring model
0.251
0.298
Trainable parameters
Scoring reference model
Feature extractor
0.135
0.227
12500
0.306
0.320
Num. training steps
5000
1250
0.311
0.329
1024
0,315
0.313
Downstream Batch Size
3072
4096
0.31
0.316
ImageNet [12]
MNIST [11]
SVHN [31]
CIFARIO [24]
CIFAR100 [24]
Food-100 101 [4]
Oxford Flowers- 102 [32]
Oxford-III Pet [33]
iWildCam [3]
SUN-397 [50]
EuroSAT [20]
Stanford Cars [23]
DTD [9]
RESISC45 [6]
GTSRB [46]
FGVC Aireraft [30]
Pascal VOC 2007 [15]
Country211 [35, 48]
Rendered SST2 [54]
FMoW [8]
27

---

Table 13: Different downstream tasks experiments using the E-FLYT-22 configuraaioooooooonn (Table 7).
Downstream tasks variation
ImageNet
Average
Imbalanced 22 tasks
0.315
0,330
Balanced 22 tasks
0.274
0.311
Only ImageNet
0.313
0,330
Table 14: SCS parameter ablations: Effects of repetition penalties and batch sizes.
Repetition penalty a
Batch size
ImageNet
Average
0.10
100K
0.395
0.369
0.15
100K
0,401
0.377
0.20
100K
0.399
0.377
0,25
100K
0.395
0.378
0.30
100K
0.396
0.374
0.40
100K
0.387
0.373
0.50
100K
0.388
0.376
0.60
100K
0.377
0.369
0.15
10K
0.399
0.373
0.15
0.398
0.371
Dollar Street [38]
STL-10 [10]
When preparing our datasets, we split them into tar files of 100 examples each webdataset format. During training, tar files at once, small shards and a large shuffle buffer help ensure diverse tasks within each training batch. The combined dataset contains 2M examples, with ImageNet making up the majority. In Table 13, we compare different dataset configurations. Using the combined dataset without balancing classes yields performance very similar using only the ImageNet dataset. When we balance the datasets to ensure equal representation (where the model sees samples from each downstream dataset the same number of times). performance deteriorates. This performance drop might be due some tasks containing as few as 1020 total training samples, potentially causing overrepresentation of these smaller datasets during training.
shuffle buffer of 50,000 examples. Since the webdataset loader reads entire
C.10
SCS ablations
With SCS, we evaluate different repetition penalties and find that performance varies smoothly with a, reaching optimal results around a 0.15 (Table 14). For sampling efficiency, introduce the batch size G, setting to 100K. This value is large enough for efficient medium scale sampling yet sufficiently small based opportunities to be selected, our most frequently sampled example appears only 56 times. To more directly validate choice of G, Table 14 shows that using values of G of different order of magnitude has little effect on performance.
empirical observation that while each sample had 1280 (128M/ /100K))
C.11 DataComp small scale results
For the small scale DataComp benchmark, we reuse the mixing model from the medium scale without retraining E-FLYT or M-FLYT. The only changes were re-tuning the repetition penalty for the smaller dataset and reducing the sampling batch size G by factor of 10 to 10K, aligning with the dataset size reduction. Figure shows that the smaller scale performs best with a higher repetition penalty, around a 0.5, compared to the medium scale which performs better with values between 0.15 0.25. higher repetition penalty reduces the number of repetitions per example, which becomes more important for smaller datasets.
28

---

ImageNet Accuracy
Average Accuracy
0.080
0.185
Accuraa) Accuracy
Average Accuracy
0.180
0.075
0.175
ImageNet
0.070
0.065
N Unique Samples (M)
N Unique Samples (M)
Soft Capp
Threshold
Comparing SCS to hard threshold filtering on the small scale DataComp benchmark. SCS was tested with a values from 0.15 I, and standard threshold with values from 10% to 30%
Figure 4:
Table 15:
Comparison between the results reported in the original papers and our reproduction using their provided UIDs. Some experiments were run using the complete DataComp medium scale dataset (128M) while our reproduction used the 11M examples marked with were performed by Wang et al. [49] on their I 110M downloaded dataset; they also report partial results using the full 128M data pool. Our best results are listed below.
were able to download. Results
ImageNet dist. shifts
Filtering Method
Pool Size
ImageNet
VTAB
Retrieval
Average
110M
0.324
0.274*
0.359*
0.263*
0.352
s C-LILLoss
119M (Reprod.)
0.333
0.273
0.361
0.251
0.352
128M
0.371
0.298
0.388
0.288
0.373
DFN
119M (Reprod.)
0.367
0.303
0.371
0.280
0.364
116M
0.382
0.303
0.393
0.306
0.379
HYPE*DEN
119M (Reprod.)
0.372
0.304
0.374
0.287
0,368
128M, 110M
0.375
0.309
0,386*
0,281
0.386
DFN+s-CLIPL.OsS
119M (Reprod.)
0.371
0.304
0,390
0.285
0.378
128M, 110M
0.382
0.314"
0.385*
0.276*
0.388
HYPEE-DEN+S-CLPLLOS
119M (Reprod.)
0.381
0.310
0.392
0.284
0.378
M-FLYT+SCS (a = 0.15) 119M
0.401
0.311
0.396
0.292
0.377
D Reproduction comparison
Table 15 compares model we train using publicly available DataComp subsets to the numbers reported in the papers that published them. The results are generally similar, but not identical, with accuracy difference of up 1%. The main driver the difference appears to be the size of the actual pool used to train the model: we were able to download only 119M of the 128M examples in the DataComp medium scale pool, while the authors of the other papers had access more complete subset the examples.
E Qualitative analysis
Figures 5 to show the top-scoring examples for M-FLYT, standard CLIP score, and DFN-FT. We observe that M-FLYT prioritizes examples containing multiple ImageNet classes. For instance, the highest-scoring example features the caption "Honey the Golden Retriever stands in tent," where both "Golden Retriever" and "tent" correspond to ImageNet classes. In contrast, the CLIP score filtering method using OpenAl's ViT-B/32 model favors images with visible text that matches the caption text.

---

Horwy Golden Baitiweet stands in tent.
four Lebratdor supples sitling bastel photo
Many ower: like this Mersche l-Sp pilmmmnn one at the 1928 cartodvad Museum Rapids, Bennett, antiers camuseels orgi
Pug Abandonee Sheel
kitten peers squarre
cesert flowers growing in monastery
Bemess Mountein Dog Seimming her Ruffhecar Lifabcket, KS Float Cook in Lake Papin on the City,
christmaa hat: Pag red chrtstmas hot holding pointing stick and showing thumbs up. sackground.
Sanchh I1
G# Retriever Dog Two Pupples Laundry
prisn: Silhowette of selider on fence and dog
RUGIE! Golden Retriever puppy log www.ggoooeooynnddddm...
Mahaganyyle Meattlyltmoon Fremed, Sine: 23Guuiii Hix 236qunnt; W 15204308
Figure 5: M-FLYT top scoring examples.
30

---

HIPSTER ARIEL
Marti
HSS odedng Occument Dreesing
Sahaddy 2015
ONLY LISTENS TO UNDER SEA ON VINYL
Platón Mateo Deflends
oely
Istans under the viny!
Raten Matuo Se Deffende (The Matt the hatt Series La Serie de Ratónn Maten) llnglish and Spanish
Saturday, April 25, 2015
hipster
Town Morea Auttion Mac 2011
"CI-TTD Projects
PROGRESSO
Ieddect the San Misertatonn
CHICK PEAS
Dowriooad Small: Town Morss I 1SMALL- MOMS Transtao, juret Market Paperbound
CaH-IT22 Projects with Sun Microsstaams Inviled
Progresso Click Pous 190z Can prodact image
Microsyttems Supercomputin 2004 Pittsburgh. PA...
And Public Senten
Predator- Prey Cycles
Academy
pray cycle Exelogy)
Blaine: With Seetch of The Life Of General John
Logan (05484501220) by Rispath, John
DOROTHY GILMAN
TOTAL QUALITY MANAGEMENT
TOOL5 OF TOM (2)
THE UNEXPECTED MRS. POLLIFAX
VANITY CASK April 2018 Heattaef
Aboula Jassin Abdulla
TOTAL QUALITY MANAGEMENT TOOLS OF TOM (2) Abdull Jassim Abdula
The Unexpected Mrs, Pellfba Audicbook. by Dorothy Giman
VANITY Aprl 2018 II unbosing in Tekugu || shadees
Figure 6: CLIP ViT-B/32 top scoring examples.
31

---

STAR
Baby Bbowee
Disneyaies
Saturi vector boowam cherry creeling cutds with spring sink blooming illuitationn flowering tempiste pirpatee decoration isatated on white background. Vicctore us
Elephant Babry Shower Ireiaalions Vellow duapp: Gray
Star Warre The Force Assabans Pin Trading Bouster Sei Danry Parts
Sauth
Shawee
wlephent butt shower
Sunfiroers Shower Vintage Invitation, Gray. Brown. Mason Sunffower, Wedding Shower, 51.45
Rairy Read Landscape
grey
tterst
heart waker splash- Sl0ck photo
Green Nature Abstract Beckground Convas Print Painting Home Decor Wall Art SPC5
person: Mechor disbility symbels blue and white illustration
2019 HAPPY NEW YEAR
Rodel Shrwej
fualy Hkge 26 2112
Inscription Happy new year 2019 on backgrou.nd nd
Wholesale Original Pointing Peimiing caruae alion print poster pichure cances oriinal paintngs
Burgundy Marsala Red Floral Autumn Invitations
Figure 7: DFN-FT top scoring examples.
32


Reviews:
[{"review_id": "J4aikxsGU9", "reviewer": "Reviewer_mNXk", "summary": "This work proposes a method to filter / curate large-scale image-caption datasets for training CLIP-like models. The method consists of two components:\n1. A learnable scoring function (M-FLYT) that estimates how useful a data example is for training. It takes as input scores from 12 existing scoring methods and learns to a linear combination of them. The scoring function is learned using a meta-learning strategy that learns the scoring function weights as as to improve the zero-shot classification performance (on ImageNet) of a reference CLIP model.\n2. A sampling approach (SCS) that selects training examples based on the learned scores (from M-FLYT). It samples with replacement but adds a penalty $\\alpha$ each time an example is selected. This prevents extensive repetitions of examples with high scores. \n\nThe method and its individual components are evalauted on the DataComp medium-scale benchmark. It outperforms previous works, achieving 40.1% accuracy on ImageNet zero-shot classification and 37.7% on average across DataComp tasks.", "questions": "Please answer my comments and questions put under the \"Weakness\" section above.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**    \n- The paper is overall easy to understand and well-written \n- The proposed meta-learning approach is conceptually simple (but hard to implement) and the idea of meta-learning the scoring function so as to explicitly improve the downstream task performance makes sense. Furthermore, the proposed sampling approach SCS is simple, easy to implement, and effective. \n- The work demonstrates that both M-FLYT approach and the SCS sampling approach improves the results compared to the baselines. \n\n**Weaknesses:**\nAlthough the proposed meta-learning-based FLYT approach is conceptually simple, implementing such meta-optimization techniques is complex (see, for example, the discussion in Appendix A), while the gains in Tables 1 and 2 appear modest. M-FLYT shows relatively small improvement over \"DFN-FT [our reproduction of 16]\" (compared to the impact of SCS). Moreover, the performance gap between M-FLYT, which mixes individual scores using a computationally heavy and complicated meta-optimization stage, and the simpler, training-free \"IN-weighted\" method (Table 2, right) is even smaller.\n\nTable 1 suggests that the SCS sampling strategy has a much larger impact on performance than M-FLYT. It is likely that using SCS with just the \"IN-weighted\" baseline (or even DFN-FT) would yield strong results, close to \"M-FLYT + SCS\", while being far simpler. Could the authors provide results without M-FLYT or E-FLYT to confirm this? Such results are necessary to judge the practical usefulness of FLYT.", "was_revised": true, "final_justification": "The rebuttal generally resolve my concerns. While the improvements from the proposed methodology are modest compared to some strong baselines, they appear consistent and worthwhile. After also reading the other reviews and authors' responses to them, I maintain my positive assessment and recommend acceptance.", "rebuttal": "We thank the reviewer for finding our paper well-written and easy to understand, for recognizing the soundness of our meta-learning approach, and for acknowledging the effectiveness of our proposed methods. Below, we address the weaknesses raised in the review.\n\n > implementing such meta-optimization techniques is complex\n\nWe agree that implementing FLYT is not trivial, but we think that this makes our implementation (which we open source) interesting to the community.\n\n> (...) M-FLYT, which mixes individual scores using a computationally heavy and complicated meta-optimization stage\n\nFirst, we note that the meta-optimization stage occurs only during the scoring model training, which we run for just 20M examples compared to the 128M used in DataComp medium scale. As detailed in Appendix C.2, the entire M-FLYT training process (the meta-optimization stage) requires 3x fewer FLOPs than training a DataComp medium scale model. At larger scales (DataComp large or Xlarge) this becomes negligible relative to the overall training cost. Moreover, our learned M-FLYT model is a linear model, which means scoring each example is very fast and computationally inexpensive.\n\n> (...) the gains in Tables 1 and 2 appear modest. M-FLYT shows relatively small improvement over \"DFN-FT [our reproduction of 16]\" (compared to the impact of SCS). Moreover, the performance gap between M-FLYT (…) and the simpler, training-free \"IN-weighted\" method (Table 2, right) is even smaller.\n\nWe invested heavily in strengthening our baselines: “ImageNet-weighted” is not a naive implementation. Specifically, the method is (as detailed in Appendix C.5):\n\n* Standardize the per-example scores produced by each method\n* Normalize the ImageNet accuracies of these methods to [0,1]\n* Add $1/(r - 1)$ to these normalized accuracies to achieve a desired min to max ratio $r$\n* Compute the weighted sum per example\n\nImageNet-weighted also requires tuning the hyper parameter $r$. It outperforms more naive baselines such as standardized sum, yet M-FLYT still outperforms it. While we agree the improvement is modest, it is nonetheless clear. When taking into account our previous point that M-FLYT is in fact not computationally expensive, as well as our next point about the built-in calibration of M-FLYT scores, we believe these improvements make M-FLYT worthwhile.\n\n> It is likely that using SCS with just the \"IN-weighted\" baseline (or even DFN-FT) would yield strong results\n\nA key advantage of the FLYT training algorithm is that it naturally produces a distribution over the training examples. Other methods such as DFN-FT or IN-weighted provide scores that are not easily translated to probability distributions.\n\nThat said, we agree with the reviewer that this claim should be supported by experimental results. We ran an experiment of SCS with a repetition penalty $\\alpha = 0.15$ on the scores produced by IN-weighted with $r=8$, converting the scores to a probability distribution by applying softmax (following the same approach as with M-FLYT), and present our results in the table below. We will include these results in our revised paper. \n\nFor easy comparison, the table also includes IN-weighted with $r=8$ without SCS, in addition to M-FLYT with and without SCS.\n\n| Experiment | ImageNet | Average |\n| :--------- | :---: | :---: |\n| IN-weighted + SCS | 0.379 | 0.362 |\n| IN-weighted | 0.353 | 0.365 |\n| M-FLYT + SCS | 0.401 | 0.377 |\n| M-FLYT | 0.359 | 0.371 |\n\nWe believe these results could be improved by introducing a new scale hyperparameter where the distribution becomes softmax(scale * scores). However, this would be significantly more computationally demanding, requiring hyperparameter sweeps over both the new scale parameter and the repetition penalty of SCS."}, {"review_id": "PYWhw1h8Ex", "reviewer": "Reviewer_sYPG", "summary": "The paper proposes a learned sample importance algorithm for filtering a dataset of varying sample quality to one that can effectively train a CLIP model. In the first phase, they train a model by simultaneously learning the CLIP model itself, along with a scoring model that predicts how important each example in the batch is, based on a downstream task loss. The scoring model is of primary interest, as once this phase finishes, they then use it to construct an N (e.g. 128M) example dataset (potentially with duplicates). To allow for duplicates, they propose SoftCap sampling, which smoothly degrades the importance of previously sampled examples, allowing them to be sampled again, but at reduced likelihood. Further variations and improvements are proposed, however the thrust of the idea remains.", "questions": "Can you explain how the \"Average\" column in Table 1 is computed? It's clearly not a simple mean over the former fields. Particularly, it seems as though \"M-FLYT+SCS\" should have the highest average, given that it beats every other method on the other categories, and yet it has a lower average score than the non-proposed methods above.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n* The method is minimal with heuristics, instead relying on the learned scoring model.\n* The proposed model does exceedingly well, beating contemporary methods not only on the downstream task (ImageNet), but also other tasks.\n* The proposed method doesn't rely on private datasets\n\n### Weaknesses\n\n* Not levied at this paper alone, but the method does directly target ImageNet by using it as the downstream task, so generalization appears to rely on the generalizability of the concepts in ImageNet. This is where the concurrently proposed MGD approach, utilizing all tasks, may result in more robust vision foundation models (of which CLIP is typically considered), since it's not overfitting to ImageNet.\n* The clarity of the paper could be improved. It would perhaps be nice to read about FLYT first, to understand the algorithm, before additional complexities such as M-FLYT and E-FLYT are introduced. For example, section 3.5 could appear before 3.3 and 3.4.", "was_revised": true, "final_justification": "The method is effective, and the paper is technically solid. it doesn't require private datasets, and is very light on added heuristics. For these reasons, I think it's suitable to accept the work.", "rebuttal": "We thank the reviewer for recognizing what we consider one of the core strengths of our work - minimal reliance on heuristics. We appreciate their acknowledgment of our method's strong performance across multiple tasks, without relying on private datasets. Below, we address the weaknesses and questions raised in the review.\n\n> Not levied at this paper alone, but the method does directly target ImageNet by using it as the downstream task, so generalization appears to rely on the generalizability of the concepts in ImageNet. This is where the concurrently proposed MGD approach, utilizing all tasks, may result in more robust vision foundation models (of which CLIP is typically considered), since it's not overfitting to ImageNet.\n\nOur method can be easily applied using any downstream task. In fact, in Appendix C.9 we experiment with E-FLYT using 22 downstream tasks training sets (similar to MGD). Our initial experiments (Table 12) show no significant difference in performance when using all 22 tasks compared to only using ImageNet.\n\nWe chose to focus on ImageNet since it represents standard practice in the field, as the reviewer notes. More importantly, an argument is usually made in the opposite direction: that using all available training sets of the evaluation downstream tasks leads to overfitting on the evaluation sets, while only using ImageNet as a proxy demonstrates better generalization by showing that we are less dependent on the specific downstream task.\n\n> The clarity of the paper could be improved. It would perhaps be nice to read about FLYT first, to understand the algorithm, before additional complexities such as M-FLYT and E-FLYT are introduced. For example, section 3.5 could appear before 3.3 and 3.4.\n\nWe thank the reviewer for the suggestion on clarity, as it is very important to us, and an outside perspective is extremely useful. While we consider Sections 3.3, 3.4, and 3.5 to be implementation choices, we agree with the reviewer that Section 3.5 (Weighted CLIP loss) is more crucial for understanding FLYT since it introduces a new loss function. In our final revision, we will move this to section 3.3. We welcome any additional suggestions or feedback the reviewer may have regarding clarity.\n\n> Can you explain how the \"Average\" column in Table 1 is computed? ... Particularly, it seems as though \"M-FLYT+SCS\" should have the highest average, given that it beats every other method on the other categories, and yet it has a lower average score than the non-proposed methods above.\n\nThe “Average” column represents the average over all 38 downstream tasks of the DataComp benchmark. Of these 38 datasets, 16 are not represented in any of the other 4 categories. These categories follow the standard evaluation protocol for filtering methods in the DataComp benchmark:\n\n* **ImageNet**: ImageNet classification only\n* **ImageNet distribution shifts**: 6 datasets including ImageNet-V2, ImageNet-O\n* **VTAB** (Visual Task Adaptation Benchmark): 13 datasets including Caltech101, CIFAR100\n* **Retrieval**: 3 retrieval datasets - Flickr30k, MSCOCO, WinoGAViL\n\nAs a concrete case, on the Camelyon17 dataset, which is only part of the average calculation, our method scores 56.8% while another method (HYPE+DFN+s-CLIPLoss) scores 69.2%. That explains how our average score can be 0.1% lower despite outperforming in all other categories."}, {"review_id": "DFV8WUaGFu", "reviewer": "Reviewer_94Jw", "summary": "This paper proposes Filter Like You Test(FLYT), a data filtering algorithm that learns to assign importance weights to each training examples based on their contribution to the downstream task. This paper also introduce Mixing-FLYT(M-FLYT), which integrates scores from multiple existing filtering methods into a unified score using learned combinations, and Soft Cap Sampling (SCS), a sampling strategy that prevents over-representation of high-scoring examples. Together, these methods achieve state-of-the-art results on the DataComp medium-scale benchmark, surpassing previous public-resource approaches.", "questions": "Please address the questions listed in the weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: \n1. The paper introduces a learning-based filtering algorithm FLYT, which use directly optimizes the scoring model based on the performance of downstream task. It can further be extend to M-FLYT, where the learnable parameters are the weight of different existing scoring methods.\n2. By combining FLYT with Mixing-FLYT and Soft Cap Sampling, the paper indeed achieves state-of-the-art results on the DataComp medium-scale benchmark, including a 5.5% improvement in ImageNet zero-shot accuracy over previous public-resource methods. It proves the effectiveness of the proposed algorithm.\n\nWeaknesses:\n1. The Soft Cap Sampling won't eliminate the repetition of sampled data. Could you explain how the proposed Soft Cap Sampling change the distribution of sampled data?\n2. The proposed FLYT algorithm appears conceptually and technically aligned with meta-learning paradigms: concept (meta learning: learning to learn) and technique (optimizing the updated parameters by second derivative). However, the paper does not acknowledge this connection or cite relevant meta-learning literature in the introduction or related work.\n3. The porposed FLYT/Mixing-FLYT may significantly increases the training time (use second-order gradients to optimize scoring model), which is not discussed in the experiments or mentioned in the limitations section of the paper.", "was_revised": true, "final_justification": "I have carefully reviewed the authors' response and acknowledge that it addresses most of my concerns.", "rebuttal": "We thank the reviewer for recognizing the effectiveness of our direct learning-based approach, especially compared to public-resource approaches. We address the weaknesses raised in the review below.\n\n> The Soft Cap Sampling won't eliminate the repetition of sampled data.\n\nWe do not aim to eliminate the repetition of sampled data. Our aim with Soft Cap Sampling is to carefully choose how many times each example is repeated during training.\n\n> Could you explain how the proposed Soft Cap Sampling change the distribution of sampled data?\n\nFigure 2 shows the histogram of the sampled examples using SCS compared to the standard top 20% threshold filtering. We observe that SCS samples more unique examples - 25.7M compared to 23.9M of threshold filtering, while the highest scoring examples are sampled up to 56 times compared to 6 times of threshold filtering. This allows for more diversity in the training distribution while upsampling the best examples.\n\n> The proposed FLYT algorithm appears conceptually and technically aligned with meta-learning paradigms (...) However, the paper does not acknowledge this connection or cite relevant meta-learning literature in the introduction or related work.\n\nIn the paragraph \"Learned data weighting\" of the related work section, we cite [37], [43] which to our knowledge are the two meta-learning papers most relevant to our work and describe their connection with our work.\n  \nHowever, we agree that we should make a clearer connection between our work and meta-learning paradigms in our related work section, and will address this in the final revision. We thank the reviewer for this observation and welcome suggestions for additional especially related papers.\n\n> The porposed FLYT/Mixing-FLYT may significantly increases the training time (use second-order gradients to optimize scoring model), which is not discussed in the experiments or mentioned in the limitations section of the paper.\n\nAs noted in Section 4.1, we discuss in detail the computational cost of our method in Appendix C.2. While second-order gradients are generally expensive, our M-FLYT scoring model requires relatively few training iterations, which overall results in 3x fewer FLOPs than training a DataComp medium scale model. At larger scales (DataComp large or Xlarge), filtering cost becomes negligible relative to the overall training cost. In the revision we will mention this under limitations, though we do not consider computational cost a significant limitation."}, {"review_id": "5ASSLLedFT", "reviewer": "Reviewer_oBzJ", "summary": "This paper presents a data filtering approach for improving CLIP pre-training by automatically identifying high-quality text-image pairs. The authors develop a learned scoring function that predicts how valuable each data point will be for CLIP model training. They implement this approach in two variants - M-FLYT and E-FLYT - which use different feature extraction techniques. Beyond scoring, they introduce SCS, a method that determines the optimal frequency for including each data point in the final training dataset. When evaluated on the DataComp benchmark, their approach outperforms all competing methods that rely solely on publicly available resources.", "questions": "1. Why optimize both the scoring model and CLIP reference model together? This seems like a non-trivial choice. What happens if you just train the scoring model while keeping the reference model frozen? I'm not convinced that jointly training both models will converge properly - as they could interfere with each other during training. It would be helpful to see a comparison with the simpler approach of fixing the reference model to understand if the joint optimization is actually necessary.\n\n2. What exactly does public-resource mean in the context of DataComp? I don't think it was explained in the paper.\n\n3. How do your embeddings differ from the original CLIP embeddings? In what cases are your embeddings better and worse than the original CLIP embeddings?\n\n4. Is there a qualitative boost in performance in addition to the quantitative gain?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper introduces a novel framework for filtering training data for CLIP pre-training.\n\n2. The authors' method demonstrates superior performance compared to other publicly-available approaches on the DataComp benchmark.\n\nWeaknesses:\n\n1. The evaluation lacks qualitative analysis that would clarify the practical implications of the boost in quantitative performance. The paper would benefit from visual demonstrations showing how the new CLIP embeddings differ from those of the original model - for instance, through image retrieval examples or t-sne embedding visualizations. \n\n2. The writing suffers from slight redundancy and could be tightened. Some concepts and findings are reiterated unnecessarily throughout the paper.", "was_revised": true, "final_justification": "After reading all of the other reviews and the authors' responses to them, I have decided to raise my score to a 4.\nAdditional analyses, as mentioned in the exchange between myself and the authors, would be required to convince me to raise my score any higher.", "rebuttal": "We thank the reviewer for recognizing the novelty of our method, and for acknowledging our method's strong performance compared to other publicly-available approaches. Below, we address the weaknesses and questions raised in the review.\n\n> The evaluation lacks qualitative analysis that would clarify the practical implications of the boost in quantitative performance. The paper would benefit from visual demonstrations showing how the new CLIP embeddings differ from those of the original model - for instance, through image retrieval examples or t-sne embedding visualizations.\n\n> How do your embeddings differ from the original CLIP embeddings? In what cases are your embeddings better and worse than the original CLIP embeddings?\n\n> Is there a qualitative boost in performance in addition to the quantitative gain?\n\nWe appreciate the reviewer's suggestions for qualitative analysis. We note that such embedding-level qualitative evaluation is not standard practice in the data filtering literature for CLIP pretraining. We would appreciate it if the reviewer could point to papers in the data filtering or CLIP pretraining literature that include such qualitative analysis.\n\nAs an alternative, we could provide a different form of qualitative analysis: examining which training examples our method prioritizes compared to existing approaches. This could provide insights as to which types of examples our learned scoring model values differently from other methods.\n\n> The writing suffers from slight redundancy and could be tightened. Some concepts and findings are reiterated unnecessarily throughout the paper.\n\nThe clarity and readability of the paper is very important to us. We would be grateful if the reviewer could point to specific instances where our writing is redundant or where concepts are unnecessarily reiterated. We are happy to revise our paper to improve clarity and conciseness.\n\n> Why optimize both the scoring model and CLIP reference model together? This seems like a non-trivial choice. What happens if you just train the scoring model while keeping the reference model frozen?\n\nWe agree with the reviewer that optimizing the reference CLIP model is a non-trivial choice. In Appendix C.7 we present E-FLYT ablation experiments including freezing the reference CLIP model. The results are shown in Table 11 under the \"Trainable parameters\" experiment, \"Only scoring model\" variation. The experiment shows that training the reference model yields considerably better performance than freezing it.\n\nIntuitively, when the reference model is frozen at a single point in parameter space, the scoring model may overfit to a data weighting that is only locally beneficial for training. In contrast, when we train the reference model, the loss function for the scoring model averages over many configurations in parameter space."}]

Author rebuttal:

Meta-review: The paper introduces a data filtering algorithm to learn a scoring function (M-FLYT) that estimates the usefulness of each data example for training a CLIP model. In the algorithm, a CLIP model is trained along with a scoring model that predicts importance of each example in a batch based on downstream task losses. The scoring model takes 12 existing filtering methods as input and learns to a linear combination with meta-learning strategy. Techniques such as Soft Cap Sampling (SCS) are leveraged to refine the sampling (e.g., preventing over-representation of high-scoring examples and smoothly reducing the importance of previously sampled examples). Improvement is showed on DataComp medium-scale benchmark and ImageNet zero-shot classification tasks.


Strength of the paper: 
- Well written
- Conceptually simple method. A novel framework to leverage meta-learning in data filtering 
- Good selection of downstream evaluation tasks supports the efficacy of proposed approach. Improvements are modest compared to some strong baselines, but appear consistent.
- Working on a popular and important area (vision language representation learning) and providing a empirically effective algorithm to the area

Reviewers align on most of the strengths of the paper from the high level and see its novelty, contribution, and relevance to NeurIPS venue. Thus we are inclined to accept this paper to the conference. However, there are a few areas  that can be improved, after most questions being addressed and clarified during the rebuttal and discussion stage

To improve:

- Although the proposed meta-learning-based FLYT approach is conceptually simple, implementing such meta-optimization techniques is complex. Leveraging downstream task or filters to select examples might also introduce uncontrolled biases and reduce model generalizability.
- Would be great to see the method being applied to different variants of CLIP (e.g., SigLIP, MetaCLIP, etc.)
- Many discussion between reviewers and authors are valuable to include in the final manuscript
Decision: Accept (poster)