Title: Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining
Venue: NeurIPS 2025
Accepted: yes
Keywords: dataset filtering

dataset curation

vision-language pretraining

CLIP

representation learning
Abstract: We introduce Filter Like You Test (FLYT), an algorithm for curating large-scale vision-language datasets that *learns* the usefulness of each data point as a pretraining example. FLYT trains a scoring model that learns to weigh each example's features using gradient signals from downstream tasks training sets. Based on FLYT, we implement Mixing-FLYT (M-FLYT), which takes the per-example scores generated by different scoring methods as features, and learns to unify them into a single score. FLYT naturally produces a distribution over the training examples, which we leverage through Soft Cap Sampling (SCS), a strategy for obtaining a filtered pretraining dataset from per-example probabilities that samples examples while preventing over-representation through a repetition penalty. Using these methods, we achieve 40.1\% ImageNet zero-shot accuracy on the DataComp medium scale filtering benchmark, a 2\% absolute accuracy increase over all previous results and a 5.5\% increase over results that---like us---use only public resources. Our approach also yields 37.7\% on the average of 38 DataComp evaluation tasks, outperforming previous public-resource approaches by 0.4\%.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a data filtering approach for improving CLIP pre-training by automatically identifying high-quality text-image pairs. The authors develop a learned scoring function that predicts how valuable each data point will be for CLIP model training. They implement this approach in two variants - M-FLYT and E-FLYT - which use different feature extraction techniques. Beyond scoring, they introduce SCS, a method that determines the optimal frequency for including each data point in the final training dataset. When evaluated on the DataComp benchmark, their approach outperforms all competing methods that rely solely on publicly available resources.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper introduces a novel framework for filtering training data for CLIP pre-training.\n\n2. The authors' method demonstrates superior performance compared to other publicly-available approaches on the DataComp benchmark.\n\nWeaknesses:\n\n1. The evaluation lacks qualitative analysis that would clarify the practical implications of the boost in quantitative performance. The paper would benefit from visual demonstrations showing how the new CLIP embeddings differ from those of the original model - for instance, through image retrieval examples or t-sne embedding visualizations. \n\n2. The writing suffers from slight redundancy and could be tightened. Some concepts and findings are reiterated unnecessarily throughout the paper.\n\nQUESTIONS:\n1. Why optimize both the scoring model and CLIP reference model together? This seems like a non-trivial choice. What happens if you just train the scoring model while keeping the reference model frozen? I'm not convinced that jointly training both models will converge properly - as they could interfere with each other during training. It would be helpful to see a comparison with the simpler approach of fixing the reference model to understand if the joint optimization is actually necessary.\n\n2. What exactly does public-resource mean in the context of DataComp? I don't think it was explained in the paper.\n\n3. How do your embeddings differ from the original CLIP embeddings? In what cases are your embeddings better and worse than the original CLIP embeddings?\n\n4. Is there a qualitative boost in performance in addition to the quantitative gain?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Filter Like You Test(FLYT), a data filtering algorithm that learns to assign importance weights to each training examples based on their contribution to the downstream task. This paper also introduce Mixing-FLYT(M-FLYT), which integrates scores from multiple existing filtering methods into a unified score using learned combinations, and Soft Cap Sampling (SCS), a sampling strategy that prevents over-representation of high-scoring examples. Together, these methods achieve state-of-the-art results on the DataComp medium-scale benchmark, surpassing previous public-resource approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. The paper introduces a learning-based filtering algorithm FLYT, which use directly optimizes the scoring model based on the performance of downstream task. It can further be extend to M-FLYT, where the learnable parameters are the weight of different existing scoring methods.\n2. By combining FLYT with Mixing-FLYT and Soft Cap Sampling, the paper indeed achieves state-of-the-art results on the DataComp medium-scale benchmark, including a 5.5% improvement in ImageNet zero-shot accuracy over previous public-resource methods. It proves the effectiveness of the proposed algorithm.\n\nWeaknesses:\n1. The Soft Cap Sampling won't eliminate the repetition of sampled data. Could you explain how the proposed Soft Cap Sampling change the distribution of sampled data?\n2. The proposed FLYT algorithm appears conceptually and technically aligned with meta-learning paradigms: concept (meta learning: learning to learn) and technique (optimizing the updated parameters by second derivative). However, the paper does not acknowledge this connection or cite relevant meta-learning literature in the introduction or related work.\n3. The porposed FLYT/Mixing-FLYT may significantly increases the training time (use second-order gradients to optimize scoring model), which is not discussed in the experiments or mentioned in the limitations section of the paper.\n\nQUESTIONS:\nPlease address the questions listed in the weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a learned sample importance algorithm for filtering a dataset of varying sample quality to one that can effectively train a CLIP model. In the first phase, they train a model by simultaneously learning the CLIP model itself, along with a scoring model that predicts how important each example in the batch is, based on a downstream task loss. The scoring model is of primary interest, as once this phase finishes, they then use it to construct an N (e.g. 128M) example dataset (potentially with duplicates). To allow for duplicates, they propose SoftCap sampling, which smoothly degrades the importance of previously sampled examples, allowing them to be sampled again, but at reduced likelihood. Further variations and improvements are proposed, however the thrust of the idea remains.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n* The method is minimal with heuristics, instead relying on the learned scoring model.\n* The proposed model does exceedingly well, beating contemporary methods not only on the downstream task (ImageNet), but also other tasks.\n* The proposed method doesn't rely on private datasets\n\n### Weaknesses\n\n* Not levied at this paper alone, but the method does directly target ImageNet by using it as the downstream task, so generalization appears to rely on the generalizability of the concepts in ImageNet. This is where the concurrently proposed MGD approach, utilizing all tasks, may result in more robust vision foundation models (of which CLIP is typically considered), since it's not overfitting to ImageNet.\n* The clarity of the paper could be improved. It would perhaps be nice to read about FLYT first, to understand the algorithm, before additional complexities such as M-FLYT and E-FLYT are introduced. For example, section 3.5 could appear before 3.3 and 3.4.\n\nQUESTIONS:\nCan you explain how the \"Average\" column in Table 1 is computed? It's clearly not a simple mean over the former fields. Particularly, it seems as though \"M-FLYT+SCS\" should have the highest average, given that it beats every other method on the other categories, and yet it has a lower average score than the non-proposed methods above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a method to filter / curate large-scale image-caption datasets for training CLIP-like models. The method consists of two components:\n1. A learnable scoring function (M-FLYT) that estimates how useful a data example is for training. It takes as input scores from 12 existing scoring methods and learns to a linear combination of them. The scoring function is learned using a meta-learning strategy that learns the scoring function weights as as to improve the zero-shot classification performance (on ImageNet) of a reference CLIP model.\n2. A sampling approach (SCS) that selects training examples based on the learned scores (from M-FLYT). It samples with replacement but adds a penalty $\\alpha$ each time an example is selected. This prevents extensive repetitions of examples with high scores. \n\nThe method and its individual components are evalauted on the DataComp medium-scale benchmark. It outperforms previous works, achieving 40.1% accuracy on ImageNet zero-shot classification and 37.7% on average across DataComp tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**    \n- The paper is overall easy to understand and well-written \n- The proposed meta-learning approach is conceptually simple (but hard to implement) and the idea of meta-learning the scoring function so as to explicitly improve the downstream task performance makes sense. Furthermore, the proposed sampling approach SCS is simple, easy to implement, and effective. \n- The work demonstrates that both M-FLYT approach and the SCS sampling approach improves the results compared to the baselines. \n\n**Weaknesses:**\nAlthough the proposed meta-learning-based FLYT approach is conceptually simple, implementing such meta-optimization techniques is complex (see, for example, the discussion in Appendix A), while the gains in Tables 1 and 2 appear modest. M-FLYT shows relatively small improvement over \"DFN-FT [our reproduction of 16]\" (compared to the impact of SCS). Moreover, the performance gap between M-FLYT, which mixes individual scores using a computationally heavy and complicated meta-optimization stage, and the simpler, training-free \"IN-weighted\" method (Table 2, right) is even smaller.\n\nTable 1 suggests that the SCS sampling strategy has a much larger impact on performance than M-FLYT. It is likely that using SCS with just the \"IN-weighted\" baseline (or even DFN-FT) would yield strong results, close to \"M-FLYT + SCS\", while being far simpler. Could the authors provide results without M-FLYT or E-FLYT to confirm this? Such results are necessary to judge the practical usefulness of FLYT.\n\nQUESTIONS:\nPlease answer my comments and questions put under the \"Weakness\" section above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}