A CLIP-Powered Framework for Robust and Generalizable Data Selection

Authors: Suorong Yang, Peng Ye, Wanli Ouyang, Dongzhan Zhou, Furao Shen
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Data selection, generalization, multimodal

Abstract:
Large-scale datasets have been pivotal to the advancements of deep learning models in recent years, but training on such large datasets inevitably incurs substantial storage and computational overhead. 
Meanwhile, real-world datasets often contain redundant and noisy data, imposing a negative impact on training efficiency and model performance.
Data selection has shown promise in identifying the most representative samples from the entire dataset, which aims to minimize the performance gap with reduced training costs. 
Existing works typically rely on single-modality information to assign importance scores for individual samples, which may lead to inaccurate assessments, especially when dealing with noisy or corrupted samples.
To address this limitation, we propose a novel CLIP-powered data selection framework that leverages multimodal information for more robust and generalizable sample selection. 
Specifically, our framework consists of three key modules—dataset adaptation, sample scoring, and selection optimization—that together harness extensive pre-trained multimodal knowledge to comprehensively assess sample influence and optimize the selection results through multi-objective optimization. 
Extensive experiments demonstrate that our approach consistently outperforms existing state-of-the-art baselines on various benchmark datasets. Notably, our method effectively removes noisy or damaged samples from the dataset, enabling it to achieve even higher performance with less data. This indicates that it is not only a way to accelerate training but can also improve overall data quality.
 The implementation is available at https://github.com/Jackbrocp/clip-powered-data-selection.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730215426373

thread_type:
official_review

<thread_object>
creator: reviewer_vc5o

<content>
summary:
The paper proposes a CLIP-powered framework for data selection that addresses the limitations of traditional single-modality data selection methods by incorporating multimodal information. This framework utilizes a pretrained vision-language model (CLIP) to integrate image and text modalities, thereby enhancing data selection accuracy and robustness. The framework is built on three modules: Dataset Adaptation, Sample Scoring, and Selection Optimization. It leverages both semantic alignment and diversity scores to refine sample selection, removing redundant and noisy data to improve training efficiency and performance on benchmark datasets. Experiments demonstrate the method's superior performance in generalization and robustness compared to existing state-of-the-art approaches, especially in noisy environments.

strengths:
1. Originality: The use of multimodal data via CLIP for data selection is innovative and distinguishes this work from single-modality approaches.
2. Quality: The methodology is backed by rigorous experiments, with a well-defined scoring system for sample selection (SAS and SDS).
3. Clarity: Visualizations effectively depict the advantages of multimodal selection over traditional methods.
4. Significance: This framework has broad applications, as it improves dataset quality and model robustness, potentially benefiting various domains using machine learning.

weaknesses:
1. Complexity: While effective, the Selection Optimization module’s complexity might hinder its application to extremely large datasets without computational resources.
2. Generality: The framework is primarily tested on vision-based datasets; extending it to text or mixed-modality datasets may reveal further limitations. Future work could focus on enhancing the framework’s versatility across other modalities​

questions:
1. Could the authors provide more detail on the computational efficiency of the Selection Optimization module for very large-scale datasets?
2. Has the framework been tested on datasets with high modality imbalance (e.g., more text than images), and how does it handle such scenarios?
3. For real-world noisy datasets, could the authors provide more information on how they define and quantify "noisy" samples?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730554881585

thread_type:
official_review

<thread_object>
creator: reviewer_hyv4

<content>
summary:
This paper proposes a novel CLIP-based data selection method leveraging multimodal information with an SGD optimization module. The experiments on several benchmarks show evident improvements over existing approaches.

strengths:
- Connecting the text and image information for the dataset selection is novel. The proposed CLIP-based method implements the idea well. 
- The SGD-based selection optimization with multi-objective is interesting. It is different from the existing sampling strategies with combinational optimization. 
- Sufficient experimental results show the effectiveness, including different datasets, different settings, and different model architectures.

weaknesses:
- The work relies on two adapters to project the CLIP features to the dataset-specific embedding space. The adapter training is based on perfect data. In fact, under the setting of noisy or corrupted data, this ideal data is inaccessible. Therefore, the experiments would be problematic. In Lines 514-515, the authors depict "is essential for effectively transferring the model’s generalization ability to target datasets". It seems the adapter has a large influence on the performance.  Will using the *imperfect* data to fine-tune the adapter degrade the method's effectiveness? This needs to be discussed.

- The class label in this work plays a role as a prototype, while previous work also uses prototypes. They [a] often use the average image features as the prototype and calculate the Euclidean distance between the embedded image and the corresponding prototype, similar to SAS (Eq. 1). Besides, they also use this distance to filter noisy labels [b]. Intuitively, we can use the average image feature to replace the text feature in Eq. 1 of the proposed method. Therefore, it is essential to discuss these results to convey to readers that introducing text is significant.  

- MoSo is NeurIPS'23 rather than NeurIPS'24. Therefore, methods from 2024 have not been compared, e.g., [c]. Besides, some typical methods good at low sampling ratios are missing, e.g., [d].

- Typo: the symbol for the learnable parameter in Figure 2 should be $\mathbf{d}$ rather than $\mathbf{w}$.

[a] Moderate coreset: A universal method of data selection for real-world data-efficient deep learning, ICLR, 2023

[b] NGC: A unified framework for learning with open-world noisy data, ICCV, 2021

[c] Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning, CVPR, 2024

[d] Submodular Combinatorial Information Measures with Applications in Machine Learning, ALT, 2021

questions:
- The text-image alignment is like using CLIP to measure the difficulty in learning the data. Noisy data or corrupted data are difficult to learn. However, will the selection also filter some important yet difficult data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730631269512

thread_type:
official_review

<thread_object>
creator: reviewer_vu6u

<content>
summary:
This paper argues that current image data selection methods are limited because they are unimodal. It proposes a multimodal method that uses the category texts from pretrained CLIP to complement images for more robust and generalized data selection. The proposed framework consists of three modules (1) dataset adaptation that integrates image and text adapters to transfer prior knowledge to the target data; (2) sample scoring that calculates the semantic alignment and diversity scores based on the multimodal features, measuring the image-text alignment as well as the local pattern variability; (3) Selection optimization that uses the two scores to select semantically representative and diverse samples, and introduces selection optimization to efficiently identify the ideal data subsets given an expected selection ratio through a multi-objective optimization strategy.

Post discussion comments:
The authors have addressed my concerns. With the additional clarifications and material, the paper is in much better shape. I am keeping my score at 8, which correctly reflects the quality of this work.

strengths:
The idea of exploiting multimodal features from the CLIP model is interesting and plausible. 

The proposed method is simple, which is a strength in my opinion. 

The method is also efficient and is able to control the alignment, diversity and selection ratio in a multiobjective optimization efficiently. 

The paper is well written and easy to follow. 

The results are good.

weaknesses:
The proposed method relies on a pretrained CLIP and hence any biases in the CLIP model will propagate to the selected dataset. 

The proposed method optimizes alignment and diversity but does it have any indirect effect on bias in the dataset?  

Is it possible to control bias in the dataset or in the subsequent models that are trained on the selected dataset? 

Does the STE cause convergence issues? 

The variable d (sample wise parameter) can be easily confused with d (feature dimension). Consider changing one of them. Also, I don’t see d in Fig. 1. Is “w: N x 1” the sample wise parameter d? Or am I missing something? 

The caption of Tab.3 needs to be corrected. You can also merge Tab.3 with Tab.2.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730639400746

thread_type:
official_review

<thread_object>
creator: reviewer_qkgn

<content>
summary:
This paper introduces a CLIP-based multimodal data selection framework that enhances the robustness and generalizability of data selection by leveraging both image and text information. Firstly, the paper trains an adapter to transfer pretrained knowledge to the target data. Then, the similarity between the image and text is used to calculate the sampling score for subset selection. The authors evaluate their results in various popularly datasets, obtaining state-of-the-art results in almost all of them.

strengths:
1. The method performs more robustly compared to other baseline data selection methods in various popularly datasets.
2. The algorithm is easy to understand and to follow.
3. The writing of this paper is overall great.

weaknesses:
Some details and motivation should be explained further.
1. The method requires training an adapter with data from the training dataset for data selection. If the noise ratio is high (such as 50%), wouldn't it be a better option not to use an adapter?
2. Does the Actual Selection Costs in Appendix G include the training time for the Adapter? Is the comparison fair?
3. From the ablation experiment, it can be seen that selection loss has a significant impact on the final result. So, what is the sensitivity of this parameter?

questions:
The authors should take the above questions into consideration: the utilization of the adapter, the comparison, and the parameter.
1. The method requires training an adapter with data from the training dataset for data selection. If the noise ratio is high (such as 50%), wouldn't it be a better option not to use an adapter?
2. Does the Actual Selection Costs in Appendix G include the training time for the Adapter? Is the comparison fair?
3. From the ablation experiment, it can be seen that selection loss has a significant impact on the final result. So, what is the sensitivity of this parameter?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732074309744

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
# General Description:
Dear Area Chairs and Reviewers,

We sincerely thank you for the time and effort in reviewing our work. We greatly appreciate the constructive feedback and the recognition from all reviewers—QkGn (R1), vU6u (R2), HyV4 (R3), and vC5o (R4)—highlighting key strengths of our work, including (1) novelty (R1, R2, R3, R4), (2) overall presentation clarity (R1, R2, R4), (3) sufficient experiments results (R2, R3), and (4) significance (R4). Besides, the concerns are mainly concentrated on (1) more experiments and analysis (R1, R3, R4), and (2) specific refinements to improve clarity (R2, R3).

# Additional Experiments and Analyses:
In the responses, we show additional experimental results and analysis, including:
1. Comparison of using and without using the adapter in high-noise conditions (R1: Q1)(Table A-1)
2. Evaluation and analysis of the dataset adaptation and selection optimization costs (R2: Q4, R4: Q3 )(Table B-1, D-1)
3. Stability analysis of the parameter of selection loss (R1: Q3)(Table A-2)
4. The effectiveness of the adapter in noisy conditions (R3: Q1)(Table C-1/2/3)
5. Validation of the significance of introducing text modality (R3: Q2)(Table C-4/5)
6. Comparison with the suggested work (R3: Q3)(Table C-6/7/8/9)

Thank you again for your thoughtful feedback and for helping us refine our work further.

Sincerely,

Authors of Submission 3088
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work proposes a new framework for data selection to address the computational overhead and the impact of noisy data when training deep learning models. The key idea is to leverage multimodal information to better select important data and this is achieved by utilising the CLIP model. A set of scores is developed and a multi-objective optimisation is implemented to realise data selection. Experimental study demonstrates the efficacy of the proposed method. Reviewers are overall positive on this work, indicating the strengths on its robust performance, easy to understand, simple and efficient, and its originality and significance. Meanwhile, the reviewers raise the issues related to the clarification of some details, the impact of bias, convergence, the presence of imperfect data, comparison with the last methods, and the complexity and generality of the method. The authors provide a high quality rebuttal and effectively address most of the issues. All the final ratings are on the positive side. By checking the submission, the reviews, and the rebuttals, AC agrees with the reviewers on their observations. Therefore, this work is recommended for acceptance.