A CLIP-Powered Framework for Robust and Generalizable Data Selection

## Abstract

Abstract Large-scale datasets have been pivotal to the advancements of deep learning models in recent years, but training on such large datasets invariably incurs substantial storage and computational overhead.
Meanwhile, real-world datasets often contain redundant and noisy data, imposing a negative impact on training efficiency and model performance.
Data selection has shown promise in identifying the most representative samples from the entire dataset, which aims to minimize the performance gap with reduced training costs.
Existing works typically rely on single-modality information to assign importance scores for individual samples, which may lead to inaccurate assessments, especially when dealing with noisy or corrupted samples.
To address this limitation, we propose a novel CLIP-powered data selection framework that leverages multimodal information for more robust and generalizable sample selection.
Specifically, our framework consists of three key modules—dataset adaptation, sample scoring, and selection optimization—that together harness extensive pre-trained multimodal knowledge to comprehensively assess sample influence and optimize the selection results through multi-objective optimization.
Extensive experiments demonstrate that our approach consistently outperforms existing state-of-the-art baselines on various benchmark datasets. Notably, our method effectively removes noisy or damaged samples from the dataset, enabling it to achieve even higher performance with less data. This indicates that it is not only a way to accelerate training but can also improve overall data quality.
The implementation will be made publicly available soon.

## 1 Introduction

Recent advancements in deep learning have been propelled by increasingly large and complex models that utilize vast datasets to achieve start-of-the-art performance Liu et al. (); Touvron et al. (). However, this success normally comes with considerable costs for data storage and computational resources, which may even limit the deployment of models to specialized infrastructure and hinder their scalability across different applications. Moreover, real-world datasets often contain redundancy and noise, which can degrade the training efficiency and performance.

To alleviate the data redundancy issue and improve the training efficiency, there are typically two kinds of methods, i.e., dynamic pruning and data selection.
Dynamic pruning methods Raju et al. (); Qin et al. () aim to reduce training costs by dynamically selecting only the most influential samples from the dataset during training. Despite their effectiveness in accelerating training, they still face the cost of large-scale data storage, and their selected samples often lack the ability to generalize well across different training processes and architectures.
In contrast, data selection methods Paul et al. (); Yang et al. (); Tan et al. (); Xia et al. () pre-select a fixed subset of the essential data points before the training begins, allowing the model to achieve performance comparable to that obtained with the full dataset.
By focusing on the most critical data points, these methods ensure
better generalization ability across various training scenarios.

Figure: Figure 1: Benefit of multimodal selection. Traditional single-modality approaches (upper part) may struggle with noisy and corrupted data, while our multimodal method (lower part) identifies class-representative samples with diversity while effectively filtering out noise and corrupted data.
Refer to caption: x1.png

Existing data selection methods typically employ carefully designed criteria via three perspectives: importance scores Paul et al. (); Tan et al. (), image data distribution Zheng et al. (); Xia et al. (), and optimization-based functions Killamsetty et al. (); Yang et al. ().
Although achieving promising results, these methods exhibit certain limitations.
On one hand, relying solely on single-modality image information can lead to ambiguities, especially when noisy samples are present, which may result in inaccurate assessments of a sample’s effect.
For instance, some methods employ difficulty-based criteria
to select data; however, distinguishing between truly difficult samples and noisy ones based solely on image modalities presents a significant challenge.
On the other hand, existing methods typically select samples with the highest or lowest scores, while the interaction between high-score and low-score samples within a group can significantly influence the overall performance, which is known as the ”group effect” Koh et al. (); Yang et al. ().
Thus, a more beneficial approach is to leverage the power of multimodal information and evaluate the collective effect of the sample group.

In this paper, we propose a CLIP-powered data selection framework that employs multimodal information for more robust and generalizable data selection, where the category text serves as strong complements to the image modality and promotes the overall performance. The framework consists of three modules, namely dataset adaptation, sampling scoring, and selection optimization module. First, the dataset adaptation module integrates image and text adapters to facilitate the transfer of pretraining knowledge to the target data. Subsequently, the sample scoring module calculates the Semantic Alignment Score (SAS) and Sample Diversity Score (SDS) based on the adapted multimodal features, which measure the image-text alignment and the variability of local patterns. Using these two scores together can select semantically representative samples while maintaining their inherent diversity. Further, in order to address the group effect, we introduce a selection optimization module to identify the ideal subsets w.r.t. the expected selection ratio through a multi-objective optimization strategy. By leveraging the multi-modal information and carefully designed supervision signals, our framework enables the selection of high-quality samples in a flexible and efficient manner.

Comprehensive evaluation across various benchmark datasets demonstrates that our approach effectively improves the performance of selected datasets,
especially on large-scale datasets such as ImageNet-1k Deng et al. ().
Moreover, the selected datasets exhibit superior cross-architecture generalization across ResNet-18/50 He et al. (), ViT Dosovitskiy et al. (), VGG-16 Simonyan & Zisserman (), etc.
Notably, since most existing methods are not robust to more complex and realistic scenarios, we further demonstrate the strong robustness of our approach in these more challenging scenes.
For instance, our proposed method can achieve an 8.13% improvement in accuracy on CIFAR-100 and a 4.41% improvement on Tiny-ImageNet compared to the leading baselines.
Meanwhile, superior performance is achieved with very high efficiency compared to other baselines.

The contributions of this work can be summarized as follows: 1) We analyze the drawbacks of previous works that rely solely on image modalities in depth, and propose a new CLIP-powered data selection framework that leverages multimodal features for robust and generalizable data selection for the first time. 2) Our framework comprises dataset adaptation and sample scoring modules to foster multi-modality knowledge transfer and comprehensive sample importance evaluation. This dual-modality design effectively removes noisy and corrupted samples from the dataset. 3) A selection optimization module is designed to identify the optimal subsets w.r.t. the expected selection ratios through multi-objective optimization, which effectively addresses the group effect while maintaining high efficiency. 4) Experimental results show that our method outperforms previous SOTA approaches in terms of performance, cross-architecture generalization, and robustness to noisy and corrupted images. Meanwhile, our approach achieves the best trade-off in performance and selection efficiency, establishing a strong baseline of data selection for future research.

## 2 Related Work

Date-efficient deep learning generally incorporates dynamic data pruning Qin et al. (); Raju et al. (), data distillation Lei & Tao (); Du et al. (); Zhang et al. (); Cazenavette et al. (), data condensation Liu et al. (); Yang et al. (); Zhao & Bilen (), and static data selection Xia et al. (); Tan et al. (); Yang et al. ().
Following the static data selection, we propose a method capable of identifying representative and diverse samples across various selection ratios.
Data selection, or static dataset pruning, aims to identify and select the most representative samples from training datasets.
Training on these selected samples can achieve comparable performance to that obtained with the full dataset while reducing training and storage costs.
Current data selection methods can be broadly divided into three categories: importance criteria Paul et al. (); Tan et al. (), dataset distribution-based methods Xia et al. (); Zheng et al. (), and optimization-based methods Nohyun et al. (); Yang et al. ().

Selection with importance criteria is the most popular type. These methods typically involve calculating importance scores for each sample and selecting samples based on these scores.
For instance, EL2N and GraNd score Paul et al. () measure the importance by calculating the expectation of the $\ell_{2}$-norm error vector and the expectation of the gradient norm, respectively.
MoSo Tan et al. () calculates the change of the optimal empirical risk when removing a specific sample from the training set.
Forgetting Toneva et al. () tracks the frequency with which a sample is misclassified after being correctly classified during training.
Similarly, Memorization Feldman & Zhang () assesses the impact of a sample’s presence or absence on the model’s ability to predict it correctly.
While importance criteria-based methods are often computationally efficient, their performance may be affected by the group effect Yang et al. (; ) and may not generalize well to complex, real-world scenarios Xia et al. ().

Dataset distribution-based methods select samples by analyzing
the geometric distribution of datasets.
For instance, Herding Welling () determines the sample importance according to samples’ distance to their corresponding class centers.
The work Ramalingam et al. () applies greedy k-center to select the coreset with good data coverage.
D2 Maharana et al. () calculates and updates the difficulty scores of each sample by incorporating the difficulty of its neighboring examples.
Moderate-DS Xia et al. () chooses samples with closer distances to the median score, while CCS Zheng et al. () balances the data distribution and the sample importance in selection.

Optimization-based methods have been proposed to select samples through various optimization techniques Tan et al. (), such as gradient matching Mirzasoleiman et al. (); Killamsetty et al. (), scalable self-supervised pruning metric Sorscher et al. (), influence function Yang et al. (); Pooladzandi et al. (), bi-level optimization Killamsetty et al. (), facility location function Mirzasoleiman et al. (); Yang et al. (), and submodularity Nohyun et al. (); Kothawade et al. (); Wei et al. ().
In contrast to these methods that only rely on image information, we leverage multimodal messages for data selection, which incorporates the semantic alignment between image data and corresponding category information.
This contributes to comprehensive assessments of sample effectiveness, particularly in complex scenarios where samples may be corrupted or wrongly labeled.

## 3 The Proposed Method

Figure: Figure 2: Our proposed method consists of dataset adaptation, sampling scoring, and selection optimization modules. The dataset adaptation module is used to learn dataset-specific knowledge. The sample scoring module computes two scores, $\bm{S}_{A}$ and $\bm{S}_{D}$, to assess sample importance, based on which the selection optimization identifies the optimal subsets w.r.t. the expected selection ratios.
Refer to caption: x2.png

Our proposed method is summarized in Figure 2.
The approach involves the use of the pretrained vision-language foundation model CLIP to construct multimodal feature spaces.
Nevertheless, there may exist domain shifts or discrepancies between the pretrained training dataset and the target dataset Liu et al. (); Alijani et al. ().
To facilitate dataset adaptation and enhance the learning of dataset-specific knowledge, we incorporate dimension-preserving adapters for both modalities.
Following this, two scores are derived to comprehensively assess the sample importance, i.e., the Semantic Alignment Score (SAS), denoted as $\bm{S}_{A}$, and the Sample Diversity Score (SDS), denoted as $\bm{S}_{D}$.
Furthermore, rather than solely based on sample scores for selection, we design a multi-objective optimization to identify the optimal subsets w.r.t. the expected selection ratios, which effectively mitigates the group effect.
We provide the detailed methodology in the subsequent sections.

### 3.1 Dataset Adaptation

To alleviate the domain shifts and discrepancies between the pretrained and target datasets, we incorporate dimension-preserving adapters to perform adaptation on the target dataset. The image and text adapters are denoted as $A_{I}$ and $A_{T}$, respectively.
Both adapters are fine-tuned for knowledge transfer while the pretrained CLIP weights are frozen.
To maintain high efficiency, both adapters utilize simple MLP.

Specifically, the fine-tuning process employs the InfoNCE loss Parulekar et al. (); Oord et al. (), which maximizes the mutual information between the image and text representations.
The text representation describes the category information using the prompt: “A photo of $[\text{CLS}]$”, where the token $[\text{CLS}]$ represents the corresponding category.
The loss ensures that the adapters effectively align and capture the relevant features from both modalities.
Furthermore, it enhances the model’s ability to distinguish between positive and negative pairs, thereby improving the robustness and accuracy of the deep representations for the specific dataset.

### 3.2 Sample Scoring

For classification datasets, the learning process for training samples is intrinsically linked to acquiring knowledge of the corresponding categories.
A training sample that more accurately represents its category is typically more effective in training deep networks.
In this way, the Semantic Alignment Score (SAS) is designed to assess the semantic similarity between training samples and their corresponding categories.
Specifically, since image and text features reside within the same embedding space Radford et al. (), the SAS is derived by calculating the cosine similarity between the embedded image and corresponding textual deep descriptions.
Accordingly, the SAS for the $i$-th sample is defined as:

$$ $\bm{S}_{Ai}=\mathrm{cos}(A_{I}(E_{I}(\bm{I}_{i})),A_{T}(E_{T}(\bm{T}_{i}))),$ (1) $$

where $\bm{I}_{i}$ is the $i$-th sample, $\bm{T}_{i}$ is the textual description of the corresponding category for $\bm{I}_{i}$, $E_{I}$ and $E_{T}$ are pretrained image and text encoders, respectively.

For selected datasets, the reduced data volume may limit the diversity of the selected data, which is important for training datasets Yang et al. ().
To address this, we introduce another diversity perspective to comprehensively assess the effect of training data.
The Sample Diversity Score (SDS) is defined as the average distance between each sample and its $k$ neighbor samples of the same class:

$$ $\bm{S}_{Di}=\frac{1}{k}\sum_{\bm{I}_{j}\in\mathrm{KNN}(\bm{I}_{i})}\|A_{I}(E_{ I}(\bm{I}_{i}))-A_{I}(E_{I}(\bm{I}_{j}))\|,$ (2) $$

where we use the KNN algorithm to obtain the neighbor samples for each sample, the distance metric is based on the $\ell_{2}$ norm, and $k$ is usually set to 10% of the number of samples per class.
In this way, the SDS can be understood as the local density of training samples in the feature space.
If a sample has a larger number of neighbors with closer distances (i.e., lower SDS), its training efficacy may be more easily substituted by other neighbor samples.
Therefore, selecting samples with higher SDS is generally more advantageous.

Figure: Figure 3: Illustration of the effectiveness of SAS and SDS. The circle and cross represent the normal and noisy samples, respectively. Different colors correspond to the selection results. SDS (b) selects diverse samples but may include noises. SAS (c) could avoid the noisy samples but potentially miss broader category information. Using both scores (d) can select category-representative while maintaining high diversity.
Refer to caption: x3.png

The effects of SAS and SDS are depicted in Figure 3. SDS contributes to the diversity of samples, but the selected data points may contain noise (Figure 3(b)). On the other hand, SAS can select samples that are semantically appropriate, as these points are basically around the sample center (Figure 3(c)). However, these samples may be too concentrated and thus lack diversity. When using SDS and SAS together (Figure 3(d)), we can cover the entire category space with fewer data and select samples that are both semantically representative and diverse, thereby boosting the effectiveness of data selection.

### 3.3 Selection Optimization

Instead of relying on combinatorial optimization functions for sample selection Killamsetty et al. (), our method determines the selection through SGD multi-objective optimization, which improves computational efficiency and accelerates convergence.
Specifically, we introduce a sample-wise parameter $\bm{d}$ to denote the selection decision, where elements of 1 indicate the selection while 0 indicates otherwise.
Although binary parameters are difficult to optimize in neural networks due to the absence of gradient, we employ the $\mathrm{sigmoid}(\cdot)$ function to push the continuous values of $\bm{d}$ towards approximate binarization.
After optimization, $\bm{d}$ is strictly binarized to explicitly indicate the final sample selection.
Initially, $\bm{d}$ is initialized with all 1s.

To guide the optimization process, we introduce three loss items.
The first item, $\mathcal{L}_{sa}$, is designed to prioritize samples with high SAS since these samples are more representative of their corresponding categories, which is defined as follows:

$$ $\mathcal{L}_{sa}=-\frac{1}{N}\sum_{i}^{N}\mathrm{sigmoid}(\bm{d})*\bm{S}_{Ai},$ (3) $$

where $N$ is the total number of samples.
$\mathcal{L}_{sa}$ punishes samples with low SAS and encourages the selection of samples with better semantic alignment.

In addition, we introduce another loss term, $\mathcal{L}_{sd}$, to encourage the selection of more diverse samples characterized by higher SDS, which is defined as:

$$ $\mathcal{L}_{sd}=-\frac{1}{N}\sum\mathrm{sigmoid}(\bm{d})*\bm{S}_{Di}.$ (4) $$

To mitigate the group effect, we optimize the selected datasets w.r.t. specific selection ratios, aiming to identify the optimal subsets.
We introduce a selection loss term, $\mathcal{L}_{s}$, to ensure the selection process adheres to the target ratio.
However, deriving exact selection rates from continuous real-valued parameter optimization is difficult.
While strictly binarized values facilitate explicit sample selection, they are challenging to optimize through gradient backpropagation.
To address this, we utlize the straight-through estimator (STE) Bengio et al. () to estimate the actual selection rate and derive gradients.
STE allows gradients to pass through the discrete decisions during backpropagation, effectively combining the benefits of both continuous and binary parameters for efficient optimization and accurate sample selection.
In this way, $\mathcal{L}_{s}$ is defined as:

$$ $\mathcal{L}_{s}=\sqrt{\left[\frac{1}{N}\sum_{i}\mathrm{STE}\left[\mathbb{I} \left(\mathrm{sigmoid}(\bm{d}_{i})_{\mathrm{i}}>0.5\right)\right]-s_{r}\right] ^{2}},$ (5) $$

where $\mathbb{I}$ is an indicator function, and $s_{r}$ denotes the expected selection ratio.
The loss term guides the parameter $\bm{d}$ toward near-binary values, ensuring the count of ones aligns with the expected sample size.
Since the selection is guided by adaptive optimization, the final selection ratio may deviate slightly from the target.
To minimize this deviation, we constrain $\mathcal{L}_{s}$ with a threshold $\theta$, which in our work is set to $5\times 10^{-4}$, ensuring the actual selection ratio differs by less than $\pm 0.05\%$ from the expected value.
Finally, the overall loss function is formulated as:

$$ $\mathcal{L}=\mathcal{L}_{sa}+\alpha\mathcal{L}_{sd}+\beta\mathcal{L}_{s},$ (6) $$

where $\alpha$ and $\beta$ are coefficients that adjust for numerical differences among the loss terms and can be set conveniently.

#### Complexity Analysis

The proposed method comprises three main components. 1) The dataset adaptation involves fine-tuning the image and text adapters. Since the adapters consist of simple linear layers, the number of parameters is small, and both the forward passes and backward passes are computationally efficient. 2) In the sample scoring process, the complexity of calculating the SAS and SDS is $O(N)$ and $O(K*d)$, respectively, where $K$ is the number of categories and $d$ is the feature dimension, typically 512. The complexity of the KNN algorithm is $O(N_{k}*d)$, where $N_{k}$ is the number of samples per class.
Given that $K$ and $d$ are constants and $N_{k}$ is usually much smaller than $N$, the overall complexity of this process is approximately $O(N)$.
3) The selection optimization of $\bm{d}$ does not involve deep models and is a numerical optimization process. The complexity is proportional to the number of parameters, i.e., $O(|w|)=O(N)$.

## 4 Experiment

### 4.1 Experimental Setup

Datasets and network architectures.
Consistent with previous works Tan et al. (); Xia et al. (); Zheng et al. (), we evaluate the effectiveness of our proposed method on various popularly used benchmark datasets, including CIFAR-10/100 Krizhevsky et al. (), Tiny-ImageNet Chrabaszcz et al. (), and ImageNet-1k Deng et al. ().
To evaluate the generalization performance of our selected datasets, we study the effectiveness of our proposed method on a wide range of network architectures, including ResNet-18/50 He et al. (), Vision Transformer (ViT) Dosovitskiy et al. (), Swin-Transformer Liu et al. (), VGG-16 Simonyan & Zisserman (), and DenseNet-121 Huang et al. ().

Baselines. We compare our proposed method with ten most representative SOTA baselines, i.e., (1) Random, (2) MoSo Tan et al. (), (3) Glister Killamsetty et al. (), (4) Herding Welling (), (5) Forgetting Toneva et al. (), (6) GraNd and (7) EL2N Paul et al. (), (8) Self-sup.-selection (SSP) Sorscher et al. (), (9) CG-Score Nohyun et al. (), and (10) Moderate-DS Xia et al. ().

Parameter settings.
The parameters in our proposed method can be easily set.
The coefficient $\alpha$ is proportional to the expected selection rate $s_{r}$, balancing the importance of dataset diversity.
For instance, $\alpha$ can be set equivalent to $s_{r}$.
The coefficient $\beta$ is set to 2 on all other datasets to adjust the numerical differences among loss items.

Figure: (a) CIFAR-100
Refer to caption: x4.png

### 4.2 Comparison with the State-of-the-arts

Performance Comparisons Consistent with prior works Xia et al. (); Sorscher et al. (), we report top-1 accuracy on CIFAR-100 and Tiny-ImageNet, and top-5 accuracy on ImageNet-1k. Note that the methods Glister and CG-Score are not compared on ImageNet-1k due to the heavy computation costs. Specifically, Glister obtains iterative solving of the bi-level optimization problem Killamsetty et al. (), while CG-Score involves the calculation of large Gram matrix inversions, making them impractical for large-scale datasets.

As illustrated in Figure 4, our method consistently achieves the best accuracy across all datasets.
Particularly, on the more challenging Tiny-ImageNet and ImageNet-1k datasets, our approach outperforms other methods by a notable margin.
While existing approaches yield relatively marginal accuracy improvements on the small-scale CIFAR-100 dataset, the gains brought by our method are more substantial.
Additionally, with relatively high selection ratios on these datasets, such as 90%, our selected datasets exhibit nearly lossless or even higher performance compared with the original full datasets and other baselines. The results indicate that our method can not only reduce training costs but may also serve as a way for data cleaning.

**Table 1: Test accuracy (%) on Tiny-ImageNet. VGG-16 and DenseNet-121 are exploited.**
| Method / Selection Ratio | VGG-16 | Densenet-121 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 70% | 80% | 90% | 100% | 70% | 80% | 90% | 100% |  |
| Random | 47.39$\pm$2.72 | 49.38$\pm$0.23 | 51.15$\pm$0.64 | 57.23$\pm$1.08 | 59.55$\pm$0.20 | 60.78$\pm$0.18 | 61.03$\pm$0.22 | 62.22$\pm$0.23 |
| EL2N | 48.30$\pm$2.95 | 48.75$\pm$1.65 | 49.01$\pm$1.31 | 57.23$\pm$1.08 | 59.61$\pm$0.00 | 60.38$\pm$0.04 | 61.16$\pm$0.47 | 62.22$\pm$0.23 |
| GraNd | 50.79$\pm$1.26 | 46.84$\pm$1.38 | 54.73$\pm$0.49 | 57.23$\pm$1.08 | 59.62$\pm$0.02 | 60.84$\pm$0.09 | 61.10$\pm$0.05 | 62.22$\pm$0.23 |
| MoSo | 50.47$\pm$1.01 | 50.12$\pm$0.83 | 50.07$\pm$0.43 | 57.23$\pm$1.08 | 59.27$\pm$0.33 | 59.86$\pm$0.07 | 60.00$\pm$0.37 | 62.22$\pm$0.23 |
| Herding | 48.59$\pm$0.07 | 45.77$\pm$0.12 | 50.77$\pm$1.24 | 57.23$\pm$1.08 | 59.00$\pm$0.28 | 60.03$\pm$0.35 | 61.15$\pm$0.12 | 62.22$\pm$0.23 |
| Glister | 48.74$\pm$2.29 | 50.05$\pm$0.02 | 49.42$\pm$1.81 | 57.23$\pm$1.08 | 59.98$\pm$0.01 | 60.62$\pm$0.34 | 61.28$\pm$0.18 | 62.22$\pm$0.23 |
| CG-Score | 48.73$\pm$2.70 | 48.49$\pm$1.88 | 49.62$\pm$1.08 | 57.23$\pm$1.08 | 59.74$\pm$0.15 | 60.55$\pm$0.20 | 61.14$\pm$0.11 | 62.22$\pm$0.23 |
| Self-sup. prototypes | 48.38$\pm$1.38 | 49.98$\pm$1.49 | 54.71$\pm$0.84 | 57.23$\pm$1.08 | 59.56$\pm$0.03 | 60.22$\pm$0.12 | 60.91$\pm$0.29 | 62.22$\pm$0.23 |
| Forgetting | 47.50$\pm$2.43 | 48.59$\pm$1.77 | 49.82$\pm$0.62 | 57.23$\pm$1.08 | 58.54$\pm$0.15 | 60.39$\pm$0.46 | 61.12$\pm$0.10 | 62.22$\pm$0.23 |
| Moderate-DS | 50.78$\pm$0.93 | 49.31$\pm$0.41 | 49.25$\pm$0.77 | 57.23$\pm$1.08 | 59.41$\pm$0.18 | 60.42$\pm$0.14 | 61.44$\pm$0.11 | 62.22$\pm$0.23 |
| Ours | 53.40$\pm$3.20 | 52.25$\pm$0.58 | 56.34$\pm$2.93 | 57.23$\pm$1.08 | 60.12$\pm$0.06 | 60.93$\pm$0.03 | 61.59$\pm$0.03 | 62.22$\pm$0.23 |

Figure: Table 2: Experimental results (accuracy, %) on CIFAR-100 and Tiny-ImageNet with noisy labels. 20% of labels are disturbed. We report mean $\pm$ standard deviation.

Figure: (a) 5% Corruption Rate
Refer to caption: x7.png

Figure: Figure 6: Effectiveness vs. efficiency on C-100. Results are reported with R-50 under 30% selection ratio on a 4-2080TI-GPU server.
Refer to caption: extracted/5927124/pictures/efficacy-efficiency-epoch-25.png

Generalization Comparisons on Different Architectures  In this section, we evaluate the generalization effectiveness of our selected datasets on deep architectures different from those used in the selection process.
Specifically, we employ the VGG-16 and DenseNet-121 models to train on the selected datasets from Tiny-ImageNet.
As shown in Table 1, the results indicate that our method surpasses all baseline methods on both architectures, demonstrating its desirable architectural generalization ability.
This suggests that the selected datasets are broadly applicable, irrespective of the specific network architecture.

Training Efficiency Comparisons  To evaluate the selection efficiency of various methods, we present an analysis of the balance between effectiveness and efficiency.
As shown in Figure 7,
our method presents the best performance with desirable efficiency.
Herding, EL2N, and GraNd obtain the lowest selection costs because they rely on predefined metrics or select samples in very early training. Our method is slightly slower than them but exhibits higher accuracy. Compared with the optimization-based approaches, like MoSo and Glister, our method enjoys both lower costs and better performance. The results verify the effectiveness of our method in balancing selection efficiency and accuracy.

### 4.3 Comparison of Robustness

Robustness on Noisy Labels  Real-world datasets often involve label noise, where some sample labels are incorrectly flipped, resulting in mislabeled data.
Unfortunately, creating clean and diverse datasets is time-consuming and expensive.
Therefore, it is necessary to evaluate the performance of selection methods under such complex scenarios.
In this study, we introduce symmetric noises Li et al. () to generate mislabeled data on both C-100 and T-ImageNet, with a 20% noise rate.

As can be seen in Table 4.2, our approach exhibits superior robustness to noisy labels, significantly outperforming other baselines by a large margin. Specifically, our approach yields improvements of over 10.12% on CIFAR-100 and 4.41% on Tiny-ImageNet compared to previous leading methods.
Additionally, Table 4.2 shows the distribution of noisy data within the selected datasets, where our method selects only 0.24% noisy samples, considerably fewer than other baselines. This reduction in noise underscores the potential of our method to improve overall data quality.

We argue that the robustness of our approach can be attributed to $\bm{S}_{A}$ in Eq. 1, which assesses the semantic alignment between image content and its labels.
In the presence of label noise, this alignment is disrupted, resulting in a lower SAS, which in turn reduces the likelihood of such samples being selected during optimization.
In contrast, most baseline methods rely solely on image features for selection, which may result in performance degradation when faced with noisy labels.
In some cases, the performance of these methods is even worse than that of random selection.
While Moderate also selects a relatively low proportion of noisy data, its performance is worse than ours.
This discrepancy highlights the effectiveness of our method in making more strategic selections in noisy environments, thereby not only minimizing noises but also optimizing the selected datasets for improved generalization.

Robustness on Corrupted Images  We further evaluate the performance of our proposed method on real-world noise corruptions that are frequently encountered Singh et al. ().
To simulate such corruptions, we employ the following five types of realistic noises Xia et al. (), including Gaussian noise, random occlusion, resolution, fog, and motion blur. The corruption rate is set to 5%, 10%, and 20%, respectively.

As shown in Figure 5, compared with prior baselines, our approach consistently presents greater robustness to corrupted images across varying corruption rates, demonstrating strong generalization in these challenging scenarios.
Notably, even at a high corruption rate of 20%, our method maintains desirable generalization performance.
This robustness is primarily attributed to the integration of text modality into the selection process, alongside the image modality.
The SAS defined in Eq. 1 measures the alignment between the image features and their corresponding category features.
When images are corrupted, this alignment is disrupted, thereby reducing the SAS and correspondingly decreasing the likelihood of selecting those images.
In contrast, methods such as Forgetting tend to prioritize difficult training samples, potentially making corrupted images more likely to be selected, as these images are typically more difficult to correctly classify. As a result, these methods are less robust to corrupted images, leading to a deterioration in generalization performance.

### 4.4 Dataset Selection Improves Generalization

Visualization Analysis  To demonstrate the generalization of the selected datasets, we train two models using the original dataset and the selected dataset (90% selection ratio), respectively, and obtain their embedding results on the CIFAR-10 test set.
To visualize the dataset distribution, we apply t-SNE to the embeddings generated by both models.
The visualization in Figure 7 shows that the model trained on the selected dataset produces better embedding results: a better inter-cluster separation and intra-cluster compactness. For a quantitative analysis, we use the Dunn Index (DI) Ncir et al. () to evaluate the clustering results (the higher, the better). After removing 10% of the data, the DI increases by 43%, presenting better clustering results.

Figure: Table 4: Performance and saved costs (%) on ImageNet-1k across Swin-T, ViT-B, and ViT-L on a 4-A100-GPU serve.

Generalization to More Advanced Architectures We further employ the selected datasets to train more advanced ViT-based architectures, including Swin Transformer, ViT-Base, and ViT-Large.
From Table 4, and corroborated by the results from previous sections, our selected datasets consistently achieve lossless performance across both CNN-based and Transformer-based architectures with reduced training costs.
This demonstrates that our approach obtains highly generalizable datasets applicable to a wide range of network architectures.

Generalization to More Challenging Benchmark Datasets To further evaluate the generalization and robustness of models trained on our selected datasets, we conduct experiments using ResNet-18 and ResNet-50 models, training on both the full datasets and our selected datasets.
These models are then tested on more challenging benchmarks, including ImageNet-Hard Taesiri et al. (), ImageNet-R Hendrycks et al. (), and ImageNet-A Hendrycks et al. ().
The results, shown in Table 5, demonstrate that models trained on our selected data consistently exhibit superior generalization and robustness on these harder ImageNet benchmarks compared to those trained on the original datasets.
Notably, this improved performance is achieved with reduced training costs, further highlighting the efficacy of our approach.

**Table 6: Evaluation of our components on CIFAR-100 (C-100) and Tiny-ImageNet (T-IN).**
|  | w/o adapter | w/o $\mathcal{L}_{sa}$ | w/o $\mathcal{L}_{sd}$ | w/o $\mathcal{L}_{s}$ | w/o adp&$\mathcal{L}_{sa}$ | w/o adp&$\mathcal{L}_{sd}$ | w/o adp&$\mathcal{L}_{s}$ | w/o adp&$\mathcal{L}_{sd}$&$\mathcal{L}_{sa}$ | Ours |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-100 | 78.20$\pm$0.18 | 78.42$\pm$0.46 | 78.85$\pm$0.05 | 78.48$\pm$0.32 | 78.11$\pm$0.16 | 78.21$\pm$0.07 | 77.10$\pm$0.29 | 77.47$\pm$0.31 | 78.98$\pm$0.09 |
| T-IN | 46.68$\pm$0.12 | 46.79$\pm$0.39 | 49.14$\pm$0.09 | 46.01$\pm$0.38 | 47.23$\pm$0.06 | 46.70$\pm$0.33 | 45.79$\pm$0.11 | 45.69$\pm$0.10 | 49.30$\pm$0.12 |

### 4.5 Ablation Study

Effect of Dataset Adaptation  To assess the effect of dataset adaptation, instead of using fine-tuned image and text adapters, we directly utilize the per-trained CLIP model to derive the scores $\bm{S}_{A}$ and $\bm{S}_{D}$.
The experimental results, presented in Table 6 with a 90% selection ratio, show a significant decline in accuracy, with drops exceeding 2% on Tiny-ImageNet.
Thus, dataset adaptation is essential for effectively transferring the model’s generalization ability to target datasets.
This is particularly crucial for datasets that differ substantially from the pre-training datasets, such as CIFAR, where variations in image sizes and domain are distinct.
Additionally, the adapters are implemented as MLP and fine-tuned with just 25 epochs, ensuring high efficiency.

Effect of Loss Terms  In Table 6, we evaluate the effect of each loss term and their combination in Eq. 6.
The overall loss function achieves the highest accuracy.
When $\mathcal{L}_{sa}$ is omitted, the selection process tends to prefer more diverse samples, but some class-representative samples may not be selected, which deteriorates the model performance severely.
Without $\mathcal{L}_{sd}$, the selection emphasizes the most category-representative samples.
Although the resulting performance drop is slightly smaller, the diversity of the selected datasets is compromised.
Therefore, the incorporation of $\mathcal{L}_{sd}$ ensures a balanced representation of the selected dataset.
Without $\mathcal{L}_{s}$, since we can not obtain the binarized selection decisions w.r.t. the expected selection ratios, we directly sort the scores in $\bm{d}$ and select the samples with higher scores.
However, this degrades our method into a totally score-based selection and fails to address the group effect, leading to a noticeable drop in performance.

## 5 Conclusion

This paper proposes a novel CLIP-powered data selection framework that leverages multimodal information for more robust and generalizable sample selection.
To achieve this, our proposed framework incorporates three modules: dataset adaptation, sample scoring, and selection optimization modules.
These modules assess the data effectiveness in model training and optimize the selection results w.r.t. the expected results.
As a result, our framework is capable of selecting the most representative samples with high diversity.
Extensive experiments demonstrate the effectiveness and efficiency of our approach, especially in terms of generalization performance on large-scale datasets and robustness in more challenging scenarios, such as noisy and corrupted images.
Future work will explore the application of our method in multimodal datasets and adapt it for other computer vision tasks, such as object detection.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: The authors should take the above questions into consideration: the utilization of the adapter, the comparison, and the parameter.
1. The method requires training an adapter with data from the training dataset for data selection. If the noise ratio is high (such as 50%), wouldn't it be a better option not to use an adapter?
2. Does the Actual Selection Costs in Appendix G include the training time for the Adapter? Is the comparison fair?
3. From the ablation experiment, it can be seen that selection loss has a significant impact on the final result. So, what is the sensitivity of this parameter?
soundness: 3
strengths: 1. The method performs more robustly compared to other baseline data selection methods in various popularly datasets.
2. The algorithm is easy to understand and to follow.
3. The writing of this paper is overall great.
summary: This paper introduces a CLIP-based multimodal data selection framework that enhances the robustness and generalizability of data selection by leveraging both image and text information. Firstly, the paper trains an adapter to transfer pretrained knowledge to the target data. Then, the similarity between the image and text is used to calculate the sampling score for subset selection. The authors evaluate their results in various popularly datasets, obtaining state-of-the-art results in almost all of them.
weaknesses: Some details and motivation should be explained further.
1. The method requires training an adapter with data from the training dataset for data selection. If the noise ratio is high (such as 50%), wouldn't it be a better option not to use an adapter?
2. Does the Actual Selection Costs in Appendix G include the training time for the Adapter? Is the comparison fair?
3. From the ablation experiment, it can be seen that selection loss has a significant impact on the final result. So, what is the sensitivity of this parameter?

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: See above.
soundness: 4
strengths: The idea of exploiting multimodal features from the CLIP model is interesting and plausible. 

The proposed method is simple, which is a strength in my opinion. 

The method is also efficient and is able to control the alignment, diversity and selection ratio in a multiobjective optimization efficiently. 

The paper is well written and easy to follow. 

The results are good.
summary: This paper argues that current image data selection methods are limited because they are unimodal. It proposes a multimodal method that uses the category texts from pretrained CLIP to complement images for more robust and generalized data selection. The proposed framework consists of three modules (1) dataset adaptation that integrates image and text adapters to transfer prior knowledge to the target data; (2) sample scoring that calculates the semantic alignment and diversity scores based on the multimodal features, measuring the image-text alignment as well as the local pattern variability; (3) Selection optimization that uses the two scores to select semantically representative and diverse samples, and introduces selection optimization to efficiently identify the ideal data subsets given an expected selection ratio through a multi-objective optimization strategy.

Post discussion comments:
The authors have addressed my concerns. With the additional clarifications and material, the paper is in much better shape. I am keeping my score at 8, which correctly reflects the quality of this work.
weaknesses: The proposed method relies on a pretrained CLIP and hence any biases in the CLIP model will propagate to the selected dataset. 

The proposed method optimizes alignment and diversity but does it have any indirect effect on bias in the dataset?  

Is it possible to control bias in the dataset or in the subsequent models that are trained on the selected dataset? 

Does the STE cause convergence issues? 

The variable d (sample wise parameter) can be easily confused with d (feature dimension). Consider changing one of them. Also, I don’t see d in Fig. 1. Is “w: N x 1” the sample wise parameter d? Or am I missing something? 

The caption of Tab.3 needs to be corrected. You can also merge Tab.3 with Tab.2.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - The text-image alignment is like using CLIP to measure the difficulty in learning the data. Noisy data or corrupted data are difficult to learn. However, will the selection also filter some important yet difficult data?
soundness: 3
strengths: - Connecting the text and image information for the dataset selection is novel. The proposed CLIP-based method implements the idea well. 
- The SGD-based selection optimization with multi-objective is interesting. It is different from the existing sampling strategies with combinational optimization. 
- Sufficient experimental results show the effectiveness, including different datasets, different settings, and different model architectures.
summary: This paper proposes a novel CLIP-based data selection method leveraging multimodal information with an SGD optimization module. The experiments on several benchmarks show evident improvements over existing approaches.
weaknesses: - The work relies on two adapters to project the CLIP features to the dataset-specific embedding space. The adapter training is based on perfect data. In fact, under the setting of noisy or corrupted data, this ideal data is inaccessible. Therefore, the experiments would be problematic. In Lines 514-515, the authors depict "is essential for effectively transferring the model’s generalization ability to target datasets". It seems the adapter has a large influence on the performance.  Will using the *imperfect* data to fine-tune the adapter degrade the method's effectiveness? This needs to be discussed.

- The class label in this work plays a role as a prototype, while previous work also uses prototypes. They [a] often use the average image features as the prototype and calculate the Euclidean distance between the embedded image and the corresponding prototype, similar to SAS (Eq. 1). Besides, they also use this distance to filter noisy labels [b]. Intuitively, we can use the average image feature to replace the text feature in Eq. 1 of the proposed method. Therefore, it is essential to discuss these results to convey to readers that introducing text is significant.  

- MoSo is NeurIPS'23 rather than NeurIPS'24. Therefore, methods from 2024 have not been compared, e.g., [c]. Besides, some typical methods good at low sampling ratios are missing, e.g., [d].

- Typo: the symbol for the learnable parameter in Figure 2 should be $\mathbf{d}$ rather than $\mathbf{w}$.

[a] Moderate coreset: A universal method of data selection for real-world data-efficient deep learning, ICLR, 2023

[b] NGC: A unified framework for learning with open-world noisy data, ICCV, 2021

[c] Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning, CVPR, 2024

[d] Submodular Combinatorial Information Measures with Applications in Machine Learning, ALT, 2021

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Could the authors provide more detail on the computational efficiency of the Selection Optimization module for very large-scale datasets?
2. Has the framework been tested on datasets with high modality imbalance (e.g., more text than images), and how does it handle such scenarios?
3. For real-world noisy datasets, could the authors provide more information on how they define and quantify "noisy" samples?
soundness: 3
strengths: 1. Originality: The use of multimodal data via CLIP for data selection is innovative and distinguishes this work from single-modality approaches.
2. Quality: The methodology is backed by rigorous experiments, with a well-defined scoring system for sample selection (SAS and SDS).
3. Clarity: Visualizations effectively depict the advantages of multimodal selection over traditional methods.
4. Significance: This framework has broad applications, as it improves dataset quality and model robustness, potentially benefiting various domains using machine learning.
summary: The paper proposes a CLIP-powered framework for data selection that addresses the limitations of traditional single-modality data selection methods by incorporating multimodal information. This framework utilizes a pretrained vision-language model (CLIP) to integrate image and text modalities, thereby enhancing data selection accuracy and robustness. The framework is built on three modules: Dataset Adaptation, Sample Scoring, and Selection Optimization. It leverages both semantic alignment and diversity scores to refine sample selection, removing redundant and noisy data to improve training efficiency and performance on benchmark datasets. Experiments demonstrate the method's superior performance in generalization and robustness compared to existing state-of-the-art approaches, especially in noisy environments.
weaknesses: 1. Complexity: While effective, the Selection Optimization module’s complexity might hinder its application to extremely large datasets without computational resources.
2. Generality: The framework is primarily tested on vision-based datasets; extending it to text or mixed-modality datasets may reveal further limitations. Future work could focus on enhancing the framework’s versatility across other modalities​

Meta-review: This work proposes a new framework for data selection to address the computational overhead and the impact of noisy data when training deep learning models. The key idea is to leverage multimodal information to better select important data and this is achieved by utilising the CLIP model. A set of scores is developed and a multi-objective optimisation is implemented to realise data selection. Experimental study demonstrates the efficacy of the proposed method. Reviewers are overall positive on this work, indicating the strengths on its robust performance, easy to understand, simple and efficient, and its originality and significance. Meanwhile, the reviewers raise the issues related to the clarification of some details, the impact of bias, convergence, the presence of imperfect data, comparison with the last methods, and the complexity and generality of the method. The authors provide a high quality rebuttal and effectively address most of the issues. All the final ratings are on the positive side. By checking the submission, the reviews, and the rebuttals, AC agrees with the reviewers on their observations. Therefore, this work is recommended for acceptance.

Decision: Accept (Spotlight)