A CLIP-Powered Framework for Robust and Generalizable Data Selection
Authors: Suorong Yang
Peng Ye
Wanli Ouyang
Dongzhan Zhou
Furao Shen
Venue: ICLR 2025 Spotlight (2025)
Keywords: Data selection
generalization
multimodal
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Large-scale datasets have been pivotal to the advancements of deep learning models in recent years, but training on such large datasets inevitably incurs substantial storage and computational overhead. 
Meanwhile, real-world datasets often contain redundant and noisy data, imposing a negative impact on training efficiency and model performance.
Data selection has shown promise in identifying the most representative samples from the entire dataset, which aims to minimize the performance gap with reduced training costs. 
Existing works typically rely on single-modality information to assign importance scores for individual samples, which may lead to inaccurate assessments, especially when dealing with noisy or corrupted samples.
To address this limitation, we propose a novel CLIP-powered data selection framework that leverages multimodal information for more robust and generalizable sample selection. 
Specifically, our framework consists of three key modules—dataset adaptation, sample scoring, and selection optimization—that together harness extensive pre-trained multimodal knowledge to comprehensively assess sample influence and optimize the selection results through multi-objective optimization. 
Extensive experiments demonstrate that our approach consistently outperforms existing state-of-the-art baselines on various benchmark datasets. Notably, our method effectively removes noisy or damaged samples from the dataset, enabling it to achieve even higher performance with less data. This indicates that it is not only a way to accelerate training but can also improve overall data quality.
 The implementation is available at https://github.com/Jackbrocp/clip-powered-data-selection.

Published as a conference paper at ICLR 2025
A CLIP-POWERED FRAMEWORK FOR ROBUST AND GENERALIZABLE DATA SELECTION
Suorong Yang Peng Ye Wanli Ouyang', Dongzhan Zhou? Furao Shen'
National Key Laboratory for Novel Software Technology. Nanjing University
2 Shanghai Artificial Intelligence Laboratory
The Chinese University of Hong Kong
ABSTRACT
Large-scale datasets have been pivotal to the advancements of deep learning mod- els in recent years, but training on such large datasets inevitably incurs substan- tial storage and computational overhead. Meanwhile, real-world datasets often contain redundant and noisy data, imposing negative impact on training ef- ficiency and model performance. Data selection has shown promise iden- tifying the most representative samples from the entire dataset, which aims minimize the performance gap with reduced training costs. Existing works typi- cally rely ual samples, which may lead to inaccurate assessments, especially when dealing with noisy or corrupted samples. To address this limitation, propose novel CLIP-powered data selection framework that leverages multimodal information for more robust and generalizable sample selection. Specifically, our framework consists of three key modules-datasee dataset adaptation, sample scoring, and selection optimization that together harness extensive pre-trained multimodal knowledge to comprehensively assess sample influence and optimize the selection results through multi-objeccive optimization. Extensive experiments demonstrate that our approach consistently outperforms existing state-of-the-as baselines various benchmark datasets. Notably, our method effectively removes noisy or damaged samples from the dataset, enabling it to achieve even higher performance with less data. This indicates that it is not only a way to accelerate training but can also improve overall data quality. The implementation is available at httpp:: ///gihhub.ccmm/accooro///// powered data seleeleee selection
monlltymdallty information to assign importance scores for individ-
INTRODUCTION
Recent advancements in deep learning have been propelled by increasingly large and complex mod- els that utilize vast datasets to achieve of performance Liu et al. (2024b): Touvron et al. (2023). However, this success normally comes with considerable costs for data storage and compu- tational resources, which may even limit the deployment of models to specialized infrastructure and hinder their scalability across different applications. Moreover, real-world datasets often contain redundancy and noise, which can degrade the training efficiency and performance.
To alleviate the data redundancy issue and improve the training efficiency, there typically two kinds of methods, i.e., dynamic pruning and data selection. Dynamic pruning methods Raju et al. (202T) Qin al. (2024) aim to reduce training costs by dynamically selecting only the most influ- ential samples from the dataset during training. Despite their effectiveness in accelerating training, they still face the cost of large-scale data storage, and their selected samples often lack the ability to generalize well across different training processes and architectures. In contrast, data selection methods Paul al. (202T): Yang et al. (202bb); Tan et al. (2024): Xiaa et al. (202) pre-select fixed subset of the essential data points before the training begins, allowing the model to achieve performance comparable to that obtained with the full dataset. By focusing on the most critical data points, these methods ensure better generalization ability across various training scenarios.
This work was done during his internship at Shanghai Artificial Intelligence Laboratory. Corresponding authors.

---

Published as a conference paper at ICLR 2025
Traditional Method
Image modality
Vision Model
Single Modality
Selection
[torch]
[goldfish]
[teapot]
Noisy samples
Noisy samples with disturbed labels
Selected
Method
Image modality
Vision Model
Multi-modal
Selection
[torch]
[goldfish]
[teapot]
Text modality
Text
Encoder
Noisy samples
Noisy samples with disturbed labels
Not Selected
Benefit of multimodal selection. Traditional single-modalne approaches (upper part) may struggle with noisy and corrupted data, while our multimodal method (lower part) identifies class- representative samples with diversity while effectively filtering out noise and corrupted data.
Figure I:
Existing data selection methods typically employ carefully designed criteria via three perspectives: importance scores Paul et al.] (2021); Tan et al. (202)). image data distribution Zheng et al. (2023); Xia et al. 2023), and optimization--a based functions Killamsetty et al. (2021b); Yang et al. (202bb).
Although achieving promising results, these methods exhibit certain limitations. On one hand, lying solely on
image information can lead to ambiguities, especially when noisy samples present, which may result in inaccurate assessments of sample's effect. For instance, some methods employ difficulty-bart based criteria to select data; however, distinguishing between truly difficult samples and noisy ones based solely image modalities presents significant challenge. On the other hand, existing methods typically select samples with the highest or lowest scores, while the interaction between high-score and low-score samples within a group can significantly influence the overall performance, which is known as the "group effect" Koh et al. (2019); Yang et al. (2023b). Thus, more beneficial approach is to leverage the power of multimodal information and evaluate the collective effect of the sample group.
In this paper, we propose a CLIP-powerde data selection framework that employs multimodal infor- mation for more robust and generalizable data selection, where the category text serves as strong complement the image modality and promotes overall performance. The framework consists of three modules, namely dataset adaptation, sampling scoring, and selection optimization module. First, the dataset adaptation module integrates image and text adapters to facilitate the transfer of pretraining knowledge to the target data. Subsequently, the sample scoring module calculates the Se- mantic Alignment Score (SAS) and Sample Diversity Score (SDS) based on the adapted multimodal features, which measure the image-text alignment and the variability of local patterns. Using these two scores together can select semantically representative samples while maintaining their inherent diversity. Further, in order to address the group effect, we introduce selection optimization module to identify the idcal subsets w.r.t. the expected selection ratio through multi multi--bjeectve optimization strategy. By leveraging the multi-modal information and carefully designed supervision signals, our framework enables the selection of high-quality samples in a flexible and efficient manner.
Comprehensive evaluation across various benchmark datasets demonstrates that our approach ef- fectively improves the performance of selected datasets, especially on large-scale datasets such as ImageNet- 1k Deng et al. (2009). Moreover, the selected datasets exhibit superior cross- architcrciee generalization across Nes He et al.] (2016), VIT Dosovitskiy et al.) (2020). VGG-16 Si- monyan & Zisserman (2014). etc. Notably, since most existing methods are not robust to more complex and realistic scenarios, challenging scenes. For instance, our proposed method can achieve an 8.13% improvement in accu- racy CIFAR- 100 and 4.41% improvement on Tiny-ImmgeN compared to the leading baselines. Meanwhile, superior performance is achieved with very high efficiency compared to other baselines.
further validate the strong robustness of our method in more
The contributions of this work can be summarized follows: 1) We analyze the drawbacks of pre- vious works that rely solely on image modalities in depth, and propose a new CLIP CLIP-powered data selection framework that leverages multimodal features for robust and generalizable data selection
2

---

Published as a conference paper at ICLR 2025
for the first time. 2) Our framework comprises dataset adaptation and sample scoring modules to knowledge transfer and comprehensive sample importance evaluation. This dual-modality design effectively removes noisy and corrupted samples from the dataset. 3) A se- lection optimization module is designed to identify the optimal subsets w.r.t. the expected selection ratios through multi-objective optimization, which effectively addresses the group effect while main- taining high efficiency. 4) Experimental results show that our method outperforms previous SOTA approaches in terms of performance, cross-architecture generalization, and robustness to noisy and corrupted images. Meanwhile, our approach achieves the best trade-off in performance and selection efficiency, establishing strong baseline of data selection for future research.
foster
RELATED WORK
Date-efficient deep learning generally incorporates dynamic data pruning Qin et al. (2024); Raju et al. (202I), static data selection Xia et al. (2023) Tan et al. (2024): Yang et al. (2023b). data
distillation Lei & Tao (023): Du et al. (2023); Zhang et al. (2023); Cazenavette et al. (2022). and data condensation Liu et al. (2023; Yang et al. (2023a). Following the static data selection, we propose a method capable of identifying representative and diverse samples across various selection ratios. Data selection, static dataset pruning, aims to identify and select the most representative samples from training datasets before training begins. Training on the selected subsets can achieve comparable performance to that obtained with the full dataset while reducing training and storage costs. Current data selection methods can be broadly divided into three categories: importance criteria Paul al. (2021); [Tan al. (2024), dataset distribution-based methods Xia al. (2023); Zheng et al. 2023) and optimization-baser methods Nohyun et al. (003)): Yang et al. (2023c).
Selection with importance criteria is the most popular type. These methods typically involve calculating importance scores for each sample and selecting samples based on these scores, For in- stance, EL2N and GraNd score Paul et al. (202T) measure the importance by calculating the expec- tation of the fz-norm eтror vector and the expectation of the gradient norm, respectively. MoSo Tan al. (2024) calculates the change of the optimal empirical risk when removing a specific sample from the training set. Forgetting Toneva et al. (2018) tracks the frequency with which sample is misclassified after being correctly classified during training. Similarly, Memorization Feldman & Zhang (2020) assesses the impact of sample's 's presence or absence on the model's ability to pre- dict it correctly. While importance criteria-based methods are often computationally efficient, their performance may be affected by the group effect Yang et al. (203bb c) and may not generalize well to complex, real-world scenarios Xia et al. (2023).
Dataset distribution-base methods select samples by analyzing the geometric distribution of datasets. For instance, Herding [Welling (2009) determines the sample importance according to samples' distance their corresponding class centers. The work |Ramalingam et al. (2023) applies greedy k-center to select the coreset with good data coverage. D2 Maharana et af. (2023) calculates and updates the difficulty scores of each sample by incorporating the difficulty of its neighboring examples. Moderate- DS Xia et al. (2023) choose samples with closer distances to the median score, while CCS Zheng et al. (2023) balances the data distribution and the sample importance in selection.
Optimization-based methods select samples through various optimization techniques, such as tem- poral dual-depth scoring Zhang et al. (2024). gradient matching MMrzzaoleeimnn et al. (2020b); Kil- lamsetty et al. (2021a). scalable self-supervised pruning metric SSorchher et al. (2022). influence function Yang et al. (2023b); Pooladzandi et al. (2022), bi-level optimization Killamsetty et al. (2021b). facility location function Mirzasoleima et al. (2020)); Yang et al. (2023d) ) and submodu- larity et al. (2021): Nohyun et al. (2023); Kothawade et al. (2022): Wei et al. (2015). In contrast to these methods that only rely on image information, we leverage multimodal messages for data lection, which incorporates the semantic alignment between image data and corresponding category information. This contributes to comprehensive assessments of sample effectiveness, particularly in complex scenarios where samples may be corrupted or wrongly labeled.
3 THE PROPOSED METHOD
Our proposed method is summarized in Figure 2 The approach involves the use of the pretrained vision-language foundation model CLIP to construct multimodal feature spaces. Nevertheless,
3

---

Published as a conference paper at ICLR 2025
Madility
AnG
Sample Scoring
Sa
Sample Diveriity Losi
Syt NOXE
Subset
I-Encoder
I-Adapter
Semaniic Alignment
Subwer
Leuualia Farauaar
Sulsers
Prining
Aphota af (CSS) Wid
Selection Optimization
T-Encoder
T-Adupter
Bemertess: mi multpiictioo
Tremal Modallty
Pre-trained Vision-
Language Model
Dataset Adaption
expested selection
the Exracted festures of modiiltty
Figure 2: Our proposed method consists of dataset adaptation, sampling scoring, and selection optimization modules. The dataset adaptation module is used to learn dataset-specific knowledge. The sample scoring module computes two scores, SA and Sp, to assess sample importance, based on which the selection optimization identifies the optimal subsets w.r.t the expected selection ratios.
there may exist domain shifts or discrepancies between the pretrained training dataset and the tar- get dataset [Lu al. (2024a); [Alijani et al. (2024). To facilitate dataset adaptation and enhance the learning of dataset-specific knowledge, we incorporate cimennsion-preservnng adapters for both modalities. Following this, two scores are derived to comprehensively assess the sample impor- tance, i.c., the Semantic Alignment Score (SAS), denoted as Sa. and the Sample Diversity Score (SDS), denoted as Sp- Furthermore, rather than solely based sample scores for selection, design a multi-ohjective optimization to identify the optimal subsets w.r.t. the expected selection ratios, which effectively mitigates the group effect. We provide the detailed methodology in the subsequent sections.
3.1 DATASET ADAPTATION
To alleviate the domain shifts and discrepancies between the pretrained and target datasets, we in- corporate dimension- pereeervingge adapters to perform adaptation the target dataset. The image and text adapters are denoted as Ay and Ar, respectively. Both adapters are fine-tuned for knowledge transfer while the pretrained CLIP weights frozen. To maintain high efficiency, both adapters utilize simple MLP.
Specifically, the fine-tuning process employs the InfoNCE loss Parulekar al. 2023 Oord al. (018), which maximizes the mutual information between the image and text representations. The text representation describes the category information using the prompt: "A photo of [CLS]", where the token [CLS) represents the corresponding category. The loss ensures that the adapters effectively align and capture the relevant features from both modalities. Furthermore, it enhances the model's ability to distinguish between positive and negative pairs, thereby improving the robustness and accuracy of the deep representations for the specific dataset.
3.2 SAMPLE SCORING
For classification datasets, the learning process for training samples is intrinsically linked acquir- ing knowledge of the corresponding categories. A training sample that more accurately represents its category is typically more effective in training deep networks. In this way, the Semantic Align- ment Score (SAS) is designed to assess the semantic similarity between training samples and their corresponding categories. Specifically, since image and text features reside within the same embed- ding space Radford et al.] (2021), the SAS is derived by calculating the cosine similarity between the embedded image and corresponding textual deep descriptions. Accordingly, the SAS for the i-th sample is defined as:
S cos (Ar(EE((L))) AT AT(Er(T:i))
(1)
where I is the i-th sample, T, is the textual description the corresponding category for I, Es and E, are pretrained image and text encoders, respectively.
For selected datasets, the reduced data volume may limit the diversity the selected data, which is important for training datasets Yang al. 2024a).. To address this, introduce another diversity
4

---

Published as a conference paper at ICLR 2025
Batt
Solected
Not Selected
Random selection
(b) With SDS
With
(d) With both SDS and SAS
Figure 3: Illustration of the effectivvness of SAS and SDS. The circle and cross represent the normal and noisy samples, respectively. Different colors correspond to the selection results. SDS (b) selects diverse samples but may include noises. SAS (c) could avoid the noisy samples but potentially miss broader category information. Using both scores (d) can select category represettteee while maintaining high diversity.
perspective to comprehensively assess the effect of training data. The Sample Diversity Score (SDS) is defined as the average distance between each sample and its k neighbor samples of the same class:
1 E |Ar(E,(I)) - (E) (L)
D
IJEKNN(L,)
where we use the KNN algorithm to obtain the neighbor samples for each sample, the distance metric is based the &g norm, and is usually set to 10% of the number of samples per class. In this way, the SDS can be understood as the local density of training samples in the feature space. If sample has a larger number of neighbors with closer distances (i.c., lower SDS), its training efficacy may be more easily substituted by other neighbor samples. Therefore, selecting samples with higher SDS is generally more advantageous.
The effects of SAS and SDS are depicted in Figure 3 SDS contributes to the diversity of samples, but the selected data points may contain noise (Figure b))). On the other hand, SAS can select samples that are semantically appropriate, as these points are basically around the sample center (Figure (()) However, these samples may be too concentrated and thus lack diversity. When using SDS and SAS together (Figure3(d))
cover the entire category space with fewer data and se- lect samples that are both semantically representative and diverse, thereby boosting the effectiveness of data selection.
3.3 SELECTION OPTIMIZATION
Instead of relying on combinatorial optimization functions for sample selection Killamsetty et al. (2221), method determines the selection through SGD multi-objective optimization, which im- proves computational efficiency and accelerates convergence. Specifically, we introduce a sample- wise parameter d to denote the selection decision, where elements of indicate the selection while 0 indicates otherwise. Although binary parameters are difficult to optimize in neural networks due to the absence of gradient, we employ the sigmoid(-) function push the continuous values of towards approximate binarization. After optimization, d is strictly binarized to explicitly indicate the final sample selection. Initially, is initialized with all 1s.
To guide the optimization process, we introduce three loss items. The first item, Lea- is designed to prioritize samples with high SAS since these samples are more representative of their corresponding categories, which is defined as follows:
1 N Σ sigmoid(d) SAL.
Lea
(3)
where N is the total number of samples. Lsa punishes samples with low SAS and encourages the selection of samples with better semantic alignment.
In addition, we introduce another loss term, Led, encourage the selection of more diverse samples characterized by higher SDS, which is defined as:
Led
N Σ sigmoid(d) Spi.
(4)
To mitigate the group effect, we optimize the selected datasets w.r.t. specific selection ratios, aim- ing to identify the optimal subsets. We introduce selection loss term, L., ensure the selection
5

---

Published as a conference paper at ICLR 2025
80
50
91.0
75
90.5
Test Accuracy (96)
Test Accuracy (%)
Test Tost Accuracy (96
40-
65
Herding
Herding
Torgetting
Forger mg
89.0
Rontor
Crahd
Graldd
Harsing
60-
2L2N
Buay
Porpetans
Seft cnlectioon
Sa Prup exlectio
88.5
Gmakd
Nestrak
ELah
55
00.0
Matorate
Mado
Mide
Milao
50
70
Dum
86
90
100
25
87.5
70
90 100
Selection Rate (%)
Selection Rate (%)
Selection Rate (%)
(a) CIFAR-100
(b) Iiage-m
(c) ImageNet-1K 1k
Figure 4: Illustratioon of comparing our method with various data selection baselines on CIFAR-100 (a). Tiny-lmageNet (b), and ImageNet- Ik (c).
process adheres to the target ratio. However, deriving exact selection rates from continuous real- valued parameter optimization is difficult. While strictly binarized values facilitate explicit sample selection, they are challenging to optimize through gradient backpropagation. To address this, we utlize the straigh-tthrouge estimator (STE) |engii et al. (2013) to estimate the actual selection rate and derive gradients. STE allows gradients to pass through the discrete decisions during backprop- agation, effectively combining the benefits of both continuous and binary parameters for efficient optimization and accurate sample selection. In this way, Lg is defined as:
C.
N Σ STE [I (sigmoidd()) > 0.5)] Sy
where I an indicator function, and 8, denotes the expected selection ratio. The loss term guides the parameter d toward near-binary values, ensuring the count of ones aligns with the expected sample size. Since the selection is guided by adaptive optimization, the final selection ratio may deviate slightly from the target. To minimize this deviation, constrain L, with a threshold 0, which in our work is set to 5 x 10-4, ensuring the actual selection ratio differs by less than +0.05% from the expected value. We also provide theoretical analysis of on the actual selection ratio gap in Appendix A Finally, the overall loss function is formulated as:
L= L aLed + BC.,,
where a and 3 are coefficients that adjust for numerical differences among the loss terms and can be set conveniently. The complete workflow is outlined in Algorithm I in Appendix B
Complexity Analysis The proposed method comprises three main components. 1) The dataset adaptation involves fine-tuning the image and text adapters. Since the adapters consist of simple lin- ear layers, the number of parameters small, and both the forward passes and backward passes are computationally efficient. 2) In the sample scoring process, the complexity of calculating the SAS and SDS is O(N) and O(K * fa), respectively, where is the number of categories and fa is the feature dimension, typically 512. The complexity of the KNN algorithm is O(N₂ fa), where N₂. is the number of samples per class. Given that K and fa are constants and N₂ is usually much smaller than N, the overall complexity of this process is approximately O(N). 3) The selection optimiza- tion of dl does not involve deep models and is a numerical optimization process. The complexity is proportional the number of parameters, i.e., O(ww)) = O(N).
EXPERIMENT
4.1 EXPERIMENTAL SETUP
Baselines. We compare our proposed method with ten most representative SOTA baselines. i.c., (1) Random, (2) MoSo Tan et al. (2024), (3) Glister Killamsetty et al. (2021b), (4) Herding Welling

---

Published as a conference paper at ICLR 2025
Table I: Test accuracy (%) on
VGG-16 and DenseNet- are exploited.
VGG
Densenet
Method / Selection Ratio
70%
80%
90%
100%
701%
M0P
90%
100%
Random
47.3
29.38
51.15
57.23
59.55 39.55
60.78
61.03_022
62.22.0.2)
F1,2N
48.30
48.75
41.01
57.23
59.61
60.38
61.16-04
62.22
GraNd
50.79
46.84
54.73
57.23=1.08
59. 39.62=a.2
60.84200
61.10=0m
62. 02.222+0.35
MoSo
10.47=1011
50.12=083
50.07:00.3
57.23+1108
59.27
59. 39.86=a077
60.00=037
62.22+0:3
Herding
48 20077
45.77
50.77
57.23
59.00
60.03:a35
61.15:0.2
62.22:002
Glister
48.74.1179
50.05
49.42.1181
57.23
1em
60.62.18
62.22.1021
CG-Score
48.73+2.70
48, 1..49
19.62.a1.05
57.23+1.06
59.74
60.55
61.14
62.22::025
Sclf-sup. prototypes
48.38+1135
49.98=a149
54.71
57.23+1106
99.5640022
60.22+a11
60.91=029
62.22+6:3
Forgetting
47.50=2243
48.59
49.82.4062
57.23
58.54:0.15
60.39
61.12:0.10
62.2240.23
Moderate DS
50.78
2943
19.25.107
57.23
19.41
60.42..0
61.44..01
62.22,102
53,40
52.25
56.34:293
57.23+106
60.12:0.86
60.93:003
61.59
62.224003
Table 2: Experimental results (accuracy, %,, mean + std) on CIFAR-10M and Tiny-ImageNet with noisy labels. 20% of labels are disturbed. We also report the numerical analysis of the proportion (%) of noisy data in the selected CIFAR-100 datasets.
CIFAR- 100 (label noise)
Tiny-ImageNet (label noise)
Noise ratios
Method/ Selection Ratio
20%
30%
20%
30%
20%
30%
Random
34.47.064
43,261121
17.782044
23.88
20.80
19.83
MoSo
=1,01=067
43.73-20.44
21.55
27.80=0.16
7.78
8.82
Moderate-DS
40.25.1012
41.531160
19.64.10.0
24.96.1030
0.31
Glister
28.51=146
43.16=131
21.61
25.45=2023
21.21
21.95
Herding
42.29=1755
50.522138
28
24.23:10.29
35.00
30.56
Forgetting
36.53=1111
45.78=104
13,20=03s
21.79=0043
23.00
21.76
GraNd
31.72.1067
42.8010.30
18,28
23.72.10.8
5.00
5.14
FI.2N
29.82=1119
33.62+2.35
13.93=mm
18.57=2031
22.00
21.80
Self-sup. prototypes
31.08
41.87:1003
10:00
21.01:1036
21.70
20.21
CG-Score
6.82+1.00
20.07
8,35
15.31
45.09
39.69
Ours
46.05
58.34
26.09110.1
33.13-1025
0.25
2009),
(5) Forgetting Toneva al.
(018)),
(6) GraNd and (7) EL2N Paul al. (2021), (8) Self-
sup.-selection (SSP) Sorscher et al. 2022). (9) CG-Score Nohyun et al. (203)). and (10) Moderate- DS Xia et al. (2023
Parameter settings. The parameters in our proposed method can be easily set. The coefficient proportional the expected selection rate sy, balancing the importance of dataset diversity, i.c., can be set equivalent to 8,. The coefficient / is set to 2 across datasets to adjust the numerical differences among loss items. For more details, please refer to Appendix
4.2 COMPARISON WITH THE
Performance Comparisons Consistent with prior works Xia et al. (2023): Sorscher et al. 2022), report top-1 accuracy CIFAR-100 and Tiny-lmageNet, and top-5 accuracy ImageNet- 1k. Note that the methods Glister and CG-Score are not compared on ImageNet- due to the heavy computation costs. Specifically, Glister obtains iterative solving of the bi-level optimization problem Killamsetty et al. (2021b), while CG-Score involves the calculation of large Gram matrix inversions. making them impractical for large-scale datasets.
As illustrated in Figure our method consistently achieves the best accuracy across all datasets. Particularly, on the more challenging Tiny-ImageNet and ImageNet- Ik datasets, our approach out- performs other methods by notable margin. While existing approaches yield relatively marginal accuracy improvements on the small-scale CIFAR- 100 dataset, the gains brought by our method are more substantial. Additionally, with relatively high selection ratios these datasets, such 90%, our selected datasets exhibit nearly lossless or even higher performance compared with the original full datasets and other baselines. The results indicate that our method can not only reduce training costs but may also serve as a way for data cleaning.
Generalization Comparisons on Different Architectures In this section, evaluate the gen- eralization effectiveness of our selected datasets on deep architectures different from those used in the selection process. Specifically, the selected datasets from Tiny- ImageNet. As shown in Table the results indicate that our method surpasses all baseline methods both architectures, demonstrating its desirable architectural gen-
employ the VGG-16 and DenseNet- 121 models to train

---

Published as a conference paper at ICLR 2025
45
45
45
Test Ac curuary
35
curacy
35
35
Bardem harding Forgening
Ransom Hording
Rascom
Hard
Forpett ng
30
Forpet ng
Grvld
Grald
Grand
25
E.2%
SSF
25
Bul
Test
25
BLN
Modorate
20
OuT
15
15
20
30
40
60
80
20
30
40
60
80
20
30
40
Selection Rate (%))
Selection Rate (96
Selection Rate (%)
(a) 5% Corruption Rate
(b) 10% Corruption Rate
(c) 20% Corruption Rate
Figure 5: Comparison of robustness to corrupted images.
Woderate
CG-Score
Accuracy
protap
Herding
Forgatting
B.2N
Avg. Pruning Costs (h)
(a) baseline DI=3.0 x 10-5
(b) 90% datasct
DI=4.3 x 10
Figure 6: Effectiveness efficiency C- 100. Results are reported with R-50 under 30% selection ratio 4-2080TI GPU.
Figure 7: Visualization of the test set distri- bution. DI: Dunn Index.
cralization ability. This suggests that the selected datasets are broadly applicable, irrespective of the specific network architecture.
Training Efficiency Comparisons To evaluate the selection efficiency of various methods, we present analysis of the balance between effectiveness and efficiency. As shown in Figure 6 our method presents the best performance with desirable efficiency. Herding, EL,N,, and GraNd obtain the lowest selection costs because they rely predefined metrics select samples in very early training. Our method slightly slower than theirs but exhibits higher accuracy. Compared with the optimization-bate based approaches, like MoSo and Glister, our method enjoys both lower costs and better performance. The results verify the effectiveness of our method in balancing selection efficiency and accuracy.
4.3 COMPARISON ROBUSTNESS
Robustness on Noisy Labels Real-world datasets often involve label noise, where some sample labels are incorrectly flipped, resulting in mislabeled data. Unfortunately, creating clean and diverse datasets is time-consuming and expensive. Therefore, it is necessary evaluate the performance of selection methods under such complex scenarios. this study, we introduce symmetric noises Li ct al. (222) to generate mislabeled data both C-100 and T-ImageNet, with a 20% noise rate.
As can be seen Table 2 our approach exhibits superior robustness to noisy labels, significantly outperforming other baselines by large margin. Specifically, our approach yields improvements of over 10.12% on CIFAR-100 and 4.41% on Tiny-ImageNe compared to previous leading methods. Additionally, Table 2 shows the distribution of noisy data within the selected datasets, where our method selects only 0.24% noisy samples, considerably fewer than other baselines. This reduction in noise underscores the potential of our method improve overall data quality.
8

---

Published as a conference paper at ICLR 2025
Table 3: Performance and saved costs (%) on ImageNet-1k lk across Swin-T, ViT-B, and ViT-L on 4-1100-GPU serve.
Table 4: Generalization evaluation on more chal- lenging ImageNet- Ik benchmark datasets.
Dataset
Model
80%
90%
Full Data
R-18
10.89
11.33
10.85
ImageNet-Hard
Model
80%
9X76
Full Data
R-50
14.75
14.98
14.75
WTTB
81.13
81.46
81.46
ImageNet-A
R-18
1.65
2.04
1.12
VIT-
84.37
84.74
84.59
R-50
3.17
3.31
3.09
Swin-T
78.05
78.63
78.31
ImageNet-R
R-18
32.99
33.70
33.03
Saved (56)
20.62%
10.31%
R-50
36.60
37.11
36.16
We argue that the robustness of our approach can be attributed to SA in Eq. [ which assesses the semantic alignment between image content and its labels. In the presence of label noise, this align- ment is disrupted, resulting in lower SAS, which in turn reduces the likelihood of such samples being selected during optimization. In contrast, most baseline methods rely solely on image features for selection, which may result in performance degradation when faced with noisy labels. In some cases, the performance of these methods is even worse than that of random selection. While Mod- erate also selects relatively low proportion of noisy data, its performance is worse than ours. This discrepancy highlights the effectiveness of our method in making more strategic selections in noisy environments, thereby not only minimizing noises but also optimizing the selected datasets.
Robustness on Corrupted Images We further evaluate the performance of our proposed method on real-world noise corruptions that frequently encountered (Sngh et al. (2024); Wei et al. (024). To simulate such corruptions, we employ the following five types of realistic noises Xia et al. 2023 including Gaussian noise, random occlusion, resolution, fog, and motion blur. The corruption rate is set to 5%, 10%, and 20%, respectively.
As shown in Figure [ compared with prior baselines, our approach consistently presents greater robustness to corrupted images across varying cortuption rates, demonstrating strong generalization in these challenging scenarios. Notably, even at high corruption rate of 20%, our method maintains desirable generalization performance. This robustness is primarily attributed to the integration of text modality into the selection process, alongside the image modality. The SAS defined in Eq. measures the alignment between the image features and their corresponding category features. When images are corrupted, this alignment is disrupted, thereby reducing the SAS and correspondingly decreasing the likelihood of selecting those images. In contrast, methods such as Forgetting tend to prioritize difficult training samples, potentially making corrupted images more likely to be selected, as these images typically more difficult to correctly classify. As result, these methods are less robust to corrupted images, leading to a deterioration in generalization performance.
4.4 DATASET SELECTION IMPROVES GENERALIZATION
Visualization Analysis To demonstrate the generalization of the selected datasets, we train two models using the original dataset and the selected dataset (90% selection ratio), respectively, and obtain their embedding results the CIFAR-10 test To visualize the dataset distribution, apply t-SNE to the embeddings generated by both models. The visualization in Figure 7 shows that the model trained on the selected dataset produces better embedding results: a better inter- cluster separation and intra-cluster compactness. For quantitative analysis, we use the Dunn Index (Cir al. (222)) to evaluate the clustering results (the higher, the better). After removing 10% of the data, the DI increases by 43% presenting better clustering results.
Generalization More Advanced Architectures We further employ the selected datasets to train more advanced ViT-based architectures, including Swin Transformer, ViT-Base, and ViT- Large. From Table and corroborated by the results from previous sections, our selected datasets consistently achieve lossless performance across both CNN-based and Transformer based architec- tures with reduced training costs. This demonstrates that our approach obtains highly generalizable datasets applicable to wide range of network architectures.
Generalization to More Challenging Benchmark Datasets To further evaluate the generaliza- tion and robustness of models trained our selected datasets, conduct experiments using ResNet- 18 and ResNet-50 models, training on both the full datasets and our selected datasets. These models then tested more challenging benchmarks, including ImageNet-Hard Taesiri

---

Published as a conference paper at ICLR 2025
Table 5: Evaluation of our components on CIFAR-I00 100 (C-100) and Tiny-Imaged (T-IN).
Wio adapter
w/o
wio Cas
wio C.
wio sopk....
we adpee.,.
wio S.
w/o L,
Ours
C-100 T-IN
78 20aus
78.42
78.35=uu
78,48 08aum
78. T
77.47
7838: am
46168,412
46.79.00
49.14.
46.01
47.25
46.70
45.79_..1
45.69
49.30
et al. 2024), ImageNet-R Hendryykks al. (2021a), and ImageNet-A [Hendryeks al. (2021b)
The results, shown in Table 4 demonstrate that models trained on our selected data consistently exhibit superior generalization and robustness on these harder ImageNet benchmarks compared to
those trained on the original datasets. Notably, this improved performance is achieved with reduced training costs, further highlighting the efficacy of our approach.
4.5 ABLATION STUDY
Effect of Dataset Adaptation To assess the effect of dataset adaptation, instead of using fine- tuned image and text adapters, we directly utilize the per-trained CLIP model to derive the scores SA and Sp. The experimental results, presented in Table [ with 90% selection ratio, show a significant decline in accuracy, with drops exceeding 2% on Tiny-ImageNee Thus, dataset adapta- tion is essential for effectively transferring the model's 's generalization ability target datasets. This is particularly crucial for datasets that differ substantially from the pre-training datasets, such as CIFAR, where variations in image sizes and domain are distinct.
Effect of Text Modality Previous works Xia et al. (2023) use the average image features as the prototype and calculate the Euclidean distance between the embedded image and the corresponding prototype, which is used to filter noisy labels Wu et al. (202T). To evaluate the effect of our text modality, we also leverage this distance to assess the noisy samples and replace the text feature in [q With 20% noisy ratio, the accuracy drops from 46.05% to 16.39% with 20% selection ratio and from 58.34% to 38.35% with a 30% selection ratio, validating the effectiveness of introducing textual modality. Due the limited space, full results can be in Appendix L
Effect of Loss Terms In Table 5 we evaluate the effect of each loss term and their combination in Eq. (6] The overall loss function achieves the highest accuracy. When L.e is omitted, the selec- tion process tends to prefer more diverse samples, but some class-representative samples may not be selected, which deteriorates the model performance severely. Without Led- the selection empha- samples. Although the resulting performance drop is slightly
sizes the most galegory--rpreseenn smaller, the diversity of the selected datasets compromised. Therefore, the incorporation of Ced ensures a balanced representation of the selected dataset. Without L,, since we can not obtain the binarized selection decisions w.r.t. the expected selection ratios, we directly sort the scores in d and select the samples with higher scores. However, this degrades our method into totally score-based selection and fails to address the group effect, leading to noticeable drop in performance.
5 DISCUSSION AND CONCLUSION
Limitation and future work. In this section. we discuss some potential limitations and future work for method. 1). While the pretrained CLIP model contributes selecting the most repre- sentative samples, the potential biases in the CLIP model may propagate to the selected dataset AI- abduimohsin et al. (2024). Future work may explore bias mitigation strategies by fine-tuning CLIP with bias-aware loss functions. 2). Our work primarily focuses on vision-based datasets for data selection, using textual modalities to guide sample selection, where both modalities are balanced. However, datasets with high modality imbalance may bring challenges since it is difficult to as- sess the modality alignment. Thus, future work could consider leveraging dynamic modality weight allocation mechanisms and augmentation strategies for modality-inbalanced dataset selection.
This paper proposes a novel CLIP-powered data selection framework that leverages multimodal in- formation for more robust and generalizable sample selection. To achieve this, our proposed frame- work incorporates three modules: dataset adaptation, sample scoring, and selection optimization modules. These modules assess the data effectiveness in model training and optimize the selection results w.r.t. the expected results. As a result, our framework is capable of selecting the most rep- resentative samples with high diversity. Extensive experiments demonstrate the effectiveness and efficiency of our approach, especially in terms of generalization performance on large-scale datasets and robustness in more challenging scenarios, such noisy and corrupted images.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENT
This work supported by the STI 2030-Major Projects of China under Grant 20212D0201 the Fundamental Research Funds for the Central Universities under Grant 2024300394, the National Natural Science Foundation of China under Grant 62276127. This work is supported by the Shang- hai Municipal Science and Technology Major Project. This work is supported by Shanghai Artificial Intelligence Laboratory.
REFERENCES
Ibrahim Alabdulmohsin, Xiao Wang, Andreas Steiner, Priya Goyal, Alexander D' Amour, and Xi- aohua Zhai. Clip the bias: How useful is balancing data in multimodal learning? arXiv preprint arXiv:2403.04547, 2024.
Shadi Alijani, Jamil Fayyad, and Homayoun Najarrn. Vision transformers in domain adaptation and domain generalization: study of robustness. Neural Computing and Applications, pp. 1-29, 2024.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: arXiv: 338. 3432, 2013.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun- Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 4750-4759, June 2022.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.0888 08019, Aug 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. Ieee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Jiawei Du, Yidi Jiang, Vincent YF Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumu- lated trajectory error to improve dataset distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3749-3758, 2023.
Vitaly Feldman and Chiyuan Zhang- What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881- 2891, 2020.
Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra. Vissl. /ttpts:://tttttiee htt.eaaac...ceeeceooooeeeeeeee/a
2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. 11X.v:2111.06377, 2021.
Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A criti- cal analysis of out-of-distribution generalization. Proceedings of the IEEE/CVF international conference on computer vision, pp. 8340-8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- fion, pp. 1-262-152777 2021b.
11

---

Published as a conference paper at ICLR 2025
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Rishabh Iyer, Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. Submodular combinatorial information measures with applications in machine learning. In Algorithmic Learning Theory, pp. 722-754. PMLR, 2021.
Krishnateja Killamsetty, Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pp. 5464-5474. PMLR, 2021a.
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Generalization based data subset selection for efficient and robust learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 8110-8118 2021b.
Pang Wei W Koh, Kai-Siang Ang, Hubert Teo, and Percy S Liang. On the accuracy of influ- ence functions for measuring group effects. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buce E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Sys- lems, volume 32. Curran Associates, Inc., 2019. URL https: / ngs in. neurips.. (tpaper/2119//iee////33333 Ttt.a///................002.1111
Suraj Kothawade, Vishal Kaushal, Ganesh Ramakrishnan, Jeff Bilmes, and Rishabh Iyer. Prism: A unified framework of parameterized submodular information measures for targeted data subset selection and summarization. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI, 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Shiye Lei and Dacheng Tao. A comprehensive survey to dataset distillation. arXiv preprint arXiv:2301.0560. 2023.
Shikun Li, Xiaobo Xia, Shiming Ge, and Tongliang Liu. Selective-supervised contrastive learning with noisy labels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 316-325,20 2022.
Fan Liu, Tianshu Zhang, Wenwen Dai, Wenwen Cai, Xiaocong Zhou, and Delong Chen. Few- shot adaptation of multi-modal foundation models: A survey, arXiv preprint arXiv:2401.01736, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 26296 26306, 2024b.
Songhua Liu, Jingwen Ye, Runpeng Yu, and Xinchao Wang. Slimmable dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 3759-3768, 2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Adyasha Maharana, Prateck Yadav, and Mohit Bansal. D2 pruning: Message passing for balancing diversity and difficulty in data pruning. arXiv preprint arXiv:2310.079))) 20223...
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6960.
PMLR, 2020a.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950-6960. PMLR, 2020b.
12

---

Published as a conference paper at ICLR 2025
Chiheb-Edding Ben Ncir, Abdallah Hamza, and Waad Bouaguel. Parallel and scalable dunn index for the validation of big data clusters. Parallel Computing, 102:102751, 2021.
Ki Nohyun, Hoyong Choi, and Hye Won Chung. Data valuation without training of model. In The Eleventh International Conference on Learning Representations, 2023.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.037445 2018.
Advait Parulekar, Liam Collins, Karthikeyan Shanmugam, Aryan Mokhtari, and Sanjay Shakkot- tai. Infonce loss provably learns cluster-preserving representations. In The Thirty Sixth Annual Conference on Learning Theory, 1914-1961. PMLR, 2023.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Find- ing important examples early in training. Advances in Neural Information Processing Systems, 34:20596-20607, 2021.
Omead Pooladzandi, David Davini, and Baharan Mirzasoleiman. Adaptive second order coresets for data-efficient machine learning. In International Conference on Machine Learning, pp. 17848- 17869, PMLR, 2022.
Ziheng Qin, Kai Wang, Zangwei Zheng, Jianyang Gu, Xiangyu Peng, xu Zhao Pan, Daquan Zhou, Lei Shang, Baigui Sun, Xuansong Xie, and Yang You. Infobatch: Lossless training speed up by unbiased dynamic data pruning. In The Twelfth International Conference on Learning Represen- fations, 2024. URL. htt.sttss://
net/ (oforumiddCCe sk..5k6
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Ravi Raju, Kyle Daruwalla, and Mikko Lipasti. Accelerating deep learning with dynamic data pruning, arXiv preprint arXiv:2111.12622 2021.
Srikumar Ramalingam, Pranjal Awasthi, and Sanjiv Kumar. A weighted k-center algorithm for data subset selection. arXiv preprint arXiv:23/2.10602. 2023.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po- Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys (CSUR), 54(4):1-34, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.155: 2014.
Krishnakant Singh, Thanush Navaratnam, Jannik Holmer, Simone Schaub-Meyer and Stefan Roth. Is synthetic data all we need? benchmarking the robustness of models trained with synthetic im- ages. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2505 2515, 2024.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Alice H. Oh, Alekh Agar- wal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Mohammad Reza Taesiri, Giang Nguyen, Sarra Habchi, Cor-Paul Bezemer, and Anh Nguyen. Imagenet-haid hard: The hardest images remaining from a study of the power of zoom and spatial biases image classification. Advunces Neural Information Processing Systems, 36, 2024.
Haoru Tan, Sitong Wu, Fei Du, Yukang Chen, Zhibin Wang. Fan Wang, and Xiaojuan Qi. Data pruning via moving ooaping-onn-smmll--ouuut Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXtv:1812.051553 2018.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 13977 2023.
Kai Wei, Rishabh Iyer, and Jeff Bilmes, Submodularity in data subset selection and active learning. In International conference on machine learning, pp. 1954-1963, PMLR, 2015.
Tong Wei, Hao-Tian Li, Chun-Shu Li, Jiang-Xin Sing-Xin&&o Shi, Yu-Feng Li, and Min-Ling Zhang, Vision- language models are strong noisy label detectors. arXiv preprint arXiv:2409. 19696, 2024.
Max Welling, Herding dynamical weights leam. In Proceedings of the 26th Annual International Conference Machine Learning, pp. 1121-1128, 2009.
Zhi-Fan Wu, Tong Wei, Jianwen Jiang, Chaojie Mao, Mingqian Tang, and Yu-Feng Li. Nge: A unified framework for learning with open-world noisy data. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 12-71,20211.
Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In The Eleventh International Conference Learning Representations, 2023.
Enneng Yang, Li Shen, Zhenyi Wang, Tongliang Liu, and Guibing Guo. An dataset conden- sation plugin and its application to continual learning. Advances in Neural Information Processing Systems, 36, 2023a.
Shuo Yang. Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reduc- ing training data by examining generalization influence. In International Conference on Learning Representations, 2023b.
Suorong Yang, Hongchao Yang. Suhan Guo, Furao Shen, and Jian Zhao. Not all data matters: An end-to-end adaptive dataset pruning framework for enhancing model performance and efficiency. arXiv preprint 23Xiv:23/2.05599 2023c.
Suorong Yang, Suhan Guo, Jian Zhao, and Furao Shen. Investigating the effectiveness of data aug- mentation from similarity and diversity: An empirical study. Patlern Recognition, 148:1 10204,, 2024a.
Suorong Yang, Furao Shen, and Jian Zhao Entaugment: Entropy-driven adaptive data augmentation framework for image classification. In European Conference Computer Vision, pp. 197-214. Springer, 2024b.
Yu Yang, Hao Kang, and Baharan Mirzasoleiman. Towards sustainable learning: Coresets for data- efficient deep learning. In International Conference on Machine Learning, pp. 39314-39330. PMLR, 2023d.
Lei Zhang, Jie Zhang, Bowen Lei, Subhabrata Mukherjee, Xiang Pan, Bo Zhao, Caiwen Ding, Yao Li, and Dongkuan Xu. Accelerating dataset distillation via model augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11950-11959, 2023.
Xin Zhang, Jiawei Du, Yunsong Li, Weiying Xie, and Joey Tianyi Zhou. Spanning training progress: Temporal dual-depth scoring (tdds) for enhanced dataset pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26223-26232, 2024.
Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage- coreset selection for high pruming rates. In The Eleventh International Conference Learning Representations, 2023,
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018
14

---

Published as a conference paper at ICLR 2025
A ANALYSIS ON THE THRESHOLD 0
We present the relationship between the value of and the actual selection ratio gap. Suppose that have L. V & E. STE (sigmoid(d.); > 0.5)] Sr S 0. Then, we have,
|d]l S 8, +0
In this way, the actual selection ratio be constrained into the 0 gap. We set 0 as 5 10 For instance, on CIFAR datasets with 50000 samples, if the expected selected number is 40000, the actual selected number can be between 39975 and 40025.
B THE SPECIFIC ALGORITHM WORKFLOW
To better understand the workflow of our proposed method, present the detailed algorithm in Algorithm beloow.
Algorithm The general workflow.
Input: dataset D, total number of training samples N, total number of epochs T, selection ratio
8r, a threshold 0, the pretrained image and text encoders are Fl and Er, the fine-tuned image and text adapters A, and Ar, respectively.
1: d 1
2: s s
3: for i=0:N-1 do
4: Calculate the SAS SA according to Eq.
5: Calculate the K neighbors x' of a; using the KNN algorithm
6: Calculate the SDS Sp according to Eq.
7:
end for
8:
for t=0:T-1 do
9:
Calculate the loss L. according to Eq.
10:
Calculate the loss Lad according to Eq.
Calculate the pruning loss L. according Eq. Calculate the overall loss according to Eq. 6
12:
Update d based on the loss C and SGD with momentum
14:
if L. < then
break 16: end if
end for
18: Return d
C IMPLEMENTATION DETAILS
Datasets and Network Architectures. Consistent with previous works Tan et al. (2024); Xia al. (2023): Zheng et al. (2023), we evaluate the effectiveness of proposed method vari- ous popularly used benchmark datasets, including CIFAR- 10/100 Krizhevsky et al.] (2009), Tiny- ImageNet Chrabaszez al. (2017), and ImageNet-Ik Deng al. (2009). To evaluate the gener- alization performance of our selected datasets, we study the effectiveness of our proposed method on wide range of network architectures, including ResNet-18/50 He et al. (2016), Vision Trans- former (ViT) Dosovitskiy et al. (2020). Swin Transformer Liu et al. (2021). VGG-16 Simonyan &
Zisserman 2014). and DenseNet- 121 Huang et al. (0017).
Training on the Selected Datasets Closely following previous works al. 2023 Yang al. (003c);; Sorscher et al.) (022). for experiments on (IFAR-10/100) we adopt a batch size of 128, an SGD optimizer with momentum of 0.9, weight decay of 5e 4, an initial learning rate of 0.1, and total training epoch of 200. The learning rate is divided by 5 after the 60th, the 120th, and the 160th epoch. For experiments ImageNn,,,,,,, adopt batch size of 256, SGD optimizer
15

---

Published as a conference paper at ICLR 2025
with momentum of 0.9, weight decay of 1r-4, an initial learning rate of 0.1, and total epoch of 90. The learning rate is divided by 10 after the 30th and the 60th epoch. For experiments on ImageNet- 1k. following Xia et al. (2023); [Sorscher et al. (2022); Yang et al. (2024b), the VISSL library Goyal et al. (2021) is exploited. We adopt a base learning rate of 0.01, a batch size of 256, an SGD optimizer with momentum of 0.9, weight decay of 1e-3, and a total epoch of 105. All experiments are conducted by three individual runs with different random seeds, while on ImageNet-Ik, due to the huge training costs, the experiment in each case is performed once. Unless specified, the network architecture used is the ResNet-50 model. All hyperparameters and experimental settings for training on different selected datasets are kept the same.
Fine-tuning the Adapters We adopt the initial learning rate of 1 x 10-4, the Adam optimizer with step size of 30 epochs, a decay factor of 0.1, and total epoch of 30. The batch size set to 256 on CIFAR-10/100, 64 on Tiny-ImageNet, and 512 on ImageNet-Ik.
Selection Optimization We adopt SGD optimizer with momentum of 0.9, an initial learning rate of x 10-3, and a total training iteration of x 10 Since this optimization merely involves numerical optimization on parameter using vanilla SGD without introducing any deep networks, the optimization can be very efficiently completed.
D EXPERIMENTAL RESULTS ON CIFAR-10
Table 6: Test accuracy (%) on CIFAR-10 10 with ResNet-50.
Method/ Selection ratio
30%
60%
70%
90%
100%
Random
84.12
90.34
92.71
94,43 37
95 12.02aa
95.55a014
95
96. 12
EL2N
70.32=0044
87 87.48aco.
(9.23
94,43=027
95. 17=027
35.55
96.01=020
12-12
MoSo
83. 83.33
89.17 17.114
92.47=0.4
04269
95. 50:5000
95.93
96.26sa.2
96.12 96:12:011
GraNd
79. 23.223
8.88800
92. 17
14
95. 19
35.35
95.96 75.964805
96. 12
Glister
79. =3..53
78.8844.9
92. 17:0.3
95.0 13
95. =0m6
95.98
96. 06.34x002
96.12+a012
Herding
78.42=0078
87.77a4.65
89.40sa54
89. 19.122
92.1
93.92:035
95.50sau3
96.12
CG Scure
80.50
89.35
92.73
95. 05.
05.8777017
95. 15.99
96.16
96.12::011
Forgetting
67.58 is
88. 1::1
13.6
95. 12.2
95.85_000
95.461622
95.85
96. 12
Moderate-DS
81.75.00
90.941em
92.791a31
94.69_03:
95.26.203
95.73
12400
Self-sup. prototypes
84.60=1m
90.07a1.14
92.64aa93
94, 22
94, 94.98:001
87.87
95.95a0.4
16.12=a12
Ours
85.70=0m
91.10=0.26
93.8
94.85=006
95,43=021
96.11
96.53#0.12
96.12#0.12
Due to the space constraint on the main paper, we provide the experimental results on CIFAR-10 in Table 6 It can be seen that our proposed method achieves superior performance across vari- ous selection ratios on CIFAR-10. 10. Since CIFAR- 10 is relatively simple classify for ResNet-50 models, various methods show only marginal accuracy differences, particularly at higher selection ratios. However, our method consistently outperforms existing baselines by relatively larger mar- gin. This highlights the effectiveness of our multimodal data selection strategy in improving model performance.
E GENERALIZATION ON VISION TRANSFORMER
To further demonstrate the generalization performance of method various architectures, employ the Vision Transformer. The implementation based on the public Github repository Specifically, we utilize the ViT-small train on the selected datasets. The experimental results are presented in Table 7 It can be observed that our method achieves the best performance compared to other baselines using VIT, validating the superior generalization performance on ViT architectures. Combined with the results on VGG-16 16 and DenseNet- 121 in Section 4.2) we demonstrate that our selected datasets obtain a wide range of applications regardless of specific architectures.
16

---

Published as a conference paper at ICLR 2025
Table 7: The accuracy (%) on CIFAR-IO with ViT-small.
Method/Selection Ratio
60%
70%
80%
90%
100%
Random
78.9820.28
80.30
81.3320010
82.63=0.18
84.00+0.32
EL2N
79.35
80.73::008
81.62.10.08
82.90=0.09
84.00:0.32
MoSo
19.45
80.27
81,82+0.15
82.92=0.34
84.00
GraNd
79.22:#0.06
80.59
81.533
82.72=0.08
84.00+0032
Glister
78.33
79.84
81.33..0.05
82,65 =0009
84,00
Herding
:6.08
78.53 45
80.31
82.08=0.02
84.00+0032
CG-Score
=9.09
80.9620.0
82,02=0023
82.91=0am
84.00+0.2
Forgetting
70.87
80.86
81.90#0.31
82.69=0.20
84,00
Moderate-Da DS
79.54-10.19
81.28
81.98
82.61=0.27
84,00
Self-sup. prototypes
19.24.10.16
80.34
81 81.66.1025
82.86
84.00
Ours
80.21
82.13
82.32.10.29
84.26
84.00
F IMPLEMENTATION DETAILS OF TRAINING VIT-BASED MODELS ON IMAGENET-IK
We train ViT- T-Base and ViT-Large on the selected ImageNet- Ik datasets based on the implementa- tion of He et al. (202T), and we train Swin Transformer based on the implementation of Liu et al. 2021). Specifically, we fine-tune the pre-trained ViT-B and ViT-L with a batch size of 32, a base learning of 5e 4, layer decay of 0.65, and weight decay of 0.05. We fine-tune the Swin Transformer with weight decay of Le 8, a base learning rate of 2e 5, a warmup learning rate of 2e 8, and batch size of 64.
G ILLUSTRATION OF IMAGE CORRUPTION TYPES
(a) Origin
(b) fog
(c) Gaussian
(d) motion blur
(e) occlution
(f) resolution
Figure 8: Illustration of the image corruption types, including fog, Gaussian noise, motion blur, random occlution, and resolution.
H VISUALIZATION OF OUR METHOD IN NOISY CONDITIONS
Noby des Bily
Mely care
Silitcc Sara Naoo
(a) Random selection
(b) Ours
Figure 9: Illustration the selected data in noisy conditions. The noisy ratio and selection ratio are 20%.
17

---

Published as a conference paper at ICLR 2025
Table 8: Comparison of actual selection costs (h) of various methods.
Method
Random
Herding
MoSo
Moderate
Glister
CG-Score
Forgetting
GraNd
EL2N
Ours
CIFAR-10 10
0
0
2:21
0.75
2.55
1.24
1.23
0.12
0.12
2.69
0.40
Tiny ImageNet
COMPARISON OF ACTUAL SELECTION COSTS
In Section 4.2 we present the comparison of tradeoffs in effectiveness and efficiency. In this section, better understand the actual high efficiency of our method, we further present direct compar- ison of the actual selection costs of different selection methods. 'he devices used are 4 NVIDIA RTX2080TI GPUs and an Inter(R) CPU E5-26778 2.50GHz.. We report the average costs across five independent runs and various selection ratios in Table [ consisting of the costs of both fine- tuning the adapter and selection optimization. Consistent with the complexity analysis Section our approach achieves superior efficiency compared to other baselines.
J TRADEOFF BETWEEN THE SELECTION COSTS AND ACCURACY
In this section, present the tradeoff between curacy and training costs across various selection ra- tios on CIFAR-I00. The devices used are NVIDIA RTX208OTI GPUs and an Intel(R) CPU ES-2678 2.GOG The results are presented in Figure TO] where the baseline model uses the original dataset for training. can be seen that with the increase the selection ratios,
80
baselim
70%
75
Accuracy (%)
70
65
60
the training costs approximate that of the baseline model.
55
Notably, when the selection ratios are above 80%, the se- lected datasets can achieve lossless accuracy with lower training costs.
1.5
2.0
2.5
Training Costs (h)
Figure 10: Relationship between aceu- racy and actual training costs.
It is important to emphasize that the practical gains from this approach are particularly significant in scenarios re- quiring the training of numerous models, such neural architecture search (NAS) Ren al. 202 Zoph et al. (2018). In such applications, the reduction in training time and computational resources result in substantial efficiency improvements, further amplifying the impact of our data selection framework.
K MORE ANALYSIS ON THE THRESHOLD
Table 9: Effect of the threshold 0
60%
70%
80%
90%
100%
5 10
44.29%
46.02%
47.41%
49.30%
49.36%
x 10
44.25%
45.98%
47.86%
49.17%
49.36%
Threshold 0 adjusts the gap between the actual and expected selection ratios. Employing smaller makes weights d closer to the expected selection ratios while may increase the training costs for convergence. In our main results, typically set 0 5 x 10-4 In this section, we employ smaller threshold on Tiny- ImageNet. i.e., 5 10 10-5, 55 which indicates that the difference in the final sample numbers is less than 5. In Table 9 we select 60%-90% of the data from ImageNe. It can be seen that the performance is robust to the change in 6. With a tighter bound, the performance exhibits minimal differences.
18

---

Published as a conference paper at ICLR 2025
Table 10: Comparison noise reduction performance using features (Ours) vs. average image features under varying noise and selection ratios. Noise proportion means the introduced noise ratio in the selected datasets.
Noise Ratio (%)
20
.0
70
Selection Ratio (%)
20
30
20
30
20
30
Noise Proportion (%)
16.39
25.35
20.00
29.95
20.22
30.16
Avg. Image Feat.
Accuracy (%)
28.42
38.35
16.56
23.19
11.18
14.61
Noise Proportion (%)
0.24
0.32
0.43
0.68
0,80
4.30
Ours
Accuracy (%)
46.05
58.34
52.56
60.72
51.50
56.80
L EFFECTIVENESS OF TEXT MODALITY
Based on single-modal features, existing methods Xia et al. 2023) use the average image features as the prototype and calculate the Euclidean distance between the embedded image and the corre- sponding prototype. This distance is used to filter noisy labels. To further evaluate the effectiveness of our multi multi-modal framework, we use the average image feature to replace the text feature in Eq. [ which is then used for selection optimization in Eq.
In Table TO we evaluate the noise robustness across various noise and selection ratios. It can be that using average image features leads to higher noise ratios compared to method. This validates the effectiveness of complementary textual information provided by text features, leading superior denoising performance.
M NUMERICAL ANALYSIS ON THE SELECTED DATASETS
Table 11: Actual selection ratios on both Tmny-ImageNet and CIFAR-100.
Expected Ratios
20%
30%
40%
60%
70%
80%
90%
20.03%
30.03%6
39.95 39.98%
59.97%
70.01%
80.02%
89 $9.99%.
CIFAR-100
19.95%
30.02%
40.00%
60.04%
70.03%
80.04%
90.04%
We present the numerical analysis of the actual selection ratios Tiny- Imageet in Table [1 It can be seen that the actual selection ratios may be slightly higher or lower than the expected values, with minimal deviations. These deviations actual selection ratios are below the expected ones, our method selects fewer samples compared to other baselines. Despite this reduction in training data volume, our method still achieves the best performance, highlighting the effectiveness of the selection strategy. Consequently, the performance improvements achieved not due the slight increases (e.g., less than 0.4% on average) in sample numbers but rather due to the strategic selection of informative samples, demonstrating the efficacy of our selection algorithm in optimizing the selected samples.
within the theoretical gap bounds. Notably, when the
19


Reviews:
[{"review_id": "zB0aj0QAlU", "reviewer": "Reviewer_QkGn", "summary": "This paper introduces a CLIP-based multimodal data selection framework that enhances the robustness and generalizability of data selection by leveraging both image and text information. Firstly, the paper trains an adapter to transfer pretrained knowledge to the target data. Then, the similarity between the image and text is used to calculate the sampling score for subset selection. The authors evaluate their results in various popularly datasets, obtaining state-of-the-art results in almost all of them.", "questions": "The authors should take the above questions into consideration: the utilization of the adapter, the comparison, and the parameter.\n1. The method requires training an adapter with data from the training dataset for data selection. If the noise ratio is high (such as 50%), wouldn't it be a better option not to use an adapter?\n2. Does the Actual Selection Costs in Appendix G include the training time for the Adapter? Is the comparison fair?\n3. From the ablation experiment, it can be seen that selection loss has a significant impact on the final result. So, what is the sensitivity of this parameter?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method performs more robustly compared to other baseline data selection methods in various popularly datasets.\n2. The algorithm is easy to understand and to follow.\n3. The writing of this paper is overall great.", "weaknesses": "Some details and motivation should be explained further.\n1. The method requires training an adapter with data from the training dataset for data selection. If the noise ratio is high (such as 50%), wouldn't it be a better option not to use an adapter?\n2. Does the Actual Selection Costs in Appendix G include the training time for the Adapter? Is the comparison fair?\n3. From the ablation experiment, it can be seen that selection loss has a significant impact on the final result. So, what is the sensitivity of this parameter?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bi0UTLE3Gb", "reviewer": "Reviewer_vU6u", "summary": "This paper argues that current image data selection methods are limited because they are unimodal. It proposes a multimodal method that uses the category texts from pretrained CLIP to complement images for more robust and generalized data selection. The proposed framework consists of three modules (1) dataset adaptation that integrates image and text adapters to transfer prior knowledge to the target data; (2) sample scoring that calculates the semantic alignment and diversity scores based on the multimodal features, measuring the image-text alignment as well as the local pattern variability; (3) Selection optimization that uses the two scores to select semantically representative and diverse samples, and introduces selection optimization to efficiently identify the ideal data subsets given an expected selection ratio through a multi-objective optimization strategy.\n\nPost discussion comments:\nThe authors have addressed my concerns. With the additional clarifications and material, the paper is in much better shape. I am keeping my score at 8, which correctly reflects the quality of this work.", "questions": "See above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The idea of exploiting multimodal features from the CLIP model is interesting and plausible. \n\nThe proposed method is simple, which is a strength in my opinion. \n\nThe method is also efficient and is able to control the alignment, diversity and selection ratio in a multiobjective optimization efficiently. \n\nThe paper is well written and easy to follow. \n\nThe results are good.", "weaknesses": "The proposed method relies on a pretrained CLIP and hence any biases in the CLIP model will propagate to the selected dataset. \n\nThe proposed method optimizes alignment and diversity but does it have any indirect effect on bias in the dataset?  \n\nIs it possible to control bias in the dataset or in the subsequent models that are trained on the selected dataset? \n\nDoes the STE cause convergence issues? \n\nThe variable d (sample wise parameter) can be easily confused with d (feature dimension). Consider changing one of them. Also, I don’t see d in Fig. 1. Is “w: N x 1” the sample wise parameter d? Or am I missing something? \n\nThe caption of Tab.3 needs to be corrected. You can also merge Tab.3 with Tab.2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nwfZ5tyFuY", "reviewer": "Reviewer_HyV4", "summary": "This paper proposes a novel CLIP-based data selection method leveraging multimodal information with an SGD optimization module. The experiments on several benchmarks show evident improvements over existing approaches.", "questions": "- The text-image alignment is like using CLIP to measure the difficulty in learning the data. Noisy data or corrupted data are difficult to learn. However, will the selection also filter some important yet difficult data?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Connecting the text and image information for the dataset selection is novel. The proposed CLIP-based method implements the idea well. \n- The SGD-based selection optimization with multi-objective is interesting. It is different from the existing sampling strategies with combinational optimization. \n- Sufficient experimental results show the effectiveness, including different datasets, different settings, and different model architectures.", "weaknesses": "- The work relies on two adapters to project the CLIP features to the dataset-specific embedding space. The adapter training is based on perfect data. In fact, under the setting of noisy or corrupted data, this ideal data is inaccessible. Therefore, the experiments would be problematic. In Lines 514-515, the authors depict \"is essential for effectively transferring the model’s generalization ability to target datasets\". It seems the adapter has a large influence on the performance.  Will using the *imperfect* data to fine-tune the adapter degrade the method's effectiveness? This needs to be discussed.\n\n- The class label in this work plays a role as a prototype, while previous work also uses prototypes. They [a] often use the average image features as the prototype and calculate the Euclidean distance between the embedded image and the corresponding prototype, similar to SAS (Eq. 1). Besides, they also use this distance to filter noisy labels [b]. Intuitively, we can use the average image feature to replace the text feature in Eq. 1 of the proposed method. Therefore, it is essential to discuss these results to convey to readers that introducing text is significant.  \n\n- MoSo is NeurIPS'23 rather than NeurIPS'24. Therefore, methods from 2024 have not been compared, e.g., [c]. Besides, some typical methods good at low sampling ratios are missing, e.g., [d].\n\n- Typo: the symbol for the learnable parameter in Figure 2 should be $\\mathbf{d}$ rather than $\\mathbf{w}$.\n\n[a] Moderate coreset: A universal method of data selection for real-world data-efficient deep learning, ICLR, 2023\n\n[b] NGC: A unified framework for learning with open-world noisy data, ICCV, 2021\n\n[c] Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning, CVPR, 2024\n\n[d] Submodular Combinatorial Information Measures with Applications in Machine Learning, ALT, 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hcAZd4FF3e", "reviewer": "Reviewer_vC5o", "summary": "The paper proposes a CLIP-powered framework for data selection that addresses the limitations of traditional single-modality data selection methods by incorporating multimodal information. This framework utilizes a pretrained vision-language model (CLIP) to integrate image and text modalities, thereby enhancing data selection accuracy and robustness. The framework is built on three modules: Dataset Adaptation, Sample Scoring, and Selection Optimization. It leverages both semantic alignment and diversity scores to refine sample selection, removing redundant and noisy data to improve training efficiency and performance on benchmark datasets. Experiments demonstrate the method's superior performance in generalization and robustness compared to existing state-of-the-art approaches, especially in noisy environments.", "questions": "1. Could the authors provide more detail on the computational efficiency of the Selection Optimization module for very large-scale datasets?\n2. Has the framework been tested on datasets with high modality imbalance (e.g., more text than images), and how does it handle such scenarios?\n3. For real-world noisy datasets, could the authors provide more information on how they define and quantify \"noisy\" samples?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Originality: The use of multimodal data via CLIP for data selection is innovative and distinguishes this work from single-modality approaches.\n2. Quality: The methodology is backed by rigorous experiments, with a well-defined scoring system for sample selection (SAS and SDS).\n3. Clarity: Visualizations effectively depict the advantages of multimodal selection over traditional methods.\n4. Significance: This framework has broad applications, as it improves dataset quality and model robustness, potentially benefiting various domains using machine learning.", "weaknesses": "1. Complexity: While effective, the Selection Optimization module’s complexity might hinder its application to extremely large datasets without computational resources.\n2. Generality: The framework is primarily tested on vision-based datasets; extending it to text or mixed-modality datasets may reveal further limitations. Future work could focus on enhancing the framework’s versatility across other modalities​", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)