ELFS: Label-Free Coreset Selection with Proxy Training Dynamics
Authors: Haizhong Zheng
Elisa Tsai
Yifu Lu
Jiachen Sun
Brian R. Bartoldson
Bhavya Kailkhura
Atul Prakash
Venue: ICLR 2025 Poster (2025)
Keywords: Coreset Selection
Data pruning
Label free coreset selection
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose a novel label-free coreset selection methods (ELFS) that outperforms existing baselines on four vision datasets.

Abstract: High-quality human-annotated data is crucial for modern deep learning pipelines, yet the human annotation process is both costly and time-consuming. Given a constrained human labeling budget, selecting an informative and representative data subset for labeling can significantly reduce human annotation effort. Well-performing state-of-the-art (SOTA) coreset selection methods require ground truth labels over the whole dataset, failing to reduce the human labeling burden. Meanwhile, SOTA label-free coreset selection methods deliver inferior performance due to poor geometry-based difficulty scores. In this paper, we introduce ELFS (Effective Label-Free Coreset Selection), a novel label-free coreset selection method. ELFS significantly improves label-free coreset selection by addressing two challenges: 1) ELFS utilizes deep clustering to estimate training dynamics-based data difficulty scores without ground truth labels; 2) Pseudo-labels introduce a distribution shift in the data difficulty scores, and we propose a simple but effective double-end pruning method to mitigate bias on calculated scores. We evaluate ELFS on four vision benchmarks and show that, given the same vision encoder, ELFS consistently outperforms SOTA label-free baselines. For instance, when using SwAV as the encoder, ELFS outperforms D2 by up to 10.2% in accuracy on ImageNet-1K. We make our code publicly available on GitHub.

Published as a conference paper at ICLR 2025
ELFS: LABEL-FREE CORESET SELECTION WITH PROXY TRAINING DYNAMICS
Haizhong Zheng"', Elisa Tsai"!, Yifu Lu', Jiachen Sun', Brian R. Bartoldson', Bhavya Kailkhura", Atul Prakash 'University of Michigan
Lawrencc Livermore National Laboratory
(hzzheng, eltsai, yifulu, jiachens, aprakash) edu (bartoldson, kailkhural eellnl.gon.
ABSTRACT
High-quality human-annotated data crucial for modern deep learning pipelines, yet the human annotation process is both costly and time-consuming. Given a constrained human labeling budget, selecting an informative and representative data subset for labeling can significantly reduce human annotation effort. Well- performing state-of-ther oftth--a---e-rtttt (SOTA) coreset selection methods require ground truth labels over the whole dataset, failing to reduce the human labeling burden. Mean- while, SOTA label-free coreset selection methods deliver inferior performance due to poor geometry-basee difficulty scores. In this paper, we introduce ELFS (Effective Label-Free Coreset Selection), a novel label-free coreset selection method. ELFS significantly improves label-free coreset selection by addressing two challenges: 1) ELFS utilizes deep clustering to estimate training dynamics- based data difficulty scores without ground truth labels; 2) Pseudo-labels intro- duce distribution shift in the data difficulty scores, and we propose simple but effective double-end pruning method to mitigate bias on calculated scores. We evaluate ELFS four vision benchmarks and show that, given the same vision encoder, ELFS consistently outperforms SOTA label-free basclines. For instance, when using SwAV as the encoder, ELFS outperforms D2 by up 10.2% in accu- racy on ImageNet-I K. We make our code publicly available on GitHub
INTRODUCTION
Modern machine learning systems, par- ticularly deep learning frameworks, are increasingly data-intensive and computa- tionally demanding (Touvron et al. [0033 Achiam et al.] (202).. High-quality la- beled data is crucial in the deep learn- ing pipeline. Given the same model ar- chitecture, improving the quality of train- ing data can significantly boost model per- formance (Zheng et al. 2023 Maharana 2023; Xia et al. 2023] Choi al.
Entire Dañaset
Performance
Comparable
label-free
T Performance
Coreset
Unlabeled Detaset
Unlabeled
Annotation
Training
Coreset
Figure 1: Label-free Coreset Selection Scheme. The goal of label-free coreset selection is to identify in- formative and representative subset the data without relying ground truth labels, minimizing human notation efforts in deep learning pipelines.
2023
However, generating high-quality
labeled data typically requires costly hu- man annotation efforts. Selecting an informative and representative subset for labeling can sub- stantially reduce these costs [Bartoldson et al.) (023), thereby optimizing the use of resources and enhancing the overall efficiency of the data annotation process.
State-of-the-art (SOTA) coreset selection methods use data difficulty scores derived from training dynamics to select coresets (Pleiss et al. 2020 Toneva et al., 2018 Paul et al. 2021 Zheng al.
Equua contribution
https//gituu..co//tssaaaaa

---

Published as a conference paper at ICLR 2025
Maharana et al. 2023 Xia et al. 2023 Choi et al. 2023). However, these methods require ground truth labels over the whole dataset to calculate difficulty scores, which fails to achieve the purpose of reducing human annotation efforts. Existing label-free coreset selection methods cal- culate importance scores based on geometric properties in the embedding space (Sorscher et al.) 2022a Maharana al. (023). However, compared to training dynamies scores, these geometry- based scores often fail to accurately estimate data difficulty and lead to poor coreset selection performance-even often worse than random sampling at some pruning rates (Maharana et al.) 2023). To utilize more accurate scores calculated by training dynamics in the label-free coreset selection setting, there are two primary challenges that need be solved:
2023
1. How to estimate training dynamics without ground truth labels?
2. How to select high-quality coresets with proxy training dynamics?
In this paper, propose a novel coreset selection algorithm, ELFS (Effective Label-Free Coreset Selection) to enhance label-free coreset selection by addressing those two challenges. For the first challenge, we show that pseudo-labels provide a good estimation for training dynamics-based diffi- culty scores. More specifically, ELFS utilizes deep clustering (Van Gansbeke et al.) 2020 Adaloglou jet al.] 2023) to assign pseudo-labed to each data point. We empirically show that training dynamic scores derived from pscudo-labels provide good proxy measure data difficulty for coreset se- lection (Section 4.3). For the second challenge, we observe that directly applying existing selection methods like CCS (Zheng et al. 2023) results in a performance gap when compared to coreset selec- tion with ground truth labels. Our further analysis shows that this gap is due to the distribution shift in the difficulty scores caused by inaccurate pseudo-labels This distribution shift leads traditional sampling methods to select more easy examples, resulting in inferior performance (Section 4.3). To mitigate this distribution shift issue, we propose simple yet effective double-end pruning method for the macudo-label-basee difficulty scores. This double-end pruning significantly reduces the num- ber of selected easy examples and improves the coreset selection performance.
To verify the effectiveness of ELFS, we evaluate it on four vision benchmarks: CIFARIO. CI- FAR100 (Krizhevsky al. 2009), STL10 (Coates et al. 201I), and ImageNet-IK (Deng et al. 2009]. The evaluation results show that, given the same vision encoder, ELFS consistently outper- forms all label-free baselines on all datasets at different pruning rates. For instance, when using SwAV as the encoder, ELFS outperforms D2 by up to 10.2% in accuracy on ImageNet-1K KK. Simi- larly, when employing DINO as the encoder, ELFS outperforms FreeSel with an accuracy gain of up to 3.9% on ImageNet-Iee IK. For some pruning rates (c.g.. 30% and 50% for ImageNet- IK), ELFS even achieves comparable performance compared to the best supervised coreset selection performance. In addition to performance comparisons, we also conduct comprehensive ablation studies analyze the contribution of each component of ELFS.
2 RELATED WORK
2.1 CORESET SELECTION
Supervised Coreset Selection. Coreset selection aims to select a representative subset of the train- ing data in one-shot manner, which be used to train future models while maintaining high accuracy. Supervised coreset selection can be categorized into two primary categories (Guo et al.] 2022). (1) Optimization--usel methods aim select subsets that produce gradients similar to the entire dataset (Mirzasoleiman el al.] 2020) Killamsetty et al., 2021a). (2) Difficulty score methods select examples based on metrics such as forgetting scores Toneva et al. 2018), area under the margin (AUM) (Pleiss et al. 2020). EL2N (Paul et al.) 2021). and Entropy Cooemann et al., 2019). These metrics often leverage training dynamics, as they provide insight into how certain data points are learned or forgotten during training, reflecting the inherent difficulty of the data. Unlike scores computed from a single model, training dynamics-based scores capture more information across multiple training epochs, leading more accurate estimation of data difficulty. A more detailed discussion of training dynamic metrics is presented in Section 3.2 Besides data difficulty metrics, recent studies have refined the difficulty score methods by proposing new sampling strategies that ensure the diversity of selected examples, rather than just selecting challenging ones (Zheng et al.
2023 Maharana al. 2023 Xia et al. 2023
2

---

Published as a conference paper at ICLR 2025
Label-free Coreset Selection. While supervised coreset selection methods achieve good perfor- mance, exploration in label-free select settings is limited. Most label-free coreset selection methods fall under the category of geometry-based methods, which select samples by leveraging geometric properties in the embedding space. This includes distances to centroids (Xia et al., 2023), to other
examples (Sener & Savarese 2017). or the decision boundary (Ducoffe & Precioso 2018 Mar-
gatina et al. (0021./Sorschii et al. 2022a) propose to compute a self-supervised pruning metric by performing k-means clustering in the embedding space of SWaV (Caron et al., 2020). Similarly, Xie et al. 2024) selects data via distance- based sampling of semantic patterns from the intermediate model feature. Maharana et al. (2023) introduce method that starts with uniform data difficulty score, which is then refined through forward and reverse messages passing on the dataset graph. Despite those efforts, label-free coreset selection still has a large performance gap compared to su- pervised coreset selection methods (See Figure D A potential reason for this gap is that SOTA label-free coreset selection methods are unable to utilize training dynamics to calculate difficulty scores, ground truth labels are required to train models and collect these dynamics. This makes the calculation of high-quality difficulty scores a key challenge label-free coreset selection.
2.2 DEEP CLUSTERING
Deep clustering aims to group unlabeled data into clusters based on learned features and has made significant strides with contrastive learning techniques (Chen et al. 2020 He et al. al.| 2020 Van Gans- beke et al. 2020 Grill al., 2020). Recent methods like TEMI (Adaloglou
al. 2023) and
SIC (Cai et al., 2023) leverage self-supervised Vision Transformers (ViT') and vision- langugge models, such as CLIP (Radford et al.) (02T). DINO (Caron et al., 2021). and DINO v2 (Oquab) et al. 2023). These approaches have set new deep clustering baselines on datasets such CI- FAR10/100 and ImageNet, showcasing the efficacy of leveraging rich, pretrained representations from self-supervised and multi-modal models for the deep clustering task.
2.3 ACTIVE LEARNING
Active learning optimizes model accuracy with minimal labeled data by iteratively requesting labels from an oracle (e.g. a human annotator) for the most informative samples (Killamsetty et al.) [021b]] Ash et al., 2019 Hacohen et al. 22022.. While it shares some similarities with our one-shot label- free coreset selection setting, several key differences distinguish the two directions: (1) Selection strategies. One-shot label-free coreset selection aims to select the coreset for human annotation in single pass before training. However, active learning requires repeated annotation during the training process. For instance, BADGE (Ash et al. 2019) needs annotation after each training iteration. This iterative annotation setting makes it less practical as it needs continuous human involvement for each iteration. (2) Model independence: One-shot coreset selection aims to find small, model- agnostic subset for training new models from scratch. In contrast, active learning selects instances tailored the current model's 's needs, making it inherently model-dependent (Attenberg & Provost, 2011 Jelenić et al.) 2023). Those two differences make label-free one-shot coreset selection a more practical technique for data efficiency compared to active learning. However, for a more comprehensive comparison, we include BADGE (Ash et al.] (2019). a pool-based active learning method, as a label-free coreset selection baseline in our evaluation.
PRELIMINARY
In this section, we provide brief introduction to label-free coreset selection problem definition, commonly used data difficulty scores, and sampling methods for coreset selection.
3.1 PROBLEM FORMULATION
The goal of label-free coreset selection is to minimize the human annotation cost by selecting an informative and representative subset while achieving good model performance. Given unlabeled dataset D- = {I1, F2 ÆN} belonging to C' classes and a budget & < N, we want to choose subset S C D and [S| k for labeling that maximizes the test accuracy of models trained on this labeled subset. This unsupervised one-shot coreset selection problem can be formulated as the following optimization problem:
3

---

Published as a conference paper at ICLR 2025
S" arg min Ez y/ SCD:|S|=k
(I)
where P is the underlying distribution of D, I the loss function, and hg is the model trained with labeled S. We assume the knowledge of the number of classes C. as it's needed to create an annotation scheme for human annotators and is generally assumed by other label-free data selection methods (Maharana et al., 2023 Sorscher et al. 2022a).
3.2 TRAINING DYNAMIC-BASED DATA DIFFICULTY SCORES
The data difficulty score of an example measures how difficult model learns this example. For instance, the forgetting score (Toneva et al.) 2018) quantifies how often a sample, once cor- rectly classified, is subsequently misclassified during training. EL2N (Paul et al. 202I) quanti- fies data difficulty by calculating L2 training loss of example in the first several epochs. Be- sides, the area under the margin (AUM) (Pleiss et al., 2020) measures data difficulty by accumu- lating margin across different training epochs. The margin for example (x, y) at training epoch t is defined as: M y) = 3y (x) maxişy 2/ (x), where 2 (x) is the prediction likeli- hood for class i training epoch t. AUM is the accumulated margin across all training epochs:
AUM(x. y) + Ei= (x, y). However, all those metries are computed by training a model
with those data, which requires ground truth labels.
3.3 CORESET SAMPLING METHODS
Early coreset selection works Toneva al. 2018 Paul et al., 2021 Coleman al. 2019) advo- cated selecting hard examples to form coresets, which is inspired by SVM: hard examples are closer to the decision boundary, thus playing a more important role in training to determine the decision boundary. However, recent studies (Zheng et al., 2023 Maharana et al. 2023) find that data cover- age is also a crucial factor for coreset selection. For instance, CCS (Zheng et al. 2023) introduces method that combines hard example pruning with stratified sampling to jointly consider data diffi- culty and coverage, significantly improving coreset selection performance. Similarly, D2 (Maharanal et al. 2023) refines difficulty scores by considering the difficulty of neighboring examples within dataset graph, allowing the selection of coresets that capture data difficulty and diversity.
METHODOLOGY
In this section, we first formulate the label-free coreset problem studied in this paper. Then, present a novel coreset selection algorithm, ELFS (Effective Label-Free Coreset Selection), aiming select high-quality coresets without ground truth labels.
4.1 METHOD OVERVIEW
As discussed in Section 2.1 the primary limitation of existing label-free data difficulty scores lies in their failure incorporate training dynamics, which better capture the model's learning behavior and provide more accurate estimation of data difficulty compared to geometry based metrics. However, existing methods for calculating training dynamic scores require ground truth labels, as these scores are derived from training models with labeled data.
To address two primary challenges discussed in Section we propose ELFS: 1) We propose using pseudo-labels to approximate training dynamic scores, and we employ deep clustering to generate pseudo-labels (Adaloglou et al., (022) (Section (43). 2) We find that simply using existing sampling algorithms tends to choose too many easy examples, which leads to inferior performance. We pro- pose a simple but effective double-end pruning algorithm to mitigate this bias. The pipeline of ELFS is illustrated in Figure
4.2 LABEL-FREE TRAINING DYNAMICS BASED SCORES ESTIMATION
Deep clustering is a technique that combines deep learning with clustering methods to group data into clusters without explicit supervision. The goal of deep clustering to learn representations
4

---

Published as a conference paper at ICLR 2025
①
Label-free Training Dynamics Based Scores Estimation
Selection with Inaccurate Scores
Desp Clister
Calou a00 Scores
AJM
Forgetting
EL2N
Annotate
Unlabeled
Dataset
Encoder
Vision
Embeediigss
beeuo--abelldd
Cataset
Difficulty
Double Pruning
Unlabeled
"raining
Figure 2: ELFS pipeline: (1) Training dynamics score estimation: ELFS begins with calculating image embeddings and nearest neighbors using vision encoder and then assigns pseudo-labels to unlabeled data via deep clustering algorithms. The pseudo-labeled dataset is then used to compute training dynamics scores. (2) Coreset selection with proxy difficulty scores: With the pseudo-label- based scores, ELFS performs double-end pruning to select the unlabeled coreset. Subsequently, Humans annotate the selected coreset. This labeled coreset is used for later training.
that capture the underlying structure of the data, which are then used to assign meaningful clus- ter labels. In our approach, we utilize Teacher wnsihhle-weighted pointwise Mutual Information (TEMI) (Adaloglou al.) [023]. state-of-the-arr deep clustering method, to generate yseudo-labell for the unlabeled dataset. We also conduct an ablation study of using different clustering methods
e.g. k-means and SCAN (Caron et al. 2020) in Section 5.3.1
TEMI uses vision encoder models (like SWAV (Caron et al. 2020) and DINO (Caron et al., 202 to convert all images into embedding space. Then, for each example, TEMI calculates the k nearest neighbor set N2- After getting the nearest neighbor sets for all examples, TEMI trains an ensemble of classification heads to assign pseudo labels: TEMI uses ensemble of student head (-) and a teacher head (.) that share the same architecture but differ in their updates. Each instance x is processed through both types of heads, producing probabilistic classifications qs (c(.)) and (clar), respectively, where e E {1, (1,..., C} denotes the class label. The student head's parameters 0, are updated via backpropagation, while the teacher head's parameters 0, are updated via an exponential moving average (EMA) of the student's parameters, ensuring more stable target distributions.
Loss function. Given H clustering heads, TEMI aims minimize the following accumulated weighted pointwise mutual information (PMI) objective to encourage the model to align instances that are likely to belong to the same cluster while penalizing those that are less likely be related:
H
(TEM((T)
wa(a, (pmi"(x,x") + (mi"(x',a)) (x',a))
(1)
2H
h= T'EN
where pmi(x, x²) = log (EC,, ((
approximates the pointwise mutual informa-
ge(c)
tion between pairs (z, 2²), and w(I Σ de (cl=));(c(2²) weights each pair to reflect the probability that they belong to the same cluster.
With trained classification heads, TEMI assigns the pseudo label ğ by aggregating these predictions and selecting the class with the highest combined probability:
1
Σ (c|2))
arg max cE(I H
1=1
Pseudo-labels generated by deep clustering make it feasible to calculate training dynamics scores with supervised training. However, the clustering-baste based pseudo-labeling can introduce many label noises (Table which causes the distribution shift of difficulty scores. In the next part, discuss how we address this distribution shift issue with a simple but effective double-pruning algorithm.
2/3 is hyperparameter for positive pair alignment, follow TEMI it 0.6 to avoid degeneration.
5

---

Published as a conference paper at ICLR 2025
4.3 CORESET SELECTION WITH PROXY DIFFICULTY SCORES
4.3.1
OBSERVATION: IMPORTANCE SCORE DISTRIBUTION SHIFT
Given the difficulty scores calculated with pseudo labels, straightforward approach is to directly apply existing coreset selection methods, such as CCS (Zheng et al., 2023), to select coreset with useudo-label-base scores. Surprisingly, despite pseudo-labels are not always correct, simply ap- plying CCS on pseudo-label-based score (AUM) outperforms SOTA label-free coreset selection methods (green curve in Table B3. However, there is still performance gap between CCS with pseudo-label scores (green curve) and CCS with ground truth label scores (purple dashed curve).
Biased coverage of CCS with pseudo-labels data difficulty scores. To better understand the reason be- hind this gap. we compared the ground truth AUM dis- tribution of coresets selected by CCS (supervised) and
Test Accuracy (9)
CCS (pseudo-label), In Figure 4(a) we plot the dis- tribution of AUM calculated with the ground truth la-
66
CCS (Supervised)
bels and the distribution of examples selected by CCS on this ground truth distribution. To make the com-
60-
(Pseudo Label)
Best Label-Free (D2)
parison, we plot the distribution of ground truth AUM for the examples chosen by CCS using pseudo-label- based AUM. From the figure, we can observe a signif- icant distribution shift between CCS (supervised) and Specifically, pseudo- label-based green region) selects more easy examples (high-AUM) compared to CCS using ground truth la- bels (purple region), which can be potential reason leading to the performance drop.
ELFS (Pseudo Label)
0.0
o's
07
08
Data Pruning Rate
CCS (pseudo-labell)
Figure 3: Performance comparison of su- pervised CCS (Zheng et al. 2023). label- free CCS, best label-free baseline (label-
CCS (striped green
free D2 (Mabarana et al. (023)). and our method ELFS on CIFAR100.
4.3.2 PRUNING WITH PSEUDO- BEEUO--LABEL-BSEE SCORES
To address this distribution gap of selected coresets, we propose a simple but effective double-end pruning method select data with pseudo-label-based scores, which aims reduce the number of selected easy examples. Double-end pruning consists of two steps: 1) Prune 3 hard examples
first. where 3 a hyperparameter (which also used in (Maharana al. 2023 Zheng et al. 2023 Mindermann et al. 2022)). ). 2) Continue pruning easy examples until the budget is met.
key challenge in our double-end pruning algorithm is determining the hard pruning rate B. Recent SOTA
Ground-Truth AUM
Ground-Truth AUM
1500
CCS (Supervised)
1500
CC5 (Supervised)
CCS (Pscudo Label)
ELFS (Psoudo Label)
coreset selection works (Zheng al.]
*Example
eExamgle
2023 Maharana et al.] (023) show
1000
1000
that hard example pruning plays crucial role in improving the core- set selection performance, since it helps ensure the selected coresets het- ter cover high-density areas (Zheng et al. perform a grid search select the op- timal B. In supervised settings, meth- ods like CCS (Zheng et al.) (202)) and D2 (Maharana et al., 2023) can grid search the best 3 with ground truth labels. However, in label-free coreset selection, the absence of ground truth labels makes the 3 grid search a challenge.
750
500
500
250
250
0
50
100
150
200
50
1.00
150
200
Area under the margin (AUM)
Area under the margin (AUM)
a 2023). A common practice is to
(a) Label-free CCS coreset.
(b) ELFS coreset.
Figure 4: Ground truth label AUM distribution of different coreset on CIFAR 100. The pruning rate for coreset is 50%. After applying double-end (ELFS) pruning on data with pseudo-label-basee scores, ELFS covers more hard data in the selected coreset when mapped onto ground truth AUM.
To address this challenge, we show that pseudo-labels generated in the deep clustering step provide a good estimation for this 3 search: After generating the seeudo-labeled dataset, we split it into 90% for training and 10% for validation. We use the validation set to determine the optimal B. Once the optimal B is identified, ELFS selects coresets from the entire training dataset. Our evaluation shows that 3 grid search with pseudo-labela provides good estimation for the optimal B. With this

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of pretrained vision models' performance. ELFS consistently outperforms baselines across all encoders. Full dataset training results: CIFARIO 95.5%, CIFARI00 78.7%, ImageNet- IK 73.1%. Badge represents the active learning (AL) baseline, while the remaining baselines are based on coreset selection. Red-highlighter cells indicate performance below random sampling. The top-performing result for each encoder bolded.
CIFARIO
CIFAR100
ImageNet--l IK
Encodor
Pruning Rate
30%
50%
70%
80%
90%
30%
50%
70%
80%
90%
30%
50%
70%
80%
90%
Supervised
95.7
94.9
93.3
91.4
87.1
78.2
75.9
70.5
56.9
72.9
Randomn
94.3
93.4
90.9
88.0
79.0
74.6
71.1
65.3
57.4
44.8
72.2
70.3
66.7
62.5
52.3
Badge (AL)
93.6
93.0
91.0
87.9
81.6
74.7
71.8
65.2
58.9
47.8
70.4
65.8
61.7
53.4
Prototypicality
94.7
92.9
90.1
84.2
70.9
74.5
69.8
61.1
48.3
32.1
70.9
60.8
54.6
41.9
30.6
SWAV
D2
94.3
93.8
91.6
85.1
71.4
75.3
71.3
66.0
56.2
42.1
72.3
65.6
55.6
50.8
43.2
(Ours)
94.3
91.8
FreeSel
94.5
93.8
91.7
88.9
82.4
75.0
70.5
65.6
57.6
44.8
72.2
70.0
65.4
61.0
51.1
DINO
ELFS (Ours)
95.5
95.2
93.2
90.7
87.3
76.8
73.6
68.4
62.3
54.9
73.5
71.8
67.2
63.4
54.9
pseudo-label- based / search, ELFS guarantees that ELFS does not use any ground fruth labels in the coreset selection process. We conduct more detailed analysis on B search in Section 5.3.6
Despite its simplicity, we find that our double-end pruning scheme significantly reduces the number of selected easy examples (striped green region in Figure 4(b) and improves the performance of label-free coreset selection (purple curve in T
EXPERIMENTS
In this section, we evaluate ELFS on four vision benchmarks. Our evaluation results show that ELFS outperforms existing label-free coreset selection baselines at all pruning rates. When using SwAV as the encoder, ELFS outperforms D2 by up 10.2% in accuracy ImageNet. Similarly, when using DINO as the encoder, ELFS outperforms FreeSel with an accuracy gain of up to 3.9% on ImageNet.
5.1 EXPERIMENT SETTING
Label-free coreset baselines. We compare ELFS with the following label-free coreset selection baselines: 1) Random:
Randomly sampled coreset. 2) Prototypicality (Sorscher et al. 2022b):
Prototypicality generates embeddings from a pretrained vision model (SwAV (Caron et al. 2020)) to perform k-means clustering. The difficulty of each data point is quantified by its Euclidean distance to the nearest cluster centroid, with those further away being preferred during coresct selection. 3) Label-free D2 (Maharana et al., 2023): D2 adapts supervised D2 pruning, which uses difficulty scores derived from training dynamics and selects coresets through forward and backward message passing in a dataset graph. In a label-free scenario, a uniform initial difficulty score applied, and the dataset graph is initialized with embeddings from SWAV (Caron et al.) 2020). 4) FreeSel (Xie) ct al. 2024): distance-based sampling on patterns extracted from inter-mediate features of pretrained models. 5) BADGE (Ash et al. 2019): An active learning method that selects new samples in each epoch using k-means++ initialization within the gradient vector space. It is important to note that BADGE itself is not a coreset selection method, and we include it as reference for comparison with active learning approaches.
Implementation. We report ELFS results using DINO CCaron et al., 202T) and SwAV (Caron et al.) 2020) as the pretrained vision model to extract embeddings. We use the area under the margin (AUM) (Pleiss et al. 2020) the training dynamics metric for all datasets. Our ablation study in Section 5.3.3 show that the choice of training dynamic metrics only has a marginal impact on ELFS's performance. We use ResNet-34 (He et al. 2016) for ImageNet- 1K, and ResNet-18 (He al. 2016) for the rest. Due to the space limitation, we include more experimental details in Appendix reported results are based training from scratch using the selected coreset.
All
7

---

Published as a conference paper at ICLR 2025
5.2 LABEL-FREE CORESET SELECTION PERFORMANCE COMPARISON
To verify the effectiveness of ELFS, we evaluate and compare ELFS with other baselines on four vision benchmarks. We report the comparison results on CIFARIO, CIFARIO0 and ImageNet- 1 in T (STL10 results can be found in Table in the Appendix). As discussed in Section vision encoders are widely adopted in label-free coreset selection methods. To guarantee a fair comparison, we report the performance of ELFS with the same encoders used by other baselines in their papers.
As shown in T given the same vision encoder, ELFS consistently outperforms existing label- free coreset selection baselines. For instance, when using SwAV as the encoder, ELFS outperforms D2 by up to 10.2% in accuracy on ImageNet-IK, Similarly, when employing DINO as the encoder, ELFS outperforms FreeSel with an accuracy gain of up to 3.9% on ImageNet-IKK IK.. Moreover, for some pruning rates (c.g., 30% and 50% for ImageNet- IK) ELFS even achieves comparable per- formance compared the best supervised coreset selection performance. We highlight in red the cells where the selected coreset performs worse than random sampling. Notably, random sam- pling remains strong baseline, particularly at higher pruning rates (80% to 90%). However, ELFS consistently outperforms both random sampling and other existing baselines. To the best of our knowledge, ELFS is the first label-free approach to outperform random sampling all pruning rates. Even though active learning and label-free one-shot coreset selection are two different types of methods (as discussed in Section (.3). For more comprehensive comparison, we also compare our method with an active learning baseline, Badge (Ash et al.. 2019). For STLIO, we observe similar Indings-El.FS outperforms all existing label-free baselines (Table in Appendix C
5.3
ABLATION STUDY & ANALYSIS
5.3.1 1 ABLATION STUDY: CLUSTERING METHOD
Table 2: Ablation study different pscudo-label generation methods (1) deep clustering: TEMI and SCAN, and (2) K-Means. All clustering methods use the DINO embeddings.
Pseudo-Labe ACC
CIFAR1O
CIFAR100
CIFARIO CIFAR100
Pruning Rate
30%
50%
70%
80%
90%
30%
50%
70%
80%
90%
92.5%
66.3%
ELFS (DINO + TEMI)
95.5
95.2
93.2
90.7
87.3
76.8
73.6
68.4
62.3
54.9
93.7%
60.6%
ELFS (DINO + SCAN)
95.3
95.2
94.0
90.8
87.3
75.9
72.5
67.8
61.6
53.7
88.3%
57.7%
ELFS (DINO K-Means)
95.3
94.1
93.3
90.3
85.5
75.2
73.2
67.4
54.9
29.0
Best Label-Free
94.7
93.8
91.7
88.9
82.4
75.3
71.8
66.0
58.9
44.8
Random
94.3
93.4
90.9
88.0
79.0
74.6
71.1
65.3
57.4
44.8
To understand the impact of different clustering methods on ELFS, we select two deep clustering methods-SCC.. (Van Gansbeke et al.) (020)) and TEMI (Adaloglou et al. 2023) along with K- Means to evaluate their influence on the performance of ELFS. As shown in Table / the results reveal that the quality of pseudo-labels decreases, the performance of the selected coreset also degrades. Both TEMI and SCAN show strong performance as the clustering methods for ELFS.
5.3.2 ABLATION STUDY: SAMPLING METHOD
Table 3: Ablation study on different pseudo-label training dynamics sampling methods on CI- FAR The coresets are selected using three sampling methods- CCS, D2, and ELFS (double-end pruning) on the AUM score derived from pseudo-label training dynamics.
Pruning Rate
30%
50%
70%
80%
90%
CCS (Pseudo-Label)
75.5
72.9
67.2
60.9
51.8
(Pseudo-Label)
74.8
70.9
61.2
49,6
27.7
ELFS (Pseudo-Label)
76.8
73.6
68.4
62.3
54.9
CCS (Ground-Trute Label)
77.1
74.5
68.9
64.0
56.0
directly applying CCS sampling leads to performance drop due to the score distribution shift caused by inaccurate pseudo-labels. In this section, we conduct an in-depth ablation study to compare the performance of different sampling methods CIFAR100 with labuds generated by DINO-TEMI. As shown in Table 3 double-end pruning used in ELFS consistently achieves comparable and better performance than other sampling methods.
As we discussed in Section 4.3
8

---

Published as a conference paper at ICLR 2025
5.3.3 ABLATION STUDY: DATA DIFFICULTY METRIC
Table 4: Ablation study on the data difficulty scores. dynamic metrics -UM,, forgetting, and EL2N.
We report results using different training
CIFARIO
CIFAR100
Pruning Rate
30%
50%
70%
80%
90%
30%
50%
70%
80%
90%
ELFS (AUM)
95.5
95.2
93.2
90.7
87.3
76.8
73.6
68.4
62.3
54.9
ELFS (Forgetting)
95.4
94.9
93.3
90.1
86.9
76.8
74.0
68.4
61.9
54.9
ELFS (EL22N
95.6
95.2
93.7
91.0
86.5
76.5
73.6
68.2
62.0
52.8
In Section 5.2 we use AUM (Pleiss al. 2020) for ELFS. To understand how other data difficulty scores influence the performance of ELFS, we conduct an ablation study of the data difficulty score metries on CIFARIO and CIFAR 100. Table shows that ELFS with different data difficulty metrics achieves very similar coreset selection performance, which indicates that using different score metric, like forgetting (Toneva et al. 2018) and EL2N (Paul al. 2021), only has a marginal impact on ELFS performance.
5.3.4 PSEUDO-LABEL QUALITY
Table 5: Pseudo-labeling statistics. Reported metries include accuracy (ACC), adjusted random
index (ARI), and normalized mutual information (NMI) in %.
CIFARIO
CIFAR100
ImageNet-IK
ACC (%)
NMI (9)
(RI(%)
ACC (%)
NMI (%)
ARI (%)
ACC (%)
NMI (%)
ARI (5)
TEMI (SwAV)
60.7
54.6
43.4
39.8
$7.7
26.5
43.1
73.5
29.5
TEMI (DINO)
92.5
86.5
85.1
66.3
76.5
53.0
58.8
81.4
44.8
Pseudo-labels generated by deep clustering are not guaranteed to be correct. In Table we report the accuracy of pseudo-labels on CIFAR10, CIFARIOM 100, and ImageNet- As shown in the table, pseudo-labels contain a considerable amount of label noise. For ImageNee- IK, accuracy is below 60% even using DINO as the vision encoder. This indicates that ELFS has strong robustness to label noise. Despite the label noise, ELFS still achieves better performance than all baselines.
5.3.5 VISION ENCODER TRAINED FROM SCRATCH ON THE SAME UNLABELLED DATASET
Table 6: Ablation study results comparing model performance when public pretrained models are unavailable. We use a self-supervised learning method (Caron et al. 2020 to train vision encoders from scratch with all unlabeled data.
CIFARIO
ImageNet IK
Pruning Rate
30%
50%
70%
80%
90%
30%
50%
70%
80%
90%
ELFS (DINO)
95.5
95.2
93.2
90.7
87.3
73.5
71.8
67.2
63.4
54.9
Random
94.3
93.4
90.9
88.0
79.0
72.2
70.3
66.7
62.5
52.3
FreeSel (DINO)
94.5
93.8
91.7
88.9
82.4
72.2
70.0
65.4
61.0
51.1
D2 (SwAV)
94.3
93.8
91.6
85.1
71.4
72.3
65.6
55.6
50.8
43.2
ELFS (Self-Trained Encoder)
94.8
94.6
92.4
88.9
82.8
73.2
71.4
66.8
62.7
53.4
In our previous evaluation, we adopt public pretrained encoders extract embeddings from images (to make fair comparison with existing baselines). In this section, we study how ELFS performs without a pretrained encoder: we use self-supervised learning method (Caron et al.) 2020) train an encoder with all unlabeled data and report results in Table [ Our evaluation results show that, despite that ELFS performance declines after replacing DINO with the self-trained encoders, ELFS with the self-trained encoder still maintains strong performance and outperforms other label-free coreset selection baselines using pretrained encoder. We argue that with the increasing availability of publicly released pretrained models, high-quality vision encoders have become more accessible. However, our study suggests that even without a public pretrained encoder, self-supervised learning on the same unlabeled dataset can be alternative obtaining vision encoder.

---

Published as a conference paper at ICLR 2025
5.3.6
HARD PRUNING RATE 3 SEARCH WITH LABELS
Pseudo-La Label Val Acc (%)
Orade Test Accuracy (%)
90%
30
90%
32
80%
80%
70%
50%
30%
30%
Selected
0.0
022
3.3
0.4
5.5
0.5
0.1
0.2
0.4
05
06
0.7
Hard Pruning Rate B
Hard Pruning Rate 8
DINO pseudo-lahal validation accuracy.
(b) DINO ground truth label test accuracy.
Figure 5: (a) reports the pseudo-label validation accuracy on different 3 on CIFARIOG 100. (b) reports the ground truth (oracle) label test accuracy on different B. Curves with different colors stand for different pruning rates. indicates the selected by the best pseudo-label validation accuracy, which equal to or close to the optimal [.
As discussed in Section 4.3 ELFS has no access to ground truth labels for searching the optimal B. To address this challenge, propose using pseudo-labels to choose B. In this section, conduct an in-depth study on the quality of 3 selected by pseudo-labels Figure/Jcomparee the pseudo-labe accuracy (a) and ground truth accuracy (b) under different hard pruning rates B with 0.1 search step. o in that 3 selected by pseudo-labels provides good estimation for the optimal / selected by ground truth labels. Our results on CIFAR 100 show that, when pruning rate is 30%, 70%, 80%, and 90%, pseudo-label search selects the same accuracy of the model trained with the coreset chosen with pseudo-labels is only 0. % lower than the coreset chosen with ground truth labels.
indicates the 3 selected by the best pseudo-label validation accuracy. We find
the ground truth label search. At 50% pruning rate, the
5.3.7 TRANSFERABILITY
Table 7: Performance of different architectures CIFARIO varying pruning rates, trained with coreset selection based on forgetting scores from ResNet18 DINO pseudo-label training dynamics. We present the top-performing results from various baselines as the best SOTA performance.
Pruning Rate
30%
50%
70%
80%
90%
Random (ResNet18)
94.3
93.4
90.9
88.0
Best SOTA Label-Free
94.7
93.8
91.7
88.9
82.4
ELFS (ResNet18)
95.5
95.2
93.2
90.7
87.3
ELFS (ResNet34)
95.4
95.3
93.1
90.9
87.6
ELFS (ResNet50)
95.4
94.8
93.1
90.3
87.3
One application of coreset selection is that a selected coreset can also be used for training other models. In this section, we evaluate coreset transferability CIFAR10 10 using coresets selected dur- ing ResNet18 training. As shown in Table 7 these coresets exhibit good transfcrability to ResNet34 and ResNet50, demonstrating robustness and generalizability across different architectures.
CONCLUSION
In this paper, we present novel label-free coreset selection method, ELFS, to select high-quality coresets without ground truth labels. ELFS first estimates data difficulty scores with heeudo-labels generated by deep clustering and then performs double-end pruning to select a coreset with these in- accurate data difficulty scores. By evaluating ELFS on four vision benchmarks, we show that ELFS consistently outperforms SOTA label-free coreset selection baselines and, in some cases, nearly matches the performance of supervised coreset selection methods, which demonstrates the effec- tiveness of ELFS. Label-free coreset selection can significantly reduce data collection costs. We believe that our work provides strong baseline and will inspire future work in this area.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work was partially supported by Cisco Research Grant. Any opinions, findings, and conclu- sions or recommendations expressed in this material are those of the author(s) and do necessarily reflect the views of our research sponsors.
Prepared by LLNL under Contract RE-AAC2-077A22344 and supported by the LLNL-LDRD Pro- gram under Project No. 24-ERD-00 1-0000000-- and Project No. 23-ERD-030 (LLNL-CONF-22001 This manuscript has heen authored by Lawrence Livermore National Security, L.I.C under Contract No. DE-AC52--0NN22344 with the U.S. Department of Energy. The United States Government retains, and the publisher, by accepting the article for publication, acknowledges that the United States Gov- emmment retains non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes.
REPRODUCIBILITY
We include implementation details in Section 5.1 and Section B in the Appendix. The implementa- tion of ELFS is available at lttps::/github.ccm/lllaiiiiii
ETHICS STATEMENT
Our work on coreset selection using pseudo-labels and double-end pruning aligns with the ICLR Code of Ethics. By reducing reliance labeled data, our method lowers annotation costs, enhances scalability, and promotes sustainable Al by improving training efficiency and reducing the carbon footprint of large-scale models. We also recognize the broader societal impact of this work, particu- larly in democratizing access to sfate-of-the-art machine learning techniques. By reducing the need for extensive labeled datasets, approach can be adopted by researchers in resource-constrained environments.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.00 08774, 2023.
Nikolas Adalogiou, Felix Michels, Hamza Kalisch, and Markus Kollmann. Exploring the limits of deep image clustering using pretrained models. arXiv preprint arXi::2303.178% 2023.
Jordan Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.036777 2019.
Josh Attenberg and Foster Provost. Inactive learning? difficulties employing active learning in practice. Acm Sigkdd Explorations Newsletter, 12(2):36 41, 2011.
Brian Bartoldson, Bhavya Kailkhura, and Davis Blalock. Cofpute-efficient deep learning: Algo- rithmic trends and opportunities. J. Mach. Learn. Res., 24: 122112 2023.
Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in neural information processing systems, 11114489-18999 2020.
Shaotian Cai, Liping Qiu, Xiaojun Chen, Qin Zhang, and Longteng Chen. Semantic-enhanced image clustering. In Proceedingy the AAAI conference artificial intelligence, volume 37, pp. 6869-6878, 2023.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9924, 2020.

---

Published as a conference paper at ICLR 2025
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervied vision transformers. Proceedings of the IEEE/CVF international conference computer vision, pp. 9650-96600 2021.
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Hoyong Choi, Nohyun Ki, and Hye Won Chung. Bws: Best window selection based sample scores for data pruning across broad ranges. 2023.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In International Conference on Learning Representations, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Melanie Ducoffe and Frederic Precioso. Adversarial active learning for deep networks: a margin based approach. arXiv preprint a 0t..//022.09994 2018
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selec- tion in deep learning. In International Conference on Database and Expert Systems Applications, pp. 181-195. Springer, 2022.
Guy Hacohen, Avihu Dekel, and Daphna Weinshall, Active learning on budget: Opposite strategies suit high and low budgets. arXiv preprint arXiv:2202.02794, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2020.
Fran Jelenić, Josip Jukić, Nina Drobac, and Jan Snajder. On dataset transferability active learning for transformers. arXiv preprint arXiv:2305.00988 2023.
Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pp. 5464-5474. PMLR, 2021a.
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Generalization based data subset selection for efficient and robust learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 8110-8118, 2021b.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
12

---

Published as a conference paper at ICLR 2025
Siyuan Li, Weiyang Jin, Zedong Wang, Fang Wu, Zicheng Liu, Cheng Tan, and Stan Li. Semire- ward: general reward model for cemi-supervised learning. arXiv preprint arXiv:2310.03013, 2023.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, In ICLR, 2017a.
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017b.
Adyasha Maharana, Prateek Yadav, and Mohit Bansal. D2 pruning: Message passing for balancing diversity and difficulty in data pruning. arXiv preprint arXiv:2310.07931 2023.
Katerina Margatina, Giorgos Vernikos, Loic Barrault, and Nikolaos Aletras. Active learning by acquiring contrastive examples. arXiv preprint arXiv:2109.088 03764, 2021.
Sören Mindermann, Jan M Brauner, Muhammed Razzak, Mrinank Sharma, Andreas Kirsch, Win- nie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Confer- on Machine Learning, pp. 15630-15649, PMLR, 2022.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovee. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950-6960.
PMLR, 2020.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199, 2023.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Find- ing important examples early in training. Advances in Neural Information Processing Systems, 34, 2021.
Gcoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems, 33:17044-17056, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR, 2021.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint trXiv:1708.004559 2017.
Jae-hun Shim, Kyeongbo Kong, and Suk-Ju Kang. Core-set sampling for efficient neural architecture search. arXiv preprint arXiv:2107.006 06869, 2021.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neu- ral scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19533-19536, 2022a.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neu- ral scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 15:19233-19566. 2022b.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2018.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
13

---

Published as a conference paper at ICLR 2025
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, and Lue Van Gool. Scan: Learning to classify images without labels. In European conference on computer vision, pp. 268-285. Springer, 2020.
Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang, Zhi Zhou, Lan-Zhe Guo, et al. Usb: A unified semi-supervised learning benchmark for classification. Advances in Neural Information Processing Systems, 35:3938-3966 2022.
Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning, In The Eleventh International Conference on Learning Representations, 2023.
Yichen Xie, Mingyu Ding, Masayoshi Tomizuka, and Wei Zhan. Towards free data selection with general-purpose models. Advances in Neural Information Processing Systems, 36, 2024.
Jachong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang, Online coreset selection for rehearsal-based continual learning. arXiv preprint arXiv:2106.01033 2021.
Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high pruning rates. In The Fleventh International Conference Learning Representations, 2023.
14

---

Published as a conference paper at ICLR 2025
A OVERVIEW
In Section B we provide details on datasets, best hyperparameters for our models, and other exper- iment settings.
In Section we report additional evaluation results.
Datasets. CIFARIO, CIFARI00 (Krizhevsky al.
2009
Custom Licence
ImageNet-1K (Deng
et al. 2009): : Custom Licence STL10 (Coates et al.
2011
Custom Licence
B DETAILED EXPERIMENT SETTING
B.1 PSEUDO-LABEL GENERATION
For pseudo-label generation, we use the training settings recommended in TEMI (Adaloglou et al, 2023). For CIFARIO. CIFAR100 (Krizhevsky et al.] 2009) and STL10 (Coates et al.) 2011), calculate the 50-nearest neighbors (50-NN) for each example using cosine distance in the pretrained model's embedding space. For clustering, 50 heads epochs with a batch size of 512, using an Adam optimizer @Loshchilov & Hutter (017b) with learning rate of 0.0001 and a weight decay of 0.0001. For ImageNet (Deng et al. 2009). all the parameters are the same, except the nearest neighbor search is adjusted to 25-ncarest neighbors (25-NN).
trained. Training is conducted over 200
B.2 CORESET SELECTION WITH PSEUDO-LABEL
Table 8: Comparison of pretrained vision models' performance on STL10. ELFS consistently out- performs baselines across all encoders. Red-highlighted cells indicate performance below random sampling. The top-performing result for each encoder is bolded.
STI.10
Encoder
Pruning Rate
30%
50%
70%
80%
90%
Random
75.6
70.2
64.0
$7.9
45.4
Badge (AL)
68.4
63.7
57.9
51.0
44.1
Prototypicality
66.5
60.5
43.2
37.6
20.8
SWAV
D2
76.2
69.2
58.2
52.7
40.2
ELFS (Ours)
79.6
77.9
72.5
71.6
49.2
FreeSel
77.3
72.4
62.5
55.8
45.3
DINO
ELFS (Ours)
78.2
73.9
65.7
61.3
52.7
Dataset Specitic Training Details. We benchmark our method ELFS using identical training set- tings as recommended in CCS (Zheng et al. (2023) and D2 (Maharana et al. (023). CIFARIO and CIFAR100: We a ResNet18 model for 40,000 iterations, with a batch size of 256 and SGD op- timizer settings that include 0.9 momentum and 0.0002 weight decay. The initial learning rate is set 0.1 with cosine annealing learning rate scheduler (Loshchilov Hutter 201 20) ImageNet-1K: train ResNet34 model for 300,000 iterations, maintaining the same optimizer parameters and initial learning rate. STI.10: We utilize the labeled portion of the STL.10 dataset for training. For all coresets with different pruning rates, train models with 6,400 iterations with a batchsize of 256 (about 320 epochs when pruning rate is zero). We use the SGD optimizer (0.9 momentum and cosine
0.0002 weight decay) with 0.1 initial learning rate. Identical other datasets, annealing learning rate scheduler (Loshchilov Hutter 2017a) with 0.0001 minimum learning rate.
Coreset Selection. For pruning ratio a = {30%, 50%.70% 80%, 90%), we report for the best coreset start point selected in Table on variety of datasets.
15

---

Published as a conference paper at ICLR 2025
6.16
(a) SwAV CIFARIO.
(b) DINO CIFARIO.
10
70
80
60
20
30
30
70
(c) SWAV CIFARI00.
(d) DINO CIFARI00.
Figure 6: Class-wise misclassification rate on CIFAR10/100 using pseudo-labels from different
pretrained models: SwAV and DINO. See Table for the overall misclassification rate.
Table 9: Dataset specific pruning rates and their corresponding chosen 3.
Dataset
Encoder
Pruning Rate
30%
50%
70%
80%
90%
SwAV
0.1
0.2
0.2
0.3
0.4
CIFAR-10
DINO
0
0
0.1
0.1
0.4
SwAV
0.1
0.2
0.4
0.5
0.6
CIFAR-100
DINO
0.0
0.2
0.4
0.5
0.7
ImageNet-IK
DINO
0.0
0.1
0.2
0.4
0.5
SWAV
0.4
0.6
DINO
0.2
0.2
0.4
0.6
0.7
STL10
SWAV
0.1
0.2
0.5
Table 10: The misclassification rate pseudo-labels vs ground-truth labels, adjusted random index (ARI), and normalized mutual information (NMI) using TEMI.
DINO
SWAV
ACC (%)
NMI (%)
ART (%)
ACC (%)
NMI (%)
ART (%)
STL10
93.0
89.6
86.1
89.6
82.0
79.3
C ADDITIONAL EVALUATION RESULTS
C.I STL-10 EVALUATION
In this section, we present evaluation results on STL10 (see Table 8 On STL10, our observa- tions align with the results from CIFAR 10/100 and ImageNet. Our method, ELFS, consistently outperforms other label-free coreset selection methods across various pruning rates. We provide the pseudo-labe label misclassification rate, normalized mutual information (NMI), and adjusted random in- dex (ARI) in Table I0] Additionally, Table the class-wir wise misclassification rate on CIFAR10 and CIFARION 100, highlighting ELFSSs effectiveness with imbalanced class assignments.
16

---

Published as a conference paper at ICLR 2025
C.2 ABLATION STUDY WITH CLIP ENCODER
We conduct additional experiments using CLIP (Radford et al. 2021) as the encoder on CIFARIO
and CIFARIO0, as shown Table TT The pseudo-label accuracy generated by CLIP (85 2% for CIFARIO and 54.0% for CIFAR 100) is better than Sw (60.7% for CIFARIO and 39.8% for CI- FART00) but worse than DINO (92.5% for CIFARIO and 66.3% for CIFAR100), These results align with the trends observed in TEMI (Adaloglou al. 2023). Importantly, all methods achieve pseudo-label accuracy significantly better than SOTA label-free approaches, further highlighting the robustness of ELFS.
For instance, in CIFAR 100, DINO achieves the highest performance across all pruning rates, main- taining robust coreset quality even as pruning increases, owing to its superior pseudo-label accuracy. CLIP, while outperforming SwAv, shows a decline in performance at higher pruning rates, emphasiz- ing the importance of encoder quality in maintaining coreset effectiveness. These findings confirm the ablation results in Section 5.3.1 and Section 5.3.4, which demonstrate that better encoders result in higher-quality pseudo-labels and consequently improve the quality of the selected coresets.
Overall, these results validate that our method benefits from pseudo-labels generated by strong en- coders like DINO and CLIP, outperforming SOTA label-free methods, and showing the importance of encoder quality in achieving optimal performance in label-free free coreset selection.
Table 11: Ablation study on encoders with CLIP (Radford et al.. 2021 on CIFAR 10 and CIFAR 100.
We also report pseudo-label accuracy (PL-ACC) using different encoders.
CIFARIO
CIFAR100
Enender
PT.-
30%
50%
70%
80%
90%
PL.- ACC
30%
50%
70%
80%
90%
ACC
DINO
92.5%
95.5
95.2
93.2
90.7
87.3
66.3%
76.8
73.6
68.4
62.3
54.9
CLIP
85.2%
95.0
94.5
92.7
90.1
84.5
54.0%
75.5
73.1
66.5
58.4
50.6
SwAv
60.7%
95.0
94.3
91.8
89.8
82.5
39.8%
76.1
72.1
65.5
58.2
49.8
Best Label-Free
94.7
93.8
91.7
88.9
82.4
75.3
71.8
66.0
58.9
44.8
C.3
IMAGENET-IK TRANSFERARILITY STUDY
Table 12: Imagenet ResNet-50 Transferability
Pruning Rate
30%
50%
70%
80%
90%
ResNet-34
73.5
71.8
67.2
63.4
54.9
ResNet-50
76.2
74.0
69.5
64.6
56.4
Table 12 presents the results of the transferability experiment on ImageNet- IK, where coresets were selected using the ResNet-34 and evaluated on both ResNet-34 and ResNet-50. The results show that ResNet-50 consistently outperforms ResNet-34 across all pruning rates, with performance dif- ferences increasing at higher pruning rates. For example, at a 70% pruning rate, ResNet-50 achieves 69.5% accuracy compared to 67.2% for ResNet-34, and at 90%, ResNet-50 maintains 56.4% accu- racy compared to 54.9%. These findings highlight the robustness and generalizability of the selected coresets, as they effectively transfer across architectures.
C.4 DATA DIFFICULTY METRIC CASE STUDY VISUALIZATION
In this section, we visualize the hard/easy data chosen by the area under the margin (AUM) met- ric (Pleiss et al. 2020). AUM measures data difficulty by accumulating margin across different training epochs. The margin for example (x, v) training epoch is defined as: M (x, y)
Zy (x) - maxxyy ={ (x), where (1) (x) is the prediction likelihood for class at training epoch I.
AUM is the accumulated margin across all training epochs: AUM(X y) 7 Σ.= M (x. v)-
Following Pleiss et al. 2200)), we illustrate the distinction between hard and easy examples using AUM in Figure 7 The graphs depict logit values (model predictions before the softmax activation) over training epochs for one hard-to-learn image and one casy-to-leam image. For both examples, the green line represents the logit value for the correct label, while the red line represents the largest
17

---

Published as a conference paper at ICLR 2025
Hard Example
Easy Example
20
20
15
1.5
1.0
CAT
1.0
CAT
Logit
Logit
0
25
50
75
100 Epoch
125
150
175
200
0
25
50
75
100
125
150
175
200
Hard-to-leam example.
(h) Fasy-to-leam example.
Figure 7: Illustration of the hard and easy example using the AUM metrics (Pleiss et al. 2020 The green line represents the logit for the correct label, and the red line represents the largest other logit. The green region represents positive AUM. The logits are from training ResNet- 18 Model on the CIFARIO dataset.
Ground-Truth AUM
1750
Ground-Truth AUM
1500
CCS (Supervised)
1500
CCS (Supervised)
Random Sampling
EExamples
ELFS (Pseudo Label)
1000
2000
500
500
50
100
150
200
%
50
100
1500
200
Area under the margin (AUM)
Area under the margin (AUM)
(a) Random sampling coresct.
ELFS coresct.
Figure 8: Ground-truth label AUM distribution of different coresets on CIFAR100 with pruning rate of 50%. Random sampling selects more easy-to-learn data due to its higher density in the dataset. contrast, ELFS has less coverage on easy-to-learn examples. This difference explains performance enhancement observed with ELFS (73.6%) compared to random sampling (71.1%).
logit value among incorrect labels. The gray line shows other logits. The green region indicates positive AUM, where the correct label consistently has the highest logit value, signifying confidence the prediction.
In the hard example (Figure aa), the model struggles to confidently predict the correct label throughout training, as evidenced by fluctuations and overlap between the green and red lines. In contrast, the easy example (Figure 7b)) shows the correct label (green line) maintaining clear margin above the incorrect predictions (red line), indicating that the model quickly and confidently learns the example.
C.5 DATA DISTRIBUTION COMPARISON BETWEEN ELFS AND RANDOM SAMPLING
Figure compares the distribution of ground-truth label AUM across coresets selected by different methods on CIFAR100 with 50% pruning rate. Random sampling, as shown the green bars in Figure a(), tends to select more easy-to-learn examples, which are higher in density in the dataset. These examples have higher AUM values, indicating that they are easier for the model to predict. However, this results in an underrepresentation of -o-der-to-learn examples date points with lower AUM values that are crucial for improving the model's robustness and generalization. In contrast, ELPS (green bars) in Figure (b) selects more balanced coreset, covering more hard examples with lower AUM values by leveraging pseudo-label based scores and double-end pruning. This targeted selection explains why ELFS achieves better performance (73.6%) compared random sampling (71.1%). The graph highlights the importance of data difficulty diversity in the coresct to ensure a more effective and robust training process.
18

---

Published as a conference paper at ICLR 2025
ELFS
Random
FreeSel
90% pruning rate
80% pruning rate
70% pruning rate
100
Test Accuracy (96
95
85
2
10
2
10
of classes
Number
Figure 9: Coreset-based continual learning evaluation on CIFARIO. In the class incremental setting with 2 incoming classes per stage, ELFS outperforms other sampling methods such FreeSel (best label-free on CIFAR10) and random sampling under different storage budgets (10% 20%, and 30%).
C.6 DIFFERENCE BETWEEN LABEL-FREE CORESET SELECTION AND REMI-SUPERVISSS LEARNING
Semi-supervised learning (SSL) (Wag et al. 2022 Li et al. 2023) is another efficient way to ad- dress the challenge of high labeling costs in machine learning. SSL leverages a small, randomly sampled labeled dataset, often little 0.2%
0.02% of the full dataset (Wang al.) 2022 ei al., (023). along with large pool of unlabeled data, to train models more effectively. However, the output of SSL and coreset selection methods are fundamentally different. While SSL focuses on training a model by utilizing both labeled and unlabeled data, coreset selection outputs not only trained model but also a compact, nigh-quality labeled subset (the coreset). This selected core- set can be reused for a variety of downstream tasks, such as continual learning (Yoon et al. 2021 Borsos al. 2020 or neural architecture search (NAS) (Shim et 2021
Similar to Yoon et al. (2021). we conduct an additional evaluation of coreset- based methods in mass-incremental continual learning setting to demonstrate the effectiveness ELFS in downstream tasks, as shown in Figure In this experiment, the training process of CIFARIO is divided into five stages, with two new classes introduced incrementally at each stage. At every stage, fixed coreset budget (10% 20%, or 30%) is used to train a model. The results demonstrate that ELFS consistently outperforms other sampling methods, including FreeSel (the best-performing label-free method on CIFAR 10) and random sampling, across all storage budgets. These findings highlight the robustness and efficiency of ELFS in preserving performance under constrained memory while adapting to new data incrementally. This emphasizes that ELFS is suitable for real-world continual learning scenarios with limited computational resources and memory.
D ADDITIONAL DISCUSSION ON PRUNING HARD EXAMPLES AND GRID SEARCH ON HARD PRUNING RATE
pruning hard example helps improve the selected coreset quality. In this section, we will discuss how pruning hard examples help improve the quality of coreset and the overhead of identifying optimal 3 to prune hard examples.
As discussed in Section 4.3.2
The impact of hyperparameter 3 in coreset selection. Similar to previous work Zheng et al. 2023): ; Maharana al. (2023) in this area, ELFS involves hard pruning rate 3 as a hyperparameter, which aims to control the percentage of hard examples to be pruned. Higher 33 means that more hard data pruned. The reasoning that we prune hard examples is based on two insights: (1) Mislabeled examples often also have higher difficulty scores, which are harmful to training Pleiss et al. 20200)
and (2) pruning hard data can allocate more budget to cover high-density areas Zheng et al. 2023)
Grid searching 33 is also common practice in SOTA baselines Zheng al. (023 Maharana al. 2023).
Grid search overhead B. Grid searching 3 is common practice in SOTA coreset selection work Zheng et al. (2023); Maharana et al. (2023) and can introduce additional overhead for coreset selection. However, we argue that the overhead of the grid search is feasible, for million-level
19

---

Published as a conference paper at ICLR 2025
large-scale datasets. For instance, evaluate ELFS on the ImageNet- IK dataset, which contains over one million images. The grid search time for single pruning rate is approximately 17 hours using four A6000 GPUs. Given that primary goal is reduce human annotation costs, this computational overhead is relatively small compared to the significant savings in labeling efforts.
More efficient selection. One potential way optimize the choice of / is to leverage the convex relationship between beta and pseudo-label accuracy, as shown in Figure [ This convexity allows stop searching once the accuracy starts drop, which reduces the search space. However, we believe that there can be more efficient and effective way to search the optimal B and further enhance our method.
Malinois
teapot
Pyrenean Mountain
dough baguette
bikini
German Shepherd Dog
stopwatch
seashore
mobile phone plunger
orange pomegranate
analing clack
Dandie Thrrier dragonily damseltly
stopwatch
English Sheepdog
honeycomb apiary
television
pineapple
spotlight
Cocker Spaniels Retriever
split-rali fence
artichoke
helmet
picket
swimsuit
electric guitar
sarong pajamas
sports convertible
shi pwrrck
American
pineapple
European green
albatross goose
plunger
cowboy hat crayfish
poncho cardigan
notebook computer
bassinet
Baxer
harmonica flute
stingray great white shark
brambling junco
monitor
cradle
Staffordshire Bull Terrier
hare cottantail rabbit
castle
hourglass
radiator
car wheel
gas mask
monastery
digital watch
space heater
taxicab
traffic sign
European green lizard
Malinois
parking meter switch
prison
mop woal
Nile crocodile
Carolina anole German Shepherd Dog
turnstile
slug
Figure 10: Randomly sampled ImageNet mislabeled examples. Each image features caption that reads "Ground Truth Label - DINO Pseudo-Labell at the top. The pseudo-labels are generated using the Hungarian match algorithm (Kuhn 1955).
20


Reviews:
[{"review_id": "F9SDI2aRY7", "reviewer": "Reviewer_sVc5", "summary": "The paper presents ELFS (Effective Label-Free Coreset Selection), a method designed to improve label-free coreset selection by estimating data difficulty scores without requiring ground truth labels. The authors tackle challenges in label-free selection by employing pseudo-labels from deep clustering to approximate training dynamics and mitigate distribution shifts with a double-end pruning technique. ELFS shows superior performance over existing label-free methods across various vision benchmarks (e.g., CIFAR10, CIFAR100, and ImageNet-1K) and achieves results close to those of supervised selection methods.", "questions": "1. Given the grid search used to determine optimal hyperparameters, could a deeper analysis reveal why certain values work best for measuring sample difficulty? Specifically, how do these parameters influence the balance between easy and hard examples selected for the coreset, and could this inform a more consistent method for tuning them?\n\n2. ELFS currently uses SwAV and DINO as feature extractors for clustering. Would more powerful encoders, such as CLIP, improve the quality of pseudo-labels or provide more stable performance across datasets? Additionally, what effect might these alternative encoders have on the distribution of selected hard and easy examples?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. ELFS effectively addresses the limitations of previous label-free coreset selection approaches, providing a feasible solution that leverages deep clustering for pseudo-labeling.\n\n2. By employing double-end pruning, ELFS improves the selection of informative samples, achieving consistent performance improvements over baselines, even in challenging scenarios.\n\n3. The evaluation across multiple datasets and pruning rates, along with an ablation study, showcases ELFS's flexibility and robustness, which may benefit a range of vision tasks.\n\n4. The authors show that including more challenging samples enhances model performance, with ELFS effectively prioritizing hard examples through double-end pruning.", "weaknesses": "1. The experiments involve numerous hyperparameters, optimized through grid search. A more in-depth analysis of the underlying reasons behind these optimal values would strengthen the understanding of how different parameters affect the measurement of sample difficulty, offering clearer insights into the importance of hard examples.\n\n2. The approach heavily relies on feature extractors like SwAV and DINO for clustering. It remains unclear if using more advanced encoders, such as CLIP, could further improve performance or stability, suggesting potential limits in ELFS's generalizability with different encoders.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jOQCwHzYyT", "reviewer": "Reviewer_ChTP", "summary": "This paper proposes a new label-free coreset selection algorithm called ELFS to relieve the costly human annotation efforts. ELFS utilizes the deep clustering to generate pseudo-labels and estimate data difficulty scores. Afterwards, a double-end pruning method is introduced to mitigate the bias of data difficulty scores. Experiments show that ELFS can surpass previous label-free coreset selection baselines on several benchmarks.", "questions": "Please consider responding to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It is an elegant and effective idea to estimate the data difficulty score through deep clustering. This handles the challenge to measure the prediction uncertainty and sample difficulty without any human labels.\n\n2. The proposed method is evaluated on multiple classification benchmark, showing notable performance gain compared with state-of-the-arts.The design of each module is well justified through ablation studies.", "weaknesses": "1. My major concern lies in the selection of hyper-parameter $\\beta$. I can understand they require some grid search for hyper-parameters. However, according to Fig. 5, the optimal value is different for multiple datasets or sampling ratios, which is quite inefficient. For example, if there is a large dataset with millions of images, it is infeasible to do grid search on it.\n\n2. Based on Tab. 7, it is quite strange that ResNet50 cannot outperform ResNet18 on the selected subset. I assume it reasons from the simplicity of CIFAR10. Maybe the authors can do the transferability experiments on complex datasets like ImageNet since it is a main difference between corset selection and active learning. \n\n3. For Sec. 4.1, I assume the formulation of label-free coreset selection is already covered in previous work. It may be moved to Sec. 3 for clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M05RvGQb0C", "reviewer": "Reviewer_NwUk", "summary": "The paper introduces a novel method called ELFS (Effective Label-Free Coreset Selection) for selecting coresets without relying on labeled data. This approach uses pseudo-labels derived from deep clustering to approximate training dynamics, enabling the estimation of data difficulty scores. These scores help identify coresets that can be labeled for training high-performance models while minimizing human annotation costs. ELFS addresses the significant performance gap typically found in label-free coreset selection by introducing a double-end pruning technique to manage the distribution shift caused by pseudo-label inaccuracies. This method shows notable improvements in various vision benchmarks over existing label-free methods, demonstrating its ability to approximate the effectiveness of supervised coreset selection.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "ELFS presents a compelling label-free coreset selection method that reduces the need for extensive and costly labeled datasets while achieving accuracy close to supervised methods. By effectively utilizing pseudo-labels, ELFS not only significantly outperforms other label-free baselines but also exhibits strong performance despite the inherent inaccuracies and noise associated with pseudo-labels. Moreover, the method demonstrates robustness and versatility, showing good transferability across different datasets and model architectures, thereby enhancing its applicability in diverse machine learning tasks.", "weaknesses": "The ELFS method is quite effective, but it mainly builds on familiar techniques like pseudo-labeling and coreset selection. This might make it seem less novel or groundbreaking to those familiar with the field. Despite this, it does a great job using these methods to ensure high accuracy and reliability.\n\nMoreover, to really show how well ELFS works and to expand its use, it would be beneficial to test it on a wider variety of datasets. This includes tackling larger and more complex datasets such as ImageNet, as well as datasets with uneven distributions or long tails. Testing ELFS in these contexts would help validate its effectiveness across different challenges and environments.\n\nPotential Application Areas for ELFS: Beyond vision tasks, are there other types of data or tasks where ELFS could be effectively applied? Exploring its adaptability to different domains like text, audio, or even structured data could open up new applications.\n\nExplanation of Hard and Easy Examples in Section 4.4.2: Could a visual representation or graph be used to clarify the difference between hard and easy examples as discussed in the section? Visual aids could help illustrate how ELFS handles these types of data, enhancing understanding of its approach.\n\nAnalysis of Data Distribution in Table 1: Is it possible to analyze further how the data distribution of the coreset selected by Random compares to that selected by ELFS? Understanding the differences in selection criteria and resulting coreset characteristics could provide deeper insights into the strengths and limitations of ELFS compared to simpler random sampling methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1H16P654c4", "reviewer": "Reviewer_u4aX", "summary": "This paper proposes a new policy to sample a core subset for deep models. It introduces a deep clustering with the pseudo-labelling to estimate the score for each sample. Meanwhile, they try to fix the bias issue of pseudo-labelling. Experiments demonstrate the effectiveness of the proposed method.", "questions": "1. Please explain why double-end pruning helps the performance.\n2. Do you fine-tune the model with the coreset? Or do you train the model from scratch?\n3. Do you use only the coreset to train the model? It would be better to show the result of using the coreset as the labelled set and the rest data as the unlabelled set to train a model with a semi-supervised learning algorithm such as SemiReward[1]. If, with the help of semi-supervised learning, a randomly sampled labelled set achieves good performance, and the labelled set selected by your model yields similar performance, then the benefits of using a coreset to train the model need to be clarified.\n```\n[1] SemiReward: A General Reward Model for Semi-supervised Learning, Siyuan Li and Weiyang Jin and Zedong Wang and Fang Wu and Zicheng Liu and Cheng Tan and Stan Z. Li, ICLR 2024\n```", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of this paper is solid, and the topic of this paper exactly matches ICLR.\n2. The writing of introduction clearly delivered the motivation and idea.\n3. The experiment result looks good. It's interesting that many methods even cannot beat random sampling as suggested in Tab.1.\n4. The ablation study is extensive.", "weaknesses": "1. Some sentences are redundant, such as these two questions proposed in the paper.\n2. It would be better to move sec 4.1 to sec 3 to give readers an overview of the problem you are solving.\n3. My **main concern** is that more benchmarks in different distributions should be evaluated. As described in the paper, this method relies on a pretrained vision encoder to get the visual features for each sample. Then, a deep clustering algorithm is introduced to get the pseudo labels and scores. However, the evaluated datasets in this paper are too easy for pretrained vision encoders. I believe that much of the data in the evaluation datasets is included during pretraining. If we use a dataset in a different distribution, such as a medical image dataset, without a good visual feature, will this method still work?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)