Label-Free Coreset Selection with Proxy Training Dynamics

## Abstract

Abstract High-quality human-annotated data is crucial for modern deep learning pipelines, yet the human annotation process is both costly and time-consuming. Given a constrained human labeling budget, selecting an informative and representative data subset for labeling can significantly reduce human annotation effort. Well-performing state-of-the-art (SOTA) coreset selection methods require ground-truth labels over the whole dataset, failing to reduce the human labeling burden. Meanwhile, SOTA label-free coreset selection methods deliver inferior performance due to poor geometry-based scores. In this paper, we introduce ELFS , a novel label-free coreset selection method. ELFS employs deep clustering to estimate data difficulty scores without ground-truth labels. Furthermore, ELFS uses a simple but effective double-end pruning method to mitigate bias on calculated scores, which further improves the performance on selected coresets. We evaluate ELFS on five vision benchmarks and show that ELFS consistently outperforms SOTA label-free baselines. For instance, at a 90% pruning rate, ELFS surpasses the best-performing baseline by 5.3% on CIFAR10 and 7.1% on CIFAR100. Moreover, ELFS even achieves comparable performance to supervised coreset selection at low pruning rates (e.g., 30% and 50%) on CIFAR10 and ImageNet-1K.

## 1 Introduction

Figure: Figure 1: Label-free Coreset Selection Scheme. The goal of label-free coreset selection is to identify a high-performing subset of the data without relying on ground-truth labels, minimizing human annotation efforts.
Refer to caption: x1.png

Modern machine learning systems, particularly deep learning frameworks, are increasingly computationally demanding and data-intensive [, ].
High-quality labeled data is crucial in the deep learning pipeline.
Given the same model architecture, improving the quality of training data can significantly boost model performance [, , , ]. However, generating high-quality labeled data typically requires costly human annotation efforts.
Selecting an informative and representative subset for labeling can substantially reduce these costs [], thereby optimizing the use of resources and enhancing the overall efficiency of the data annotation process.

State-of-the-art (SOTA) coreset selection methods use data difficulty scores derived from training dynamics to select coresets [, , , , , , ].
However, these methods require ground-truth labels over the whole dataset for model training and calculating difficulty scores, which defeats the purpose of reducing human annotation efforts.
Existing label-free coreset selection methods, on the other hand, calculate importance scores based on geometric properties in the embedding space [, ].
However, compared to training dynamics scores, these geometry-based scores often fail to accurately estimate data difficulty and lead to poor coreset selection performance—even often worse than random sampling at some pruning rates [].

In this paper, we hypothesize that high-quality data difficulty scores can be calculated *without ground-truth labels*.
To calculate high-quality data difficulty scores and select representative coresets, we propose a novel coreset selection algorithm, Effective Label-Free Coreset Selection (ELFS).
Instead of using embedding space geometry-based scores, ELFS first employs deep clustering [] to assign a pseudo-label to each data point.
Then, the pseudo-labeled dataset is used to calculate training dynamic scores.
Even though pseudo-labels provided by deep clustering do not guarantee label accuracy, we empirically find that training dynamic scores derived from pseudo-labels still provide a good proxy to measure data difficulty for coreset selection (Section 3.4).

Directly applying traditional selection methods like CCS [] to pseudo-label difficulty scores outperforms existing label-free baselines.
However, a performance gap still exists between the pseudo-label based coreset selection and the supervised coreset selection.
Our analysis (Section 3.4) reveals that CCS sampling selects lots of easy data with pseudo-label difficulty scores, which provide less information compared to hard data.
To address this, we propose a simple but effective double-end pruning method for the data difficulty scores calculated with pseudo-labels.
Double-end pruning significantly reduces the number of selected easy examples and improves the coreset selection performance.

To verify the effectiveness of ELFS, we evaluate it on five vision benchmarks: CIFAR10, CIFAR100 [], CINIC10 [], STL10 [], and ImageNet-1K []. ELFS consistently outperforms all label-free baselines across all datasets at varying pruning rates. For datasets like CIFAR10 and ImageNet-1K, we even observe performance that matches supervised coreset selection methods at low pruning rates. For instance, with a 10% pruning rate on ImageNet-1K, ELFS achieves a 73.5% test accuracy, while the best-supervised method (D2 []) achieves 72.9%. Notably, ELFS is the first label-free approach to consistently and significantly outperform random sampling at high pruning rates. Furthermore, we empirically demonstrate the robustness of ELFS to pseudo-label noise. For example, even with a pseudo-label misclassification rate of 33.69% using DINO [] on CIFAR100, ELFS achieves 54.9% at a 90% pruning rate, outperforming other baselines by at least 7.1%.

## 2 Related Work

### 2.1 Coreset Selection

Coreset selection aims to select a representative subset of the training
data in a one-shot manner, which can be used to train future models while maintaining high accuracy.
Coreset selection methods are categorized into three primary categories []. (1) Geometry-based methods select samples by leveraging geometric properties in the embedding space. This includes distances to centroids [, , , ], to other examples [], or to the decision boundary [, ]. (2) Optimization-based methods aim to select subsets that produce gradients similar to the entire dataset [, ]. (3) Difficulty score methods select examples based on metrics such as forgetting scores [], area under the margin (AUM) [], EL2N [], and Entropy []. Recent studies have refined the difficulty score methods by proposing new sampling strategies that ensure the diversity of selected examples, rather than just selecting challenging ones [, ].

While most coreset selection techniques are developed for supervised settings, exploration in the self-supervised (label-free) contexts is limited [, ]. Sorscher et al. [] propose to compute a self-supervised pruning metric by performing k-means clustering in the
embedding space of SWaV [], while Maharana et al. [] introduce a method that starts with a uniform data difficulty score, which is then refined through forward and reverse message passing on the dataset graph.

### 2.2 Deep Clustering

Deep clustering aims to group unlabeled data into clusters based on learned features and has made significant strides with SOTA contrastive learning techniques  [, , , ].
Recent methods like TEMI  [] and SIC [] also leverage self-supervised Vision Transformers (ViTs) and vision-language models, such as CLIP [], DINO [], and DINO v2 []. These approaches have set new deep clustering baselines on datasets such as CIFAR10/100 and ImageNet, showcasing the efficacy of leveraging rich, pretrained representations from self-supervised and multi-modal models for the deep clustering task.

### 2.3 Active Learning

Active learning optimizes model accuracy with minimal labeled data by iteratively requesting labels from an oracle (e.g. a human annotator) for the most informative samples [, , ].
While it shares some similarities with our one-shot label-free coreset selection setting, several key differences distinguish the two directions:
(1) Selection strategies.
One-shot label-free coreset selection aims to select the coreset for human annotation in a single pass before training.
However, active learning requires repeated annotation during the training process.
For instance, BADGE [] needs annotation after each training iteration. This iterative annotation setting makes it less practical as it needs continuous human involvement for each iteration.
(2) Model independence: One-shot coreset selection aims to find a small, model-agnostic subset for training new models from scratch. In contrast, active learning selects instances tailored to the current model’s needs, making it inherently model-dependent [, ].
Those two differences make label-free one-shot coreset selection a more practical technique for data efficiency compared to active learning.
However, for a more comprehensive comparison, we include BADGE [], a pool-based active learning method, as a baseline in our evaluation as a label-free coreset selection baseline.

## 3 Methodology

### 3.1 Problem Formulation

The goal of label-free coreset selection is to minimize the human annotation cost by selecting an informative and representative subset while achieving good model performance.
Given an unlabeled dataset $\mathcal{D}=\{x_{1},x_{2},...,x_{N}\}$ belonging to $C$ classes and a budget $k\leq N$, we want to choose a subset $\mathcal{S}\subset\mathcal{D}$ and $|S|=k$ for labeling that maximizes the test accuracy of models trained on this labeled subset.
This unsupervised one-shot coreset selection problem can be formulated as the following optimization problem:

$$ $S^{*}=\operatorname*{arg\,min}_{\mathcal{S}\subset\mathcal{D}:|S|=k}\mathbb{E} _{x,y\sim P}[l(x,y;h_{\mathcal{S}})],$ (1) $$

where $P$ is the underlying distribution of $\mathcal{D}$,
$l$ is the loss function,
and $h_{\mathcal{S}}$ is the model trained with labeled $\mathcal{S}$.

### 3.2 Method Overview

The primary limitation of SOTA label-free data difficulty scores lies in their failure to incorporate training dynamics. Training dynamics better capture the model’s learning behavior during training, which provides a more accurate estimation of data difficulty compared to geometry-based metrics.
However, existing methods for calculating training dynamic scores require ground-truth labels, as these scores are derived from training models with labeled data. For example, the forgetting score [] quantifies how often a sample, once correctly classified, is subsequently misclassified during training.
Average Uncertainty Margin (AUM) [] quantifies model confidence by summing the margins during training, where the margin is the difference between the logits for the correct label and the second-highest logits.

Our key hypothesis is that *if training dynamics can be estimated without labels, it is possible to compute higher quality scores for estimating data difficulty.* To this end, we propose using pseudo-labels to approximate training dynamic scores.
As shown in Figure 2, we employ deep clustering to generate pseudo-labels []. Deep clustering involves organizing data points based on their representations learned by a neural network, enabling label assignment in an unsupervised manner.
Although pseudo-labels may not be perfectly accurate, our empirical results show that training dynamic scores derived from these pseudo-labels still serve as a robust proxy for measuring data importance (see Section 3.4).
We introduce our methods below:

### 3.3 Pseudo-Label Generation with Deep Clustering

Figure: Figure 2: ELFS pipeline: (1) *Pseudo-label generation:* ELFS begins with calculating image embeddings and nearest neighbors using a pretrained model and then assigns pseudo labels to unlabeled data via deep clustering algorithms; (2) *Coreset selection*: ELFS computes training dynamics scores, like AUM and forgetting score, with pseudo-labels. Next, a coreset will be selected based on the pseudo-label data difficulty score. (3) *Human labeling and training*: Humans annotate the selected coreset. This labeled coreset is used for later training.
Refer to caption: x2.png

In our approach, we employ Teacher Ensemble-weighted pointwise Mutual Information (TEMI) [], a SOTA deep clustering method, to assign pseudo-labels to an unlabeled dataset.

Step 1: Mining nearest neighbors in pretrained model embedding space. Various publicly available pretrained vision models [, , , ] provide rich semantic representations for deep clustering. For a dataset $\mathcal{D}=\{x_{1},x_{2},...,x_{N}\}$, we use a pretrained model $f(\cdot)$ to compute the $k$ nearest neighbors for each $x\in\mathcal{D}$ in the feature space using cosine similarity. The nearest neighbors for each $x$ are grouped into a set $\mathcal{N}_{x}$. These neighbors, often belonging to the same semantic cluster [], serve as candidates for clustering.

Step 2: Pseudo-label generation via deep clustering: Our next step is to assign a pseudo-label $\tilde{y_{i}}$ to each $x_{i}$ using information from its nearest neighbors $\mathcal{N}_{x_{i}}$. Recent studies [, , ] focus on training clustering heads $h_{\eta}(\cdot)$, which are softmax layers that output probability distributions over $C$ classes, ensuring consistency between a data point and its nearest neighbors.

TEMI uses an ensemble of student head $h_{s}(\cdot)$ and a teacher head $h_{t}(\cdot)$ that share the same architecture but differ in their updates. Each instance $x$ is processed through both types of heads, producing probabilistic classifications $q_{s}(c|x)$ and $q_{t}(c|x)$, respectively, where $c\in\{1,\ldots,C\}$ denotes the cluster class. The student head’s parameters $\theta_{s}$ are updated via backpropagation, while the teacher head’s parameters $\theta_{t}$ are updated via an exponential moving average (EMA) of the student’s parameters, ensuring stable target distributions.

Loss function. Given $H$ clustering heads, TEMI aims to minimize the following accumulated weighted pointwise mutual information (PMI) objective:

$$ $\mathcal{L}_{\text{TEMI}}(x)=-\frac{1}{2H}\sum_{h=1}^{H}\sum_{x^{\prime}\in \mathcal{N}_{x}}w_{h}(x,x^{\prime})\cdot\Big{(}{\text{pmi}^{h}}(x,x^{\prime})+ {\text{pmi}^{h}}(x^{\prime},x)\Big{)}\quad\quad\quad(1)$ $$

where ${\text{pmi}}(x,x^{\prime})=\log\left(\sum_{c=1}^{C}\frac{(q_{s}(c|x)q_{t}(c|x^
{\prime}))^{\beta}}{q_{t}(c)}\right)$ (^1^11$\beta$ is a hyperparameter for positive pair alignment, we follow TEMI to set it as 0.6 to avoid degeneration.) approximates the pointwise mutual information between pairs $(x,x^{\prime})$, and $w(x,x^{\prime})=\sum_{c=1}^{C}q_{t}(c|x)q_{t}(c|x^{\prime})$ weights each pair to reflect the probability that they belong to the same cluster.

Pseudo-labels generation: We assign pseudo-labels to the unlabeled dataset using a trained model with $H$ clustering student heads. For each data point $x$, predictions are obtained from all $H$ heads. The pseudo-label $\tilde{y}$ is then assigned by aggregating these predictions and selecting the class with the highest combined probability:

$$ $\tilde{y}=\arg\max_{c\in\{1,\ldots,C\}}\frac{1}{H}\sum_{h=1}^{H}q_{h}(c|x)$ $$

This pseudo-labeling method can introduce noise (see Table 2 and Table 10 in Section 9 for clustering results using different pretrained models across different datasets). Despite this, it allows for the effective use of semantic features from unlabeled data, enhancing model performance when labeled data is limited.

In the next section, we show that, despite the potential noise of pseudo-labels,
training dynamic scores derived from pseudo-labels still provide a good proxy to measure data importance for coreset selection.

### 3.4 Coreset Selection with Pseudo-Labels

Figure: Figure 3: Performance comparison of supervised CCS [], label-free CCS, best label-free baseline (label-free D2 []), and our method ELFS on CIFAR100.
Refer to caption: x3.png

Pseudo-labels generated by deep clustering make it feasible to calculate training dynamics scores with supervised training.
We find that, simply employing existing coreset selection methods, such as CCS [], we can select a coreset that outperforms SOTA label-free coreset selection methods (green curve in Figure 3).
However, there is still a performance gap remaining between CCS with pseudo-label scores (green curve) and CCS with ground-truth label scores (purple dashed curve).

Biased coverage of CCS with pseudo-labels data difficulty scores.
To better understand the reason behind this gap,
we compared the ground-truth AUM distribution of coresets selected by CCS (supervised) and CCS (pseudo-label).
AUM quantifies the model’s confidence in each data point throughout training, and higher AUM values indicate easier examples.

As shown in Figure 4(a), there is a significant distribution shift between CCS (supervised) and CCS (pseudo-label). Specifically, CCS using pseudo-labels (striped green region) selects easier (high-AUM) examples compared to CCS using ground-truth labels (pink region), leading to a performance drop.
To address this distribution gap, we propose a simple but effective double-end pruning method to select data with scores calculated with pseudo-labels, which aims to reduce the number of easy examples in selected coresets.
Double-end pruning consists of two steps:
*Step 1:* Prune $\beta$ hard examples first, where $\beta$ is a hyperparameter (as in [, , ]).
*Step 2:* Continue pruning easy examples until the budget is met.
Despite its simplicity, we find that double-end pruning significantly reduces the number of selected easy examples (striped green region in Figure 4(b)) and enhances the performance of label-free coreset selection (red curve in Figure 3).
We also conduct an in-depth study to discuss the difference between CCS and double-end pruning in Section 4.3.4.
It is worth noting that a recent concurrent study [] also discusses double-end pruning in supervised coreset selection, but our research is the first to demonstrate its substantial benefits in a label-free setting.

Figure: (a) Label-free CCS coreset.
Refer to caption: x4.png

Hard pruning rate $\beta$ search with pseudo-labels.
Another challenge for label-free coreset selection is determining an optimal hard pruning rate $\beta$. Pruning $\beta$ percent of the hardest examples first is a common practice in SOTA coreset selection methods (e.g., CCS [] and D2 []), as it helps ensure the selected coresets better cover high-density areas []. In supervised settings, CCS and D2 can use ground-truth labels to choose the best $\beta$. However, in a label-free setting, we do not have access to any ground-truth labels.

To address this, we use pseudo-labels for $\beta$ search. After generating the pseudo-labeled dataset, we split it into 90% for training and 10% for validation. We use the validation set to determine the optimal $\beta$. Once the optimal $\beta$ is identified, ELFS selects coresets from the entire training dataset. We empirically show that this approach enhances the performance of label-free coreset selection. We conduct a more detailed analysis for $\beta$ search in Section 4.3.2.
$\beta$ search with pseudo-labels guarantees that ELFS does not use any ground-truth labels in the coreset selection process.

## 4 Experiments

In this section, we evaluate ELFS on five vision benchmarks. Results show that ELFS outperforms SOTA label-free coreset selection baselines at all pruning rates. At a 90% pruning rate, ELFS surpasses the best baseline by 5.3% on CIFAR10 and 7.1% on CIFAR100. On CIFAR10 and ImageNet-1K, ELFS even achieves comparable performance to supervised coreset selection at low pruning rates.

### 4.1 Experiment Setting

We compare ELFS to three label-free coreset baselines (Prototypicality, BADGE, label-free D2) on five vision benchmarks (CIFAR10, CIFAR100 [], CINIC10 [], STL10 [], and ImageNet-1K []).
We use ResNet-34 [] for ImageNet-1K, and ResNet-18 [] for the rest.

Label-free coreset baselines. We compare ELFS with following label-free coreset selection baselines: A) Random: Randomly sampled coreset. B) Prototypicality []: Prototypicality generates embeddings from a pretrained vision model (SwAV []) to perform k-means clustering. The difficulty of each data point is quantified by its Euclidean distance to the nearest cluster centroid, with those further away being preferred during coreset selection. C)
BADGE []: An active learning method that selects new samples in each epoch using k-means++ initialization within the gradient vector space. D)
Label-free D2 []: D2 adapts supervised D2 pruning, which uses difficulty scores derived from training dynamics and selects coresets through forward and backward message passing in a dataset graph. In a label-free scenario, a uniform initial difficulty score is applied, and the dataset graph is initialized with embeddings from SwAV [].

Implementation.
We report ELFS results with DINO [] as the pretrained vision model to extract embeddings. We also conduct an ablation study on pretrained models in Table 1.
For deep clustering, we calculate 25-nearest neighbors (25-NN) for ImageNet-1K, and 50-nearest neighbors (50-NN) for the rest.
Due to the space limitation, we include more experimental details in section 8

### 4.2 Label-free Coreset Selection Performance Comparison

Figure: (a) Evaluation on CIFAR10.
Refer to caption: x6.png

We report the performance comparison between ELFS and other baselines in Figure 5.
We use the recommended hyperparameters in CCS [] and D2 [] for reporting the best-supervised baseline.
As shown in Figure 5, our proposed method ELFS significantly and consistently outperforms SOTA schemes as well as the random sampling at all pruning rates.
Random remains a strong baseline, especially with a high pruning rate ($80\%$ to $90\%$). Except for BADGE, we observe that all other label-free baselines are worse than random on CIFAR10, CIFAR100, and ImageNet-1K at high pruning rates.
Our method is the first label-free approach to outperform random sampling at all pruning rates.
For instance, at $90\%$ pruning rate, ELFS outperforms random by more than $10\%$ on CIFAR100, and $2.6\%$ on ImageNet-1K.
Notably, ELFS even demonstrates comparable performance to supervised methods on CIFAR10 and ImageNet-1K when the pruning rate is equal to 30% and 50%.
Due to the page limitation, we report the results on CINIC10 and STL10 in Table 9 in section 9.
For those two datasets, we observe similar findings—ELFS outperforms all SOTA label-free baselines.

### 4.3 Analysis & Ablation Study

#### 4.3.1 Impact of Pretrained Vision Models on Coreset Selection

**Table 1: Performance with different pretrained vision models. Our method ELFS consistently outperforms other baselines with all pretrained models.**
|  | CIFAR10 | CIFAR100 |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Pruning Rate | 0% | 30% | 50% | 70% | 80% | 90% | 0% | 30% | 50% | 70% | 80% | 90% |
| Random | 95.5 | 94.3 | 93.4 | 90.9 | 88.0 | 79.0 | 78.7 | 74.6 | 71.1 | 65.3 | 57.4 | 44.8 |
| Prototypicality [] | - | 94.7 | 92.9 | 90.1 | 84.2 | 70.9 | - | 74.5 | 69.8 | 61.1 | 48.3 | 32.1 |
| Badge [] | - | 93.6 | 93.0 | 91.0 | 87.9 | 81.6 | - | 74.7 | 71.8 | 65.2 | 58.9 | 47.8 |
| D2 (label free) [] | - | 94.3 | 93.8 | 91.6 | 85.1 | 71.4 | - | 75.3 | 71.3 | 66.0 | 56.2 | 42.1 |
| ELFS (SwAv) | - | 95.0 | 93.6 | 92.5 | 89.0 | 82.0 | - | 73.5 | 73.1 | 66.5 | 58.4 | 50.6 |
| ELFS (CLIP) | - | 94.5 | 94.7 | 92.8 | 89.6 | 84.8 | - | 75.3 | 71.2 | 65.9 | 58.3 | 48.8 |
| ELFS (DINO) | - | 95.3 | 94.9 | 93.3 | 90.1 | 86.9 | - | 76.8 | 74.0 | 68.4 | 61.9 | 54.9 |

**Table 2: Pseudo-labeling results after deep clustering. Reported metrics include the misclassification rate, adjusted random index (ARI), and normalized mutual information (NMI) in %. The misclassification rate of other datasets is reported in Table 10.**
|  | CIFAR10 | CIFAR100 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Miscls (%) | NMI (%) | ARI (%) | Miscls (%) | NMI (%) | ARI (%) |
| TEMI (SwAv) | 39.33 | 54.61 | 43.39 | 60.23 | 57.73 | 26.51 |
| TEMI (CLIP) | 14.87 | 75.53 | 70.93 | 45.95 | 66.43 | 40.18 |
| TEMI (DINO) | 7.50 | 86.49 | 85.07 | 33.69 | 76.48 | 52.95 |

Pretrained model ablation study. Pretrained vision model embeddings are crucial for effective label-free coreset selection, as high-quality embeddings yield more accurate pseudo-labels, enhancing coreset selection performance. We conduct an ablation study to evaluate the impact of different pretrained models on ELFS.

We first study how different pretrained vision models impact ELFS performance.
Besides DINO [], we also evaluated the ELFS with SwAv [] and CLIP [], as shown in Table 1. The deep clustering pseudo-label misclassification rates are reported in Table 2. NMI (Normalized Mutual Information) measures the similarity between clustering assignments and ground truth labels, while ARI (Adjusted Rand Index) evaluates the agreement between data clustering. Higher NMI and ARI reflect more accurate clustering. We find that a pretrained model that aligns better with the dataset, like DINO, provides more accurate pseudo-labels, which leads to better ELFS performance.
Notably, even though some pretrained vision models, like SwAv, produce pseudo-labels with a high misclassification rate (39% for CIFAR10), ELFS still achieves comparable or better performance than other label-free coreset selection baselines.

Similarly, as shown in Figure 10 and Figure 5(c), despite a 41.2% pseudo-label misclassification rate on ImageNet-1K, ELFS outperforms all label-free baselines. This demonstrates that ELFS performance remains robust and effective even with label noises.

**Table 3: Performance evaluation at varying pruning rates using pseudo-labels generated by different pretrained models for coreset selection methods.**
|  | DINO | SwAv |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Pruning Rate | 30% | 50% | 70% | 80% | 90% | 30% | 50% | 70% | 80% | 90% |
| Prototypicality | 94.9 | 93.8 | 86.6 | 75.7 | 42.6 | 94.7 | 92.9 | 90.1 | 84.2 | 70.9 |
| Label-free D2 | 94.2 | 93.1 | 91.5 | 88.2 | 82.8 | 94.3 | 93.8 | 91.6 | 85.1 | 71.4 |
| ELFS | 95.3 | 94.9 | 93.3 | 90.1 | 86.9 | 95.0 | 93.6 | 92.5 | 89.0 | 82.0 |

Label-free baselines with DINO. D2 and Prototypicality both use SwAv in their paper.
For a fair comparison, we also evaluate D2 and Prototypicality with DINO.
As shown in Table 3, ELFS consistently outperforms D2 and Prototypicality when using DINO, which further demonstrates the effectiveness of ELFS.

#### 4.3.2 Hard Pruning Rate β 𝛽 \beta italic_β Search with Pseudo-Labels

Figure: (a) DINO pseudo-labal validation accuracy.
Refer to caption: x9.png

As discussed in Section 3.4, ELFS has no access to ground-truth labels and thus cannot perform hyperparameter tuning to identify the optimal hard pruning rate $\beta$.
To address this challenge, we propose using pseudo-labels for the hard pruning rate $\beta$ search. In this section, we conduct an in-depth study to determine if the $\beta$ selected based on pseudo-label accuracy also performs better than other $\beta$ values when evaluated with ground-truth labels. Figure 6(a) presents the pseudo-label validation accuracy on various potential coresets, sampling from the forgetting score derived from pseudo-label training dynamics. The pseudo-label validation accuracy guides us in selecting the coreset for human annotation. Figure 6(b) shows the corresponding oracle (ground truth) test accuracy on the potential coresets. Our results on CIFAR100 with a $10\%$ step size demonstrate that the selected coreset yields the best oracle performance when the pruning rate is 30%, 70%, 80%, and 90%  despite a pseudo-label misclassification rate of $33.69\%$. At a $50\%$ pruning rate, the chosen coreset is only 0.1% ($74.0\%-73.9\%$) less effective than the best potential coreset. This shows that pseudo-label test accuracy is a reliable proxy to select the coreset when ground-truth labels are not available.

#### 4.3.3 Transferability

**Table 4: Performance of different architectures on CIFAR10 at varying pruning rates, trained with coreset selection based on forgetting scores from ResNet18 DINO pseudo-label training dynamics.**
| Pruning rate | 30% | 50% | 70% | 80% | 90% |
| --- | --- | --- | --- | --- | --- |
| Random (ResNet18) | 94.3 | 93.4 | 90.9 | 88.0 | 79.0 |
| ELFS (ResNet18) | 95.3 | 94.9 | 93.3 | 90.1 | 86.9 |
| ELFS (ResNet34) | 95.4 | 95.3 | 93.1 | 90.9 | 87.6 |
| ELFS (ResNet50) | 95.4 | 94.8 | 93.1 | 90.3 | 87.3 |

We evaluate coreset transferability on CIFAR10 using coresets selected during ResNet18 training. As shown in Table 4, these coresets exhibit good transferability to ResNet34 and ResNet50, demonstrating robustness and generalizability across different architectures.

#### 4.3.4 Sampling Methods Comparison

**Table 5: Performance comparison of label-free CCS, supervised CCS, ELFS, and label-free D2 on CIFAR100, using forgetting scores derived from DINO pseudo-label training.**
| Pruning rate | 30% | 50% | 70% | 80% | 90% |
| --- | --- | --- | --- | --- | --- |
| D2 (Best Label-Free) | 75.3 | 71.3 | 66.0 | 56.2 | 42.1 |
| CCS (Pseudo Label) | 75.5 | 72.9 | 67.2 | 60.9 | 51.8 |
| ELFS | 76.8 | 74.0 | 68.4 | 61.9 | 54.9 |
| CCS (Supervised) | 77.1 | 74.5 | 68.9 | 64.0 | 56.0 |

In Section 3.4, we show that CCS sampling causes a distribution shift when sampling with scores calculated using pseudo-labels. To better understand this effect, we compare our method with label-free CCS [], i.e. stratified sampling on the difficulty score derived from pseudo-label training with an optimal hard pruning-off for difficult examples. As shown in Table 5, our method consistently outperforms CCS by 1% to 3% across various pruning rates. This is expected, as our approach more effectively includes difficult data points chosen by the supervised CCS coreset (see Section 3.4).

#### 4.3.5 Computational Overhead

**Table 6: Computational Overhead for ELFS on various datasets, measured from the runtime on NVIDIA 2080Ti GPUs, on a Linux server with 128G CPU memory.**
| Dataset | CIFAR10 | CIFAR100 | STL10 | CINIC-10 | ImageNet-1K |
| --- | --- | --- | --- | --- | --- |
| Pseudo-Label Generation |  |  |  |  |  |
| Embedding and K-NN | 2 min | 2 min | 2 min | 14 min | 4 hour 50 min |
| Clustering Head Training | 41 min | 41 min | 12 min | 4 hours 46 min | 34 hours |
| Data Difficulty Calculation |  |  |  |  |  |
| Training Dynamic Collection | 65 min | 65 min | 50 min | 80 min | 31 hours |
| Data Difficulty Calculation | 1 min | 1 min | 1 min | 2 min | 40 min |

Computational overhead of our coreset selection method. We present the computational overhead of our coreset selection method across various datasets, measured using NVIDIA 2080 Ti GPUs, as shown in Table 6. Except for the training dynamic collection for ImageNet-1K, which uses two GPUs, all other tasks were executed on a single GPU. We do not report the computational overhead of grid searching for $\beta$, as the search time varies with the pruning rate, step size, and the search strategy. Instead, this is discussed in Section Limitations. It is worth mentioning that while BADGE, the active learning baseline, demonstrates acceptable performance compared to other baselines, it incurs significantly higher computational overhead. For example, on CINIC10, BADGE takes approximately 98 hours to complete the selection on a 2080 Ti GPU with 10% query budget (the smallest coreset size in our setting), whereas ELFS completes it in about 6.5 hours.

## 5 Conclusion

In this paper, we tackle the challenge of high annotation costs in deep learning pipelines by introducing ELFS, a novel label-free coreset selection method. Our approach leverages deep clustering to assign pseudo-labels to an unlabeled dataset, thereby enabling the calculation of training dynamic data importance scores without the need for ground-truth labels.
ELFS overcomes the limitations of geometry-based scoring methods, which often result in poor coreset selection performance, especially given high pruning rates. We empirically demonstrate the effectiveness of ELFS across five vision benchmarks: CIFAR10, CIFAR100, CINIC10, STL10, and ImageNet-1K. ELFS consistently outperforms existing label-free coreset selection baselines, and in some cases, nearly matches the performance of supervised coreset selection methods. Our findings underscore the potential of pseudo-labeling combined with training dynamics to enhance coreset selection, paving the way for more efficient and scalable high-quality data generation in deep learning frameworks.

## Acknowledgements

We thank Cisco Research for providing a grant to support this work. We are also thankful to the National Science Foundation and U.S. Department of Energy for their support. This material is based upon work supported by the National Science Foundation under Grant 2039445 and by the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344 by the LLNL-LDRD Program under Project No. 24-ERD-010 and Project No. 23-ERD-030. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of our research sponsors.

## Broader Impact

Our work on coreset selection using pseudo-labels and double-end pruning advances label-free coreset selection by reducing reliance on labeled data, thereby lowering annotation costs and enhancing scalability. This method promotes sustainable AI practices by improving training efficiency and reducing the carbon footprint of large-scale models. Furthermore, it democratizes access to advanced machine learning techniques, enabling researchers in resource-constrained environments to leverage state-of-the-art methods.

## Limitations

Access to pretrained models. Our approach assumes access to pretrained models. While this is generally feasible, it may not always be practical, especially for specific tasks where pretrained models are not readily available. Hyperparameter Tuning for $\beta$. Selecting the optimal $\beta$ value can incur extra overhead. In our implementation, we use binary search instead of a linear grid search, reducing the time complexity to $O(\log n)$. However, this still presents an additional computational burden. Despite this, our method demonstrates superior accuracy compared to SOTA label-free methods at all coreset rates. Efficiently fine-tuning $\beta$ remains an open area for future research, aiming to streamline the process while maintaining performance gains.

## 6 Overview

In Section 7, we discuss code release, licence, and reproducibility.

In Section 8, we provide details on datasets, best hyperparameters for our models, and other experiment settings.

In Section 9, we report additional evaluation results.

## 7 Code Release, Licence, and Reproducibility Statement

Codebase. We will publicly release our code and models. Our code base is built upon [TEMI](https://github.com/HHU-MMBS/TEMI-official-BMVC2023) (Apache) and [CCS](https://github.com/haizhongzheng/Coverage-centric-coreset-selection) (MIT). We include detailed experimental settings in Section 8.

Datasets. CIFAR10, CIFAR100 []: [Custom Licence](https://www.cs.toronto.edu/~kriz/cifar.html). ImageNet-1K []: [Custom Licence](https://image-net.org/download). STL10 []: [Custom Licence](https://cs.stanford.edu/~acoates/stl10/#:~:text=please%20cite). CINIC10 []: [Custom Licence](https://github.com/BayesWatch/cinic-10).

## 8 Detailed Experiment Setting

### 8.1 Pseudo-Label Generation

For pseudo-label generation, we use the training settings recommended in TEMI []. For CIFAR10, CIFAR100 [], STL10 [] and CINIC10 [], we calculate the 50-nearest neighbors (50-NN) for each example using cosine distance in the pretrained model’s embedding space. For clustering, 50 heads are trained. Training is conducted over 200 epochs with a batch size of 512, using an AdamW optimizer [] with a learning rate of 0.0001 and a weight decay of 0.0001. For ImageNet [], all the parameters are the same, except the nearest neighbor search is adjusted to 25-nearest neighbors (25-NN).

### 8.2 Coreset Selection with Pseudo-Label

**Table 7: Dataset specific pruning rates and their corresponding chosen $\beta$.**
| Dataset | Label Type | Pruning Rate |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | 30% | 50% | 70% | 80% | 90% |
| CIFAR-10 | pseudo-label (SwAV) | 0.1 | 0.2 | 0.2 | 0.3 | 0.4 |
| pseudo-label (CLIP) | 0.1 | 0.2 | 0.2 | 0.2 | 0.5 |  |
| pseudo-label (DINO) | 0 | 0 | 0.1 | 0.1 | 0.4 |  |
| CIFAR-100 | pseudo-label (SwAV) | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 |
| pseudo-label (CLIP) | 0.1 | 0.1 | 0.2 | 0.2 | 0.5 |  |
| pseudo-label (DINO) | 0.1 | 0.2 | 0.4 | 0.5 | 0.7 |  |
| ImageNet-1K | pseudo-label (DINO) | 0.0 | 0.1 | 0.2 | 0.4 | 0.5 |
| CINIC10 | pseudo-label (DINO) | 0.0 | 0.2 | 0.3 | 0.3 | 0.4 |
| STL10 | pseudo-label (DINO) | 0.2 | 0.2 | 0.4 | 0.6 | 0.7 |

Dataset Specific Training Details.
We benchmark our method ELFS using identical training settings as recommended in CCS [] and D2 []. CIFAR10 and CIFAR100: We use a ResNet18 model for 40,000 iterations, with a batch size of 256 and SGD optimizer settings that include 0.9 momentum and 0.0002 weight decay. The initial learning rate is set at 0.1 with a cosine annealing
learning rate scheduler []. ImageNet-1K: we train a ResNet34 model for 300,000 iterations, maintaining the same optimizer parameters and initial learning rate. CINIC10: We follow the same training setting in CCS []. STL10: We utilize the labeled portion of the STL10 dataset for training. For all coresets with different pruning rates, we train models with 6,400 iterations with a batchsize of 256 (about 320 epochs when pruning rate is zero). We use the SGD
optimizer (0.9 momentum and 0.0002 weight decay) with a 0.1 initial learning rate. Identical to other datasets, we use a cosine annealing learning rate scheduler [] with a $0.0001$ minimum learning rate.

Coreset Selection. For pruning ratio $\alpha=\{30\%,50\%.70\%,80\%,90\%\}$, we report $\beta$ for the best coreset start point selected in Table 7 on a variety of datasets.

## 9 Additional Evaluation Results

**Table 8: Performance evaluation varing pruning rates for ImageNet-1K dataset.**
| Pruning Rate | 0% | 30% | 50% | 70% | 80% | 90% |
| --- | --- | --- | --- | --- | --- | --- |
| Random | 73.1 | 72.2 | 70.3 | 66.7 | 62.5 | 52.3 |
| Prototypicality [] | - | 70.9 | 60.8 | 54.6 | 41.9 | 30.6 |
| Badge [] | - | 71.7 | 70.4 | 65.8 | 61.7 | 53.4 |
| D2 (label free) [] | - | 72.3 | 65.6 | 55.6 | 50.8 | 43.2 |
| ELFS (DINO) | - | 73.5 | 71.8 | 67.2 | 63.4 | 54.9 |

**Table 9: Performance evaluation at pruning rate $\alpha=\{30\%,50\%,70\%,80\%,90\%\}$ for CINIC10 and STL10 datasets.**
|  | CINIC10 | STL10 |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Pruning Rate $\rightarrow$ | 0% | 30% | 50% | 70% | 80% | 90% | 0% | 30% | 50% | 70% | 80% | 90% |
| Random | 89.9 | 88.9 | 87.6 | 85.1 | 82.6 | 77.3 | 79.3 | 75.6 | 70.2 | 64.0 | 57.9 | 45.4 |
| Prototypicality [] | - | 88.8 | 87.4 | 83.7 | 80.9 | 72.0 | - | 66.5 | 60.5 | 43.2 | 37.6 | 20.8 |
| Badge [] | - | 87.3 | 86.5 | 84.2 | 82.6 | 78.1 | - | 68.4 | 63.7 | 57.9 | 51.0 | 44.1 |
| D2 (label free) [] | - | 89.1 | 87.8 | 85.6 | 80.6 | 66.4 | - | 76.2 | 69.2 | 58.2 | 52.7 | 40.2 |
| ELFS (DINO) | - | 89.9 | 87.8 | 85.7 | 84.9 | 80.2 | - | 77.0 | 73.9 | 65.7 | 61.3 | 52.7 |

Figure: (a) SwAV CIFAR10.
Refer to caption: x11.png

**Table 10: The misclassification rate of pseudo-labels vs ground-truth labels, adjusted random index (ARI), and normalized mutual information (NMI) using TEMI [] (DINO []).**
|  | STL10 | CINIC10 | ImageNet-1K |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Miscls (%) | NMI (%) | ARI (%) | Train Mis (%) | NMI (%) | ARI (%) | Miscls (%) | NMI (%) | ARI (%) |
| TEMI (DINO) | 7.02 | 89.59 | 86.06 | 33.94 | 16.67 | 9.55 | 41.2 | 81.35 | 44.85 |

Figure: Figure 8: Randomly sampled ImageNet mislabeled examples. Each image features a caption that reads “Ground-Truth Label → DINO Pseudo-Label” at the top. The pseudo-labels are generated using the Hungarian match algorithm [].
Refer to caption: x17.png

In this section, we present evaluation results on ImageNet-1K (see Table 8), CINIC10, and STL10 (see Table 9). On CINIC10 and STL10, our observations align with the results from CIFAR10/100 and ImageNet. Our method, ELFS, consistently outperforms other label-free coreset selection methods across various pruning rates. We provide the pseudo-label misclassification rate, normalized mutual information (NMI), and adjusted random index (ARI) in Table 10. The results demonstrate that ELFS is reliable across a wide range of pseudo-label misclassification rates, from 7.02% to 41.2%. Additionally, Figure 7 shows the class-wise misclassification rate on CIFAR10 and CIFAR100, highlighting ELFS’s effectiveness with imbalanced class assignments.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. Given the grid search used to determine optimal hyperparameters, could a deeper analysis reveal why certain values work best for measuring sample difficulty? Specifically, how do these parameters influence the balance between easy and hard examples selected for the coreset, and could this inform a more consistent method for tuning them?

2. ELFS currently uses SwAV and DINO as feature extractors for clustering. Would more powerful encoders, such as CLIP, improve the quality of pseudo-labels or provide more stable performance across datasets? Additionally, what effect might these alternative encoders have on the distribution of selected hard and easy examples?
soundness: 4
strengths: 1. ELFS effectively addresses the limitations of previous label-free coreset selection approaches, providing a feasible solution that leverages deep clustering for pseudo-labeling.

2. By employing double-end pruning, ELFS improves the selection of informative samples, achieving consistent performance improvements over baselines, even in challenging scenarios.

3. The evaluation across multiple datasets and pruning rates, along with an ablation study, showcases ELFS's flexibility and robustness, which may benefit a range of vision tasks.

4. The authors show that including more challenging samples enhances model performance, with ELFS effectively prioritizing hard examples through double-end pruning.
summary: The paper presents ELFS (Effective Label-Free Coreset Selection), a method designed to improve label-free coreset selection by estimating data difficulty scores without requiring ground truth labels. The authors tackle challenges in label-free selection by employing pseudo-labels from deep clustering to approximate training dynamics and mitigate distribution shifts with a double-end pruning technique. ELFS shows superior performance over existing label-free methods across various vision benchmarks (e.g., CIFAR10, CIFAR100, and ImageNet-1K) and achieves results close to those of supervised selection methods.
weaknesses: 1. The experiments involve numerous hyperparameters, optimized through grid search. A more in-depth analysis of the underlying reasons behind these optimal values would strengthen the understanding of how different parameters affect the measurement of sample difficulty, offering clearer insights into the importance of hard examples.

2. The approach heavily relies on feature extractors like SwAV and DINO for clustering. It remains unclear if using more advanced encoders, such as CLIP, could further improve performance or stability, suggesting potential limits in ELFS's generalizability with different encoders.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please consider responding to the weaknesses.
soundness: 3
strengths: 1. It is an elegant and effective idea to estimate the data difficulty score through deep clustering. This handles the challenge to measure the prediction uncertainty and sample difficulty without any human labels.

2. The proposed method is evaluated on multiple classification benchmark, showing notable performance gain compared with state-of-the-arts.The design of each module is well justified through ablation studies.
summary: This paper proposes a new label-free coreset selection algorithm called ELFS to relieve the costly human annotation efforts. ELFS utilizes the deep clustering to generate pseudo-labels and estimate data difficulty scores. Afterwards, a double-end pruning method is introduced to mitigate the bias of data difficulty scores. Experiments show that ELFS can surpass previous label-free coreset selection baselines on several benchmarks.
weaknesses: 1. My major concern lies in the selection of hyper-parameter $\beta$. I can understand they require some grid search for hyper-parameters. However, according to Fig. 5, the optimal value is different for multiple datasets or sampling ratios, which is quite inefficient. For example, if there is a large dataset with millions of images, it is infeasible to do grid search on it.

2. Based on Tab. 7, it is quite strange that ResNet50 cannot outperform ResNet18 on the selected subset. I assume it reasons from the simplicity of CIFAR10. Maybe the authors can do the transferability experiments on complex datasets like ImageNet since it is a main difference between corset selection and active learning. 

3. For Sec. 4.1, I assume the formulation of label-free coreset selection is already covered in previous work. It may be moved to Sec. 3 for clarity.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 2
questions: See weaknesses.
soundness: 3
strengths: ELFS presents a compelling label-free coreset selection method that reduces the need for extensive and costly labeled datasets while achieving accuracy close to supervised methods. By effectively utilizing pseudo-labels, ELFS not only significantly outperforms other label-free baselines but also exhibits strong performance despite the inherent inaccuracies and noise associated with pseudo-labels. Moreover, the method demonstrates robustness and versatility, showing good transferability across different datasets and model architectures, thereby enhancing its applicability in diverse machine learning tasks.
summary: The paper introduces a novel method called ELFS (Effective Label-Free Coreset Selection) for selecting coresets without relying on labeled data. This approach uses pseudo-labels derived from deep clustering to approximate training dynamics, enabling the estimation of data difficulty scores. These scores help identify coresets that can be labeled for training high-performance models while minimizing human annotation costs. ELFS addresses the significant performance gap typically found in label-free coreset selection by introducing a double-end pruning technique to manage the distribution shift caused by pseudo-label inaccuracies. This method shows notable improvements in various vision benchmarks over existing label-free methods, demonstrating its ability to approximate the effectiveness of supervised coreset selection.
weaknesses: The ELFS method is quite effective, but it mainly builds on familiar techniques like pseudo-labeling and coreset selection. This might make it seem less novel or groundbreaking to those familiar with the field. Despite this, it does a great job using these methods to ensure high accuracy and reliability.

Moreover, to really show how well ELFS works and to expand its use, it would be beneficial to test it on a wider variety of datasets. This includes tackling larger and more complex datasets such as ImageNet, as well as datasets with uneven distributions or long tails. Testing ELFS in these contexts would help validate its effectiveness across different challenges and environments.

Potential Application Areas for ELFS: Beyond vision tasks, are there other types of data or tasks where ELFS could be effectively applied? Exploring its adaptability to different domains like text, audio, or even structured data could open up new applications.

Explanation of Hard and Easy Examples in Section 4.4.2: Could a visual representation or graph be used to clarify the difference between hard and easy examples as discussed in the section? Visual aids could help illustrate how ELFS handles these types of data, enhancing understanding of its approach.

Analysis of Data Distribution in Table 1: Is it possible to analyze further how the data distribution of the coreset selected by Random compares to that selected by ELFS? Understanding the differences in selection criteria and resulting coreset characteristics could provide deeper insights into the strengths and limitations of ELFS compared to simpler random sampling methods.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Please explain why double-end pruning helps the performance.
2. Do you fine-tune the model with the coreset? Or do you train the model from scratch?
3. Do you use only the coreset to train the model? It would be better to show the result of using the coreset as the labelled set and the rest data as the unlabelled set to train a model with a semi-supervised learning algorithm such as SemiReward[1]. If, with the help of semi-supervised learning, a randomly sampled labelled set achieves good performance, and the labelled set selected by your model yields similar performance, then the benefits of using a coreset to train the model need to be clarified.
```
[1] SemiReward: A General Reward Model for Semi-supervised Learning, Siyuan Li and Weiyang Jin and Zedong Wang and Fang Wu and Zicheng Liu and Cheng Tan and Stan Z. Li, ICLR 2024
```
soundness: 3
strengths: 1. The motivation of this paper is solid, and the topic of this paper exactly matches ICLR.
2. The writing of introduction clearly delivered the motivation and idea.
3. The experiment result looks good. It's interesting that many methods even cannot beat random sampling as suggested in Tab.1.
4. The ablation study is extensive.
summary: This paper proposes a new policy to sample a core subset for deep models. It introduces a deep clustering with the pseudo-labelling to estimate the score for each sample. Meanwhile, they try to fix the bias issue of pseudo-labelling. Experiments demonstrate the effectiveness of the proposed method.
weaknesses: 1. Some sentences are redundant, such as these two questions proposed in the paper.
2. It would be better to move sec 4.1 to sec 3 to give readers an overview of the problem you are solving.
3. My **main concern** is that more benchmarks in different distributions should be evaluated. As described in the paper, this method relies on a pretrained vision encoder to get the visual features for each sample. Then, a deep clustering algorithm is introduced to get the pseudo labels and scores. However, the evaluated datasets in this paper are too easy for pretrained vision encoders. I believe that much of the data in the evaluation datasets is included during pretraining. If we use a dataset in a different distribution, such as a medical image dataset, without a good visual feature, will this method still work?

Meta-review: This paper proposes label-free coreset selection by using deep clustering to estimate data difficulty scores w/o gt labels and this paper proposes a double-end pruning method to mitigate bias on these calculated scores. After carefully considering the reviewers' feedback and the author's rebuttal, the Area Chair (AC) agrees with the positive assessment provided by the four reviewers. All reviewers have reached a consensus to accept the paper, recognizing its merits. The AC has thoroughly reviewed the comments and rebuttal and concurs with the overall evaluation. Consequently, the AC recommends accepting this paper for publication.

Decision: Accept (Poster)