Rethinking Coreset Selection: The Surprising Effectiveness of Soft Labels
Authors: Saumyaranjan Mohanty
Deexitha Vattivella
Konda Reddy Mopuri
Venue: Accepted by TMLR ()

Abstract: Data-efficient deep learning is an emerging and powerful branch of deep learning that focuses on minimizing the amount of labeled data required for training. Coreset selection is one such method, where the goal is to select a representative subset from the original dataset, which can achieve comparable generalization performance at a much lower computation and disk space overhead. Dataset Distillation (DD), another branch of data-efficient deep learning, achieves this goal through distilling a small synthetic dataset from the original dataset. While DD works exploit soft labels (probabilistic target labels instead of traditional one-hot labels), which have yielded significant improvement over hard labels, to the best of our knowledge, no such study exists for coreset selection. In this work, for the first time, we
study the impact of soft labels on generalization accuracy for the image classification task for various coreset selection algorithms. While soft labels improve the performance of all the methods, surprisingly, random selection with soft labels performs on par or better than existing coreset selection approaches. Our findings suggest that future coreset algorithms should benchmark against random selection with soft labels as an important baseline.

Published in Transactions on Machine Learning Research (03/2026)
Rethinking Coreset Selection: The Surprising Effectiveness of Soft Labels
Saumyaranjan Mohanty
ai23resch04001 @att0.0@hhhho.....
Deportment of Artificial Intelligence Indian Institute of Technology Hyderabad
Decxitha Vattivella
es22btech1 1036@ithh.e.
Department of Engincering Science
Indian Institute of Technology Hyderobad
Konda Reddy Mopuri
krmopuri@oi. wwt.uuiiiiiiittt...oo
Department of Artificial Intelligence Indian Institute of Technology Hyderabad
Reviewed on OpenReview: https: openreview. net/ met/forre forum? L178KAR1
Abstract
Data-efficient deep learning is an emerging and powerful branch of deep learning that focuses on minimizing the amount of labeled data required for training. Coreset selection is one such method, where the goal is to select a representative subset from the original dataset, which can achieve comparable gencralization performance at a much lower computation and disk space overhead. Datasel Distillation (DD), another branch of data-efficient deep learning, achieves this goal through distilling a small synthetic dataset from the original dataset. While DD works exploit soft labels (probabilistic target labels instead of traditional one- hot labels), which have yielded significant improvement over hard labels, to the best of our knowledge, no such study exists for coreset selection. In this work, for the first time, we study the impact of soft labels on generalization accuracy for the image classification task for various coresct selection algorithms. While soft labels improve the performance all the methods, surprisingly, random selection with soft labels performs on par or better than existing coreset selection approaches. Our findings suggest that future coreset algorithms should benchmark against random selection with soft labels as an important baseline.
1 Introduction
The escalating scale and complexity of contemporary datasets and deep learning models have necessitated the use of increasingly intensive computational resources. While large-scale datasets have driven the stunning success of deep learning (Alzubaidi et al., 2023; Aach et al., 2023; Zhou et al., 2025), increasing computing power has become central to performance improvements, which in turn has led to an alarming rise in the environmental cost of deep learning (Thompson et al., 2022; Patterson et. al., 2021). Moreover, achieving Srate-OFThho-Ae (SOTA) results on complex architectures requires extensive hyperparamctor tuning, which in turn consumes substantial computational resources (Zoph & Le, 2017; Li et al., 2018).
Coreset selection is one of the data-efficient deep learning mechanisms, which aims to select smaller but representative subset of the original large-scale dataset (Mirzasoleiman et al., 2020) without sacrificing generalization performance much. Approximating the learning characteristics of the full dataset, such as gradient information and data redundancy, enables learning a representative subset, significantly reducing training time and computational requirements (Feldman, 2020). Various coresct selection methods have been developed based on these principles. These include geometry-bast (Chen et al., 2010; Sener & Savarese, 2018; Zheng et al., 2023), loss-based (Toneva al., 2019; Paul al., 2021), decision boundary based (Ducoffe

---

Published in Transactions on Machine Learning Research (03/2026)
& Precioso, 2018; Margatina et al., 2021), gradient based (Killamsetty et al., 2021; Mirzasoleiman et al., 2020; Mohanty et al., 2025).
Most image classification tasks utilize hard labels, where each sample is assigned a particular class. Szegedy et al. (2016) demonstrated that label smoothing using soft target distributions instead of one-hot labels improves model generalization and calibration. In their work on knowledge distillation, Hinton et al. (2015) used soft labels generated through a teacher model (a sophisticated neural network) transfer its knowl- edge to a student model (a relatively simpler neural network), improving the student model's performance substantially.
In the case of a hard label, the target class has a probability of 1, and the rest of the classes have a probability 0. While with soft labels, all the classes assume non-zero probabilities, with the highest probability assigned to the target class. In knowledge distillation, soft labels encode the teacher model'ss uncertainty arising from class similaritics, guiding the student network to learn nuanced inter-class relationships.
Dataset Distillation (DD) is another data-efficient deep learning method, which aims to generate small number of synthetic datasets (much smaller than the original dataset), which can encompass information from the original dataset. While coresct selection seleets a subset of the real datasct, there is no such constraint on dataset distillation as it generates synthetic images, which can be visually indiscernible as compared to the real dataset. Hence, coreset methods are constrained to the original data manifold. However, distillation methods can encode rich priors (including teacher kuowledge) into learned synthetic data and soft labels, often at the cost of more complex optimization procedure to generate those distilled examples.
Recent SOTA works on Dataset Distillation (DD) methods (Yin et al., 2023; Du et al., 2024; Sun et al., 2024) have utilized soft labels to achieve significant improvements over methods using hard labels. Qin et al. (2024) explored the contribution of soft labels in improving generalization performance by these DD works compared to the contribution of the methods. They have highlighted the crucial role of soft labels in distillation, demonstrating that bascline random-sclection methods with soft labels perform on par with, or better than, existing distillation methods. Their study raises a natural question, whether a similar behavior is expected in the case of coreset selection? Crucially, dataset distillation and coreset selection reduce training cost by very different means: distillation synthesizes new (often optimized) images and labels, while coresct selection chooses subset of existing images. Unlike dataset distillation, where sophisticated optimization may encode teacher priors into the synthetic samples, coreset selection operates under the stricter constraint of choosing real examples, making it fundamentally different and underexplored setting for evaluating the rolc of soft labels.
In this paper, we explore soft labels' impact on the performance of coreset selection methods and ask when training on real image subsets, do improvements come mainly from enriched label information or from smorter subset selection?
We have conducted extensive ablation studies on benchmark image classification datasets using various CNN and transformer-basee architectures. Our major contributions in this paper are:
Comprehensive study on the impact of utilizing soft labels with coresct selection methods in the context of image classification.
Comparative analysis of the performance gains with soft labels on random selection and coreset selection methods.
Empirical demonstration that soft labels with random selection perform on par better than coreset selection methods.
2
Related Works
2.1 Coreset selection
Geometry-based methods (Agarwal et al., 2020; Chen et al., 2010: Sener & Savarese, 2018; Zheng et al., 2023; Xia et al., 2023) aim to select the coreset that preserves the geometric structure of the original dataset.
2

---

Published in Transactions on Machine Learning Research (03/2026)
while removing data points that are close to each other in the feature space. Decision boundary-basee methods (Ducoffe & Precioso, 2018; Margatina et al., 2021) exploit the fact that data points nearer the decision boundary generally carry more information value for the model than points far away from it.
Uncertainty-based methodologies posit that samples assigned lower confidence scores by a model exert greater influence on enhancing generalization compared to those with higher confidence. The Selection via Proxy framework (Coleman et al., 2020) employs a reduced-scale proxy model in lieu of the full target model. These computationally efficient models serve as effective surrogates, offering reliable signals for coreset se- lection grounded in both uncertainty and representativennss. Furthermore, commonly adopted criteria for quantifying sample uncertainty include least-confidence mcasures (Shen et al., 2018) and entropy-based met- rics (Settles, 2012). Loss/Ean Errorrbasedd methods (Toneva et al., 2019; Paul et al., 2021) select coresets based on each sample's contribution towards the loss function during model training. Nagaraj et al. (2025) proposed to construct the coresct by considering samples that have a higher correlation the loss difference between the training and validation trajectory.
Zheng et al. (2023) proposed Coverage-centric Coreset Selection (CCS), which selects coreset by utilizing both data coverage and sample importance. D) Pruming method (Maharana et al., 2024) combines both diversity and difficulty score through message passing on dataset graph to select the most representative subset. Cho et al. (2025) introduced Difficully and Uncertainty Aware lightweight (DUAL) score to combine example difficulty and prediction uncertainty to select a coreset during the early training stage.
Gradient Matching- based methods (Mirzasoleiman et al., 2020; Killamsetty et al., 2021) utilize gradients from the full training dataset and select a coreset whose weighted gradients yield the smallest difference. An intuitive coresct selection method termed 'Noisc-free Loss Gradients' based the similarity of loss gradients proposed by Mohanty et al. (2025), which composes a coreset of samples that have the maximum nunber of neighbours with higher cosine similarity among their gradients.
2.2 Use of soft labels
In their seminal work on knowledge distillation, Hinton et al. (2015) showed that, by reflecting output distributions of a well-trained model, soft-labels can encode usable information in a way that one-hot labels can not. Szegedy et al. (2016) introduced soft labels through a technique called label smoothing regularization. This work showed that blending true one-hot labels with soft label scheme of uniform distribution is an effective regularizer for the model.
Soft labels help reduce overfitting by encouraging the network to be less confident in any class, stabilizing training, and improving calibration predicted probabilitics. Lukasik et al. (2020) have shown that, under noisy labels conditions, label smoothing is quite effective.
3 Methodology
In this section, we detail the proposed study. We first train a teacher network on the given large dataset with hard labels. Then, obtain the coresct for the large dataset. Later, we obtain the corresponding soft labels using one of the two methods (sec. 3.1 and sec. 3.2) described below. Finally, we use the coreset and soft labels to train new classifier accordingly (sec. 3.3). We perform this analysis across various coreset selection methods and compare their performance with that of their hard label counterparts.
3.1 Generating soft labels via Teacher Network
The Knowledge Distillation (KD) (Hinton et al., 2015) process requires soft labels to transfer the learned knowledge from a sophisticated teacher onto a simpler student network. For this purpose, the softmax output of the trained teacher used as the soft labels train the student network. The rich information these soft labels enables the simpler student to achieve a better generalization performance compared training it with hard labels.
3

---

Published in Transactions on Machine Learning Research (03/2026)
In study, approach to generating soft labels is to trained teacher network. Let z zeacher be the logits predicted by the teacher for sample in the coreset. gleacher E RK where K is the number of classes. Logits refer to the unnormalized output values produced by the final layer of a neural network before applying the softmax (Goodfellow et al., 2016) operation. Let T be the temperature parameter. The leacher-based soft. labels are given by:
exp(z feacherr /T)
keacher
(1)
N= exp(zk feacher ( /T)
3.2 Generating soft labels via label smoothing
We explored Label smoothing. introduced by Szegody et al. (2016) as another way to obtain soft labels. The standard one-hot vector is replaced by softened version, where the larget becomes a mixture of the original ground truth and uniform distribution, It can be represented
smooth
-
if i = ground truth label, otherwise.
(2)
where [0, 1] is the label smoothing parameter. It may be noted that label smoothing does not add any priors, unlike Knowledge Distillation, and is primarily used as regularizer to address overfitting and overconfidence during training deep neural networks.
3.3
Training the classifier the coreset
The next step after obtaining the soft labels is training the classifier freshly on the coreset in order to assess its effectiveness in representing the original dataset.
3.3.1
With based soft labels (y teacher)
Similar to the teacher, logit output of the student (25) also processed with temperature (r) to obtain its prediction (y):
ý: exp(z /T)
(3)
E exp(z5/T)
The classifier training objective over the soft labels is the KL-divergence loss function, which is given by:
teacher
LKD KKy (5) 2 yk
teacher log yk
(4)
ÿk
In the case of hard labels, the standard cross-entropy loss is used to train the network.
LCE
Yk log(Ek)
(5)
Where y E (0, 1 is the ground truth one-hot label for the sample. Finally, a weighted combination of soft and hard label losses trains the student network.
L = (I - 2) LOE A LKD
(6)
Where À is the weight assigned to the soft label loss. Figure 1 provides an overview of the classifier training coreset with teacher-based soft labels.
4

---

Published in Transactions on Machine Learning Research (03/2026)
3.3.2 With label-smoothing based soft labels
If we obtain the soft labels for the coreset via label smoothing (Eq. 2), we train the target classifier optimizing the cross-entropy loss (Eq. 5). our experiments, we have used the PyTrrc implemenlation of cross-entropy Joss with label smoothing
Teacher Network
Soft labels
Distilled
Knowledge
Coreset
Criecd
Cosch
Student Network
Hard labels
Figure 1: Classifier training with teacher-based soft labels over the coreset. Note that the Leacher is trained the large dataset with hard labels. It provides the soft labels and is frozen during the student network's training.
3.4 Analysis soft vs hard labels from Information Theory viewpoint
From an information-theorele viewpoint, soft labels increase entropy, as each label carries greater uncer- tainty and more mutual information about the class structure. Soft labels act a regularizer by replacing the empirical one-hot targets with smoother distribution that better approximates the population distri- bution 2, This reduces the KT. divergence between the model's predictive distribution and the population distribution, compared to training with hard labels.
A hard label for an N-class classification problem a one-hot vector, where each output E {0, 1}. Entropy of y is given by H (y) = 0, as all probability mass is concentrated on the target class.
A soft label probability distribution, and its entropy is given by:
H(y) = Σ ploo((y))
The richer supervision, due to the strictly positive entropy, encodes teacher uncertainty and the inter-class similarity structure. This additional entropy per sample acts as an informational regularizer, reducing overfitting (Pereyra et al., 2017). In low-data. regimes typical of coreset selection, each example must carry more learning signal. Hard labels collapse all probability mass onto single class, discarding inter-class similarity. By contrast, soft labels increase per-sample entropy and encode richer structure, effectively raising the information budget per example.
4 Experimentation
Applications. We have evaluated the impact of soft labels with coreset selection for image classification across various benchmark datascts.
hnttps://docs htt////c..pytrrrre pytorch. wttg///cs//ttalleggareaaatee IIt. refers the underlying, true distribution of classifications for given sample, often aggregated from a population of human
wrw.sentropyLoss.oooooo html

---

Published in Transactions on Machine Learning Research (03/2026)
Datasets. We worked with benchmark datasets for image classification, such CIFAR-100 (Krizhevsky, 2009), Tiny ImageNet (Te & Yang, 2015), and ImageNet-1K (Deng et al., 2009). To study the impact of soft labels when selecting a coreset from dataset with significant label noise, we have also considered CIFAR-10ON (Wei ct al., 2022).
Classifier. We trained randomly initialized ResNet-18 (He et al., 2016), ResNet-50 (He et al., 2016), VGG- 16 (Simonyan & Zisserman, 2015) and VIT (Liu et al., 2021) architectures as the target classifiers
Teacher networks. Unless otherwise specified, well-trained ResNet-18 model (trained the full dataset) is used as the teacher network for the teacher-based soft label generation. Coreset selection methods. We have considered a variety of popular coreset selection methods, such as CRAIG (Mirzasoleiman et al., 2020), Forgetting (Toneva et al., 2019), GradMatch (Killamselty et al., 2021), GraNd (Paul et al., 2021), GraphCut (Wei et al., 2015), Moderate (Xia et al., 2023), CCS (Zheng et al., 2023), D pruning (Maharana et al., 2024), DUAL (Cho et al., 2025), CLD (Nagaraj et al., 2025), and Noiso-free gradients (NFG) (Mohanty et al., 2025) on which the proposed study is carried out. Importantly, we have also studied the impact of soft labels on randomly selected subset as the coreset.
We have utilized the DeepCore (Guo et al., 2022) library 3 for the implementation of CRAIG, Forgetting, GradMatch, GraNd, and GraphCut methods. We have used publicly available codebase released by the authors of respective papers for the implementation of Moderate 4 CCS 5 D2 pruning 6 DUAL 7, NFG and CLD 9, The CCS and D² pruning methods use iterations instead epochs for training over a selected coreset. The number of epochs varies based on the size of the coreset chosen. Thus, we have presented the results obtained on CCS and D>2 pruning separately in subsection 4.2.
4.1 Results
We have observed that soft labels generated through the teacher network outperform those generated using label smoothing. This expected as teacher-generaled soft labels encode reliable information than randomly assigned probabilities through label smoothing. Hence, all our experimental results reported in the main paper are based teacher-generate random seeds and have reported the mean and standard deviation of the accuracy values obtained. For completeness, the results obtained with the label smoothing-basel soft. labels are provided in the appendix. The best performances are highlighted in bold, while the second- best performances are underlined.
soft labels. We have carried out each experiment 5 times with different
4.1.1 Result CIFAR-100
Table presents the test set accuracies obtained with hard (HL) and soft labels (SL) for the CIFAR-100 dataset with ResNet-18 architecture. Two noteworthy observations are (i) Soft labels improve generalization accuracy for all the coreset selection methods, (ii) random selection with soft labels performs par and sometimes outperforms the existing coreset selection methods.
4.1.2 Result on Tiny ImageNet
Figure shows test set accuracy values obtained on ResNet-18 and ViT architectures for the Tiny ImageNet dataset. The coreset selection methodologies we have considered are Moderate (Xia et al., 2023), Noise Free Gradients (NFG) (Mohanty et al., 2025). It can be noted that while random selection with hard labels performs less than the coreset selection methods, random selection with soft labels performs on par with NFG and better than the moderate selection method.
httttss//gthaa..cc///aatiikk/// htttpps//gthuub.co///mmbb////// ICLR. Moderate-DS
Shttps: $https://githubb. htttpp//gghbbbbbccmm////uugggggg http:///www.......ooocoo////ccca
hhttterr/ggtaabboomm//yyaaaaaaan /2pruning
8 httte///gihuubcomm////eeehhoonon /Noiios free gradient
fradg
9 hhttp:////tthuti https///ghhubb.omm/mmmmmagggggg CCrreett._CCosssssssscoooaaa from Trajectories
Coresets

---

Published in Transactions on Machine Learning Research (03/2026)
Table I: Performance of the coreset selection methods with hard and soft labels for the CIFAR-100 dataset with the ResNet-18 architecture. Various Coresct sizes are considered from 0.5% 20% of the full dataset. The best performance in each column is presented in bold, and the second best is underlined. While soft labels improve test set accuracy all the methods, random selection with soft labels performs par with other coreset selection methods. Test accuracy with model trained on full dataset is 77.41%.
Methods
0.5%
1%
5%
10%
20%
GraiMatch (HL)
4.15 0.25
11.07 1: 0.03
28.88
1.45
40.04 +: 1.39
56.18 i 0.82
GradMatch (SL)
10.84 = 0.07
23.02 0.49
57.11
0.02
59.24 = 0.08
60.80 0.12
Cmig (HL)
6.44 + 0.38
10.19 1 0.15
19.64
0.99
26.45 1 0.59
45.14 1 0.45
Craig (SL.
11.41 + 0.44
19.77 + 0.10
57.34 + 0.11
59.26 + 0.07
60.34 + 0.08
GraphCut (HL)
8.18 = 0.41
12.52 0.07
30.30 0.62
41.05 0.80
56.78 0.32
GraphCut (SL)
13.69 1 0.19
24.08 0.15
57.22
59.41 1: 0.22
60.70 0.04
GraNd (HL)
2.38 + 0.23
3.71 + 0.05
7.92 = 0.33
12.83 + 0.10
32.02 = 1.02
GraNd (SL)
9.68 + 0.22
18.87 + 0.53
56.37 +: 0.08
58.12 1 0.05
60.06 + 0.08
Forgetting (HL)
0.43
9.45 + 0.07
0.22
37.63 0.09
56.09 0.31
Forgetting (SL)
11.33 + 0.35
27.13 = 0.09
55.26 0.26
56.83 = 0.15
58.07 = 0.13
Moderate (HL)
5.36 = 0.12
7.78 + 0.20
18.62 0.33
32.57 + 1.40
54.40 = 0.19
Moderate (SL.)
9.26 + 0.09
14.67 + 0.21
51.18 0.31
$7.61 + 0.27
58.35 + 0.32
DUAL (HL.)
11.13 + 0.35
15.10 + 0.71
38.43
0.52
53.37 + 0.23
57.63 + 0.45
DUAL (SL)
14.87 + 0.48
20.37 = 0:22
49.19 + 0.15
61.95 + 0.23
63.12 + 0.69
CLD (HL)
7.83 0.17
8.73 1 0.51
23.06
0.39
36.04 1 0.47
52.32 0.28
CLD (SL)
12.23 + 0.21
14.15 0.19
40.04 0.46
56.97 + 0.51
66.22 0.12
NFG (HL)
12.00 + 0.11
18.95 + 0.08
37.67 + 0.10
46.48 + 0.28
58.06 + U.34
NFG (SL.)
11.98 + 0.80
26.34 + 0.20
56.88 + 0.50
58.51 + 0.30
59.38 + 0.10
Random (HL)
7.97
0.03
12.82
0.16
31.93
0.15
41.00 0.04
52.75
0.20
Random (SL)
16.26 0.91
36.07 + 0.15
57.35
0.27
58.45 1 0.08
59.24 0.08
60
-((H)
((H))
C(H]
DH
60
-EHH)
-F(S]
-A(S -B(S)
((S
15)
(5)
Test Test Accuracy 1%)
40
40
Testt Accuracy
20
20
Random (HL)
Moderate (HL)
NFGJSL)
Randum/(U)
Modeerate(SU)
0.5
10
20
30
50
60
Selection percentage
Selection Percentage
Figure 2: Test set accuracy obtained on Tiny ImageNet Dataset with hard label and soft label with knowledge distillation. The left panel shows results obtained on the ResNet-18 architecture (Methods used are, A: GraphCut, B: GradMatch, C: Facility Location, Moderate, E: NFG, F: Random), and the right panel shows results obtained the ViT architecture. HL refers to training with hard labels, while SL refers to training with soft labels.
4.1.3 Result on ImageNet-1K
Table compares the results obtained with hard labels and soft labels on the ImageNet-1K datasct with the ResNet-18 18 architecture. Results for ViT architecture are shown Figure Soft labels enhance the perfor- mance of the student network quite significantly. However, similar to other experiments, random selection

---

Published in Transactions on Machine Learning Research (03/2026)
with soft labels outperforms other coreset selection methods at 5% and 10% selection ratios, underscoring the importance of soft labels when training the student network on limited dataset.
Top-1 accuracics for ImageNet-IK dataset with ResNet-18 architocture. Soft labels are generated through the teacher network. As can be seen, soft labels can enhance the performance of the student network quite significantly. However, similar to other experiments, the random selection with soft labels is performing almost par with other methods, underlying the significance of soft labels during the training of the student network with a limited dataset. Test accuracy with model trained on full dataset is 69.76%.
Table 2:
Methods
0.1%
0.5%
1%
5%
10%
GradMatch (HL)
0.93 + 0.04
5.20 + 0.22
12.28 + 0.49
40.16 + 2.28
45.91 + 1.73
GradMatch (SL)
2.13 + 0.12
13.85 + 0.31
28.54
0.24
52.56 0.23
58.14 0.47
Craig (HL)
1.13 + 0.08
5.44 = 0.32
9.40 + 1.69
32.30 + 1.24
38.77 = 0.56
Craig (SL)
2.23 1 0.12
16.52 1 0.37
32.67 1 0.21
56.92 1 0.34
59.38 1 0.17
GraphCut (HL)
1.21 +: 0.09
7.66
16.43 0.53
42.23
0.60
50.53 0,42
GraphCut (SL)
2.24 + 0.23
14.84 + 0.37
31.08 + 0.17
56.56 0.21
59.48 0.17
Forgetting (HL)
0.76 + 0.01
4.69 +: 0.17
14.02 +: 0.13
47.64 1 0.03
55.12 0.13
Forgetting (SL)
2.02 + 0.13
15.87 + 0.15
32.12 0.31
56.65 0.24
59.50 0.18
NFG (HL)
1.88 + 0.10
11.58 + 0.15
22.82 + 0.07
43.50 + 0.41
49.51 + 0.26
NFG (SL)
4.70 +: 0.17
21.22 = 0.14
37.77 + 0.23
55.50 + 0.27
58.60 + 0.41
Random (HL)
0.76 + 0.01
3.78
8.85 +
40.09 0.21
52.10 0.22
Random (SL)
1.97 +: 0.20
15.78 + 0.14
32.26 + 0.23
56.97 + 0.18
59.61 + 0.31
GradMatch (HL)
GradMatch (SL)
Forgetting (HL)
Forgetting (SL)
Random (HL)
Random (SL)
Test Accuracy (%)
30
15
10
Selection percentage
Figure 3: Test set accuracy comparison for various coreset selection methods with hard and soft labels, ViT architecture and ImageNet-1K dataset.
4.2 Result for CCS and 2 pruning
Table compares the performance of CSS and 22 method in hard and soft label settings for the CIFAR- 100 dataset. with the ResNet-18 architecture. Comparing with Random selection with soft labels,

---

Published in Transactions on Machine Learning Research (03/2026)
observe that random selection with soft labels is performing almost par with these two coreset selection methods, beating them narrowly for 50% selection ratio. Figure 4 compares the performance of CCS and D² methods with hard and soft labels and random selection with soft labels for ImageNet- IK dataset with ResNet-34 architecture. Random selection with soft labels performs at par with CCS with soft labels while oulperforming D with soft. labels for lower selection Tractions.
Table 3: Comparison of test set accuracies on CLFAR-100 dataset with ResNet-18 architecture with hard and soft labels with CSS and D² pruning methods. Random selection with soft labels is also provided. As can be seen, random selection with soft labels performing on par with coreset selected through these two methods for majority of selection ratios. Test accuracy with model trained full dataset. is 77.41% %.
Selection ratio
Hard Label
Soft Label
Random SL
CCS
D
CCS
D
0.01
12.0 + 0.7
11.1
0.1
17.1 + 0.6
17.2 + 0.8
14.8 + 0.5
0.05
40.6
0.3
31.1
1.2
56.4
+ 0.3
50.1 + 1.4
49.0
55.0 + 0.7
56.8
0.2
05.4 0.6
66.2 +
63.5 + 0.3
0.2
63.2 + 0.5
64.9
+ 0.7
71.4 0.8
72.4 + 0.2
71.3 + 0.2
0.3
68.8 + 0.1
70.2
+ 0.5
74.5 0.2
74.6 + 0.7
74.2 + 0.2
0.5
73.8 + 0.2
75.3
0.4
76.4 0.7
75.4 +
76.6 = 0.3
75
60
test test accuracy (%))
ACS_AUM (HL)
CCS_AUM (SL)
30
(HL)
(SL)
Random (SL)
15
0.05
0.1 Selection fraction
0.15
0.2
Figure 4: Comparison of test sot accuracies ImageNet dataset with ResNet-34 architecture with hard and soft labels with CSS and D² pruning methods. Random selection with soft labels performs at par with CCS + soft labels and outperforms the D + soft labels.
4.3
Ablation Studies
In this section, we perform ablation studies analyze the impact of various hyperparameters on generaliza-
tion accuracy.
4.3.1 Impact of the weight the soft labels towards generalization
Eq. describes how both hard label loss (cross entropy loss) and soft label loss (KL-divergence) are weighted by the À to constitute the total loss function. Figure shows the impact of the weight on the soft label loss the test. set. accuracy. As the soft label loss weightage increases, accuracy also increases.

---

Published in Transactions on Machine Learning Research (03/2026)
800
softt label loss
haadd labal loss
Accaraty (In
600
-
0.0.44
Training
400
-.00.8
1=.1.
200
0
60
Epochs
0.5
fraction
Figure Comparison of training loss progression with soft label and hard label. Results are shown for the Tiny ImageNet dataset, with random selection baseline for 20% coreset selection. Training with soft labels converges much faster than training with hard labels.
Figure 5: Impact of soft label loss weightage on test set accuracy. The graph shows that using pure soft labels (corresponding À 1.0) gives the best sults, while pure hard label loss gives the poorest results. Results are obtained for the Tiny ImageNet dataset on the ResNet-18 architecture with Noise Free Gradients (Mohanty et al., 2025) as the coreset selection method.
4.3.2 Training dynamics comparison
Figure 6 shows the progression of the training loss while training with hard and soft labels. Keeping all the hyperparameters equal, training with soft labels converges faster than training with hard labels. It signifies potential for training for a lesser number of epochs to obtain better results. It also shows that soft labels possess more learnable information than hard labels.
4.3.3 Impact of temperature
The lemperature parameter is vital in smoothing the soft. labels (Mäller el al., 2019). Table 4 tabulates test set accuracy vs the value of r.
Table 4: Impact of temperature parameter on test-set accuracy. Results are obtained on ImageNet- IK datasot at various selection percentages with ResNet-18 architecture. Test set accuracy first increases with temperature, then decreases further.
Temperature
Accuracy (0.5%)
Accuracy (1%)
19.22 +:2 U.91
28.76 + U.11
20.77 + 0.21
30.79 + 0.12
10
21.52 + 0.13
31.10 + 0.27
20
20.75 1 0.22
28.49 i 0.15
20.67 1 0.16
27.45
4.4 Cross architecture performance study
In this section, we study the cross-architccture performance of various coresct sclection methods and compare against a random selection baseline, under training with soft. labels. The coreset selection is carried out by the ResNet-15 architecture for Forgetting, GradMatch and NFG methods, while ResNet-34 architecture is used for CCS and 2 coresct selection method. Cross-architcctural performance tested on VGG-16 and ViT architectures, on the ImageNet-1K dataset. The results are provided in Table 5, Figure and Figure We observe that random selection with soft labels performs par with other coreset selection methods.
10

---

Published in Transactions on Machine Learning Research (03/2026)
Table 5: Cross architecture evaluation for various coreset selection methods with hard labels and soft labels. The coresets are generated through the ResNet-18 architecture and evaluated on the VGG-16 and ViT architectures. In the cases of soft labels, ResNet-18 is used as the teacher model.
Target Architecture-+
VGG-16
VIT
Selection % +
0.5%
1%
5%
10%
0.5%
1%
5%
10%
Forgetting (HL)
=
0.2
48.3 0.2
57.5 0.7
0.1
8.5 =
14.3 0.2
Forgetting (SI.)
24.0 0.1
0.3
0.1
0.2
0.2
10.5 0.3
20.5 0.7
GradMatch (HL)
0.7 +: 0.2
8.7 + 0.3
39.4 0.1
51.3 0.2
1.1 + 0.1
1.5 + 0.2
10.3 0.3
16.0 0.6
GradMatch (SL)
22.7 0.2
37.6 0.2
37.5
0.4
61.2 0.9
2.8
+ 0.2
4.0
+ 0.4
16.4 1: 0.3
23.4 1 0.8
NFG (HL)
10.9 0.4
18.0 + 0.4
48.4
0.3
54.9 0.7
3.6 + 0.2
6.1
+ 0.6
18.1
0.3
29.5 0.7
NFG (SL)
25.4 + 0.3
42.5 + 0.7
58.0 + 0.1
61.3 0.9
5.1+ 0.3
7.1
0.4
20.3 + 0.2
30.2 + 0.8
Random (IIL)
+ 0.2
+: 0.2
47.3 0.4
57.2 0.3
2.5 0.3
0.3
17.4 0.5
29.0 1 0.3
Random (SL)
23.6 0.3
41.0 0.1
57.9 0.5
61.4 0.4
3.7 0.2
0.2
20.6 1 0.4
31.4 0.7
30
1est Accuracy
Accurscy
200
40
10
Random (H) -Rnndom CSS
Random (1) -anddom (S) CCS
CCS (9)
02 (H]
-02 (S)
-OCS (5)
-02 (H)
-02
10
Selection Percentage
Selection Percentage
Figure 7: Cross architecture evaluation for varions coreset selection methods with hard and soft labels. The coresels are generated through the Res Vel-34 architecture and evaluated on the VGG-16 architec- ture. In the case of soft labels, ResNet-34 is used as the teacher model.
Figure 8: Cross architecture evaluation for various coresct selection methods with hard and soft labels. The coresets are generated through the ResNet-34 architecture and evaluated on the ViT architecture. In the case of soft labels, ResNet-34 used as the teacher model.
4.5 Impact on dataset with label noise
We have studied the impact of soft labels with coreset selected from the CIFAR-100N dataset, which is dataset specifically generated to study the impact of label noise introduced by human annotators. Table 6 provides results for the CIFAR-100N dataset with the ResNet-18 architecture. Soft labels are generated through the Leacher nelwork. Random selection with soft labels outperforms other coreset selection hods for lower selection percentages, while performs second-best. for 10% and 20% selection.
4.6 Impact of teacher accuracy on students' performance
multiple checkpoints of the teacher model to train the student model for various percentages of coreset selection. Figure 9 shows that the student network performs better the same coreset when the teacher network accuracy is higher. The results are shown for ImageNet-11K for 0.5%, 1%, and 5% of coreset. selection. A more accurate leacher generates soft labels that better capture true inter-class relationships and uncertainty, rather than noisy or misleading distributions. These high-quality soft labels act a stronger supervisory signal, guiding the student toward more robust decision boundarics even when trained on small coresets. Consequently, the student benefits from the teacher's superior knowledge, leading to improved generalization performance.
In this experiment,
11

---

Published in Transactions on Machine Learning Research (03/2026)
Table 6: Results for CIFAR-1OON dataset. ResNet-18 architecture is used. While soft labels are able to improve test accuracy on all the methods, random selection with soft labels performs on par with other coreset selection methods.
Methods
0.5%
1%
5%
10%
20%
GradMatch (HL)
4.96 +:
8.89
19.91
0.41
21.15 0.11
32.21 0.52 40.99 0.12
GradMatch (SL.)
9.53 + 0.39
14.61 + 0.59
30.98 + 0.05
38.22 0.23
Craig (HL)
5.83 + 0.13
7.98 + 0.40
11.12 + 0.11
19.00 + 0.25
21.29 +: 0.12
Craig (SL)
9.92 + 0.02
12.95 = 0.24
27.60 0.71
37.15 0.12
40.02 0.02
GraphCut (HL)
6.84 + 0.11
9.84 + 0.33
20.07 = 0.07
22.27 + 0.04
33.61 1.22
GraphCut (SL)
10:22 +: 0.26
12.45 + 0.12
27.88 + 0.55
37.18 + 0.29
40.55 1 0.03
NEG (HIL)
7.71 + 0.27
11.34 + 0.30
30.87 = 0.92
40.88 U.54
49.21 0.16
NFG (SL)
9.15 + 0.43
22.91 + 0.36
41.72 + 0.21
42.91 + 0.17
50.01 + 0.24
random (HL)
7.06 1 0.52
8.59
0.28
22.97 T 1.15
29.65
0.31
42.81
0.21
random (SL)
16.9 + 0.40
29.77 = 0.21
42.31 + 0.23
42.86 0.26
43.04 0.15
45
Student accuracy (%6
fraction=00005
30
fraction=0.01
fraction=0.05
15
0
15
30
45
60
Teacher accuracy (%)
Figure 9: Impact of teacher model accuracy on performance of the student network for various percentages of coresct selection on ImageNet-111 -K dataset. As we can observe, the student performs better when the teacher's accuracy improved.
5 Discussion
Our results reveal a striking and consistent trend: introducing soft labels substantially improves the perfor- mance of models trained on coresets, the extent that random selection with soft labels often matches surpasses sophisticated coresct selection algorithms with hard labels. This finding carries several important implications.
Rethinking evaluation baselines. Traditionally, coreset selection methods are benchmarked against random selection with hard labels. Our results show that this baseline underestimates (at least on existing image recognition benchmarks) what can be achieved with minimal selection bias once label information is enrichod. Furthermore, may be noted that the teacher model required to produce informative soft labels is trained on the full dataset. In other words, a well-trained teacher improves students' performance better argue that random selection with soft labels should be
than poorly trained teacher. Consequently, adopted as a new standard bascline when evaluating future coresot methods.
Interpreting the role of soft labels. Soft labels encode richer information about inter-class similarity and teacher uncertainty. When coresets drastically reduce the available training data, this additional structure mitigates overfitting and improves calibration, providing robustness especially in noisy-label regimes such CIFAR-10ON, Our ablations on À (soft label weight) and 7 (temperature) confirm that carefully tuned soft labels can drive consistent generalization gains.
12

---

Published in Transactions on Machine Learning Research (03/2026)
Beyond image classification. Although experiments focus standard image classification bench- marks, the principle is not domain-specifie. We expect similar dynamics to emerge in text and audio tasks where inter-class structure is informative. Extending this investigation to other modalities and downstream applications is a promising direction for future work.
Limitations. Our findings do not imply that subset selection obsolete. Subset choice continues to matter in scenarios where teacher models are unavailable or too costly (need train on the entire dataset) and storage/memory memory constraints prohibit dense soft-label distributions.
In summary, this study demonstrates that the perceived advantage of coreset methods can be largely neu- tralized once soft labels are introduced. Any future evaluation of coreset selection should include random selection with soft. labels as a baseline.
6 Conclusion
In this work, for the first time, we systematically examined the role of soft labels in coreset selection for data-cfficient learning. We found that soft labels consistently enhance performance across multiple datasets, architectures, and coreset algorithuns. Remarkably, even simple random selection combined with soft labels better than the sate-of-theeart coreset selection methods using
often achieves accuracy comparable hard labels.
Our study also highlights broader insight: investing in improved teacher models or belter soft-label gen- eration methodology may provide greater benefits than devising increasingly complex selection methods. At the same time, coresets remain relevant in scenarios where teacher models are unavailable or computa- tional/storage budgets prohibit them. Overall, our study suggests that the benchmark for coresct methods should be redefined to include random selection with soft labels as stronger baseline.
Looking ahead, we believe our study opens two promising avenues: (1) extending the analysis of soft labels with coresets beyond image classification into object detection, text, audio, and multimodal domains, and (2) exploring hybrid hods that jointly optimize subset selection and label softening.
13

---

Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora. Contextual diversity for active learn- ing. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XVI, pp. 137-153, Berlin, Heidelberg, 2020. Springer- Springer-Verlag, ISBN 978-3-030-58516-7, doi: 00.1007/978-3-000- Published in Transactions on Machine Learning Research (03/2026)
References
Marcel Aach, Eray Inanc, Rakesh Sarma, Morris Riedel, and Andreas Lintermann. Large scale performance analysis of distributed deep learning frameworks for convolutional neural networks. Journal of Big Dala, 10(1), June 2023. ISSN 1115. doi: 00.1886//445377-023-00766 URL http://dx. doi org/
9. URL https: wtt.://...0///01100000 000100000000/7---0000-8---------
Laith Alzubaidi, Jinshuai Bai, Aiman Al-Sabaawi, Jose Santamaría, A. S. Albahri, Bashar Sami Nayyef Al-dabbagh, Mohammed A. Fadhel, Mohamed Manoufali, Jinglan Zhang, Ali H. Al-Timemy, Ye Duan, Amjed Abdullah, Laith Farhan, Yi Lu, Ashish Gupta, Felix Albu, Amin Abbosh, and Yuantong Gu. A survey deep learning tools dealing with data scarcity: definitions, challenges, solutions, tips, and applications. Journal of Big Data, 10(1):46, Apr 2023. ISSN 2196-1115. doi: 10.1186/s403377-02-000777 URL https://ddi.orr//10.. 11186//00577-023-00777----
Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. In Proceedings of the Twenty-Sian Sizth Conference on Uncertainty in Artificial Intelligence, UAI'10, pp. 109 116, Arlington, Vir- ginia, USA, 2010. AUAI Press. ISBN 9780974903965.
Yeseul Cho, Baekrok Shin, Changmin Kang, and Chulhee Yun. Lightweight dataset pruning without full training via example difficulty and prediction uncertainty. In Forty-second International Conference Machine Learning, 2025. URL http:://pppaeeiiioooooooooo/o/// net/forruridddrrriiizzz
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matci Zaharia. Selcction via proxy: Efficient data selection for deep learning. In International Conference on Learning Represenlations, 2020. URL https: http://ppeeeevieeeee net/forum?id= HJg2bOVYDr.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pottern Recognition, pp. 248-255. IEEE, 2009.
Jiawei Du, Xin Zhang, Juncheng Hu, Wenxin Huang, and Joey Tianyi Zhou. Diversity-driven synthesis: En- hancing dataset distillation through directed weight adjustment. In The Thirty-eighth Annual Conference on Newral Information Processing Systems, 2024. URL httpss///penrrvveew net/forum??d-uuSaDll. id=wwSaDDLII....
Melanie Ducoffe and Frederic Precioso. Adversarial active learning for deep networks: margin based approach, 2018. URL https: bestps//aaxxin http://xxxvv....g/e. org wwwo//ggg/a.oooooooooooo 0t////802.0090
Dan Feldman. Introduction to corc-scts: updated survey, 2020. URL https://arx/v.org/aaa 09384.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, Cambridge, MA, 2016.
URL attp://wwa deeplearningbook. org.
Chengeheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selection in deep learning. In Database and Expert Systems Applications: 33rd International Conference, DEXX 2022, Vienna, Austria, August 22 24, 2022, Proceedings, Part 1, pp. 181 191,, Berlin, Heidelberg, 2022. Springer- Verlag. ISBN 078-3-031-12442--8 doi: 00.1007/978-33001-1122233 14. URL tttps://doi..o 10. 11-11079978----1--222------
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: 10. /V.100//VPP..201..000
14

---

Published in Transactions on Machine Learning Research (03/2026)
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531,, 2015. URL https://api.senant semanti cacholar..og//oorpsttttt 722003347.
Krishnateja Killamsetly, Durga Sivasubramanian, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad- match: Gradient matching based data subset selection for efficient deep model training, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto, Toronto, ON, Canada, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. httpp::/lttbyyimggeettt.eene herokuapp. com/, 2015. Accessed: 2025-08-13 13.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, (185):1-52, 2018. URL http://jmlr otgppapprrs/v1/11--55555 html.
Yahui Liu, Enver Sangincto, Wei Bi, Nicu Sebe, Bruno Lepri, and Marco De Nadai. Efficient training of visual transformers with small datasets. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In Proceedings of the 37th International Conference on Machine Learning, TCMT.20. JMMR..rr, 2020.
Adyasha Maharana, Prateek Yadav, and Mohit Bansal. $)mathbb(D)""$ pruning: Message passing for balancing diversity difficulty in data pruning, In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttps::///ppeeeeeeeee wwwpaareiiww..nttoouuoeee hotp/oorrmm/dddtabrooAoo.
Katerina Margatina, Giorgos Vernikos, Loic Barrault, and Nikolaos Alctras. Active learning by acquiring contrastive examples. In Proceedings of the 2021 Conference on Empirical Melhods in Natural Language Processing, pp. 650-663, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL https://aclanthholgg/.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovee. Coresets for data-efficient training of machine learning models. In Proceedings of the 37th International Conference on Machine Learning, ICML/20. JMLR.org, 2020.
Saumyaranjan Mohanty, Chimata Anudeep, and Konda Reddy Mopuri. Noise-free loss gradients: A sur- prisingly effective baseline for coreset selection. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreevie. http:///ppparreeeeee..eettrrtrre nett//oorumm1ddd0PPPttteee
Rafael Müller, Simon Korublith, and GeolTrey Hinton. When does label smoothing help? Curran Associates Inc., Red Hook, NY, USA, 2019.
Manish Nagaraj, Deepak Ravikumar, and Kanshik Roy. Coresets from trajectories: Selecting data via correlation of loss differences. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL
https://openreview
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training, 2021. URL https:
//arxiv.org/aate wttp://///.bbb/.....eegg 10350.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Tn A. Beygelkimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL http:ps://opeeeeeeeeeeeeeeeeeeee id=Uj7pF-D-Yvt
Gabriel Percyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions, 2017. URL https://openreviikt net/forum?id HkCjNN5ex.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Tian Qin, Zhiwei Deng, and David Alvarez-Melis. A label is worth thousand images in dataset distillation.
In The eiirlh- Annual Conference on Neural Informalion Processing Systems, 2024. URL https: //opnneevie. www. mett/Eorrumm1ddoooMMR001
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. Tn
International Conference on Learning Representations, 2018. URL http:::://oppneeeeeeweee////rroe H1aIuk-RW.
Burr Scttles. Active Learning. Springer International Publishing, 2012. ISBN 9783031015601. doi: 10.1007/ 978-3-031-01560--1 URL http://dx. doi arrg/10.1007/978-3333
Yanyao Shen, Hyokun Yun, Zachary C. Lipton, Yakov Kronrod, and Animashree Anandkumar. Deep active learning for named entity recognition, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR), 2015. URL /ttps://arxiv.org/aaa 1409.1556.
Pong Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In Proceedings of the (EEE/CVP Conference on Compuler Vision and Pattern Recognition (CVPR), 2024.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In 2016 IEFE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2818 2826, 2016. doi: 10.1109/CVPP P01...006...
Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F. Manso. The computational limits of deep learning, 2022. URL https://arxiv. .
055588.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Ge- offrey J. Gordon. An empirical study of example forgetting during deep neural network learning. International Conference on Learning Representations, 2019. URL /ttpp:://opnrreviex net/forumm? BJ1xm30cKm.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations, 2022. URL https://openneever.
Kai Wei, Rishabh Tyer, and Jelf Bilmes. Data subset selection via graph cuts. In Proceedings of the 32nd International Conference on Machine Learning (ICML), pp. 1077-1085. PMLR, 2015.
Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-cfficient deep learning. In The Eleventh International Conference Learning Representations, 2023. URL
Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Datasct condensation at imagenct scale from new perspective. In Thirty-seventhh Conference Neural Information Processing Systems, 2023. URL https://openrevieer http://wwwww.eeennneeieeeeeeeeee wwt.://w......oer.reee//gogaaaaa
Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high pruning rates. In The Eleventh International Conference on Learning Representations, 2023. URL https:// openreview
Qing Zhou, Junyu Gao, and Qi Wang. Scale efficient training for large datasets, In CVPR, 2025.
Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning. In International Confer- ence on Learaing Representations, 2017. URL http:////e.eeeeeeee...ee.e//ee/e
16

---

Published in Transactions on Machine Learning Research (03/2026)
A Appendix
In this appendix, we provide some additional insights and results to aid the analysis provided in the main paper.
A.1 Results for VGG16 architecture CIFAR-100 dataset
Table 7: Results for CTFAR-100 dataset. VGG16 architecture used. While soft labels are able improve test set accuracy on all the methods, random selection with soft labels performs on par with other coresct selection methods.
Methods
0.5%
1%
5%
10%
20%
GradMatch (HL)
1.87 + 0.28
4.32
1.29
18.52
0.43
31.90
0.38
45.8 + 0.32
GradMatch (SL)
6.12 + 0.11
12.15 + 0.04
41.73 + 0.46
49.80 +: 0.37
55.12 +: 0.01
Craig (HL)
1.94 + 0.33
3.39 = 0.26
18.91 = 0.93
30.37 + 0.09
50.2 = 0.25
Craig (SL)
6.08 + 0.10
12.89 = 0.08
46.14 0.58
52.16 0.01
56.76 0.01
GraphCut (HL)
2.42 + 0.21
7.13 + 0.03
27.49 + 1.07
40.10 + U.26
32.72 + 0.15
GraphCut (SL)
5.75 + 0.33
14.91 + 1.08
45.72 = 0.33
51.75 + U.17
55.77 + 0.05
GraNd (HL)
1.73 1 0.29
1.94
8.75 1:
18.68 1.37
39.07 0.27
GraNd (SL)
5.54 1 0.51
12.24 1 0.09
40.12 1 0.47
49.49 1 0.37
54.92 1 0.02
Moderate (HL)
2.00 + 0.10
2.10 + 0.23
13.70 + 0.91
28.10 + 1.60
49.90 + 0.30
Moderate (SL.
5.15 + 0.41
8.02
41.64 0.18
52.33 0.42
56.58 0.45
random (HL)
2.73 + 0.17
4.75 + 0.2
20.18 + 0.18
24.54 + 1.99
40.59 + 1.05
random (SL)
5.88 1: 0.02
12.64 1: 0.49
46.45 + 0.08
52.94 + 0.19
56.98 + 0.16
A.2 Comparison between soft labels generated through label smoothing and teacher network
Table 8 compares results obtained by soft labels generated from label smoothing and through teacher network the Tiny ImageNet dataset with the ResNet-18 architecture. Soft labels generated through the teacher network perform much better compared to soft labels generated through label smoothing, which is expected, as teacher aetwork-generatee soft labels encode more features than randomly assigned probabilities through label smoothing.
Table 8: Comparative analysis of performance for soft labels generated through label smoothing (LS) and teacher network (TN). The results are obtained for training the RosNet-18 network on the Tiny ImageNet dataset with the Noise Free Gradient coreset selection method.
Selection %
LS
TN
0.5
6.84 + 0.02
13.37 + 0.14
1.0
12.60 + 0.14
21.28 + 0.08
5.0
26.16 + 0.01
37.55 + 0.17
10.0
31.80 + 0.20
45.55 + 0.23
20.0
38.50 + 0.15
51.61 - 0.21
30.0
41.93 = 0.11
53.89 + 0.34
40.0
45.49 + 0.23
55.32 + 0.21
A.3 Impact of e in label smoothing
The hyperparameter e controls the label smoothing (Eq. 2). Table 9 shows the impact of on test set accuracy for 10% and 20% coreset selection on the Tiny ImageNet datasct with the ResNet-18 architecture. Similar to the temperature hyperparameter for Seacher-generated soft label generation, test accuracy first increases with e, but then it decreases.
17

---

Published in Transactions on Machine Learning Research (03/2026)
Table 9: Impact of label smoothing parameter e on test set accuracy for 10% and 20% coreset selection. Results are obtained on the Tiny ImageNet dataset with the ResNet-18 architecture.
E 4. Selection Ratio +
10%
20%
0
31.3 = 0.2
37.6 0.2
0.05
31.5 + 0.1
37.8 = 0.3
0.1
31.4 + 0.1
37.8 = 0.3
0.15
31.8 + 0.2
38.5 + 0.1
0.2
31.4 + 0.1
38.3 = 0.3
0.3
31.5 + 0.1
37.8 = 0.4
A.4 Impact and
In this experiment, we study how various combinations of temperature parameter T and soft label loss weightage impact test set accuracy. Figure 10 shows the impact of various combinations of and X on test set accuracy. The combination of higher temperature (r) and soft label weight (A) yields optimal accuracy because it maximizes the entropy and inter-class similarity encoded in soft labels, providing richer supervision than hard labels. In low-data coreset regimes, this enhanced information per sample reduces overfitting and improves generalization. The combination flatter soft label distribution and higher emphasis on this during training results better performance.
KD Performance Grid: Temperature vs X
1-
30.9
32.5
32.8
34.0
34.1
47
30.1
38.7
39.4
(lemperature)
10
31
31.3
34.2
39.2
43.6
45.3
0.7
0.5
0.8
1.0
(KD loss weight)
Figure 10: Heatmap of accuracy values obtained on test dataset of Tiny ImageNet with ResNet-18 architec- ture by varying temperature 7 and soft label loss weight A. We can observe that a combination of higher temperature T and 100% soft labels provides the optimal result.
A.5 Impact of random selection from limited classes
Table 10 summarizes the drop in generalization performance when model is trained on a random subset of only 50% of the target classes, compared to the full set. As can be obscrvod, the manner of random selection does impact the generalization performance on the test set.
A.6 Compute requirement analysis
Table 11 provides comparative analysis of the memory requirements for training with hard vs. soft labels.
18

---

Published in Transactions on Machine Learning Research (03/2026)
Drop in generalization performance when a model is trained on a random subset of only 50% of the target classes, compared to the full set.
Table 10:
Selection Ratio
Random selection from all classes
Random sclection from 50% of classes
0.01
32.26
18.75
0.05
56.97
43.06
Table 11: Memory requirement comparison for training random selection with hard labels and soft labels. Training is carried out on ImageNet-1K dataset for 0.5% coresct selection, with ResNet-18 architecture. A batch size of 128 is used. The additional memory requirement resulted in increase in accuracy from 3.78% to 15. 19...%
Method
Memory requirement
Random (HL)
4.7 GB
Random (SL)
6.5 GB
A.7 Computation Experiment Information
Information regarding computing infrastructure for carrying out the experiments is provided in Table 12.
Table 12: Details of the computing infrastructure.
Description
Value
GPU
32 GB Tesla V100
OS
Ubuntu 18.04.1 LTS
Python
3.11.7
Pytorch 2.4.1
NumPy 1.24.3
19


Reviews:
[{"review_id": "nLkEeHaZAD", "reviewer": "Reviewer_R5Fn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the impact of using soft labels generated via knowledge distillation from a teacher network and label smoothing on the performance of coreset selection methods for image classification. While Dataset Distillation literature has frequently utilized soft labels, coreset selection research has predominantly relied on hard labels. The authors conduct extensive experiments across multiple datasets and architectures, applying soft labels to both random selection and various established coreset algorithms. The central finding is that soft labels significantly improve the generalization performance of all subset selection methods. More importantly, the authors demonstrate that a simple baseline of random selection combined with soft labels often performs on par with or better than sophisticated coreset selection methods trained with hard labels.\n\n**Strengths**\n\n- The authors evaluate their hypothesis across a wide range of benchmark datasets. The study benchmarks against over 10 coreset selection methods. Experiments are conducted across different model architectures.\n- The revelation that Random Selection + Soft Labels outperforms sophisticated methods like GradMatch or Craig with Hard Labels challenges the prevailing evaluation standards in the field.\n- The paper investigates why soft labels work, comparing Teacher-generated labels vs. Label Smoothing. It finds Teacher-generated labels superior, likely due to encoding inter-class similarities.\n\n**Weaknesses**\n\n- The paper argues that coreset selection aims to reduce computational requirements. However, the proposed method requires a pre-trained teacher network trained on the full dataset to generate labels. If the goal of coreset selection is to avoid training on the full dataset due to resource constraints, requiring a full-dataset-trained teacher creates a paradox. The teacher here is effectively a form of lossy compression of the full dataset, which will likely lead to unfair comparison when compared to works that are only trained on partial data. A fairer comparison for training from scratch scenarios would be Random (HL) vs. Coreset (HL).\n- Section 3.4 attempts an Information Theoretic analysis, stating that soft labels increase entropy and reduce KL divergence. However, this is a standard interpretation of KD and does not rigorously explain why the specific geometry of a random subset becomes sufficient to match the geometry of an optimized coreset when soft labels are applied. The contribution is primarily empirical benchmarking rather than a new theoretical understanding of coreset geometry under soft targets.", "claims_and_evidence_yn": "No", "claims_explanation": "The claims are generally well-supported by the extensive experimental tables. However, as mentioned in the weaknesses, there are some caveats regarding fair comparison.\n\n- The empirical claim that training on a random subset with teacher-generated soft labels yields higher accuracy than training on coresets with hard labels is well-supported by the data.\n- The claim that Random+SL is a data-efficient approach is not supported by the current evidence framing. The method requires training a teacher model on the full dataset first. If the computational cost of training the teacher is included, Random + SL is likely far more expensive than many coreset methods.\n- The implication that soft labels generally bridge the gap is weakly supported and likely confounded by information leakage. The authors show in Table 9 that Teacher-based Soft Labels drastically outperform Label Smoothing. This suggests the performance boost comes specifically from the Teacher's knowledge of the full dataset (an implicit information leak), not just the regularization effect of soft targets.", "audience_yn": "Yes", "audience_explanation": "Coreset selection and data-efficient deep learning are active areas of research. Researchers developing new coreset algorithms would need to know that they should benchmark against Random + SL, not just Random + HL.", "requested_changes": "The paper effectively compares \"Learning from scratch on a smart subset\" (Coreset HL) vs. \"Distilling a fully-trained model onto a random subset\" (Random SL). This distinction must be made explicit to support the claims of comparability.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3KnHs7s1ww", "reviewer": "Reviewer_LNKP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the authors empirically investigate the effect of soft labels on generalization accuracy in image classification across various coreset selection algorithms. Extensive experiments show that soft labels consistently improve the performance of all methods. Moreover, random selection with soft labels performs comparably to or better than existing coreset selection approaches, suggesting that future coreset algorithms should use random selection with soft labels as a key baseline.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main findings of the submission are convincing, supported by comprehensive experiments that evaluate various coreset selection methods across multiple datasets and network backbones. A series of ablation studies on key hyperparameters further strengthens the work. It would be greatly appreciated if the authors could make their benchmark code publicly available to inspire future research in the community.", "audience_yn": "Yes", "audience_explanation": "Demonstrating the surprising effectiveness of soft labels in the subfield of coreset selection is sound, and it could be beneficial for researchers working in the field.", "requested_changes": "1. Presentation: The authors mention that this evaluation draws inspiration from Data Distillation (DD). The introduction/discussion on page 2 would benefit from expansion. For example, explicitly highlighting the unique advantages of coreset methods compared to DD and explaining why a dedicated evaluation for them is valuable and necessary.\n\n2. Plots: The figures could be enhanced for better readability. Specifically, the font sizes of the tick labels, axis labels, titles, and legends in some plots (e.g., Fig 2) are kind of small. It is also recommended that the authors ensure greater consistency across all figures to achieve a more polished presentation.\n\n3. In terms of the *cross-architecture performance study*, what was the reason behind choosing 0.5% and 1% specifically as the selection percentages? In addition, could the authors please explain why all methods perform poorly on ViT when the selection rate is set to 1%?\n\n4. It would be greatly appreciated if the authors could make their benchmark code publicly available to inspire future research in the community.", "broader_impact_concerns": "N/A", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}, {"review_id": "QPdgwafZL9", "reviewer": "Reviewer_kcXS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper compares coreset selection methods with soft labeling and finds that the latter has a more significant impact on generation for image classification tasks. In particular, soft labeling with random selection already outperforms state-of-the-art coreset methods. The authors therefore recommend that future coreset selection work include soft labeling as a stronger baseline.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claim that soft labeling has a larger impact than coreset selection is supported across a wide range of coreset algorithms.", "audience_yn": "Yes", "audience_explanation": "The comparison across techniques may be of interest to practitioners.", "requested_changes": "Beyond the empirical comparison of the two techniques, could you analyze their interplay and the underlying principles that explain when and why they work? Some theoretical understanding would be helpful for drawing deeper insights.", "broader_impact_concerns": "N/A", "additional_comments": "The finding is interesting, but the methodological novelty appears somewhat limited.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)