Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
Authors: Yeseul Cho
Baekrok Shin
Changmin Kang
Chulhee Yun
Venue: ICML 2025 poster (2025)
Keywords: Dataset Pruning
Coreset Selection
Example Difficulty
Prediction Uncertainty
Primary area: general_machine_learning->supervised_learning
TL;DR: For dataset pruning, we introduce the DUAL (Difficulty and Uncertainty-Aware Lightweight) score, a new method that identifies important training examples early in the training process by considering both example difficulty and prediction uncertainty.

Abstract: Recent advances in deep learning rely heavily on massive datasets, leading to substantial storage and training costs. Dataset pruning aims to alleviate this demand by discarding redundant examples. However, many existing methods require training a model with a full dataset over a large number of epochs before being able to prune the dataset, which ironically makes the pruning process more expensive than just training the model on the entire dataset. To overcome this limitation, we introduce the **Difficulty and Uncertainty-Aware Lightweight (DUAL)** score, which aims to identify important samples from the early training stage by considering both example difficulty and prediction uncertainty. To address a catastrophic accuracy drop at an extreme pruning ratio, we further propose a pruning ratio-adaptive sampling using Beta distribution.
Experiments on various datasets and learning scenarios such as image classification with label noise and image corruption, and model architecture generalization demonstrate the superiority of our method over previous state-of-the-art (SOTA) approaches. Specifically, on ImageNet-1k, our method reduces the time cost for pruning to 66\% compared to previous methods while achieving a SOTA 60\% test accuracy at a 90\% pruning ratio. On CIFAR datasets, the time cost is reduced to just 15\% while maintaining SOTA performance.

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
Yeseul Cho
Baekrok Shin 1
Changmin Kang' Chulhee Yun
Abstract
revealed power-law relationship between the generaliza- tion capacity of deep neural networks and the size of their training data (Hestness et al., 2017; Rosenfeld et al., 2019; Gordon et al., 2021), meaning that the improvement of model performance becomes increasingly cost-inefficient as we scale up the dataset size.
Recent advances in deep learning rely heavily on massive datasets, leading to substantial stor- age and training costs. Dataset pruning aims to alleviate this demand by discarding redundant ex- amples. However, many existing methods require training model with a full dataset over large number of epochs before being able to prune the dataset, which ironically makes the pruning pro- cess more expensive than just training the model on the entire dataset. To overcome this limitation, we introduce the Difficulty and Uncertainty- Aware Lightweight (DUAL) score, which aims to identify important samples from the early train- ing stage by considering both example difficulty and prediction uncertainty. To address catas- trophic accuracy drop at an extreme pruning ra- tio, we further propose a pruning ratio-adaptive sampling using Beta distribution. Experiments on various datasets and learning scenarios such image classification with label noise and im- age corruption, and model architecture general- ization demonstrate the superiority of our method over previous state-of-the-art (SOTA) approaches. Specifically, on ImageNet- Ik, our method reduces the time cost for pruning to 66% compared to previous methods while achieving SOTA 60% test accuracy at 90% pruning ratio. On CIFAR datasets, the time cost reduced to just 15% while maintaining SOTA performance. Implementation is available at github/ dual-pruning.
Fortunately, Sorscher et al. (2022) demonstrate that the power-law scaling of error can be reduced to exponential scaling with Pareto optimal data pruning. The main goal of dataset pruning is to identify and retain the most informative samples while discarding redundant data points for training neural networks. This approach can alleviate storage and computational costs as well training efficiency.
However, many existing pruning methods require training model with full dataset over number of epochs to measure the importance of each sample, which ironically makes the pruning process more expensive than just training the model once on the original large dataset. For instance, several score-based methods (Toneva et al., 2018; Pleiss et al., 2020; Paul et al., 2021; He al., 2024; Zhang et al.. 2024) require training as they utilize the dynamics from the whole training process. Some geometry-based methods (Xia et al., 2022; Yang et al., 2024) leverage features from the penultimate layer of the trained model, therefore training a model is also required. Hybrid methods (Zheng et al., 2022; Maharana et al., 2023; Tan et al., 2025), which address the difficulty and diversity of samples simultaneously, still hold the same limitation they use existing score metrics. Having to compute the dot product of learned features to get the neighborhood information makes them become even more expensive to utilize.
In order to address this issue, we introduce the Difli- culty and Uncertainty-Awan Lightweight (DUAL) score, which can measure the importance of samples in the early stage of the training process by considering example diffi- culty and the prediction uncertainty. Additionally, for the high pruming ratio-when the selected subset is scarce we propose araning-ratio--daatiie Beta sampling, to inten- tionally include easier samples which have lower scores to achieve a better representation of the data distribu- tion (Sorscher et al., 2022; Zheng et al., 2022; Acharya et al., 2024).
1. Introduction
Advancements in deep learning have been significantly driven by large-scale datasets. However, recent studies have
"Equal contribution 'Kim Jaechul Graduate School of AI, KAIST, Scoul, South Korca, Correspondence to: Chulhee Yun <chulhee, yun @ krz>
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
96
80
Tota Score Comp tittation Time (Min)
D2
CCS
40
94
75
Dyn-Unc
Test Accuracy (%)
92
70
90
D2
D2
25
CCS
CCS
TDDS
Dyn-Unc
Dyn-Unc
EL2N
15
TDDS
55
TDDS
86
EL2N
EL2N
10
50
84
Ours
Ours
Ours
Random
Random
82
Random
0
30
40
50
60
70
80
90
30
40
50
60
70
80
90
Pruning Ratio (%)
Pruning Ratio (%)
Figure 1. Test accuracy comparison CIFAR datasets (Left: Results for CIFAR-10. 10, Right: Results for CIFAR- (00). The color represents the total computation time, including the time spent training the original dataset for score calculation, for each pruning method. Blue indicates lower computation time, while red indicates higher computation time. Our method demonstrates its ability minimize computation time while maintaining SOTA performance.
Experiments conducted on CIFAR and ImageNet datasets under various learning scenarios verify the superiority of our method. Specifically, on ImageNet-1k, our method re- duces the time cost to 66% compared to previous methods while achieving a SOTA performance, 60% test accuracy the pruning ratio of 90%. On the CIFAR datasets, il- lustrated in Figure 1, our method reduces the time cost to just 15% while maintaining SOTA performance. Especially, our proposed method shows notable performance when the dataset contains noise.
forgetting events, where correct prediction is flipped to wrong prediction during training process. Dyn- uen-Une (He et al., 2024), which strongly inspired our approach, prior- itizes the most uncertain samples rather than typical easy or hard samples during model training. The uncertainty is measured by the variation of predictions in sliding window, and the score averages the variation throughout the whole training process. TDDS (Zhang al., 2024) averages dif- ferences of Dullback-Leible divergence loss of non-target probabilities for T training epochs, where 7 highly de- pendent on the pruning ratio. The information from training dynamics proves useful for pruning because it allows one to differentiate hard but useful samples from noisy ones (He et al., 2024). However, despite its stability and effectiveness, previous methods fail to guarantee cost-effectiveness as they require at least one full training of the model on the entire dataset.
2. Related Works
Data pruning aims to remove redundant examples, keeping the most informative subset of dataset, namely the coreset. Research in this area can be broadly categorized into two groups: score-based and geometry-based methods. Score- based methods define metrics to measure the importance of data points, then prioritize high-scoring samples. On the other hand, geometry-based methods focus on presenting a better representation the true data distribution. Recent studies propose hybrid methods which incorporate the ex- ample difficulty score with the diversity of the coreset.
Geometry-based. Geometry- bbsse methods concentrate on providing a better representation by minimizing the re- dundancy of selected samples. SSP (Sorscher al., 2022) selects the samples most distant from k-means cluster cen- ters, while Moderate (Xia al., 2022) prefers samples near the median. However, these methods often compromise gen- eralization performance, since they underestimate the effect of difficult examples.
Score-based. EL2N (Paul et al., 2021) calculates L2 norms of the enror vector, as an approximation of the gradi- ent norm. Entropy (Coleman et al., 2020) quantifies the in- formation contained in the predicted probabilities at the end of training. However, the outcomes of such "snapshot" meth- ods need multiple runs to be stabilized, as shown in Figure 7, Appendix B. AUM (Pleiss et al., 2020) accumulates the gap between the target probability and the second-highed pre- diction probability. Methods that utilize training dynamics throughout the entire training offer more stable measures. Forgetting (Toneva al., 2018) score counts the number of
Recently, hybrid approaches have emerged that harmonize both difficulty and diversity. CCS (Zheng al., 2022) parti- tions difficulty scores into bins and selects an equal number of samples from each bin ensure balanced representa- tion. 2 (Maharana et al., 2023) employs message- passgeegaannggggg mechanism with a graph structure where nodes represent dif- ficulty scores and edges encode neighboring representations, facilitating effective sample selection. BOSS (Acharya et al., 2024) introduces Beta function for importance sampling
2

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
based on difficulty scores, which resembles our pruning ratio-adaptive sampling; we discuss the differences in Sec- tion 3.3. SIMS (Grosz et al., 2024) defines SIM score using class separability. data integrity, and model uncertainty, and then integrates sampling strategy. Our DUAL. pruning de- fines a score metric with difficulty and uncertainty, and it becomes a hybrid approach when the score is combined with our proposed Beta sampling.
Epoch 60
Epoch 90
1.0
Dyn Unc Spore Compo Commuu00 Rod 0t Spochh
0.4
0.2
0.0
0.0
01
0.2
03 0.4
05
6.0
63
6.
03
0.5
Standard Deviation
Standard Deviation
3. Proposed Methods
(a) Score calculated at epoch T = 60.
3.1. Preliminaries
1.0
Epoch 60
Epoch 90
Dyn Un Score Comp Rod at Epoch 90
Let D= == (() (I:: }n)) be labeled dataset of training samples, where E X C Rd and y y := are the data point and the label, respectively. C' is positive integer and indicates the number of classes. For each labeled data point (æ, y) E 'D denote Pvy(y x) as the prediction probability of y given æ, for the model trained with & epochs. Let S C D be the subset retained after pruning. Pruning ratio is the ratio of the size of D ! to D, or r = D
0.81
Mean
0.6
0.2
0.0
62 Slandard Deviation
Standard Deviation
(b) Score calculated epoch T = 90.
Figure 2. The left column ('Epoch 60") shows the prediction mean and standard deviation, computed using the predicted target prob- abilities up to epoch 60. The right column ("Epoch 90") shows corresponding values up to epoch 90. In each row, samples are colored by the normalized Dyn-Unc score computed at epoch 60 for Figure 2a and at epoch 90 for Figure 2b. The epoch at which the score was computed is indicated by bold outline for each row
The Dynamic Uncertainty (Dyn-Une) score (He et al., 2024) prefers the most uncertain samples rather than easy-to-learn or hard-to-learn samples during model training. The uncer- tainty score is defined as the average of prediction variance throughout training. They first define the uncertainty in a sliding window of length J:
K+ts( (y x)- - P
Un(e, 1
==
easy samples first, then treat harder samples later (Bengio et al., 2009; Arpit et al., 2017; Jiang et al., 2020; Shen et al., 2022). In other words, we see that the uncertainty of easy samples rises first, and then more difficult samples start to move and show increased uncertainty score.
where Pkx == Ejzo Paas(Ul)))
is the average prediction of
the model over the window [k, k + J - 1]. Then taking the average of the uncertainty throughout the whole training process leads to Dyn-Unc score:
Figure 2 further gives justification for this intuition. In Fig- ure 2a, samples with the highest Dyn-Une scores calculated at epoch 60 move upward by the end of training at epoch 90, This means that we measure Dyn-Unc score at the early stage of training, gives the highest scores to relatively easy samples rather than the most informative samples. It seems undesirable that it results in poor test accuracy on its coreset, as shown in Figure 8 in Appendix B.
1 (n( U(x,y) y)
T 1+1
3.2. Difficulty & Uncertainty-Aware Lightweight Score
Following the approach of Swayamdipta et al. (2020) and He et al. (2024), we analyze data points from ImageNet- 1k based on the mean and standard deviation of predictions during training, as shown in Figure 2. We observe data points typically flow along the moon from bottom to top. Data points starting from the bottom left region with low prediction mean and low standard deviation move to the middle region with increased mean and standard deviation, and those starting at the middle region drift toward to the upper left region with high prediction mean and smaller standard deviation. This phenomenon is closely aligned with existing observations that neural networks typically learn
To capture the most useful samples that are likely to con- tribute significantly to Dyn-Une during the whole training process (of 90 epochs) at the earlier training stage (e.g. epoch of 60), we need to target the samples located near the bottom-right region of the moon-shaped distribution, as Figure 2b illustrates. Inspired by this observation, we de- sign scoring metric that identifies such samples by taking both the uncertainty of the predictions and the prediction probability into consideration.
Here, we propose the Difficulty and Uncertainty-Aware
3

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
Lightweight (DUAL) score, measure that unites example difficulty and prediction uncertainty. We define the DUAL score of a data point (æt, y) at & E [T- J + 1] as
Technical details about Theorem 3.1 are provided in Ap- pendix D, together with an empirical verification of the time-efficiency of DUAL pruning over Dyn-Unc.
Empirically, as shown in Figure 3, the DUAL score targets data points in the bottom-right region during the early train- ing phase, which eventually evolve to the migdie-righttost part by the end of training. This verifies that DUAL pruning identifies the most uncertain region faster than Dyn-Unt both in theory and practice. The differences arise from the additional consideration of example difficulty in our method. We believe this adjustment leads to improved generaliza- tion performance compared to Dyn-Unc, as verified through various experiments in later sections.
DUAL (x. y)
|R PK+s(U m) Px]
1 1 Pk) (a)
(3)
J
(6)
where P& : Eyzd (vla²) is the average prediction of the model over the window |k; k + J - 1]. Note that AU is the product of two terms: (a) 1 - Px. quantifies the example difficulty averaged over the window; (b) is the standard de- viation of the prediction probability over the same window,
Epoch 60
Epoch 90
estimating the prediction uncertainty.
2.0
DUAL Score Con ou at Epach 60
Finally, the DUAL score of (æ, y) is defined as the mean of DUAL2 scores over all windows:
0.8
0.6
DUAL( v) = Ek=1 J+ DUAL
0.44
(4)
T-J+1
02:
The DUAL score reflects training dynamies by leveraging prediction probability across several epochs, providing reliable estimation to identify the most uncertain examples.
0.0 0.1
0.
03
0.4
0.5
0.0
63
67
0.3
04
0.5
Standard Deviation
Standard Deviation
Figure 3. Our DUAL score targets similar uncertain samples in the carly epoch of 60 (highlighted in bold). Selected samples are finally located the most uncertain region when the whole training processes are considered.
A theoretical analysis of a toy example further verifies the intuition above. Consider a linearly separable binary clas- sification task { E R", y/ E (+1))) where N with || || < (x), 02) < ||02||| Without loss of generality, we set 3/1 3/2 +1. A linear classifier, f(æ; w) w x, is employed as the model in our analysis. The parameter w is initialized at zero and updated by gradient descent. Soudry et al. (2018) prove that the parameter of linear classifiers diverges to infinity, but directionally converges to the L2 maximum margin separator. This separator is determined by the support vectors closest the decision boundary. If valid pruning method encounters this task, then it should retain the point closer to the decision boundary, which is Ii in our case, and prune Zz. Due to its large norm, 22 exhibits higher score values in the early training stage, for both Dyn-Une and DUAL scores. It takes some time for the model to make prediction on Φ; with large confidence to increase its uncertainty level as well as prediction mean, and the scores for æ eventually become larger than those for 22 as training proceeds. In Theorem 1. we show through rigorous analysis that the moment of such a flip in order happens strictly earlier for DUAL than for uncertainty.
However, score-based approaches including our method, suffer from accuracy drop at the high pruning ratio due to biased representations. To address this, we propose an additional sampling strategy that adaptively selects samples regarding the coreset size.
3.3. Pruning Ratio-Adaptive Sampling
Since the distribution of difficulty scores is dense in high- score samples, selecting only the highest-score samples may result in biased model (Zhou et al., 2023; Maharana et al., 2023; Choi et al., 2024). To address this, we design sampling method to determine the subset S C D, rather than simply pruning the samples with the lowest scores. We introduce a Beta distribution that varies with the pruning ratio. The primary objective of this method is to ensure that the selected subsets gradually include more easy samples into the coreset as the pruning ratio increases.
(1+e) (1 (1++-7)-- Let st be the standard deviation and [ be the mean of af(m w,)) within a window from time t. to 1 + -
Theorem 3.1 (Informal). Define a(2) ==
(5)
However, the concepts of "easy" and "hard" cannot be dis- tinguished solely based on uncertainty or DUAL score. To address this, we use the prediction mean again for sampling, We utilize the Beta probability density function (PDF) to define the selection probability of each sample. First, we assign each data point a corresponding PDF value based
Denote Tn, and Tom as the first time when [63 > s(2) and
(1 > (1) -H() occurs, respectively. If the learning rate is small enough, then Tem < T

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
Table 1. Comparison of test accuracy hetween the DUAL score method and existing coresct selection techniques using ResNet-18 on CIFAR-10 and CIFAR-100 100 datasets. Training the model on the full dataset achieves an average test accuracy of 95.30% on CIFAR-I0 10 and 78.91% on CIFAR- 100. The best result in each pruning ratio is highlighted in bold.
Dataset (-+)
CIFARIO
CIFAR100
Pruning Rate (-+)
30%
50%
70%
%
90%
30 %
50% %
70%
80%
90%
Random
2.
93.20
90. 10
88.2 88.2810.17
23
75. 15.15
11.68:00.31
64.8610.39
59.23:20.62
45.09=1.26
Entropy
93.4810008
92.4710.17
88.53 19.5441.18
82.57.10.6
75.20
70.9010.35
61.7010.47
56.2410.51
42.25
Forgetting
95,4810.14
94.9410.21
89,55
15.47:1122
46.64.11.00
77.52.10.2 26
10.93
49.6610.20
39.09
26.87
EL2N
95.44=0.06
95.19+0.11
91.62+0.14
74.70=0.45
38.7440.75
77.13=0.23
68.98+0.35
34.59+0.48
19.52=0079
8.89 =0.28
AUM
90.62=0.09
87.26+0.11
81.28+0.26
76.58+0..5
67.88++..5
74.34=00.44
69.57+0021
61.12+0.20
55.80=0.35
45.00=0037
Moderate
94,26=0009
92.79+0.0z
90.45+0.21
88.90+0.17
85.52++0.20
75.20=a.25
70.90+0.35
61.70+0.47
56.24=0.51
42.25=0.39
Dyn-Unc
95,491.022
95.3510.12
91.7810.65
83.3210.94
59.67.1.79
77.67
1..23
64.3010.13
55.01
34.57
TDDS
94,42..011
93.1110.14
91.02.10.19
88.2510.24
82.4910.28
75.02.00.7
7188010.33
64.61
59.88
10.93
CCS
95.31.10.2
95.0610.15
92.6810.11
91.25110.1
85.92.0.39
77.15
73.8310.21
68.6510.31
64.06.10.21
54.23.0048
D2
24.1320.20
93.2622.16 16
92.34.0.16 18
90.3822.34
86.
76.47=0.29
73.8840.28
62.9940.28
61.48:20.34
50. 1420.90
DUAL
95.25 94.95+0.22
91.75
82.02
54.95+0.42
77.43+0.08
74.6240.47
66.4140.52
56.57
34.38=1.89
D sampling 95.51
95.233:0.0
93.04.0043
91.4210.35
87.09
77.8620.12
74.6610.12
69.2510.22
64.76100.3 54.54:0000
on its prediction mean and weight this probability using the DUAL score. The weighted probability with the DUAL score is then normalized to the range [0, 1J. then used as the sampling probability. We clarify that sampling probabil- ity is for selecting samples, not for pruning. Therefore, for each pruning ratio r, we randomly select (1 r) n sam- ples without replacement, where sampling probabilities are given according to the prediction mean and DUAL score as described. The detailed algorithm for our proposed pruning method is provided in Algorithm I, Appendix C.
et al. (2022); if the initial dataset is small, the coreset is more effective when it contains easier samples, while for relatively large initial dataset, including harder samples can improve generalization performance. More descriptions for our Beta sampling are provided in Appendix C.
Remark. BOSS (Acharya et al., 2024) also uses the Beta distribution to sample easier data points during pruning, similar to our approach. However, a key distinction lies in how we define the Beta distribution's 's parameters, a, and 3,.. While BOSS adjusts these parameters to make the mode of the Beta distribution's PDF scale linearly with the pruning ratio r, we employ non-linear combination. This non- linear approach has the crucial advantage of maintaining an almost stationary PDF at low pruning ratios. This stability is especially beneficial when the dataset becomes casier where there is no need to focus on easy examples. Furthermore, unlike previous methods, we define PDF values based on the prediction mean, rather than any difficulty score, which is another significant difference.
We design the Beta PDF function to assign a sampling prob- ability concerning& a prediction mean as follows:
Br = C (1 - 1D) (1 )P)
(5)
ar = -r. By,
where C > 0 is a fixed constant, and the μD stands for the prediction mean of the highest score sample. Recalling that the mean of Beta distribution is or a. the above choice makes the mean of Beta distribution moves progressively with r, starting from pD (r ~ 0, small pruning ratio) to one. In other words, with growing r, this Beta distribution becomes skewed towards the easier region (r + 1, large pruning ratio), which in turn gives more weight to easy samples.
Remark. SIMS (Grosz et al., 2024) also proposes a ratio- adaptive sampling strategy, applying importance weights over the original score distribution. However, it assumes normal distribution of scores, which does not hold in practice (see Figure 2 of (Grosz et al., 2024)). In contrast, our sampling method, by not relying on any specific score distribution, remains robust across diverse datasets.
The tendency of evolving should be different with datasets, thus a hyperparameter cp 2 is used to control the rate of evolution of the Beta distribution. Specifically. the choice of cD depends on the complexity of the initial dataset. For smaller and more complex datasets, setting Cp smaller value retains more easy samples. For larger and simpler datasets, setting cp to a larger value allows more uncertain samples to be selected. (For your intuitive understanding, please refer to Figure 20 and Figure 21 in the Appendix C.) This is also aligned with the previous findings from Sorscher
4. Experiments
4.1. Experimental Settings
We assessed the performance of our proposed method in three key scenarios: image classification, image classifica- tion with noisy labels and corrupted images. In addition, we validate cross- arosstarchieectuee generalization on three-layer CNN, VGG-16 (Simonyan & Zisserman, 2015), ResNet-18 and ResNet-50 (He et al., 2015).
5

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
Hyperparameters For training CIFAR-10 and CIFAR- 100, we train ResNet-18 for 200 epochs with batch size of I 128. SGD optimizer with momentum of 0.9 and weight de- cay of 0.0005 is used. The learning rate is initialized as 0.1 and decays with the cosine annealing scheduler. As Zhang et al. (2024) show that smaller batch size boosts perfor- mance at high pruning rates, we also halved the batch size for 80% pruning, and for 90% we reduced it to one-fourth. For ImageNet-Ik,, ResNet-34 is trained for 90 epochs with a batch size of 256 across all pruning ratios, An SGD opti- mizer with a momentum of 0.9, weight decay of 0.0001, and an initial learning rate of 1.1 is used, combined with a cosine annealing scheduler.
Table 2. Comparison of test accuracy of DUAL score with existing coreset selection methods using ResNet34 for ImageNet- IK. The model trained with the full dataset achieves 73. % test accuracy. The best result in each pruning raio is highlighted in bold.
Pruning Rate (-)
30%
50%
70 %
80%
90% %
Random
72.2
70.3
66.7
62.5
52.3
Entropy
72.3
70.8
64.0
55.8
39.0
Forgetting
72.6
70.9
66.5
62.9
52.3
EL2N
72.2
67.2
48.8
31.2
12.9
AUM
72.5
66.6
40.4
21.1
9.9
Moderate
72.0
70.3
65.9
61.3
52.1
Dyn-Unc
70.9
68.3
63.5
59.1
49.0
TDDS
70.5
66.8
59.4
54.4
46.0
CCS
72.3
70.5
67.8
64.5
57.3
Baselines The baselines considered in this study are listed
D2
72.9
71.8
68.1
65.9
55.6
as follows:' (1) Random; (2) Entropy (Coleman et al.,
DUAL
72.8
71.5
68.6
64.7
53.1
2020); (3) Forgetting (Toneva et al., 2018); (4) EL2N (Paul et al., 2021); (5) AUM (Pleiss et al., 2020); (6) Moder- ate (Xia et al., 2022); (7) Dyn-Unc (He et al., 2024); (8) TDDS (Zhang et al., 2024); (9) CCS (Zheng et al., 2022); and (10) 2 (Maharana et al., 2023). To ensure a fair com- parison, all methods were trained using a common set of base hyperparameters (c.g.. learning rate, batch size, num- ber of epochs), while any seetho--speific hyperparameters were set to the optimal values reported for each score met- ric in their respective original works. Technical details are provided in the Appendix A.
DUAL sampling
73.3
72.3
69.4
66.5
60.0
We also evaluate our pruning method on the large-scale dataset, ImageNet- Ik. The DUAL score is computed during training, specifically at epoch 60, which is 33% earlier than the original train epoch used to compute scores for other baseline methods. As shown in Table 2, Dyn-Une performs worse than random pruning across all pruning ratios, and we attribute this undesirable performance to its limited total training epochs (only 90), which is insufficient for Dyn-Umt Unc to fully capture the training dynamics of each sample. In contrast, our DUAL score, combined with Beta sampling, outperforms all competitors while requiring the least com- putational cost. By considering both training dynamics and the difficulty of examples, DUAL can effectively identify uncertain samples early in the training process, even with limited training dynamics than full training. Remarkably, for 90% pruning on Imagenet- K, it maintains test accuracy of 60.0%, surpassing the previous state-of-the-at (SOTA) by a large margin.
4.2. Image Classification Benchmarks
Table 1 presents the test accuracy for image classification results on CIFAR-10 and CIFAR-100. Our pruning method consistently outperforms other baselines, particularly when combined with Beta sampling. While the DUAL score ex- hibits competitive performance in lower pruning ratios, its coreset accuracy degrades with more aggressive pruning. Our Beta sampling effectively mitigates this performance drop here.
Notably, the DUAL score requires training a single model for only 30 epochs for computation, significantly reducing the computational cost. In contrast, the second-best meth- ods, Dyn-Unc and CCS, rely on scores computed over 200 epochs-a full training cycle on the original dataset-which makes them significantly less efficient. Even when account- ing for subset selection, score computation, and subset train- ing, the total time remains lower than training the full dataset once, as illustrated in Figure 4. Specifically, on CIFAR- 10, our method achieves lossless pruning up to a 50% pruning ratio while saving 35.5% of total training time.
4.3. Experiments under More Realistic Scenarios
4.3.1. LABEL NOISE AND IMAGE CORRUPTION
Data affected by label noise or image corruption are difficult and unnecessary samples that hinder model learning and degrade generalization performance. Therefore, filtering out these samples through data pruning is crucial. Most data pruning methods, however, either focus solely on selecting difficult samples based on example difficulty (Paul et al., 2021; Pleiss et al., 2020; Coleman et al., 2020) or prioritize dataset diversity (Zheng et al., 2022; Xia et al., 2022), mak- ing them unsuitable for effectively pruning such noisy and corrupted samples.
IIfomax (Tan et al., 2025) was excluded as it employs dif- ferent base hyperparameters in the original paper compared to other baselines and does provide publicly available code. See Appendix A.1 for more discussion.
In contrast, methods that select uncertain samples while
6

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
70
to the optimal. Notably, when the pruning ratio is 10%, nearly all pruned samples are mislabeled data. Conse- quently, as observed in Figure 5b, DUAL pruning leads to improved test accuracy compared to training on the full dataset, even up to a pruning ratio of 70%. At lower pruning ratios, performance improves as mislabeled data are effec- tively removed, highlighting the advantage of our approach in handling label noise. Similarly, for image corruption, our method prunes more corrupted data across all corruption rates compared to other methods, as shown in Figure 15, 16 in Appendix B.2. As a result, this leads to higher test accu- racy, as demonstrated in Figure Sc.
Total Subset Training Time (Min)
Random
60
D2
EL2N
Dyn-Unc
DUAL
40
Full Training Time
15.8%
30
-35.6%
X 55.6%
45.5%
20
.75.8%
10
30
40
50
60
70
80
90
Pruning Ratio (%)
Detailed results, including exact numerical values for differ- ent corruption rates and Tiny- ImageNet experiments, can be found in Appendix B.1 and B.2. Across all experiments, DUAL pruning consistently shows strong noise robustness and outperforms other methods by a substantial margin.
Figure 4. Comparison in total time spent (full dataset training, score estimation, and subset training) on CIFAR datasets. While other methods remain ineffective as they require more than full training, our method achieves a 15.5% time reduction with only 30% pruning, approaching the efficiency of random pruning.
4.3.2. CROSS CROOS--RCCHTECTUREE GENERALIZATION
Next, we evaluate the ability to transfer scores across var- ious model architectures. Especially, we can get high quality example scores for pruning by using a simpler archi- tecture than one for the training, our DUAL pruning would become even more efficient in time and computational cost. Therefore, we focus the cross-architecture generalization from relatively small networks to larger ones with three- layer CNN, VGG-16, ResNet- 18, and ResNet-50, Results are summarized in Table 3.
considering training dynamics, such as Forgetting (Toneva et al., 2018) and Dyn-Unc (He et al., 2024), demonstrate robustness by pruning both the hardest and easiest samples, ultimately improving generalization performance, as illus- trated in Figure 5a. However, since noisy samples tend to be memorized after useful samples are learned (Arpit al., 2017; Jiang et al., 2020), there is possibility that those noisy samples may still be treated as uncertain in the later stages of training and thus be included in the selected subset.
Competitors are selected from each categorized group of the pruning approach: EL2N from difficulty-based, Dyn-Une from uncertainty- group. Standard deviations are omitted here due to space limit; please refer to Appendix B.3 for details.
The DUAL score aims to identify high- sigh-uncertaitty samples early in training by considering both training dynamics and example difficulty. Noisy data, typically under-learned com- pared other challenging samples during this phase, exhibit lower uncertainty (Figure 13, Appendix B) Consequently, our method effectively prunes these noisy samples.
and CCS from the geometry-
Table 3. Cross-architecture generalization performance on CIFAR- 100 from ResNet-18 to ResNet-50. '(R50)' marker denotes the score computed on ResNet-50, serving as a baseline. DUAL + B means our Beta sampling with DUAL scores. We report an average of five runs.
To verify this, we evaluate our method by introducing a spe- cific proportion of symmetric label noise (Patrini et al., 2017: Xia et al., 2020; Li et al., 2022) and applying five different types of image corruptions (Wang et al., 2018; Hendrycks & Dietterich, 2019; Xia et al., 2021). We use CIFAR-100 with and
ResNet-18
ResNet-50
with ResNet-34 for these experiments. On CIFAR- 100, we test label noise and im- age corruption ratios of 20%, 30%, and 40% using a model trained for 30 epochs. For Tiny-ImageNet, we use a 20% ra- tio of label noise and image corruption. We prune the label- noise-added dataset using model trained for 50 epochs dataset with model trained for
ResNet-18 18
Pruning Rate (-))
30%
50%
70%
90%
Random EL2N
77.17
73.74
66.66
40.48
79.46
74.85
58.75
16.19
Dyn-Une
79.90
75.78
61.75
25.08
CCS
77.24
73.81
66.66
40.31
DUAL
79.48
76.47
68.56
29.82
and the 30 epochs using DUAL pruning-bov significantly lower than the 200 epochs used by other methods. For detailed experimental settings, please refer to Appendix A.2.
DUAL + 3
79.53
75.08
67.54
50.34
DUAL (R50)
79.60
76.64
68.60
29.84
DUAL (R50) + 3
79.63
76.49
70.37
50.27
As shown in Figure 5, the left plot demonstrates that DUAL pruning effectively removes mislabeled data at ratio close
Specifically, when pruning 90% of the original dataset, we find that other methods all fail, showing worse test accura-
7

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
1.00
Optimal
Full Test A65
Ratio of Misiabele Data
70
Test Accuracy (
Pull Test 20st
Test Accuracy(S))
60
60
Random EL2N
Random
Random
40
50
50
Dyn-Unc
Dyn-Unc
Dyn-Unc
20
Forgetting DUAL
40
Forgetting
Forgetting
DUAL
DUAL
DUAL
DUAL
DUAL
10
20
50
60
70
60
90
30
10
20
30
60
70
80
90
20 30
40
50
60
70
00
90
Pruning (%)
Pruning
(%)
Pruning
(%)
Pruned mislabeled data ratio
(b) Test accuracy under label noise
(c) Test accuracy under image noise
Figure 5. The left figure shows the ratio of pruned mislabeled data under 20% label noise CIFAR-100 trained with ResNet- 18. When label noise is 20%, the optimal value (black dashed line) corresponds to pruning 100% of mislabeled data at 20% pruning ratio. The middle and right figures depict test accuracy under 20% label noise and 20% image corruption, respectively. Our method effectively prunes mislabeled data near the optimal value while maintaining strong generalization performance. Results are averaged over five random seeds.
cies than random pruning. However, our proposed pruning method consistently shows the powerful ability to generalize across the various network architectures, outperforming or being on par with other computationally expensive baselines. More results provided in Appendix B.3.
pling on existing score metrics. We apply our Beta sampling strategy to other score-based methods, including Forget- ting, EL2N, and Dyn-Unc, on the CIFAR1O and CIFAR100 datasets. Compared vanilla threshold pruning, which se- lects only the highest-scoring samples, we observe that pre- vious methods become competitive when Beta sampling is adjusted (see Table 4). For the case of random pruning combined with Beta sampling, we do not use any score but select samples only with Beta sampling.
4.4. Ablation Studies
Hyperparameter Analysis In this section, we investigate the robustness of our hyperparameters, T. J, and ep. We fix / across all experiments, it has minimal impact selection, indicating its robustness (Figure 9, Appendix B). In Figure 6, we assess the robustness of 7 by varying it from 20 to 200 on CIFAR-100. We find that while is highly robust in early epochs, increasing it eventually degrades generalization. This is expected, as larger T overemphasizes difficult samples due our difficulty-awwe pruning in earlier epochs (30 50) proves more effective and robust. For we vary it from 3 to 6 and observe consistent performance, indicating robustness to its choice as well.
Table 4. Comparison on CIFAR-10 and CIFAR-100 for 90% prun- ing rate. We report average accuracy with five runs. The best performance in bold in each column.
IIAA--0
Method
Thresholding
3-Sampling
selection. Thus,
Random
83.74 0.21
83.31 (-0.43) 0.14
EL2N
38.74 1: 0.75
87.00 (+48. 22) 1: 0.45
Forgetting
46.64 + 1.90
85.67 (+39.03) =0.13
Dyn-Une
59.67 +: 1.79
85.33 (+25.66) 1: 0.20
Ours
54.95 1: 0.42
87.09 (+32.14) 1: 0.36
CIFAR-100
Test Accuracy (96)
70
Method
Thresholding
3-Sampling
30%
Random
45.09 + 1.26
51.76 (+6.67) + 0.25
23-
50%
EL2N
8.89 = 0.28
53.97 (+45.08) + 0.63
80%
90%
Forgetting
26.87 + 0.73
52.40 (+25.53) = 0.43
20
30
50
100
200
Dyn- Dyn-Une
34.57 + 0.69
51.85 (+17.28) 1: 0.35
Ours
34.28 + 1.39
54.54 (+20.26) 0.09
Figure 6. Left: T varying while J 10 and ep 4. Right: cp varying while 7 = 30 and J = 10. Three runs are averaged.
Even with random pruning, our Beta sampling continues to perform well. Notably, EL2N, which performs poorly on its own, becomes significantly more effective when com- bined with our sampling method. Similar improvements are also seen with Forgetting and Dyn-Une scores. This is be-
Beta sampling with existing scores Next, we study the impact of our proposed ratooaaiiii Beta sam-
8

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
cause our proposed Beta sampling enhances the diversity of selected samples, especially when used with example difficulty-based methods. More results conducted at 80% are included in the Appendix B.6.
Impact Statement
This paper presents work on data pruning to advance ma- chine learning, with the potential for positive societal impact through improved efficiency.
References
Additional Analysis In addition to the main results pre- sented in this paper, we conducted various experiments to further explore the effectiveness of our method, which is located in Appendix B. For instance, to list a few, these investigations include: (i) calculating the Spearman rank correlation between individual DUAL scores and their aver- age DUAL score across five runs to assess score consistency; and (ii) analyzing coreset performance under time budget. The findings from these analyses are presented in Figure 7 and Figure& 8, respectively, within Appendix B.
Acharya, A., Yu, D., Yu, Q., and Liu, X. Balancing feature similarity and label variability for optimal size-aware one-shot subset selection. In Forty-first International Conference on Machine Learning, 2024.
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. A closer look at memorization in deep networks. In International conference on machine learning, pp. 233-242. PMLR, 2017.
Furthermore, detailed results on extreme cases of label noise (ranging from 20% to 40% for CIFAR- 100 and 20% for Tiny-ImageNet) are presented in Appendix B.1. Similar comprehensive results for various image corruptions can be found in Appendix B.2. The generalization performance of our method across other network architectures is further de- tailed in Appendix B.3. Additionally. results for long-tailed data classification using the CIFAR-10-LT and CIFAR-100 LT datasets are provided in Appendix B.4. Lastly, a com- parison with dynamic pruning methods such as Qin al. (2024) and Yuan et al. (2025) is provided in Appendix B.5.
Bengio, Y., Louradour, J., Collobert, R., and Weston, J. Cur- riculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Cao, K., Wei, C., Gaidon, A., Arechiga, N., and Ma, T. Learning imbalanced datasets with label-distribution- aware margin loss. Advances in neural information pro- cessing systems, 32, 2019.
Choi, H., Ki, N., and Chung, H. W. Bws: Best window selection based on sample scores for data pruning across broad ranges. arXiv preprint arXiv:2406.03057, 2024.
5. Conclusion
This paper introduces Difficulty and Uncertainty-Aware Lightweight (DUAL), a novel scoring metric for cost- effective pruning. DUAL is the first metric to combine difficulty and uncertainty into a single measure, and its effectiveness in identifying the most informative samples early in training is further supported by thcoretical analy- sis. Further, we propose pruning-ratio-adapiis sampling to consider the sample diversity when the pruning ratio is extremely high. Our proposed DUAL score, combined with Beta sampling, shows remarkable performance, partic- ularly under label noise and image corruption by effectively distinguishing noisy samples. Future work could explore extending this approach to unsupervised settings.
Coleman, C., Ych, C., Mussmann. S., Mirzasoleiman, B.,
Bailis, P., Liang, P., Leskovec, J., and Zaharia, M. Se- lection via proxy: Efficient data selection for deep learn- ing, 2020. URL htts:htt:://// //tp:///a/ivv///i org/ abs/ 1906. 11829.
Gordon, M. A., Duh, K., and Kaplan, J. Data and pa- rameter scaling laws for neural machine translation. In ACL Rolling Review May 2021, 2021. URL https: / //operreviiee ew, net/ forum? i d= IKA MML..LSS.
Grosz, S., Zhao, R., Ranjan, R., Wang, H., Aggarwal, M., Medioni, G., and Jain, A. Data pruning via separabil- ity, integrity, and model uncertainty-awer aware importance sampling. In International Conference on Pattern Recog- nition, pp. 398 413. Springer, 2024.
Acknowledgement
This work was supported by two Institute of Information & communications Technology Planning & Evaluation (IITP) grants (No. RS-2019-II 1110075,, Artificial Intelligence Grad- uate School Program (KAIST); No. RS-2024-00457882, National AI Research Lab Project) funded by the Korean government (MSIT), and National Research Foundation of funded
Gunasekar, S., Lee, J. D., Soudry, D., and Srebro, N. Implicit bias of gradient descent on linear convolutional networks. Advances neural information processing systems, 31, 2018.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition, 2015. URL https:/ // mexiv.org/bba. 11121 03385.
Korea (NRF) grant (No. by the Korean government (MSIT).
9

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
He, M., Yang. S., Huang, T., and Zhao, B. Large-scale dataset pruning with dynamic uncertainty. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7713-7722, 2024.
Simonyan, K. and Zisserman, A. Very deep convolutional
networks for large-scale image recognition, 2015. URL wtwpv/.rrg/abs//11333
https: /
Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Morcos, A. S. Beyond neural scaling laws: beating power law scaling via data pruning. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https: / / //ppereeiieww net/ www.mmmmiii....o............. wiw.imiiiinnnnnnPyemmm
Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness common corruptions and perturba- tions. arXiv preprint 1903. 2019.
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang. Y., and Zhou, Y. Deep learning scaling is predictable, empirically, arXiv preprint ar 1rXiv:171..004099 2017.
Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., and Srebro, N. The implicit bias of gradient descent on sep- arable data. Journal of Machine Learning Research, 19 (70):1-57, 2018.
Jiang, Z., Zhang, C., Talwar, K., and Mozer, M. C. Charac- terizing structural regularities of labeled data in overpa- rameterized models. arXiv preprint 2rXiv:2002.03206, 2020.
Swayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Ha- jishirzi, H., Smith, N. A., and Choi, Y. Dataset cartogra- phy: Mapping and diagnosing datasets with training dy- namics. In Proceedings of the 2020 Conference on Empir- ical Methods in Natural Language Processing (EMNLP), pp. 9205992799999 2020.
Li, S., Xia, X., Ge. S., and Liu, T. Selective- suserviie contrastive learning with noisy labels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 316-325, 2022.
Tan, H., Wu, S., Huang, W., Zhao, S., and QI. X. Data pruning by information maximization. In The Thirteenth International Conference on Learning Representations, 2025. URL https:: //opereve/ppenevieew net/ forum? id=93XT01K0ct
Maharana, A., Yadav, P., and Bansal, M. D2 pruning: Mes- sage passing for balancing diversity and difficulty in data pruning, arXiv preprint arXiv:2310.0793. 2023.
Patrini, G., Rozza, A., Krishna Menon, A., Nock, R., and Qu, L. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1944 1952, 2017.
Toneva, M., Sordoni, A., Combes, R. T. d., Trischler, A.. Bengio, Y., and Gordon, G. J. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv: /8/2.::8821000 05599995........ 2018.
Paul, M., Ganguli, S., and Dziugaite, G. K. Deep learning on a data diet: Finding important examples early in training. Advances in neural information processing systems, 34: 20596-20607, 2021.
Wang, Y., Liu, W., Ma, X., Bailey, J., Zha, H., Song, L.,, and Xia, S.-T. Iterative learning with open-set noisy labels. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8688-8696, 2018
Pleiss, G., Zhang, T., Elenberg, E. R., and Weinberger, K. Q. Identifying mislabeled data using the area un- der the margin ranking, 2020. URL https: arxiv. org/ abs/ / 2001.
Xia, X., Liu, T., Han, B., Gong. C., Wang, N., Ge, Z., and Chang, Y. Robust early-learning: Hindering the memo- rization of noisy labels. In International conference on learning representations, 2020.
Qin, Z., Wang, K., Zheng, Z., Gu, J., Peng, X., Zhou, D., Shang, L.. Sun, B., Xie, X.. You, Y., et al. Infobatch: Loss- less training speed up by unbiased dynamic data pruning. In The Twelfth International Conference on Learning Representations, 2024. URL https: httth:://ppeereviiiii //tp///eepee.....e///eep........ net/ Eorum wiw.mmiidd-lllll sklssks........
Xia, X., Liu, T., Han, B., Gong, M., Yu, J., Niu, G., and Sugiyama, M. Instance correction for learning with open- set noisy labels, arXiv preprint arXiv: 100:20000.0455 2021.
Xia, X., Liu, J.. Yu, J., Shen, X., Han, B., and Liu, T. Mod- erate coreset: A universal method of data selection for real-world data-efficient deep learning. In The Eleventh International Conference on Learning Representations, 2022.
Rosenfeld, S., Rosenfeld, A., Belinkov, Y., and Shavit, N. A constructive prediction of the generalization error across scales. arXiv preprint arXiv: 1909::990...000000 12673, 2019.
Yang, S., Cao, Z., Guo, S., Zhang, R., Luoo P., Zhang, S., and Nie, L. Mind the boundary: Coreset selection via reconstructing the decision boundary. In Forty-furs Inter- national Conference on Machine Learning, 2024.
Shen, R., Bubeck, S., and Gunasekar, S. Data augmentation as feature manipulation. In International conference on machine learning, pp. 19773-19809 PMLR, 2022.
10

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
Yuan, S., Lin, R., Feng, L.,, Han, B., and Liu, T. Instance-
dependent early stopping. International conference learning representations, 2025.
Zhang. X., Du, J., Li, Y., Xie, W., and Zhou, J. T. Spanning training progress: Temporal dual-depth scoring (tdds) for enhanced dataset pruning- In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern 2024.
Recognition, pp.
Zheng. H., Liu, R., Lai, and Prakash, A. Coverage-centric coreset selection for high pruning rates. arXiv preprinl arXiv:2210.15889, 2022.
Zhou, X., Pi, R., Zhang, W., Lin, Y., and Zhang, Prob- abilistic bilevel coreset selection, 2023. URL https: //arxiv. org/ abs/2301. 09880.

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
A. Technical Details
A.1. Details on Baseline Implementation
EL2N (Paul et al., 2021) is defined the error L2 norm between the true labels and predictions of model. Then examples with low scores are pruned out. We calculate error norm epoch 20 from five independent runs, then the average was used for EL2N score.
Forgetting (Toneva et al., 2018) is defined as the number of forgetting events, where the model prediction goes wrong after the correct prediction, up until the end of training. Rarely are unforgotten samples pruned out.
AUM (Pleiss et al., 2020) accumulates the margin, which means the gap between the target probabilities and the second largest prediction of model. They calculate the margin at every epoch and then transform it into an AUM score at the end of the training. Here samples with small margin are considered as mislabeled samples, thus data points with small AUM scores are eliminated.
Entropy (Coleman et al., 2020) is calculated as the entropy of prediction probabilities at the end of training, then the samples which have high entropy are selected into coreset.
Dyn-Une (He et al., 2024) is also calculated at the end of training, with the window length J set as 10. Samples with high uncertainties are selected into the subset after pruning.
TDDS (Zhang et al., 2024) adapts different hyperparameter for each pruning ratio. As they do not provide full information for implementation, we have no choice but set parameters for the rest case arbitrarily. The provided setting for (pruning ratio, computation epoch T, the length of sliding window K) is (0.3, 70, 10), (0.5, 90, 10), (0.7, 80, 10), (0.8, 30, 10), and (0.9, 10, 5) for CIFAR-100, and for ImageNet-(0.3,, 20, 10), (0.5, 20, 10), (0.7, 30, 20). Therefore, we set the parameter for CIFAR-10& as the same with CIFAR-100, and 80%, 90% pruning on ImageNet- 1, we set them as (30, 20), following the choice for 70% pruning.
CCS (Zheng et al., 2022) for stratified sampling method, we adapt AUM score as the original CCS paper does. They assign different hard cutoff rate for each pruning ratio, For 10, the cutoff rate (30%, 0), (50%, 0), (70%, 10%), (80%, 10%), (90%, 30%). For CIFAR 100 and ImageNet- IK,, we set them as the same with the original paper. As explicitly mentioned in Appendix B of Zheng et al. (2022), use the AUM score calculated at the end of training. This means scores are computed at epoch 200 for the CIFAR-10/100 datasets and at epoch 90 for the ImageNet- IK dataset.
D2 (Maharana et al., 2023) for ID² pruning, we set the initial node using forgetting scores for CIFAR-10 and CIFAR-100, we set the number of neighbors k, and message passing weight as the same with the original paper.
Remark. As detailed by Zhang et al. (2024) (Section 5.2), TDDS employs 90 epochs for initial full-dataset training. Subsequently, an exhaustive search is conducted to determine an optimal epoch for score computation (e.g., 30 epoch for pruning ImageNet by 70-90%). In our evaluations, we utilized these reported optimal epochs for TDDS across all pruning ratios. While this approach leads to shorter score computation period for TDDS (excluding the significant overhead of the exhaustive search itself), it is crucial to note that our proposed method consistently achieves significantly higher test accuracy. This advantage is demonstrated by both their reported results and our reproduced experiments.
Note that, Infomax (Tan et al., 2025) was excluded as it employs different base hyperparameters in the original paper compared to other baselines and does not provide publicly available code. Additionally, implementation details, such as the base score metric used to implement Infomax, are not provided. As we intend to compare other baseline methods with the same training hyperparameters, we do not include the accuracies of Infomax in our tables. To see if we can match the performance of Infomax, we tested our method with different training details. For example, if we train the subset using the same number of iterations (not epoch) as the full dataset and use a different learning rate tuned for our method, then an improved accuracy of 59% is achievable for 90% pruning on CIFAR- 100, which surpasses the reported performance of Infomax. For the ImageNet-1 K dataset, our method outperforms Infomax without any base hyperparameter tuning, while also being cost-effective.
A.2. Detailed Experimental Settings
Here we clarify the technical details in our works. For training the model on full-dataset and the selected subset, all parameters are used identically only except for batch sizes. For 10/10- we train ResNet-18 for 200 epochs with batch size of 128, for each pruning ratio (30%, 50%, 70%, 80%, 90%) we use different batch sizes with (128, 128, 128,
12

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
64, 32). We set the initial learning as 0.1, optimizer as SGD with momentum 0.9, and scheduler cosine annealing scheduler with weight decay 0.0005. For training ImageNet, we use ResNet-34 the network architecture. For all coresets with different pruning rates, we train models for 300,000 iterations with a 256 batch size. We use the SGD optimizer with 0.9 momentum and 0.0001 weight decay, using a 0.1 initial learning rate. The cosine annealing learning rate scheduler was used for training. For fair comparison, we use the same parameters across all pruning methods, including ours. All experiments were conducted using an NVIDIA A6000 GPU. We also attach the implementation in the supplementary material.
For calculating DUAL score, we need three parameters T, J, and cD, each means score computation epoch. the length of sliding window, and hyperparameter regarding the train dataset. We fix J as 10 for all experiments. We use (I' J, cp) for each dataset as followings. For CIFAR-10, 10,
(30, 10, 5.5), for CIFAR- 100, (30, 10, 4), and for ImageNet-1KG 1K, (60, 10, 11). We first roughly assign the term ep based the size of initial dataset and by considering the relative difficulty of each, we set op for CIFAR-100 smaller than that of CIFAR-10 10. For the ImageNet- IK dataset, which contains 1,281,167 images, the size of the initial dataset is large enough that we do not need to set cp to a small value to in order to intentionally sample easier samples. Also, note that we fix the value of C' of Beta distribution at 15 across all experiments. A more detailed distribution, along with a visualization, can be found in Appendix C.
Experiments with label noise and image corruption on CIFAR-100 are conducted under the same settings as described above, except for the hyperparameters for DUAL pruning. For label noise experiments, set T to 50 and J to 10 across all label noise ratio. For CD. we set it to 6 for 20% and 30% noise, 8 for 40% noise. For image corruption experiments, we set 7 to 30, / to 10, and cp to 6 across all image corruption ratio.
For the Tiny-ImageNer case, we train ResNet-34 for 90 epochs with a batch size of 256 across all pruning ratios, using a weight decay of 0.0001. The initial learning rate set to 0.1 with the SGD optimizer, where the momentum is set to 0.9, combined with a cosine annealing learning rate scheduler. For the hyperparameters used in DUAL pruning, we set ' to 60, J 10, and ep 6 for the label noise experiments. For the image corruption experiments, set T to 60, J to 10, and ep to 2. We follow the ImageNet- IK hyperparameters to implement the baselines.
13

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
B. More Results on Experiments
We evaluate our proposed DUAL score through a wide range of analyses this section. In Appendix B.1 and B.2, we demonstrate the robustness of the DUAL score through intensive experiments. In Appendix B.3, we investigate the cross- architecture performance of our method. In Appendix B.6, show the effectiveness of our Beta sampling when combined with other existing scores, compared previous sampling strategies.
We first investigate the stability of our DUAL score. We calculate the Spearman rank correlation of individual scores and the average across five rums, following Paul et al. (2021). As shown in Figure 7, snapshot-based methods such as EL2N and Entropy exhibit relatively low correlation compared to methods using training dynamics. particular, the DUAL score shows minimal variation across runs with a high Spearman rank correlation. This shows strong stability across random seeds.
Rank Correlation between Individual Scores and Overall Average
1.50
0.35
Speaman Rank Corr. Coef.
0,70
EL2N
0.65
Dyn-Unc
0.50
DUAL
20
16
100
150
250
Score Computation Epoch
Figure 7. Average of Spcarman rank correlation among independent runs and overall average of five runs. DUAL score is calculated at 30th epoch.
Next, we compute the Dyn-Unc, TDDS, and AUM scores at the 30th epoch, as we do for our method, and then compare the
test accuracy on the coreset. Our pruning method, using the DUAL score and ratio-adaptive Beta sampling, outperforms the others by a significant margin, as illustrated Figure 8. We see that using epoch of 30 results in insufficient training dynamics for the others, thus negatively impacts their performance.
CIFAR100 Performance Pruned at Epoch 30
80
NG FFüiin
70
Test Accuracy (%))
60
50
Dyn-Unc
TDDS
CCS
40
DUAL
DUAL+
10
20
30
40
50
60
70
80
90
Pruning Ratio
Figure 8, Test accuracy comparison under limited computation budget (epoch 30)
14

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
75
Test Accuracy (%)
70
65
30%
60
50%
70%
80%
55
90%
10
15
J
Figure 9. J varies from 5 to 15, showing minimal differences, which demonstrates its robustness. We fix T = 30, Cp 4. Runs averaged over three runs.
Figure 10 is a visualization of samples kept and pruned by our method. Samples kept by our method are more recognizable. The black swan on the grass and the sun-shaped balloon are rare cases, while the others are more easily recognizable. Samples pruned by our method are either typical, confusing, or mislabeled. The first and fifth examples are mislabeled, the second and third typical, and the fourth confusing.
Paper Towel
Black
Dallcon
Hamster
Cockta Shaker
(a) Samples kept by our method.
Po Towel
Black Swan
Ballcon
Hamster
Cockta il Shaker
set and apply seap.
Scrub hands and rinse.
hands thoroughly using towels.
(b) Samples pruned by method.
Figure 10. Illustrations of samples kept and pruned by our method at the pruning ratio of 30%. The pruned samples are likely either typical, confusing, or mislabeled, while the kept ones are certainly recognizable.
We compare the subset selected at the high pruning ratios by previous SOTA methods, namely CCS and D2. First, examine the total amount of overlap by counting the number of samples in the intersection of each method in Table 5. Furthermore, for intuitive understanding, we visualize the selected subset by each method for the 70%, 80% pruning cases on CIFAR-100 in Figure 11. We can see that DUAL+3 seems to include more difficult examples than others.
15

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
Table Comparison of subset overlap ratio different pruning methods at high pruning rates. Let and T denote the subsets selected by the two methods. The subsct overlap ratio is computed as
[8Л7)
(which is equal to [8h ).
1ST
Pruning Rate
CCS
DUAL+3
D2 & DUAL+B
CCS & D2
70%
0.41
0.37
0,42
80%
0.31
0.27
0.34
90%
0.19
0.17
0,21
70% Pruning
CCS
D2
DUAL
0.8
Predetio Keen
0.2
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
0.2
0.3 Prodiction Sti
Prodiction
Std
Prediction Std
80% Pruning
CCS
D2
DUAL
Predetton Mean
0.6
0.0
0.1
0.2
0.3
0.4
0.5
0.2
0.2
0.3
34
0.2
0.3
0.4
Prediction Std
Prediction Std
Prediction Std
90% Pruning
CCS
D2
DUAL
Predictin Mean
0.6
6.4
0.1
0.2
0.3
0.4
0.1
0.2
0.3
0.4
0.1
0.2
0.4
Prodiction Std
Prediction Std
Prodiction Std
Figure 11. Comparison of selected subset over different methods high pruning rates.
16

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
B.1. Image Classification Under Label Noise
We evaluated the robustness of our DUAL pruning method against label noise. We introduced symmetric label noise by replacing the original labels with labels from other classes randomly. For example, if apply 20% label noise dataset with 100 classes, 20% of the data points randomly selected, and each label is randomly reassigned to another label with probability of 1/ 1/99 for the selected data points.
Even under 30% and 40% random label noise, our method achieves the best performance and accurately identifies the noisy labels, as can be seen in Figure 12. By examining the proportion of noise removed, we can see that our method operates close to optimal.
Ratio of Pruned Mislabeld Data (%)
100
100
Optimaj
Optimal
Ratio Pruned Mislabeld Data
80
80
60
Random
Random EL2N
40
EL2N
40
Dyn-Unc
Dyn-Unc
Forgetting
Forgetting
DUAL
DUAL
DUAL
DUAL +
10
20
30
40
50
60
70
80
90
10
20
30
40
50
60
70
80
90
Pruning Ratio (%)
Pruning Ratio (%)
(a) 30% label noise
(b) 40% label noise
Figure 12. Ratio of pruned mislabeled data under 30% and 40% label noise CIFAR-100
Figure shows scatter plot of the CIFAR-106 100 dataset under 20% label noise. The model trained for epochs, and we compute the prediction mean (y-axis) and standard deviation (x-axis) for each data point. Red dots represent the 20% mislabeled data. These points remain close to the origin (0,0) during the early training phase. Therefore, pruning at this stage allows to remove mislabeled samples nearly optimally while selecting the most uncertain ones.
Full Train Set
Selected Subset
Pruned Subset
2.0
Prediction Mean
DUAL Score
1.5
0.4
1.0
0.2
0.5
0.0
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
Standard Deviation
Standard Deviation
Standard Deviation
Figure 13. Pruning ratio is set 50%. Only 116 data points 10,000 mislabeled data selected subset where red dots indicate mislabeled data.
We evaluated the performance of our proposed method across a wide range of pruning levels, from 10% 90%, and compared the final accuracy with that of baseline methods. As shown in the Table 6-9, our method consistently outperforms the competition with a substantial margin in most cases. For comprehensive analysis of performance under noisy conditions, for C10, which show results for 20%, 30%, and 40% noise, respectively. Additionally,
please refer to Tables the results for 20% label noise ImageNet are shown in Table
17

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
Table 6. Comparison of test accuracy of DUAL. pruning with existing coreset selection methods under 20% lahel noise using ResNet-18 for CIFAR-100. The model trained with the full dataset achieves 65.28 % test accuracy on average. Results are averaged over five runs.
Pruning Rate (++)
10%
20 %
30%
50%
70 %
80 %
90 %
Random
64.22 =0.37 63. 12 +0.26 61.75
#0.21
58.13
=0.22 50.11
+0.75
44.29
#1.2
32.04
#0.93
Entropy
63.51 20.25 60.59
10.23 56.75
10.37 44.90
=0.74 24.43
10.12
16.60
10.29
10.35 10.49
Forgetting
64.29 =0.26 63.40
40.14 64.00
20.27 67.51
=0.52 59.29
40.66
50.11
#0.01
32.08
11.15
EL2N
64.51 =0.35 62.67
+0.28 59.85
#0.31
46.94
=0.75 19.32
#0.87
11.02
#0.15
6.83 #0.21
AUM
64.54 =0.23 60.72
+0.22 50.38
#0.00
22.03
=0.92 5.55
#0.26
3.00
#0.18
1.68 10.10
Moderate
64.45 =0.29 62.90
40.33
61.46
+0.50
57.53
=0.61 49.50
+1.06
43,81
#0.80
29.15
70.79
Dyn-Unc
68.17 20.26 71.56
10.27 74.12
10.15
73.43
20.12 67.21
10.27
61.38
10.27
48.00
10.79
TDDS
62.86 =0.30 61.96
+1.03
61.38
+0.53
59.16
=0.44 48.93
+1.6
43.83
#1.13
34.05
+0.49
CCS
64.30 #0.21 63.24
40.24
61.91
20.45
58.24
-0.29
50.24
40.39
43.76
#1.07
30.67
#0.96
DUAL
69.78 =0.28
74.79
+0.07
75.40
#0.11
73.43
=0.16
67.57
10.18
61.46
+0.45
43.30
11.59
DUAL+B sampling
69.95 =0.60
74.68
+1.22
75.37
+1.3
73.29
=0.81
68.43
+0.77
63.74
+0.35
54.04
+0.92
Table Comparison test accuracy of DUAL pruning with existing coreset selection methods under 30% label noise using ResNet-18 for CIFAR-100. The model trained with the full dataset achieves 58.25% % lest accuracy on average. Results are averaged over five runs.
Pruning Rate (-)
10%
20% %
30%
50%
70%
80%
90 %
Random
57.67 10.52
56.29
10.55
54.70
10.60
51.41
20.88
42.67
10.80
36.86
11.01
25.64
1:0822
Entropy
55.51 =0.42
51.87
+0.36
47.16
+0.58
35.35
=0.49 18.69
20.76
13.61
#0.42
8.58 :0.49
Forgetting
56.76 =0.62 56.43
+0.28
58.84
#0.26
64.51
=0.37 61.26
+0.69
52.94
#0.68
34.99 #1.16
EL2N
56.39 =0.53 54.41
+0.68
50.29
#0.40
35.65
=0.79 13.05
+0.51
8.52
#0.40
6.16 #0.40
AUM
56.51 =0.56 49.10
+0.72
37.57
20.66
11.56
=0.46 2.79
10.28
1.87 20.24
1.43 10.12
Moderate
57.31 20.75 56.11
10.45
54.52
10.48
50.71
=0.42 42.47
10.29
36.21
11.00
24.85 11.72
Dyn-Unc
62.20 =0.41
66.48
+0.40
70.45
#0.50
71.91
=0.34 66.53
10.19
61.95
40.46
49.51 =0.52
TDDS
57.24 =0.44
55.64
40.46
53.97
40.46
49.04
=1.05
39.90
+1.21
35.02
11.34
26.99
1..0
CCS
57.26 =0.48
56.52
+0.23
54.76
#0.52
51.29
=0.32 42.33
+0.78
36.61
#1.31
25.64
#1.65
DUAL
62.42 =0.18
67.52
+0.40
72.65
10.17
71.55
=0.23
66.35
10.14
61.57
#0.11
48.70
#0.19
DUAL+$ sampling
63.02 #0.41
67.52
10.24
72.57
10.15
71.68
=0.27
66.75
10.45
62.28
10.43
52.60
10.87
Table 8. Comparison of test accuracy of DUAL pruning with existing coreset selection methods under 40% label noise using ResNet-18 for CIFAR-108 100. The model trained with the full dataset achieves 52.74% % test accuracy on average. Results are averaged over five runs.
Pruning Rate (-+)
10%
20%
30%
50%
70%
80%
90%
Random
51.13 =0.71 48.42
+0.46
46.99
20.29
43.24
-0.46
33.60
+0.50
28.28
10.81
19.52 70.79
Entropy
49.14 10.32 46.06
10.58
41.83
10.73 28.26
-0.3
15.64
10.19
12.21
10.68
8.23 10.40
Forgetting
50.98 =0.72 50.36
+0.48
52.86
40.47
60.48
=0.68
61.55
+0.58
54.57
40.86
37.68 +1.6
EL2N
50.09 =0.86 46.35
40.48
41.57
10.26 23.42
=0.80
9.00
10.25
6.80
#0.44
5.58 +0.40
AUM
50.60 =0.54 41.84
10.76
26.29
10.72 5.49
10.19
1.95
10.21
1.44
10.14
1.43 +0.24
Moderate
50.62 =0.27 48.70
+0.79
47.01
#0.21
42.73
=0.39
32.35
+1.29
27.72
1..6
19.85
#1.11
Dyn-Une
54.46 =0.27 59.02
+0.23
63.86
+0.47
69.76
10.16
65.36
10.14
61.37
+0.32
50.49
#0.71
TDDS
50.65 =0.23 48.83
40.38
46.93
20.66
41.85
=0.37
33.31
40.79
29.39
#0.35
21.09
+0.89
CCS
64.30 10.29
48.54
10.35
46.81
10.45
42.57
10.32
33.19
10.88 28.32
10.59
19.61
10.75
DUAL
54.46 10.33
58.99
10.34
64.71
10.44
69.87
0.28
64.21
10.21
59.90
10.44
49.6
10.27
DUAL+$ sampling
54.53 #0.06
59.65
+0.41
64.67
+0.34
70.09
-0.33
65. 12
+0.46
60.62
+0.30
51.51
#0.41
18

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
Table 9. Comparison of test accuracy of DUAL pruning with existing coreset selection methods under 20% label noise using ResNet-34 for ImageNe, The model trained with the dataset achieves 42.24% % test accuracy on average. Results are averaged over three runs.
Pruning Rate (-+)
10%
20 %
30 %
50%
70 %
80 %
90 %
Random
41.09 10.29 39.24
10.39
37.17
10.23
32.93 10.45
26. 12 10.63
22.11
10.42
13.88
10.60
Entropy
40.69 =0.06 38. 14 40.92 35.93
#1.56
31.24
=1.76
23.65
42.05
18.53
12.10
10.52 11.64
Forgetting
43.60 =0.65 44.82
40.20
45.65
40.48
46.05
=0.07
41.08
40.33
34.89
#0.12
24.58 #0.06
EL2N
41.05 10.35 38.88
10.63
32.91
10.39
20.89
10.80
8.08
10.24
4.92
10.32
3.12 10.07
AUM
40.20 =0.27 34.68
10.35
29.01
10.12
10.45
=0.85
2.52
10.75
1.30
10.23
0.79 10.40
Moderate
41.23 20.38 38.58
10.60 37.60
10.66
32.65
=1.18
25.68
10.40
21.74
10.63
14.15 20.73
Dyn-Une
45.67 =0.78 47.49
#0.46 49.38
#0.17
47.47
=0.32 42.49
#0.39
37.44
#0.73
28. 18.48 #0.73
TDDS
36.56 20.54
36.90
10.48
47.62
11.36
42.44
0063 34.32
10.26
24.32
10.26
17.43
10.17
CCS
40.49 20.67 39.06
40.24
37.67
40.46
30.83
1.02
22.38
40.70
19.66
10.58
12.23 10.64
DUAL
45.76
-0.67
48.20
10.20
49.94
10.17
48.19
0.27
42.80
10.74
37.90
10.59
27.80
10.49
DUAL+B sampling
45.21 10.08
47.76
10.33
48.99
10.32
46.95
0.23
43.01
10.43
37.91
1..28
28.78 10.57
19

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
B.2. Image Classification Under Image Corruption
We also evaluated the robustness of our proposed method against five different types of realistic image corruption: motion blur, fog, reduced resolution, rectangular occlusion, and Gaussian noise across the corruption rate from 20% to 40%. The ratio of each type of corruption is 4% for 20% corruption, 6% for 30% comrptiion, and 8% for 40% corruption. Example images for each type of corruption can be found in Figure 14. Motion blur, reduced resolution, and rectangular occlusion are somewhat distinguishable, whereas fog and Gaussian noise are difficult for the human eye to differentiate. Somewhat surprisingly, our DUAL pruning prioritize to remove the most challenging examples, such as fog and Gaussian corrupted images, as shown in Figure 16.
Motion
Fog
Resolution
Rectangle
Gaussian
Figure 14. Examples of the different types of noise used for image corruption. Here we consider motion blur, fog, resolution, rectangle,
Gaussian noise.
Rotio Corruptrd (96)
Ratio of Pruned Corrupteo Data (%)
Optimal
100
Optimal
Optimal
80
Rabio Pruned Corupted
60
60
60
Random EL2N
Randam EL2N
40
40
EL2N
Dyn-Unc Forgetting
Dyn-Unc Forgetting
20
20
Forgetting
DUAL
DUAL
DUAL
DUAL
8
DUAL
DUAL
10
20
50
60
70
60
90
10
20
30
40
50
60
70
90
10
20
30
40
50
60
70
80
90
Pruning Ratio (%)
Pruning Ratio (%)
Pruning Ratio (%)
(a) 20% image corruption
(b) 30% image corruption
(c) 40% image corruption
Figure 15. Ratio of pruned corrupted samples with corruption rate 20%, 30% and 40% on CIFAR-100
Ratio of Pruned Corrupted Data
Motion
Fog
Resolution
Rectangle
Gaussian
Unc
Fongetting
60
DUAL
40
D
20
60
80
20
60
80
20
40
80
20
40
60
80
40
60
80
Pruning Ratio (%)
Pruning Ratio (%)
Pruning Ratio (%6
Pruning Ratio (%6
Pruning Ratio (%6
Figure Illustration the different types of noise used for image corruption. DUAL pruning prioritizes removing most challenging corrupted images, such as fog and Gaussian noise.
We evaluated the performance of our proposed method across a wide range of pruning levels, from 10% to 90%, and compared the final accuracy with that of baseline methods. As shown in the table, our method consistently outperforms the competitors in most cases. For comprehensive analysis of performance under noisy conditions, please refer to Tables 10 to 12 for CIFAR-100 which show results for 20%, 30%, and 40% corrupted images, respectively. Additionally, the results for 20% image corruption in Imag-em are shown in Table 13.
20

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
Table 10. Comparison of test accuracy of DUAL. pruning with existing coreset selection methods under 20% image corrupted data using ResNet-18 18 for 100. The model trained with the full dataset achieves 75.45 % test accuracy on average. Results are averaged over five runs.
Pruning Rate (++
10%
20 %
30 %
50%
70 %
80
90%
Random
74.54 10.14
73.08
10.27
71.61
10.14 67.52
=0.32 59.57
10.52
52.79
10.68
38.26
1..32
Entropy
74.74 10.25 73.15
10.25
71.15
10.13 64.97
20.5 49.49
11.40
35.92
10.64
17.91
1:045
Forgetting
74.33
=0.25
73.25
+0.29
71.68
40.37 67.31
=0.23 58.93
+0.35
52.01
#0.02
38.95
= 11.24
EL2N
75.22 =0.09
74.23
#0.11
72.01
#0.18 48.19
=0.17 14.81
#0.14
8.68
#0.06
7.60 #0.18
AUM
75.26 10.25 74.47
10.31
71.96
10.22 47.50
11.39 15.35
11.79
8.98
11.37
5.47 10.85
Moderate
75.25
=0.23 74.34
40.31 72.80
20.25 68.75
=0.40 60.98
+0.39
54.21
#0.93
38.72 =0.30
Dyn-Unc
75.22
#0.25 75.51
40.22 75.09
10.23 72.02
=0.07 62.17
40.55
53.49
10.47
35.44 10.49
TDDS
73.29
=0.40
72.90
+0.31
71.83
#0.78
67.24
=0.92 57.30
#3.11
55.14
#1.21
41,58
#2.10
CCS
74.31
20.14
73.04
40.23
71.83
10.25
67.61
=0.48
59.61
40.64
53.35
10.71
39.04
#1.14
DUAL
75.95
=0.19
75.66
+0.23
75.10
20.23
72.64
=0277
65.29
40.64
57.55
#0.55
37.34
11.70
D sampling
75.50
#0.21
75.78
10.15
75.10
10.13
72.08
0.22
65.84
10.37
62.20
10.72
53.96
10.35
Table 11. Comparison of test accuracy of DUAL pruning with existing coreset selection methods under 30% image corrupted data using ResNet-18 for CIFAR-100. The model trained with the full dataset achieves 73.77% test accuracy on average. Results are averaged over five runs.
Pruning Rate (++)
10%
20 %
30%
50%
70 %
80%
90 %
Random
72.71 10.34
71.28
10.31 69.84
10.24
65.42
10.3
56.72
10.56
49.71
10.65
35.75
11.41
Entropy
72.94 =0.09
71.14
10.14 68.
78 40.20 61.34
=0.59 42.70
1.00 29.46
1.08
12.55 #0.66
Forgetting
72.67 =0.21 71.22
40.08 69.65
40.45
65.25
=0.33
56.47
40.31
49.07
#0.32
34.62
#1.15
EL2N
73.33 10.08 71.99
10.11
67.72
10.50
37.57
10.70 10.75
10.28
9.08
10.30
7.75 10.08
AUM
73.73 =0.19 72.99
10.22
70.93
10.33
57.13
=0.42 28.98
10.50
19.73
10.28
12.18 #0.46
Moderate
74.02 =0.28
72.70
10.30
71.51
10.26 67.35
=0.16 59.47
10.34
52.95
10.60
37.45
11.21
Dyn-Une
73.86
=0.21 73.78
+0.20
73.78
#0.12 71.01
=0.23 61.56
+0.46
52.51
#1.08
35,47
#1.31
TDDS
71.58
10.50
71.45
10.68
69.92
10.25
65.12
2.08
55.79
12.16
53.85
10.94
40.51 11.34
CCS
72.58
10.12
71.38
40.35
69.83
20.26
65.45
=0.23
56.65
40.45
49.75
10.90
34.63
11.79
DUAL
73.96
2020
74.07
10.43
73.74
10.18
71.23
00.88
64.76
10.32
57.47
10.51
37.93
12.38
DUAL+ sampling
73.91
10.17
73.80
10.48
73.59
10.19
71.12
20.29
65.18
10.44
61.07
10.47
52.61
10.47
Table 12, Comparison of test accuracy of DUAL pruning with existing coresct selection methods under 40% image corrupted data using ResNet-18 for CIFAR-100. The model trained with the full dataset achieves 72.16% test accuracy average. Results are averaged over five runs.
Pruning Rate (+)
10%
20 %
30 %
50%
70 %
80 %
90% %
Random
70.78 20.25
69.30
+0.29
67.98 +0.26
63.23
-0.26
53.29
+0.64
40.85 45.76
32.63 #0.61
Entropy
70.74 =0.18
68.90
20.37 66.19
#0.46 57.03
=0.60 35.62
+1.88
22.50
#1.03
7.46 #0032
Forgetting
70.54 10.10
69.17
10.30
67.41
10.28 62.77
10.15 52.89
10.38
44.94
10.66
30.48 10.49
EL2N
71.57 10.28 69.24
10.16
62.95
10.52 28.33
=0.47 9.48
10:21
8.86
10.21
7.58 10.16
AUM
71.66 =0.23
69.75
+0.30
62.10
40.46 26.56
=0.22 8.93
10.19
5.82
#0.09
4.15 #0.11
Moderate
72.10 =0.11
71.55
+0.25
69.84
40.39
65.74
=0.21 56.96
+0.32
49.04
20.74
34.87 #0.57
Dyn-Unc
71.86 10.12
71.65
10.18
71.79
10.27
69.17
20.4 59.69
10.30
51.36
10.70
34.02 10.45
TDDS
70.02 =0.43
69.27
40.74
68.03
#0.55
63.42
=0.77 55.28
41.93
51.44
#1.36
38.42 +0.80
CCS
70.84 10.41 69.08
10.41
68.11
10.09
63.36
20.16 53.21
10.54
46.27
10.52
32.72 10.52
DUAL
71.90 10.27
72.38
10.27
71.79
10.11
69.69
10.18
63.35
L0.29
56.57
11.07
37.78
1.073
DUAL+ sampling
71.96
13.13
71.92
40.22
71.69
40.1 18
69.23
10.15
63.73
+0.43
59.75
+0.32
51.51
#0.68
21

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
Table 13. Comparison of test accuracy of DUAL pruning with existing coreset selection methods under 20% image corrupted data using ResNet-34 Tiny-ImageNet. The model trained with the full dataset achieves 57.12% test accuracy on average. Results are averaged over three runs.
Pruning Rate (-+)
10%
20 %
30%
50%
70 %
80% %
90 %
Random
49.59 10.93
48.64
10.94
45.64
10.53
41.58
20.66
33.98
10.55
28.88
10.67
18.59 1.025
Entropy
50.34 10.19
48.02
40.49
44.80
40.30
36.58
=0.19
25.20
+0.53
16.55
40.40
3.32 #0.26
Forgetting
46.81 =0.26
41.16
40.28
35.58
#0.17
26.80
=0.18
17.66
40.23
12.61
#0.04
6.01 #0.19
EL2N
50.66 10.27 47.76
10.25
42.15
11.02
23.42
10.26
8.07
10.09
6.57
10.36
3.75 10.13
AUM
51.11 =0.73
47.70
40.51
42.04
10.81
20.85
=0.79
6.87
10.24
3.75
20.21
2.27 #0.11
Moderate
51.43 20.76
49.85
40.23
47.85
10.31
42.31
=0.40 35.00
10.49
29.63
10.67
19.51 10.72
Dyn-Une
51.61 =0.19
51.47
40.34
51.18
#0.58
48.88
=0.85 42.52
#0.34
37.85
#0.17
26.26
#0.70
TDDS
51.53 10.40
49.81
40.21
48.98
10.27
45.81
10.16 38.05
10.70
33.04
10.39
22.66
11.28
CCS
50.26
20.78
48.00
40.41
45.38
40.63
40.98
=0.23
33.49
40.04
27.18
10.66
15.37 20.54
DUAL
51.22
10.40
52.06
10.55
50.88
10.64
47.03
20.56
40.03
10.09
34.92
10.15
20.41
11.07
DUAL+B sampling
52.15
10.25
51.11
10.34
50.21
10.36
46.85
0.27
42.97
10.28
38.30
1:0.06
27.45 10.50
22

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
B.3. Cross-architecture generalization
In this section, investigate the cross-architecture generalization ability of our proposed method. Specifically, we calculate the example score architecture and test its coreset performance different architecture. This evaluation, with results presented in Tables 14 through 18, aims assess the transferability of these scores across diverse architectural designs.
Table 14, Cross-architecture generalization performance on CIFAR- 100 from three layer CNN to ResNet-18. We report an average of five RR8 R18' stands for computation ResNet-18, baseline.
3 3- CNN -> ResNet-18
Pruning Rate (-+)
30%
50%
70%
90%
Random
75.15 40.28
71.68 +0.31
64.86 10.39
45.09 1.26
EL2N
76.56 +0.65
71.78
+0.32
56.57 +1.32
22.84 =3.51
Dyn-Unc
76.61 +0.75
72.92
+0.37
65.97 +0.53
44.25 =2.47
CCS
75.29 +0,20
72.06
10.19
66.11 #0.15
36.98 11.47
DUAL
76.61 #0.08
73.55
+0.12
65.97
#0.18
39.00
=2.51
DUAL+B sampling
76.36 10.18
72.46
10.41
65.50 10.53
48.91 20.60
DUAL (R18~>18)
77.43 10.18
74.62
10.47
66.41
10.52
34.38
1.39
DUAL (R18~RIS +B sampling
77.86 #0.12
74.66
10.12
69.25
#0.22
54.54 =0.09
Table 15,
generalization performance CIFAR-100 100 from three layer CNN VGG-16. We report average of five runs. VV16 + V16' stands for score computation on VGG- 16, as a baseline.
3- CNN
VGG-16
Pruning Rate (-+)
30%
50%
70%
90%
Random
69.47 #0.27
65.52
+0.54
57.18
40.68
34.69 +1.97
EL2N
70.35 10.64
63.66
11.49
46.12 16.87
20.85 19.03
Dyn- Dyn-Une
71.18 #0.96
67.06
+0.94
58.87 40.83
31.57 13.29
CCS
69.56 10.33
65.26 10.50
57.60 10.80
23.92 11.85
DUAL
71.75 10.16
67.91
10.27
59.08 10.64
29.16 1:2.28
DUAL+33 sampling
70.78 10.41
67.47
#0.44
60.33 40032
43.92 1.15
DUAL (V16->V16)
73.63 40.62
69.66
+0.45
58.49
40.77
32.96 1111
DUAL (V16-+V16) +3 sampling
72.77 #0.41
68.93 +0.23
61.48 40.36
42.99=0.62
Table 16. Cross-architecture generalization performance on CIFAR-100 from VGG-16 to ResNet-18. We report an average of five runs. R18 R18' stands for computation ResNet-18, 18, baseline.
VGG-16 - ResNet-18
Pruning Rate (-+)
30%
50%
70%
90%
Random
75.15 +0.28
71.68
#0.31 64.86
=0.39
45.09
1.226
EL2N
76,42 #0.27
70.44
+0.48
51.87
1122 25.74
41.53
Dyn-Une
77.59 10.19
74.20
10.22
65.24
10.36
42.95
11.14
CCS
75.19 10.19
71.56
10.28 64.83
=0.25
46.08
+1.23
DUAL
77.40 +0.36
74.29
10.12 63.74
=0.30
36.87
42.27
DUAL+ sampling
76.67 10.15
73.
10.29
65.69
10.57
45.95 10.52
DUAL (R18->R18)
77.43 10.18
74.62
10.47
66.41
20.52
34.38 11.39
DUAL (R18-+R18) +.3 sampling
77.86 10.12
74.66
10.12
69.25
=0.22
54.54 +0.09
23

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
Table 17. Cross- arssitecche generalization performance CIFAR- 110 from R8
VGG-16 16, We report average of five runs.
V16 + V16' stands for score computation on VGG-16, 16, as a baseline.
ResNet-18
VGG-16
Pruning Rate (-+)
30%
50%
70%
90%
Random
70.99 +0.33
67.34
+0.21
60.18 40.32
41.69 +0.72
EL2N
72.43 10.54
65.36
10.68
43.35
10.81
19.92 1..89
Dyn-Unc
73.34 00.29
69,24
+0.39
57.67 #0.52
31.74 +0.80
CCS
71.18 10.16
67.35
10.38
59.77 10.43
41.06 1:1003
DUAL
73.44 10029
69.87
10.35
60.07
10.47
29.74 11.70
DUAL +3 sampling
73.50 20.27
70.43
+0.26
64.48 #0.47
49.61 +:0.99
DUAL (V16->V16)
73.63 #0.61
69.66 +0.45
58.49
40.77
32.96
#1.12
DUAL (V16-+V16)+) sampling
72.66 10.17
68.80
+0.34
60.40
40.68
41.51 #0.47
Table 18. Cross-architecture generalization performance on CIFAR 100 from VGG-16 to ResNet-50, We report an average five runs.
R550 R50' stands for computation ResNet-50, baseline
VGG-16 + ResNet-50
Pruning Rate (-+)
30%
50%
70%
90%
Random
71.13 16.52
70.31
+1.20
61.02 11.68
41.03 =3.74
EL2N
76.30 +0.69
67.11
43.09
44.88
#3.65
25,05
=1.76
Dyn-Unc
77.91 10.54
73.52
10.41
62.37
10.62
39.10 44.0
CCS
75.40 +0.64
70.44
+0.49
60.10 11.24
41.94
DUAL
77.50 40.53
71.81
10.48
60.68 1.667
34.88 -3.47
DUAL +3 sampling
76.67 #0.15
73.14
+0.29
65.69 #0.57
45.95 =0.52
DUAL (R50~>R50)
77.82 +0.64
73.66
+0.85
52. 12 +2.73
26. 1196
DUAL (R50~>550++3 sampling
77.57 +0.23
73.44 +0.87
65.17 +0.96
47.63 -2.47
24

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
B.4. Image Classification on Long-tailed Distributions
We also conducted experiments on long-tailed versions of CIFAR-IO and CIFAR-100, following the procedure of Cao et al. (2019). These long-tailed datasets were constructed using imbalance ratio, p. defined as the ratio between the sample sizes of the most frequent class (Fmax) and the least frequent class (Amin), i.e. p T /umin The class distribution in this long-tailed setup exhibits an exponential decay in sample sizes across classes. For all pruning ratios, we compared our method against several baselines: Random, EL2N, Dyn-Unc, and CCS. The results presented in Table 19 and 20 demonstrate that DUAL pruning (along with the Beta sampling) achieves superior performance compared to other baselines.
Table 19. Test accuracy on long-tailed tailed imbalance on CIFAR-10. The test accuracy on a full dataset is 89.98 (p 10) and 75.03 (p = 100).
We report the average performance three runs.
CIFAR- -O-LT
Imbalance Ratio
10
100
Pruning Rate
30%
50%
70%
80%
90%
30%
50%
70%
80%
90%
Random
42.48
28.20
18.85
10:24 10.00
40.00
10.00
28.23
19.36
#0.18 10.00
10.00
10.00 10.00
EL2N
89.42 20.20 87.59
40.97 68.15 43.44 52.90
#1.87 33.25
#0.41
72.70
11.58 66.06
#4.27 52.90 #2.88 41.79 #2.61 30.30 40.58
Dyn-Unc
89.64 20.28 87.60
20.39
67.60
14.34 53.05
20.58 39.16
21.94
74.40
11.32
70.22
=1.00 51.89 13.08 41.27
22.34
31.24 20.23
CCS
84,42
10.50 73.04
11:20 47.07
10.68 37.38
40.36
27.91
20.06
63.
18
11.56 45,46
=1333 32.66
10.63
29.38
10.71 24.10 10.97
DUAL
89.67 10.40
88.75
10.36
75.38
13.41
56.70
12.63
43.58
72.94
11.14 69.66
20.73
52.80
1:000
38.32
11:28
25.30 11:26
DUAL + B
89,49
10.21
88.12
10.61
76.00
12.79
78.31
12.26
71.27
11.44
73.81
12.06
68.89
=0.24
52.95
12,79
46.49
1.880
36.43
11.00
Table 20. Test accuracy long-tailed imbalance CIFAR-100. The test accuracy full dataset is 62.92 (p 10) and 41.67 (p= = 100). We report the average performance across three runs.
CIFAR-I00
Imbalance Ratio
Pruning Rate
30%
50%
70%
80%
90%
30%
50%
70%
80%
90%
Random
32.89 #0.23 18.79
+0.75
8.26 +0.41
5.43 +o.07
3.23 #0.05
22.88 #0.87
11.45
=0.11 5.90
#0.15
3.96 #0005
2.48 +0.02
EL2N
57.57 +0.50 47.23
+0.46 21.38
+0.33
13.92
#0.97
9.54 +0.19
37.59
#2.18 24.76
=1.87 12.33
#0.54
9.42 #0.26
6.64 +0.02
Dyn-Une
58.09 +0.85 46.68
+0.69 25.95
#2.16 20.80
+0.68
13.48 +0.50
37.82
+1.08 26.88
=0.38 15.41
#0.42
12.47
+0.55
9.52 +0.12
CCS
46.51 #0.56 34.85
+0.79 18.08
40.80 11.34
#0.30
6.06 +0.43
27.46
#0.27 17.85
=0.76 11.43
#0.33
8,25
#:066
4.34 60.53
DUAL
58,50 #0.27 54.11
+0.27 39.15
+1.43 30.10
#0.97
18.80
=1.17
36.35
#0.66 30.19
=1.88 20.47
#0.30
17.76
+0.47
12.52 #0.56
DUAL 9
58.05
#0.34 54.88
+0.36
43.53
#0.66
35.87
11.75
27.13 #1.49
37.04
#0.97
32,25
=0.46
21.94
11.27
19.38
+0.77
15.42
#0.32
25

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
B.5. Comparison with Dynamic Pruning Methods
In this section, we present several experiments comparing recent dynamic pruning methods, such as those by Yuan et al. (2025) and Qin et al. (2024), with static approaches, including DUAL pruning. We first highlight two key differences between static and dynamic data pruning.
Compared to static pruning, dynamic pruning maintains access to the entire original dataset throughout training, allowing it to fully leverage all available information in the original dataset.
While both aim to improve training efficiency, their underlying goals differ slightly. Static data pruning seeks to identify a "fixed" subset that reduces the dataset size while preserving as much information about the original dataset as possible. This subset can then serve as a new, independent dataset, reusable across various model architectures and experimental setups. In contrast, dynamic data pruning enhances training efficiency within single training session by pruning data dynamically the fly. However, this approach requires storing the entire original dataset, making dynamic pruning less memory-efficient and not reusable.
Standard Training. We conducted experiments on CIFAR-I0 10 and CIFAR- 100 with ResNet- 18, using the same hyperpa- rameters described in Section 4. We first tested dynamic random pruning, which dynamically prunes randomly selected samples from the entire dataset at each epoch. Notably, dynamic random pruning significantly outperformed all static CIFAR-10 10 and 72.8% on CIFAR-100 at a pruning ratio of 90%. We
baselines, achieving test accuracies of 91.82% also evaluated the methods in Yuan et al. (2025); Qin et al. (2024) and the results are provided in Figure 17. Overall, dynamic methods consistently outperform static baselines. However, at lower pruning ratios (e.g., on CIFAR-10), DUAL can outperform dynamic methods under a similar computational budget.
We believe this performance gap stems from differences in accessible information: static methods are limited 10% of the data, while dynamic methods use the full dataset. Consequently, the performance gap widens even further at aggressive pruning ratios. To validate this, plot how often each sample was seen during training. The plot in Figure 18 shows that static methods are confined to a subset, while dynamic ones use nearly all data data-rendering direct comparison somewhat unfair. Indeed, dynamic pruning methods might be better compared with scheduled batch-selection approaches, such as curriculum learning, rather than static pruning methods.
Label Noise Setting. We also evaluated these methods under label noise conditions. In fact, Yuan et al. (2025) conclude that IES cannot prune any samples (corrupted or not) when label noise is introduced. Similarly, InfoBatch (Qin et al., 2024) tends to retain harder (and often noisy) samples, as removes only easy examples during training. In contrast, DUAL effectively filters noisy samples, improving performance even beyond full-data training.
We conducted experiments on CIFAR-100 with a 40% label noise setting (full-train test accuracy: 52.74%) to verify this explanation. DUAL achieves over 70% test accuracy at a 50% pruning ratio as can be seen in Table 8 in Appendix B. whereas InfoBatch achieves only 51 .44 accuracy with a similar number of iterations. Under similar iterations, random dynamic pruning achieves 51.81% % test accuracy, which still outperforms random static pruning. Lastly, IES (Yuan et al., 2025) prunes only 1.7% of samples during training (consistent with the original report in their paper), resulting in $1.95% test accuracy. Furthermore, our static method can create fixed subsets which nearly all noisy samples have been removed, resulting in high- high-quality datasets that can be preserved for future use.
26

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
CIFAR-10
96
Train
94
92
Accuracy
(Dyn) Random
88
(Dyn) InfoBatch
(Dyn)
86
(Static) Random
(Static) DUAL+ +
84
(Static) CCS
120000
Total Number of Iterations
CIFAR-100
70
60
(Dyn) Random
(Dyn) InfoBatch
55
(Dyn)
(Static) Random
50
(Static) DUAL+A
(Static) CCS
45
80000
120000
Total Number of Iterations
Figure 17. This figure shows test accuracy (y-axis) versus the total number of iterations needed to fully train each subset (x-axis). The top figure corresponds CIFAR-10, and the bottom to CIFAR- 100. Results are averaged over five random seeds. For the static pruning methods, we plotted the number of iterations and test accuracy at pruning ratios of 30%, 50%, 70%, 80%, and 90%, which correspond to the markers on each line from right to left. We adjust the total training epochs InfoBatch following in order to evaluate its performance when the total number of iterations is reduced. For IES, we adjust the pruning threshold to evaluate its performance under reduced total number training iterations. Results demonstrate that dynamic pruming methods including even random dynamnc the most efficient and the best-perforning However, it does not achieve as favorable a time-performance trade-off as dynamic methods. Still, at low pruning ratios where more information available- DUAL performs comparably to dynamic approaches.
procedure in the original paper.
outherform static baselines in accuracy while requiring fewer iterations. Among static methods, DUAL is both
Dynamic Pruning
0.000700
0.001.5
0.000.00
0.000775
0.000500
0 50000 20003 20005 40002 30000
10200 20100 33200 42000 52000 Served triees
10005 20035 20000 40020 50000
20005 20020 20000 40003 50002 Sories Dala
Sucied Daca
Sented Datu
Figure 18. This figure shows the selection frequency of each sample at a high pruning ratio (90%). The x-axis represents data indices sorted descending order by selection frequency, the y-axis indicates the normalized selection ratio i.e., how often each data point was selected, divided by the total number of data points seen during training (values sum to 1). Methods like Random Dynamic Pruning. InfoBatch, and IES use nearly the entire dataset during training, making them incomparable to static pruning methods, which are restricted to only 10% of the data.
27

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
6.6. Ablation Study on Beta Sampling
We study the impact of our Beta sampling on existing score metrics. We apply our Beta sampling strategy forgetting, EL2N, and Dyn-Unc scores of CIFARIO and 100. By comparing Beta sampling with the vanilla threshold pruning using scores, with results presented in Tables 21 and 22, observe that prior score-based methods become competitive, outperforming random pruning when Beta sampling is adjusted.
Table 21. Comparison on CIFAR-10 10 and CIFAR-100 100 for 90% pruning rate. We report average accuracy with five runs. The best performance bold each column.
CIFAR-10
CIFAR-100
Method
Thresholding
B-Sampling
Thresholding
8-Sampling
Random
83.74
83.31 (-0.43) 0.14
45.09 1.26
51.76 (+6.67) +:
EL2N
38.74 0.75
87.00 (+48.26)
8.89 0.28
53.97 (+45.08) 0.63
Forgetting
46.64 1: 1,90
85.67 (+39.03) 10.13
26.87 1 0.73
52.40 (+25.53) 1 0.43
Dyn-Une
59.67 = 1.79
85.33 (432. 14) # 0.20
34.57 # 0.69
51.85 (+17.28) + 0.35
DUAL
54.95 +: 0.42
87.09 (+31.51) + 0.36
34.28 + 1.39
54.54 (+20.26) 1: 0.09
Table 22. Comparison CIFAR-10 and CIFAR-100 for 80% pruning rate. We report average accuracy with five runs. The hest
performance bold each column.
CIFAR-10
CIFAR-100
Method
Thresholding
B-Sampling
Thresholding
3-Sampling
Random
88.28 0.17
88.83 (+0.55)
59.23 =
61.74 (+2.51) + 0.15
EL2N
74.70 + 0.45
87.69 (+12.99)
19.52
63.98 (+44.46) +:2 0.73
Forgetting
75.47 1: 1,27
90.86 (+15.39) 1 0.07
39.09 1 0.41
63.29 (+24.20) 1.1.3
Dyn-Unc
83.32 = 0.94
90.80 (+7.48) = 0.30
55,01 = 0.55
62.31 (+7.30) = 0.23
DUAL
82.02 + 1.85
91.42 (49.68) = 0.35
56.57 = 0.57
64.76 (+8.46) + 0.23
We also study the impact of our pruning strategy with DUAL score combined with Beta sampling. We compare different sampling strategies: vanilla thresholding, stratified sampling (Zheng et al., 2022), and Beta sampling on CIFARIO and 100, 80% and 90% pruning rates. The results, presented in Table indicate that our proposed Beta sampling mostly performs the best, especially with the high pruning ratio.
Table 23. Comparison Sampling Strategy
CIFAR10
Pruning Rate
30%
50%
70%
80%
90%
DUAL
95.35
95,08
91.95
81.74
55.58
DUAL + CCS
95.54
95.00
92.83
90.49
81.67
DUAL + 9
95.51
95.23
93.04
91.42
87.09
CIFAR100
Pruning Rate
30%
50%
70%
80%
90%
DUAL
77.61
74.86
66.39
56.50
34.28
DUAL + CCS
75.21
71.53
64.30
59.09
45.21
DUAL + 3
77.86
74.66
69.25
64.76
54.54
28

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
C. Detailed Explanation about Beta Sampling
Here, we provide details on our choice of Beta sampling. Appendix C.1 shows the visualization the selected data using the Beta sampling. Appendix C.2 presents the full algorithm for our DUAL pruning method with the suggested Beta sampling strategy.
We begin by explaining why the Beta distribution is selected the sampling distribution. The domain of Beta distribution is [0, 1|, which naturally aligns with the range of prediction means. Moreover, the probability density function (PDF) of the Beta distribution can be shaped to decay at both tails, ensuring that samples with extreme scores are rarely selected. While other distributions, such as Gaussian, could also be considered, their support spans R. As a result, they may assign non-negligible probability to values far outside the desired range unless their standard deviation made extremely small.
We define our sampling distribution Beta(ar, B,) follows:
8, = 15 (1 (DD (1 POD)
(6)
ar 15 3r,
where μp E [0, 1] is the probability mean of the highest DUAL score training sample. To ensure stability, compute this as the average probability mean of the 10 highest DUAL score training samples. Additionally, mentioned in Appendix A.2, we set the value of C to 15 across all experiments. Recall that the variance of Beta distribution with parameters is a8 given by (a++))((aa++++))- so increasing C' leads to lower variance, an effect illustrated in Figure 19. This enables more concentrated sampling distribution in a specific region, improving the effectiveness of the sampling process by reducing unnecessary spread. In an implementation, we add I to a, to provide a more targeted sampling.
C= 15
C=
jo
0.0
Ho
3.5
0=0.5
0-0.
0-0
2.5
0=9
PDF
2.0
1.0
0.0
0.2
0.4
1.0
0.6
0.8
1.0
Prediction Mean
Prediction Mean
Figure 19. Visualization of Beta distribution for varying C. Large C enables more concentrated targeting.
Now we justify our choice of parameters a, and 3, in the Beta distribution. When the pruning ratio is set to zero, a, and 3, are configured so that the mean of the Beta distribution (which is a a+4B)) matches the prediction mean of the highest-scoring sample. This allows the sampling to focus on high-score samples at low pruning ratios. To gradually include easier samples as the pruning ratio increases, we set parameters o, and 8, depending on the pruning ratio. While BOSS (Acharya et al., 2024) adjusts these parameters such that the mode of the Beta distribution (which is with the pruning ratio r, we adopt a non-linear scaling by raising to the power of ep. This results in PDF that is remains almost stationary at low pruning ratios, but gradually shifts toward easier samples in polynomial manner the pruning ratio increases.
_--1 for a, 3 > 1) scales linearly
a+--2
The hyperparameter ep is chosen based on the relative complexity of the dataset. We assumed that a larger dataset, which may have more samples per class, tends to be relatively easier. A higher value of cp results in smaller 3r, thereby increasing the mean and reducing the variance of the Beta distribution. For more difficult datasets, sampling easier examples becomes more important, which justifies using larger cp. Figure 20 illustrates the Beta PDF for different values of cp. In both subplots, we set μD 0.25. The left subplot shows the PDF with ep 5.5, which corresponds to the value used in 10 experiments, while the right subplot shows the case where cp = 4, which is used in CIFAR-100.
29

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
Cp 5.5
Cp =
r=0.3
3.5
r=0.5
0=7.
2,5
0=0.8
PD
0=0.9
1.0
0.0
0.2
0.4
1.0
0.6
0.8
1.0
Prediction Mean
Prediction Mean
Figure 20. Visualization of Beta distribution for varying ep. Left subplot corresponds the value used in CIFAR-10, and the right subplot corresponds to the value used CIFAR-100.
30

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
C.1. Visualization of Selected Data with Beta Sampling
We provide visualizations of Beta sampling in Figure 21. This figure illustrates, in respective columns: (i) sample selection probabilities for the coreset; (ii) examples of selected samples when forming 70%, 30% and 10% coreset; and (iii) the samples pruned at these corresponding rates. The visualization shows that Beta sampling increasingly favors the selection of easier samples for the coreset as the pruning ratio increases.
Sampling Probability
70% Selected
30% Pruned
10-
06-
02-
0.1
03
0.4
0.
0.4
01
0.3
(a) CIFAR 100 with pruning ratio 30%
Sampling Probability
30% Selected
70% Pruned
1.0
0.-
02
0.0
0.1
0.2
0.3
0.1
0.0
0.1
0.2
0.3
0.4
0.0
01
0.
0.3
0.
(b) CIFAR- 100 with pruning ratio 70%
Sampling Probability
10% Selected
90% Pruned
0.
0.5
0.4
0.
0.0
0.1
0.2
0.3
CA
0.0
0.1
0.5
04
0.1
0.2
0.5
0.4
(c) CIFAR-100 with pruning ratio 90%
Figure 21. Pruning visualization on CIFAR-100.
31

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
C.2. Algorithm of Proposed Pruning Method
The detailed algorithms for DUAL pruning and Beta sampling follows:
Algorithm 1 DUAL pruning + 8 sampling
input Training dataset D, pruning ratio dataset simplicity training epoch T, window length J.
output Subset $ С DD such that S] = (1-
for (æt, wi) E DD do
for = 1,... T
+ do
Pa vii) <- 2i=0 j=0 Pk+j(U (Wi x.)) (Example Difficulty)
J-1
Us(2,, 9i) + V J-1 Σド= Pk+S (yt x.)) (a. (Prediction Uncertainty]
DUAL, (T,:,((;;:11)) M/) f (1 Ps (ct x Us(a,, end for DUAL(I;; v.) f F-J Ei= (UAL 9:)
end for 8-sampling then
for (æ,, vi) D do
P(x,, 91) + T Ek=1 (1 x;;)
@ (P())))L) t PDF value of Beta(ay, B,) from Equation (5)
(xx, t 4 (P(x,,, x DUAL(Z; V.)
end for
((x;)) Esco (0x;)
((2,)
S + Sample (1 r) [D] data points according to (((x,)
else
S + Sample (1 r) [D] data points with the largest DUAL( ms)) score
endi if
32

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
D. Theoretical Results
Throughout this section, we will rigorously prove Theorem 3.1, providing the intuition that Dyn-Unc takes longer than our method to select informative samples.
D.I. Proof of Theorem 3.1
Assume that the input and output (or label) space are X IR" and y = {=1}, respectively. Let the model f x R be of the form fæ; w) w parameterized by u E IR" with aro-initialization. Let the loss the exponential loss, {(2) = e Exponential loss is reported to induce implicit bias similar logistic loss in binary classification tasks using linearly separable datasets (Soudry et al., 2018; Gunasekar et al., 2018).
The task of the model is to learn binary classification. The dataset D consists only two points, i.e. D {( (æ2, v²)). where without loss of generality yf = 1 for i = 1, 2. The model learns from D with the gradient descent. The update rule, equipped with a learning rate > 0, is:
W00
2
Σ (f (f;; (w):
Wi+1 = wi - 1/
= w, + w 21 1 + - xxxx2)
For brevity, denote the model output of the i-th data point at the f-th epoch as 91 w)) The update rule for the parameter is simplified
Wi+11 = w; + 17 (c vha 21 + ev- x32)
(7)
We also derive the update rule of model output for each instance:
yit1 = w 01 = wr + 17
ve (1)
I1 1e (2) 22))
vi 11) + 17E mee" ") | 112 ne w (0),
(8)
(2) (2)
-v.") || ne vi (z))
Wi+1
yi
+ ne
Assume that æ: is farther from the origin in terms of distance than क is, but not too different in terms of angle. Formally, Assumption D.I. ||ج||| > 1, 4| 1 < 2(x11, x2) < ||>:| 12 Moreover, (æ), x22) <
||| || |||||
Under these assumptions, as (I1,, x22) > 0, D is linearly separable. Also, notice that æ; and ah are not parallel. Our definition of a linearly separable dataset is in accordance with Soudry et al. (2018). dataset D is linearly separable there exists w such that (æ:, w² > O, Vi.
(8) Theorem D.2. Let v1 be the variance and Pej be the mean of (B) (1) within window from time t to t + 1. (8)
Denote Tr and T um as the first time when v-(1) v/3) and v() (1 (t
(()) occurs, respectively. Under
Assumption D.I, if is sufficiently small then Tom < T...
(1) By Soudry et al. (2018), the learning is progressed as: wr, Vt and y diverges to positive infinity (Lemma 1) but w: directionally converges towards L2 max margin vector, w 2 / 1| the growth of w is logarithmic, i.e. we as r logt. We hereby note that Theorem 3 of Soudry et al. (2018) holds for learning rate 1) smaller than global constant. Since our condition requires 1) to be sufficiently small, we will make use of the findings of Theorem
(2)
w. or lim, [w. T (Theorem 3). Moreover,
(2) Lemma D.3. Ayi = Vi Vt (1)
a non-negative, strictly increasing sequence. Also, lime A oc.
Proof.
33

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
1) Since un = 0. 90
(1)
yo so
Ayo 0. By Equation (8) and Assumption D.I, Ay = yi vi (
(2)
17 [||02|12 || 122) >
2)
A441 - Ay - [0--0 - (110211- (211, x2)) + -e- (221, - -|021119)
= Kie vi + K2e -v(") > 0,
for some positive constant K1, K2. As yt = w x, would logarithmically grow in terms of t, e e-v!" is decreasing in t.
Moreover, Vi (1) w, 21 t xn log = log I, - (1 is (asymptotically) in scale of t-1 and so is Ayr-1 Ayr. Hence, } is non-negative and increases to infinity.
The notation Ay, = Vi Vi (1) for all > 0. Lemma D.4. For all t > O, V241 y.
will be used throughout this section. Next, we show that, under Assumption D.I, (11 < yi
(2)
Proof. Notice that:
(1 -| || 171 x2) (110) n|| 2| + 1 x22)
1) 1/2 < ?
(1)
(1)
(1)
|||: 1| + ne v1ah (x1,
1/2
+ ne
1) (-- + 1) || 2| 77 (e-vi") + 1) (0) x2)
< x 2||20 1| n x 2( .2)
< <,0,,, x2)) + + y/ 1
2) Assume, for t >>>>>00 0,
(1) (1)
2
(1) (1)
Ve-t ||<|| 1| + me
Mo+1 (2) 222)
(2)
Vi+2 1/41 m/e
1/1
(2) ne
Vi |x]
ne
(2)
(x1, 122)
y/ ne Vi (2))
(2)
ne |022|| VittI
(2)
By Lemma D.4, for all >>>00000000 0, (v(")
(2) 1et11
lies entirely on right-hand side of (1) (1)
without any overlap.
(
We first analyze the following term:
Observe that:
Mt41
(1)
Vi
(1)
me (1)
[ 1|2 11e v1 (01, 22)
V2+1 yi
me v! (2)
|| 1| + 1/e wi (x) x2)
(9)
| 112 + te -vv (x xz)
(z1, x22) + e Ari x 12
It is derived that the fraction is an increasing sequence in terms of t. For values b, d' > O, eta < ad' cb + cd' < ad + c'b + c'd. Taking:
it
| 2 6 (z), x22)
c e- Au (2,
d' c --- c-Au--1(21 x2)
d = e Aui |2
d' = e Agett ||
34

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
have
ad' + ob + ed'
Agc-1 -m- | 11 + Av (201, x2)² 2 e- -Aee - (x) (2) ||
<e -m- || ||=1 te Auc (2) + e Amee Aui (x) x2ه و 1
ad cbb c'd.
The inequality holds by Lemma D.3 and the Cauchy-Schwarre inequality, Taking the limit of Equation (9) as t co, the ratio converges to:
2 11
(10)
(2 222)
For the later uses, we also define the initial ratio, which is smaller than 1:
M0
| 1 + (x), x2)
Ro = (2) (2)
(S R).
(11)
(x), 22)
|| 12
1/1 Vi
Now we analyze a similar ratio of the one-step difference, but in terms of a Vii logistic function, (2) (1 =-=) Notice that d'(=)= o=) o()).
(4)
instead of yt There, o stands for the
0 (1111 a
Lemma D.5. (() =
monotonically increases to +oc.
(2) ((-1)
Proof.
(1) (1) o'
(1)))
c E (1)) (1)
V141 Vi
yit11
(1)
(for some
by the mean value theorem.)
(2) (2)
(2)
(2) E yy++1
(2)
V1+1 Vt o'
141 y
(1) (1) o'
Vc+1
(1)
d'' decreasing on R
Mit1 Vi o' v((2) ())
(2)
-2
(1) (1) e Min
+
V411 i/
1/41 2/1
(2) (2)
e e-v
(1 (1+--1)) (2)
-2
(1) (1)
V141 Vt
c
(
-1
(::
(1= E [1/4, on R+)
(2) (2)
V441 V1
Ro (2) (1)
2 ca 2111
As (2) - v1-1 = yt increasing, see Appendix D.I.I.
(2)
(1)
c
(1)
|21112 + +e
- (2
22) 0o, (t) + co. For the part that proves T/V (t) is
Notice that yV 1. Lemma D.5 implies that there exists (unique) T, > 0 such that for all f 2 To, (V) holds, or (1) (2)
(2) Vi+1
(2( Recall that the (sample) variance of finite dataset T (0), - Fn can
Ui+1) be computed as:
n-
Var
(a. x2,)²
n(n 1) i=l j=i-1
35

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
Hence, for given J, (which corresponds to the window size.) for all 1 2 T..
J-2 J(J - 1) k= 1=k+1
$ = V Var {or U/ Smm
(1)
(vi-R) ) 12
)
Uitt)
J-2
(2)
o (v )² 2
(v
J(J - 1)
Vit!
-
1=
(fo (v.²
} ++J-1
((2)
Var
T=
It is easily derived that the converse is true: If (tv(t) is increasing and t.J
V then TV (t) (t)> > 1.
We have two metrics: the first is only the variance (which corresponds to the Dyn-Unc score) and the second is the variance multiplied by the mean subtracted from I (which corresponds to the DUAL pruning score), Both the variance and the mean are calculated within a window of fixed length. At the early epoch, as the model learns 22 first, both metrics show a smaller value for 2 than that for 2/2. At the late epoch, now the model learns 21, so the order of the metric values reverses for both metrics.
Our goal is to show that the elapsed time of the second metric for the order to be reversed is shorter than that of the first metric. Let Tem be that time for our metric. We represent the mean of the logistic output within a window of length J and from epoch t, computed for i-th instance by "يjs
H+J-1
m = J Σ 0 (1/0)
(12)
Fort t 2 To, we see that the inequality still holds:
s 1 Hej
(1)
1
2-2
o (UL.. (u.))) - (1ffe)
2
1- 1- Σ σ (441)
(+J-1
J(J - 1)
k=0 1=b+1
J-2 J-1
(4J-1
0 (v(t)) - a (M )
Σ o (v191)
1 J
J(J 1) k=0 1=k+1
= (2) (1-
as for all t, (v.²²) terms for a similar analysis upon μ.;
> a (1.") ( Indeed, T Tum S Tw holds, but is Tym < T, true? To verify the question, we reshape the
$ 1 m(:
(1)
(13)
J-2 J-1
1+J-1
J[J 1)
1
(1... - - Vetk
2
j
Σ
1 - o (241)
(14)
k= 1=++1
T=t
J-2 J-1
1+J-1
>
1
Σ
Σ
(itt
(2)
- a (e+t))
2
1 j
Σ
1 - σ (1,71)
(15)
J(J 1) 1)
k= 1=
(
(1 (1))
(16)
36

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
The intuition is now clear: for any time before T.., know that the variance of #1 is smaller than that of æg. if the ratio corresponding to 1 aty) is large, the factors could be canceled out and the inequality still holds. If this case is possible, definitely Tom < T.
Now let us analyze the ratio of a vi
1-0 Vi 1-0
Lemma D.6. Yas (t)
increases to +00.
Proof.
(2)
(tx = 1 + ellt
1 + etli
A
+ ele
cau
2 AM
1 1 evll
(v())
The quantity the last line indeed diverges to infinity. We now show that 7Mr (1) is increasing.
Av
7ar(t)
Ave
1+
et
+
evt
A
11
(1)
+1
y/ (1)
-1) 0 181"
+1
< (6A00-1
1)
((tt1)
+ = (1 11.
1-0(1,")
1-o
1-o
Notice that, for <> 0, > d 0,
3.
Recall from Lemma D.5 that (t)
1-0
hence (1) (1). Moreover,
(1)
yi
11)
Vi+1
V1
(!) <
(2)
(2)
(2)
Vi+1
11
σ
Vi+1
(1) (1)
2
1641
V1
(
1
Met1
(2) (2)
1te wi
Mi+1
1/1
Mi+1 1/1
(1) (1)
# (1)
1 e Dets
1+ 12)
E (0.1).
(2)
1 é (
1
Ve+1 V/
(1) (1)
(2)
9i+1 Ut
2111 0/ Av
(2)
1 +
(2) (2)
114 V1
1 + w-
(1) (1)
1 - (2)
Mi+ 1/
-
Av
(2) (2)
1 1 1
1i+1 Vi
Re 7M (t).
37

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
Now revisit Equation (13).
2-
++J-1
(Uiti)
- (vife) (vife)
2
Σ
(
J(J - 1)
K= 1=k+1
T=t
(17)
J-2 2 J-1
++J-1
Σ
Σ
(2) Vi+t)
(vitt))
(iith)
(2)
)}
Σ
(1(2)
J(J 1) k=0 1=++
Assume, for the moment, that for some constant C 1, a (v
but ( a > Ca [1 1 (vft)) for all large t. Then the ratio of the first term of the left-hand side of Equation (17) to the first term of the right-hau hand side is greater than C² Also, the ratio of the second term of the left-hand side of Equation (17) to the second term of the right-hand side greater than C²_ If so, we observe that 1) the inequality in Equation (17) holds, 2) as the condition (t) 2 1 for T. now changed (t(()= 2 C for Tom hence Ten < T. guaranteed. It remains to find the constant C. Recall that, for all I,
Vt (1)
C-1 Jo (v - (vx (v,
1 1-o
(t) & Re (9) -bb TM (t).
(2)
If we set -ewh--w = C-3, when (t) becomes at least C-1, we have 7As (t) 2 C²2 satisfying the condition for Tem the learning rate sufficiently small, then (t) cannot significantly increase in one step, allowing (v to fall between C-1 and 1. Refer to Figure 23a to observe that the graph of y/ resembles that of a continuously increasing function.
D.I.I. MONOTONICITY OF (t(
Recall that:
(1)
yi+1
y/
(t)
(2)
(2)
1i+1
1/1
(1)
(1) o (((0)
1i+1 1/1
()) (2)
(2)
Vi+1 V1 o'
for some 50 (v1".(11)
122 ) by the mean value theorem. The first term is shown to be increasing (to R). (v( is increasing if the second term is also increasing in l.
..fli) ( E (3i²)
(2)
Let AG (.2) -. By Lemma D.4, AG
((())
(1)
21
e
((0)
e र
lte lt E
o'
1- 2
= e
1+0-5(
Define g(2, = et
The partial derivatives satisfy:
Ite-v
V V.g= (e"-e (( for x > 0ify (1ter)+
V Vog = (e 0, Vy if 0.
(1+ou)3
Notice that (
(1) (1) is (strictly) increasing and positive, if we show that g(AG,, )) Since e (vt
( Bett)
AG. is increasing in t, we are done. Our result is that, 22 vi is small for 1,2, c. ~ (vi) + vi211 /2 so AÇ (Ayt + Ay(+1) /2, which is indeed increasing.
38

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
In partieular, if assume for now) for all 1,
c E
2
(8)
M/++
(2)
Vt
(1)
2
(8)
(18)
3
3
(2)
(1)
(2)
(1)
Atft
+
1141
Vi
1/+1
Atft + y
Met1
Vi
+ A E
3
3
3
3
(2) (1)
Altt + A16+1
3/+1
9t
AG
3
3
(2) (1)
yo+1 + A41+2
Di41
96+2
(t)
3
A
(t) holds by Assumption D.I:
(t) + 41122 - Ay > 9142
(1)
y1 1I
(1)
(1) (1)
A41+1 At > Vit1
1/1
e vi ((21, x2) - ||> (1) + e -v--
(110211- (001, x2)) >
θ 1)
vi | 1| + te- vt (9)
(x)1 x2) Vt.
It remains to show Equation (18). To this end, we use Lemma D.7. Lemma D.7. Let 22 > z1(2 0) be reals and Ç E (2), be number that satisfies the following: a (22) - -0
(22 - 21) (C). Denote the midpoint of 22) as m == (#1 + 22) /2. For (1 >) > 0, if 22 21 < <0(ve) then
IC (Cml <.
Proof. Expand the Taylor series of a at m for zy:
a + o' (m) (2+ m) + 2!
o" om(mm) (2 -m)² + [mm(m) (z: m)² + O (1z: - -mm²) ml())
3!'
We have:
σ (22) a (21) o'(1n o'mm) (z2 21) +
1
9"' (m) (m)) (22 21)² + +0 (1s2 21)²)
24
o' (C) = o' d'mm) + r" (m) (22 - 21)² +0 (l2 - 21)*)
24
Now, expand the Taylor series of o' at m for C:
1
o' (c) = o')) + (m) (5 m) + o (m) (5 - mm² +0 0(( - ml
2!
Comparing the above two lines gives
(m)((m)(S m) 120" -0 m)² (m) (22 21)²+ + 0 -(22-))1)))
If (m) = 0 then 15 ml 0 (122 -
)1))) (z)))1)))
22 = ( is sufficient.
Otherwise, we can solve the above for ( - m from the fact that (²))) < for > 0:
12o""(m)( - m) = -120" (m) / (m20"(m))2 + (m) (m((m) (22 - 21)² + + 01222 (122-21)))
(m))(()²((2) 22
+ ((22 21)
2m0"(m)
39

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
The last equality is from the Taylor series VT+ 2 +0 or V + (a(xx)) have | - m/ = θ ((z2 )1
We
1 < 0 (V - v1/) This
For 14 - -v1 (v, 1 (6)
/ < (uL41 - v/")// /6, suffices to have u
generally holds for sufficiently small 7).
D.2. Experimental Results under Synthetic Setting
This section displays the figures plotted from the experiments on the synthetic dataset. We choose X = IR² and { (00.1,0.1,, 1), (10,5),11 1)}. We fix J = 10 and = 0.01 (unless specified), The total time of training T specified for each figure for neat visualization.
Weight Evolution
10000
Data points Weight
8000
8-
6000
X2(10, 5)
Epoch
4
4000
2000
(0.1, 0.1)
0
2
8
10
W1
Figure 22. Illustration of the evolution of the weight as the model learns from the woo-point dataset. Observe that the weight learns first (closer the orange dashed line), gradually moves towards æ; (closer the brown dashed line). Here 10, 000.
We also empirically validate statements of Appendix D.I.I. Figure 23 shows that (t) and Aç, are indeed increasing (6) functions. Figure 24 shows that (i) is sufficiently close to the midpoint of the interval it lies in, yi Mb411
40

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
Evolution of yit) Over Epochs
Evolution of AZ; Over Epochs
16P
107
Y/b
4
10
10-2
0-
2200
4000
6000
8000
10000
2005
4000
6000
5000 10000
Epoch
Epech
(a) TV (L) in log scale.
(b) AC
Figure 23. Empirical validations of the critical statements in Appendix D.1.1. We ran experiments and plot the results that both (v (left-in log scale) and AC, (right) are an increasing sequence terms of l. Here, we set 1) = 0.0005. The reason that if learning rate larger, ( quickly saturates to leading to possibility division by zero yV (1) and degradation in numerical stability ACe Moreover, notice that the graph of Tr (1) in the log scale closely resembles that of Aç, in the original scale.
Distance between t Midpoint
Distance between 2! and Midpoint
10-3
10-
ICP)-1yPt + +
10a4
10-3
1/.5 (//5 PP
30
10m4
10-3
10 01
10.03
2000
4000
6000
8000
10000
2000
6000
8000
10000
Tpoch
Epoch
(a) (1)
(yb + 2/2) in log scale.
(b) kk (vl + 1 in log scale,
Figure 24. Empirical validations of the critical statements in Appendix D.1.1. We ran experiments and plot the results that both c! (left) and (22) (right) extremely close the midpoint (v( (1) ) 22 and (y(²) + v/-1)/2, compared the interval length, respectively. In both plots, the blue line is the true distance while the orange line is the interval length. Here, we set r = 0.0005 for the same reasoning of Figure 23. Empirically, the noise introduced by MVT is small deny that Aç, is an increasing sequence.

---

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
We also show that can observe the "flow" of the moon plot in Figure 2 for the synthetic dataset.
Dynddne
Epech: 15
Epoeh: 100
Epech 1030
Eaceh: 10000
Epect: 10350
Metn
Ours
34
310
312
30d
Figure 25. Evolution of æ1, 22 by their mean and standard deviation in prediction probabilities different epochs. The marker "o' and 'x' stands for æt and 222 respectively. The red color indicates the sample to be selected, and blue color indicates the sample to pruned. Observe that the path that each data point draws resembles is of moon-shape. Here T = 30, 000.
42


Reviews:
[{"review_id": "BdoMXLeFzd", "reviewer": "Reviewer_3gz4", "summary": "This paper proposed a dataset pruning score named as Difficulty and Uncertainty-Aware Lightweight (DUAL). The main idea is two fold. First, it combines the data difficulty and data uncertainty into one numerical measure for pruning. This extends existing work on uncertainty based data pruning such as Dyn-Unc (He et al. 2024) by simultaneously considering both difficulty and uncertainty. Second, to mitigate the accuracy degradation on high pruning ratio, the paper proposed to use Beta distributions fitted by the DUAL score as the way to determine the likelihood to prune the data instead of directly pruning data with worse DUAL score. A theoretical analysis was conducted to prove DUAL score can prone data more efficiently compared to Dyn-Unc based pruning strategy and extensive experimental evaluation on CIFAR-10, CIFAR-100, ImageNet-1k shows DUAL not only achieves high accuracy but also needs fewer training epoches (i.e. more efficient) and more robust compared to other state-of-the-art pruning methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful and valuable feedback. We address the given concerns and questions below.\n\n\n### **1. Experiment with a more challenging dataset**\nThank you for your suggestion. Due to time constraints, we experimented on a randomly sampled 20% subset of iNaturalist 2017 on ResNet-50, achieving 30.17% test accuracy. We applied random pruning and our DUAL+Beta method to prune 50% of this subset. Selected 50% subset is re-trained from scratch, and random pruning shows 12.6% test accuracy, while our method achieved 18.07%, showing its potential on challenging datasets.\nWe additionally conducted experiments on the long-tailed CIFAR-10 and CIFAR-100, following [1]. Competitors are Random, EL2N, Dyn-Unc, and CCS for all pruning ratios. The table is attached [here](https://vo.la/UHJYgf).\n\n\n### **2. Extension to nonlinear separable cases**\nThe key insight of our analysis is that the prediction variance of uncertain samples is lower than that of easy samples at the early stage; so we use the prediction mean to diminish the score of easy ones.\nIn the non-linear case, the decision boundary is likely more complex than a hyperplane. However, the gist of our approach should still hold: samples near the decision boundary would exhibit higher variance over training epochs, while those far from it should stabilize quickly.\nOne possible extension is to analyze the behavior of the DUAL score in non-linear models, e.g., deep neural networks. One might focus on how the feature representation evolves over training and thus how uncertainty propagates. A potential approach is to generalize the method in the representation learning framework. Instead of tracking the prediction variance and mean, one could study these in an appropriate feature space, e.g., the penultimate layer of the network. Such extension is out of scope, but we consider this an interesting future direction.\n\n\n### **3. Stability of DUAL+Beta**\nWe clarify that it is different runs altogether, i.e., repeated Appendix C.2 Algorithm 1. Beta sampling selects easier samples that tend to be more typical, leading to more stable performance than selecting the most uncertain and difficult samples. This phenomenon is observed in other methods used with beta sampling (see Table 4 in our paper).\n\n\n### **4. Regarding Beta Sampling**\nWe provide a detailed explanation for beta sampling in our response to Reviewer YPaU.\n\n\n### **5. Under-performance of ResNet-50**\nWe initially used ResNet-18's training parameters due to time constraints. We retrained ResNet-50 with batch size of 256, SGD with 0.9 momentum and 5e-4 weight decay, and a 0.3 lr with 3 epochs warm-up. With these parameters, full dataset test accuracy shows 80.1%. We also add a row indicating the case from R18 to R18 [here](https://vo.la/avcZFx), reflecting your suggestion.\n\n\n### **6. Clarification of Table 4 & Figure 2.**\nSorry for the confusion. Delta value for Dyn-Unc is 25.66, and for Ours is 32.14.\nWe provide corrected Figure 2 and a detailed explanation in the response to Reviewer TH5d.\n\n\n### **7. Missed essential reference**\nThe SIM score [2] is composed of three factors: Class Separability (cluster overlap), Data Integrity ($\\ell_2$-norm of feature representations), and Model Uncertainty (prediction consistency across independent models). All factors are computed at epoch 20 using a snapshot approach. As a result, stabilizing the SIM score requires 10 independent runs, incurring high computational costs. In contrast, our method leverages training dynamics to calculate the DUAL score within a single run, requiring fewer epochs.\nSIMS also proposes a ratio-adaptive sampling strategy, applying importance weights over the original score distribution. However, it assumes a normal distribution of scores, which does not hold in practice (see Fig. 2 of [2]). In contrast, our sampling method, by not relying on any specific score distribution, remains robust across diverse datasets. We will add a citation of SIMS [2] and this comparison in the new version.\n\n### **8. Qualitative results on the pruned and kept samples**\nThanks for your suggestion. We visualize pruned and selected samples by DUAL on ImageNet-1K [here](https://vo.la/iCZkWL). DUAL tends to retain uncertain and challenging samples while discarding some typical instances and extremely difficult examples.\n\n### **Other Questions**\nQ1) First, we emphasize that DUAL prioritizes samples with high uncertainty and high difficulty. Hence, the two cases you mentioned cannot be distinguished by the DUAL score alone. However, our Beta sampling favors samples with low difficulty and high uncertainty over those with high difficulty and low uncertainty. \n\nQ3) As in our response to Review TH5d, ‘bottom-right’ is correct. \n\n---\n[1] Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss, NeurIPS 2019.\n\n[2] Data pruning via separability, integrity, and model uncertainty-aware importance sampling, ICPR. Cham: Springer Nature Switzerland 2024.", "claims_and_evidence": "1. One of the major issue of existing pruning metrics e.g. Forgetting score is that to get the pruning metrics itself require full dataset of training, which makes pruning itself become more computationally costly compared to training the image classification model itself. The proposed DUAL method showed that it can reduce the training time needed to calculate pruning score and simultaneously obtained state-of-the-art accuracy on the pruned dataset. Both theoretic and empirical evidence are provided to justify the claimed advantage of DUAL.\n2. The proposed DUAL method still suffer from severe accuracy degradation when pruning ratio is high. The paper claims that such degradation can be effectively mitigated with Beta distribution based random sampling instead of hard thresholding. This is supported by experimental evaluation where DUAL + Beta-sampling consistently improve performance with DUAL only based pruning.\n3. The proposed DUAL metric shows more robustness against label noise and image distortion. Experimental evaluation shows that DUAL method can better retain accuracy and identify noisy samples especially when pruning ratio is low (which means samples that are most likely to be pruned is also the noisy ones).\n4. The paper also demonstrated the proposed DUAL score is generalizable across different model architectures. In particular, when a simpler model (e.g. ResNet18) is used for pruning the dataset, a complex model (e.g. ResNet50) trained on the pruned dataset can still achieve good performance.", "methods_and_evaluation_criteria": "The paper utilized three different image classification datasets and conducted various experiments following the flow of pruning + classification. Mean accuracy and standard deviation are the primary metrics used to assess performance. And the metrics is measured under different pruning ratio. The paper also measured total training time vs. pruning ratio to demonstrate the efficiency of DUAL. While I believe the evaluation is comprehensive, it can be strengthened by evaluating on more challenging benchmark dataset such as iNaturalist [1], which exhibits more unbalanced and long-tail distribution of different image classes. \n\n[1] Van Horn, Grant, et al. \"The inaturalist species classification and detection dataset.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.", "theoretical_claims": "The paper provides a theoretical claim that under certain assumptions, DUAL metric takes less iterations than Uncertainty-only metric to identify the data sample that needs to be pruned. A thorough analysis is provided to prove the theorem, which is mostly sound. The claim supported empirical observation where DUAL metric can achieve higher accuracy with fewer number of training iteration. I have is the theorem assumes a linear model for classifying the data sample. I'm curious if the authors have any insights on whether the analysis can be extended to nonlinear separable cases.", "experimental_designs_or_analyses": "The paper provided a thorough analysis on the effectiveness of DUAL. This experiment includes 1) accuracy obtained on pruned dataset by different methods under different pruning ratios (accuracy); 2) data pruning on dataset with artificially introduced label noise and image distortion (robustness); 3) cross architecture generalization where pruning model and classification model is different (generalization) Ablation study is provided to justify the importance of different hyperparameters and beta distribution fitting strategy. There are a few experiment results that need further explanation.\n1. In Table 1, DUAL+\\beta sampling consistently yield smaller variance. Is the variance computed over different sampling over the same beta distribution i.e. repeated line 1398 in Appendix C.2 algorithm 1 or different pruning runs altogether i.e. repeated Appendix C.2 algorithm 1? Is there any insights on why using beta sampling yield more stable results?\n2. In Table 1 and 2, under high pruning ratio, using Beta sampling significantly improve the performance of DUAL, which is actually worse than many other methods. I'm wondering why beta sampling is so effective and if other distribution of sampling can be considered.\n3. In Table 3, it is surprising that pruning using ResNet-18 achieved better performance than pruning using ResNet-50, given the classification model is actually ResNet-50. This seems to suggest pruning using simpler model is preferable. Or there are other causes for ResNet-50 to under perform such as lack of proper selection of hyperparameters? Typically if training with full dataset, ResNet-50 should outperform ResNet-18 on CIFAR-100. So even under pruning setting, I would expect ResNet-50 pruning and classification should still be better. Perhaps another baseline where pruning and classification is done by ResNet-18 can be added. \n4. In Table 4, for CIFAR-10 the delta value in row 'Dyn-Und' and 'Ours' do not seem to be correct. Please verify. \n5. In Table 4, for both CIFAR-10 and CIFAR-100, the improvement due to beta sampling is more significant for DUAL than Dyn-Unc. Is this expected?\n6. I don't understand why each row of figure 2 has two plots. Figure 2(a) says 'score calculated at epoch T=60', so why there is another figure in the same row with T=90? Similarly, Figure 2(b) says 'score calculated at epoch T=90', why there is another figure in the same row with T=60?", "relation_to_broader_scientific_literature": "The paper has discussed and compared with a comprehensive collection of methods on data pruning. One of the main contribution of the paper is using beta-distribution based sampling to adjust pruning under high pruning ratio. Besides the BOSS paper mentioned in the paper, another paper shared similar idea is [2], which uses importance sampling as an ingredient for pruning. This paper should be compared in the paper.\n\n[2] Grosz, Steven, et al. \"Data pruning via separability, integrity, and model uncertainty-aware importance sampling.\" International Conference on Pattern Recognition. Cham: Springer Nature Switzerland, 2024.", "essential_references_not_discussed": "As mentioned in previous question, the paper SIMS [2] should be cited and compared in the paper.\n\n[2] Grosz, Steven, et al. \"Data pruning via separability, integrity, and model uncertainty-aware importance sampling.\" International Conference on Pattern Recognition. Cham: Springer Nature Switzerland, 2024.", "other_strengths_and_weaknesses": "[Other strengths]\n+ The DUAL score showing strength under noisy label and corrupted images is particularly encouraging especially under low pruning rate as it is non-uncommon to have small amount of polluted data samples in dataset so that we can use DUAL to prune a small number of samples and better \n+ Figure 4 is a powerful demonstration of the efficiency of the proposed method as it considers all the time needed from pruning to training classifier. And under 30% pruning ratio, there is still about 15% reduction of total time spent. This is a significant data point I do not see in many other data pruning literature. \n+ State-of-the-art accuracy achieved on CIFAR-10, CIFAR-100, and ImageNet-1K data pruning task. \n+ A theoretical analysis that is also supported by empirical evidence on the efficiency of the pruning metrics.\n\n[Other weaknesses]\n- There is no qualitative results on the pruned and kept samples, which can provide some intuition on the data pruned and kept by DUAL. \n- Two important ingredients: uncertainty and beta-sampling are heavily inspired by prior work. The uncertainty part uses the same formulation as Dyn-Unc. This may have put a dent on the overall novelty of the pruning metric. \n- The dataset evaluated are small-scale. As suggested earlier, a medium/large scale and challenging dataset should be considered for demonstrating the effectiveness of DUAL.", "other_comments_or_suggestions": "Typos:\n- line 407 left column, the text says 'we vary it from 3 to 7', however, the figure 6 only shows results from 3 to 6.\n- line 381 right column, 'asses' should be 'assess'.", "questions_for_authors": "1. Since DUAL score use multiplication of the two measure i.e. difficulty and uncertainty, how will DUAL metric handle high difficulty and low uncertainty, or low difficulty and high uncertainty samples? In general, do we expect difficulty and uncertainty always correlate with each other?\n2. Have the authors considered other ways to construct the beta distribution? Is there any insights on why the proposed form may be better than other design?\n3. In line 146 right column, should 'bottom-right' be 'bottom-left'?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "plI4XFymwu", "reviewer": "Reviewer_YpaU", "summary": "- This paper introduces a new method Difficulty and Uncertainty Aware Lightweight (DUAL) that combines Dyn-Unc with a measure of prediction confidence over training.\n- The authors further introduce pruning-ratio-adaptive Beta sampling, which boosts performance at all pruning ratios and particularly helps at very high pruning rates\n- The authors find that DUAL+Beta outperform existing pruning methods particularly at high pruning rates.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review and insightful suggestion.\n\nBefore we address your concerns regarding our beta sampling method, we would like to emphasize the novelty of the DUAL score lies in its time efficiency. Many existing pruning techniques require full training to estimate example difficulties, which makes the pruning process more expensive than training on the original dataset. Our DUAL can identify important samples more quickly, reducing the total time cost to less than that of a single training run. Beta sampling is applied to mitigate performance drops at high pruning ratios by selecting a larger portion of easier samples.\n\n### **Details for Beta Sampling Design**\n\n- **The choice of Beta PDF.**\n    \n    The domain of Beta distribution is [0, 1], which naturally aligns with the range of prediction means. As the PDF decreases at the tails, ensuring that samples with extreme scores have a negligible chance of being selected. While other distributions, e.g. Gaussian, could also be used for modeling, their support is $\\mathbb{R}$, which means they can assign a non-negligible probability to values far outside the desired range unless the standard deviation becomes extremely small.\n    \n- **The choice for hyperparameters of the Beta Distribution.**\n    \n    When the pruning ratio is set to 0, $\\alpha$ and $\\beta$ is configured so that the mean of the Beta distribution ($\\frac{\\alpha}{\\alpha + \\beta}$) aligns with the prediction mean of the highest-scoring sample. This helps target high score samples at low pruning ratios.\n    \n    To include easier samples at the high pruning ratios, we set parameters $\\alpha$ and $\\beta$ to depend on the pruning ratio. While BOSS changes them so that the Beta distribution's mode scales linearly with the pruning ratio $r$, we employ a non-linear scaling by raising $r$ to the power of $c_D$. This approach creates a PDF that is almost stationary at low pruning ratios and moves to the easier region polynomially as the pruning ratio increases.\n    \n    The hyperparameter $c_D$ is chosen based on the relative complexity of the dataset. We assume that the larger the dataset and the more samples per class there are, the easier the whole dataset is. Higher $c_D$ decreases $\\beta$ and thus increases the mean and decreases the variance of Beta distribution. For a more difficult dataset, easy data has to be sampled more, so $c_D$ should be larger (refer to Fig. 15 of App. C).\n    \n    Recall that the variance of the Beta function is given by $\\frac{\\alpha \\beta}{(\\alpha + \\beta)^2 (\\alpha + \\beta + 1)}$. According to our definition of the constant $C$ in Equation (5), where $C=\\alpha+\\beta$, increasing $C$ leads to a lower variance. This results in a more focused sampling distribution that concentrates on a specific region, improving the effectiveness of the sampling process by reducing unnecessary spread. The impact of the $C$ value is illustrated [here](https://vo.la/TftlTS).\n    \n    In conclusion, we fix a moderate value of $C$ as 15 across all experiments, and tune only $c_D$ based on the guidelines outlined above, which is not extensive. Furthermore, as demonstrated in Section 4.4.1 of our paper, the choice of hyperparameters including $c_D$ remain robust across a wide range of values.\n    \n\n### **Comparison with Dyn-Unc + Beta Sampling**\n\nFirst, we clarify that the performance gain by adding a beta sampling is higher for DUAL, not for Dyn-Unc. Table 4 should be corrected as Dyn-Unc+beta (+25.66) and DUAL+beta (+31.51) for CIFAR-10.\n\nBeta sampling applied to Dyn-Unc utilizes the prediction mean at epoch 200 where Dyn-Unc is computed. Here, all samples are almost learned thus their prediction means become concentrated to 1. Therefore, Dyn-Unc with Beta sampling selects a much more easier samples, resulting in an inferior performance compared to DUAL.\n\nWe illustrate selected samples [here](https://vo.la/NGuzVi). This figure clearly shows that Dyn-Unc+Beta actually selects easier samples. Furthermore, we test the Dyn-Unc+Beta for all pruning ratios and confirm that it is consistently worse than DUAL+Beta. The performance table can be found [here](https://vo.la/TAvedH).\n\n### **Correlation between DUAL and D2, CCS**\n\nWe appreciate this insightful suggestion. We visualize the selected subset by each method and examine the intersection [here](https://vo.la/BShVWs). CCS applies AUM with stratified sampling and a hard cutoff (pruning ratio, cutoff ratio): (70%, 20%), (80%, 40%), (90%, 50%). This approach completely excludes the most difficult samples while always retaining the easiest ones. D2 uses the forgetting score for graph-based sampling, applying the same hard cutoff as CCS. However, since it removes samples with the highest forgetting scores, it fails to eliminate the most difficult ones. DUAL selects uncertain and difficult samples, thereby excluding the hardest ones while simultaneously expanding coverage toward easier samples using Beta sampling.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "- The authors did a thorough job of comparing their method to previous data pruning methods on the relevant benchmarks.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, the evaluation method looks consistent with prior data pruning work.", "relation_to_broader_scientific_literature": "The authors discuss prior work well and evaluate their methods against prior methods.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "- Overall I think this is a strong contribution that is clearly explained and evaluated.\n- My main criticism is that the authors seem to emphasize the contribution of DUAL more than the contribution of the beta sampling, but looking at the performance numbers adding beta sampling to Dyn-Unc makes up most of the performance gain compared to adding the extra term that changes Dyn-Unc to DUAL. I think the authors should add Dyn-Unc + beta as a row in table 1 so that this is more clear. That result is present in the beta section and the appendix, but you should make it more clear what the benefit of each component is separately in table. I also feel in some parts of the text the importance of the beta sampling is a little understated.", "other_comments_or_suggestions": "None", "questions_for_authors": "- How were the constants of the beta distribution determined? Is there an automatic way to do this or did you empirically try many possibilities?\n- Have you looked at how correlated the particular examples selected by DUAL+beta at high pruning rates are with the examples selected by existing (but more complicated) methods like D2 and CCS? I think it would be interesting to add to your conclusion/discussion if you think these metrics are all getting at something similar from different angles or if selected example sets are more disjoint.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kAq0Fkx2C5", "reviewer": "Reviewer_TH5d", "summary": "This paper presents a dataset pruning method designed to reduce the computational burden of the pruning process. The authors introduce a strategy that leverages both difficulty and prediction uncertainty to efficiently select a coreset at an early stage of training. The effectiveness of the approach is validated across multiple benchmark datasets. However, the novelty of the method is somewhat limited, and the comparative experiments are not entirely comprehensive.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and insightful comments. Below, we address the concerns and clarify any confusion raised.\n\n# 1. Figures 2 & 3\n\nFirst, we apologize for any confusion in Figures 2 and 3. Revised figures are available [here](https://vo.la/msMJZE) (see Figure2_revised, Figure3_revised). In both figures:\n\n- The Y-axis is the mean at epoch $T$: $mean_T(\\mathbf{x},y)\\coloneqq\\bar{\\mathbb{P}}=\\frac{1}{T}\\sum_{t=1}^T\\mathbb{P}_t(y\\mid\\mathbf{x})$\n- The X-axis is standard deviation at epoch $T$: $std_T(\\mathbf{x},y)\\coloneqq\\sqrt{\\frac{\\sum_{t=1}^T[\\mathbb{P}_t(y\\mid\\mathbf{x})-\\bar{\\mathbb{P}}]^2}{T-1}}$.\n\nIn Figure 2, we mislabeled the colorbars. Both the labels are revised as ‘Dyn-Unc Score’.\n\nFigure 2a shows that if we calculate the Dyn-Unc score at epoch 60, the data points with the highest scores tend to evolve toward the **top-right region** by the end of training (epoch 90). However, this is not desirable for Dyn-Unc, as it prioritizes uncertain points (rightmost region) **at the end of training** (Figure 2b right). To effectively target that region **earlier** in training (e.g., at epoch 60), we should instead focus on the **bottom-right region**, since those points eventually move to the rightmost area by epoch 90.\n\nHence, we modify the Dyn-Unc score by multiplying it with 1-prediction mean, which helps us better sample this target region at an earlier stage, as shown in Figure 3. Also, the bold outline and the colorbar label of Figure 3 should be corrected to indicate **Epoch 60**.\n\nAgain, we sincerely apologize for the confusion this may have caused.\n\n# 2. Computational Efficiency in ImageNet\n\nFirst, we would like to clarify that CCS and TDDS require full training (90 epochs) to compute their scores on ImageNet. This is explicitly mentioned in Section 5.2 of TDDS [1] and in Appendix B of CCS [2]. The 60 epochs refer to the “post-pruning” training used for coreset, as described in [1]. For a fair comparison, we train 300,000 iterations on coresets, following the setup of CCS.\n\nFor further validation, we implemented CCS with AUM score calculated at epoch 60. As shown in the table below, the test accuracy is significantly lower than the original performance. Moreover, at low pruning ratios, the performance is even worse than random pruning (see Table 2 in our paper).\n\n| Pruning Ratio | 30% | 50% | 70% | 80% | 90% |\n| --- | --- | --- | --- | --- | --- |\n| CCS (ep 60) | 70.97 | 69.88 | 66.27 | 63.50 | 56.79 |\n\nTherefore we can assert that our method is both time-efficient and superior compared to CCS. Also, we want to emphasize that saving 30 epochs of full ImageNet dataset training is a significant improvement. Training for 90 epochs on NVIDIA A6000 takes approximately 20h, **thus reduction of 30 epochs can save about 6-7h.** \n\nThere are a few subtle points for TDDS. In TDDS, **the epochs used for full dataset training is 90, then they perform an exhaustive search** to determine that around 30 epochs is optimal for pruning ImageNet by 70-90%. We evaluated their method across all pruning ratios using the reported score computation epoch. As a result, TDDS has a shorter overall training time when we ignore the exhaustive search; however, the important thing is that both their reported results and our reproduced experiments show that **our method significantly outperforms TDDS** in test accuracy.\n\n# 3. Regarding Figure 4.\n\nWe can explicitly compare other baselines, such as CCS and TDDS, with DUAL in terms of total time spent as shown in Figure 4. However, many of the baseline plots are omitted because these methods require full training on the original dataset, which leads to **excessive overlap in the curves**. We will clarify this point more explicitly in the revision. We also plotted the test accuracy against the total time [here](https://vo.la/OeGesx), which hopefully addresses your concerns.\n\nWhile we could also create a version of Figure 4 using ImageNet to compare our effectiveness with other SOTA methods such as CCS and D2, we present the CIFAR results in Figure 4 as it highlights the improvement achieved by our method.\n\n# 4. Novelty of DUAL\n\nThe key novelty of DUAL is its time efficiency. Unlike most pruning methods that require full training to estimate sample difficulty, DUAL identifies informative samples much earlier. To address performance drops at high pruning ratios, we use Beta sampling to include easier samples. While the goal aligns with CCS and BOSS, our approach is distinct—it leverages prediction mean and a non-linear shift in the distribution mode. This allows us to prioritize harder samples at low or medium pruning ratios and easier ones at high ratios, contributing to our SOTA performance.\n\n# 5. Minor comments\n\nThank you for pointing out the typo. We will fix them in the new version.\n\n---\n\n[1] Spanning training progress: Temporal dual-depth scoring (tdds) for enhanced dataset pruning, CVPR 2024.\n\n[2] Coverage-centric coreset selection for high pruning rates, ICLR 2023.", "claims_and_evidence": "The main claim in Section 3.2 is unclear and confusing.\n\nThe authors attempt to compare the differences in the “Dyn-Unc score” at epochs 60 and 90. However, inconsistencies between the subcaptions of Figure 2, the main caption, and the color bars make it difficult to interpret their intended meaning.\n\nAdditionally, the authors do not specify how the mean and standard deviation are calculated—whether these values are computed at epoch 60 or 90 remains unclear.\n\nMoreover, some claims in this section appear to be incorrect. Specifically, the authors argue that to use the Dyn-Unc score at epoch 60 for selecting samples with high Dyn-Unc scores at epoch 90, they should target the samples in the bottom-right region. However, according to Figure 2, the correct region should be the **top-right**, not the bottom-right.", "methods_and_evaluation_criteria": "The proposed method does not actually reduce the computational cost of pruning, as it still requires training the original ImageNet for 60 epochs—equivalent to the full training process of (SOTA) methods like TDDS and CCS. This limitation likely explains why the authors only provide comparative results (e.g., Figure 4) on CIFAR rather than ImageNet.\n\nFurthermore, in Figure 4, the authors do not compare their method with all relevant baselines. The results appear to be selectively presented, raising concerns about the completeness of the comparison.", "theoretical_claims": "yes.", "experimental_designs_or_analyses": "As mentioned earlier, the results regarding computational load are incomplete. The paper does not provide a thorough comparison of training costs with other pruning methods, especially on large-scale datasets like ImageNet. Without a direct comparison to all baseline methods in terms of computational efficiency, the claim that the proposed approach reduces pruning costs remains unsubstantiated.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The novelty of this work is limited, as it primarily offers a small improvement over the existing Dyn-Unc method. Additionally, the effectiveness in reducing computational load is not significant, as the proposed approach still requires extensive training (e.g., 60 epochs on ImageNet), which is comparable to the full training process of existing SOTA pruning methods like TDDS and CCS.", "other_comments_or_suggestions": "There are some typos and misleading annotation in figures.\n\nFor example, in **Figure 2**, the caption and description are inconsistent, making it difficult to understand the intended comparison of the **Dyn-Unc score** at **Epoch 60 and 90**. \n\nAdditionally, in **Figure 4**, the **y-axis label** “total subset training time” is misleading and should be corrected to **“total time spent”**.", "questions_for_authors": "Please refer to the comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tf9bhFtybL", "reviewer": "Reviewer_rkgP", "summary": "This paper aims to pruning datasets in the early stages of training, without the need to train on the entire dataset. To achieve this, the authors propose a new scoring metric - the DUAL Score, which simultaneously considers sample difficulty and prediction uncertainty. To address potential sample distribution bias under extreme pruning rates, they introduce a Beta distribution-based adaptive sampling method for pruning ratios, ensuring that the selected subset better represents the overall data distribution.\n\n# **update after rebuttal**\nI would like to thank the authors for their sincere efforts to address my concerns. I have increase my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their valuable and constructive feedback. Below, we address the concerns raised.\n\n# Comparison with Dynamic Data Pruning Methods\n\nThank you for pointing out the relevant references, especially [1, 2]. To address your concerns, we conducted several experiments comparing recent dynamic pruning methods with static approaches, including DUAL pruning.\n\nBefore discussing the experimental results, let us first highlight two major differences between static data pruning and dynamic data pruning.\n\n1. Compared to static pruning, dynamic pruning maintains access to the entire original dataset throughout training, **allowing it to fully leverage all available information in the original dataset**.\n2. While both aim to improve training efficiency, their underlying goals differ slightly. Static data pruning seeks to identify a “fixed” subset that reduces the dataset size while preserving as much information about the original dataset as possible. This subset can then serve as a new, independent dataset, reusable across various model architectures and experimental setups. In contrast, dynamic data pruning enhances training efficiency within a **single training session** by pruning data dynamically on the fly. However, this approach requires storing the entire original dataset, making dynamic pruning less memory-efficient and not reusable.\n\n## Standard Training\n\nWe conducted experiments on CIFAR-10, 100 with ResNet-18, following the same hyperparameters as in Section 4.1 of our paper. All reported results are averaged over five runs. We first tested dynamic random pruning, which dynamically prunes randomly selected samples from the entire dataset at each epoch. Notably, dynamic random pruning significantly outperformed all static baselines, achieving test accuracies of 91.82% on CIFAR-10 and 72.8% on CIFAR-100 at a pruning ratio of 90%. We also evaluated [1, 2] and the results are provided [here](https://vo.la/XAAkQb). Overall, dynamic methods consistently outperform static baselines. However, at lower pruning ratios (e.g., on CIFAR-10), DUAL can outperform dynamic methods under a similar computational budget.\n\nAs mentioned, we believe this performance gap stems from **differences in accessible information**: static methods are limited to 10% of the data, while dynamic methods use the full dataset. Consequently, the performance gap widens even further at aggressive pruning ratios, contrary to the reviewer’s expectations. To validate this, we plotted how often each sample was seen during training. The plot [here](https://vo.la/jhIWIZ) shows that static methods are confined to a subset, while dynamic ones use nearly all data—rendering direct comparison somewhat unfair. Indeed, dynamic pruning methods might be better compared with scheduled batch-selection approaches, such as curriculum learning, rather than static pruning methods.\n\n## Label Noise Setting\n\nWe also evaluated these methods under label noise. In fact, [2] concluded that **their method cannot prune any samples (corrupted or not) when label noise is introduced**. Similarly, [1] tends to retain harder (and often noisy) samples, as it removes only easy examples during training. In contrast, DUAL effectively filters noisy samples, improving performance even beyond full-data training. \n\nWe conducted experiments on CIFAR-100 with a 40% label noise setting (full-train test accuracy: 52.74%) to verify this explanation. DUAL achieves **over 70% test accuracy** at a 50% pruning ratio, whereas InfoBatch achieves only 51.24% accuracy with a similar number of iterations. Under similar iterations, random dynamic pruning achieves 51.81% test accuracy, which still outperforms random static pruning (see Table 7 in Appendix). Lastly, IES [2] prunes only 1.7% of samples during training (consistent with the original report in their paper), resulting in 51.95% test accuracy. Furthermore, our static method can create fixed subsets in which **nearly all noisy samples have been removed, resulting in high-quality datasets that can be preserved for future use.**\n\n# Minor Comments\n\n- For details on the Beta sampling method and its hyperparameters, please refer to the response to Reviewer YpaU.\n- As suggested, we plotted test accuracy against total training time. The plots are available [here](https://vo.la/OeGesx). Results show that DUAL efficiently prunes data while achieving SOTA performance.\n\nWe will include these findings and additional discussions in the revised version.\n\n---\n\n[1] InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning, ICLR 2024.\n\n[2] Instance-dependent Early Stopping, ICLR 2025.", "claims_and_evidence": "In general, I agree with the authors' proposed motivation that \"Many existing methods require training a model with a full dataset over a large number of epochs before being able to prune the dataset, which ironically makes the pruning process more expensive than just training the model on the entire dataset.\"\n\nThe majority of the claims made in the submission are supported by clear and convincing evidence. The authors provide extensive empirical results across multiple benchmark datasets—CIFAR-10, CIFAR-100, and ImageNet-1K—as well as under challenging conditions like label noise, image corruption, and cross-architecture scenarios. Moreover, the theoretical analysis (although it appears somewhat supplementary, which I don't consider problematic) reinforces the claim that the DUAL Score can identify critical samples earlier than methods based solely on prediction uncertainty. \nHowever, while the evidence is largely compelling, a few claims could benefit from additional validation. I have provided my further recommendations in the following review.", "methods_and_evaluation_criteria": "The DUAL Score combines example difficulty with prediction uncertainty, allowing for early identification of influential samples without requiring a full training cycle. Additionally, the adaptive Beta Sampling strategy is designed to dynamically balance the selection of difficult, informative samples and easier ones, ensuring that the pruned subset remains representative of the overall data distribution even at high pruning ratios.\n\nThe evaluation criteria employed in the study are both robust and appropriate for the application. By leveraging standard benchmark datasets such as CIFAR-10, CIFAR-100, and ImageNet-1K, the authors provide a comprehensive assessment of their approach across a range of complexities and scales. The inclusion of realistic scenarios—such as experiments under label noise, image corruption, and cross-architecture generalization—further underscores the practical relevance and robustness of the method.", "theoretical_claims": "This paper is empirical research and does not make significant theoretical claims. I have reviewed what could be considered the theoretical component of the paper - the Theorem for DUAL Pruning in Appendix D. This section does not present any novel theoretical contributions; rather, the authors combine two previously explored aspects - Difficulty and Uncertainty - to facilitate efficient and effective selection.\nFurthermore, I have reservations about the Beta Sampling method proposed by the authors, as it seems to introduce an excessive number of hyperparameters that could be difficult to control (although the authors can demonstrate these hyperparameters are not sensitive).\n\nHowever, I want to emphasize that the lacking of theoretical analysis is NOT a shortcoming. I do not look the paper require novel theoretical contributions or theoretical guarantees.", "experimental_designs_or_analyses": "Generally speaking, the authors have done a good job. However, it's worth noting that I believe they should provide a graph plotting test accuracy against method runtime (with wall-time on the x-axis), to clearly demonstrate the performance differences between various methods in terms of wall-time, thereby locating the contribution of the DUAL method. \n\nIn my view, this setting provides the most intuitive representation of each method's capability to reduce computational overhead during pruning. When implementing this setting, the authors should also include [1] InfoBatch and [2] IES. The authors can compare the methods under both high and low pruning rate scenarios, selecting representative methods from Table 1 for this experiment.\n\n[1] InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning, ICLR 2024.\n\n[2] Instance-dependent Early Stopping, ICLR 2025.", "relation_to_broader_scientific_literature": "Many existing methods require training a model with a full dataset over a large number of epochs before being able to prune the dataset, which ironically makes the pruning process more expensive than just training the model on the entire dataset. Key research includes Data Pruning/Core Set studies by:\n\nHuggins et al. (2016) Coresets for scalable...\n\nPaul et al. (2021) Finding important examples early in training\n\nKrishnateja et al. (2021) ``Glister''\n\nXia et al. (2022, 2024) on moderate and refined core set approaches", "essential_references_not_discussed": "See Other Strengths And Weaknesses.", "other_strengths_and_weaknesses": "My primary concern is that the authors only discuss recent work on static dataset pruning. Despite some differences in settings, I strongly recommend comparing the proposed method with [1] InfoBatch (ICLR'24 Oral) and [2] Instance-dependent Early Stopping (IES) (ICLR'25 Spotlight). These are important works in dataset pruning that shouldn't be excluded from the baselines. Both works have straightforward code implementations and methodologies, making fair comparisons feasible. Given the rebuttal timeline, I believe it would be sufficient to include results on the two CIFAR datasets in Table 1.\n\nIn the field of dataset pruning, my general perspective is that for high pruning rates, pre-training pruning (similar to coreset) is preferable, while for low pruning rates, Dynamic Data Pruning is the better choice. If the authors' proposed method underperforms compared to the aforementioned methods at low pruning rates, that's acceptable - I suspect DUAL would achieve superior performance at high (or extremely high) pruning rates. The authors can confidently note that these Dynamic Data Pruning methods become ineffective at high removal rates, denoting such cases with \"-\" in their results.\n\nI commend the paper's clear organization and the authors' substantial effort on evaluation. In particular, I appreciate their focus on addressing an important challenge in the field: the computational inefficiency of many existing coreset selection methods. This is indeed a significant problem that warrants attention.\n\n[1] InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning, ICLR 2024.\n\n[2] Instance-dependent Early Stopping, ICLR 2025.", "other_comments_or_suggestions": "See Other Strengths And Weaknesses", "questions_for_authors": "Not applicable.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a lightweight dataset pruning framework based on the proposed DUAL score, which combines example difficulty and prediction uncertainty to identify informative samples early in training, significantly reducing computational overhead compared to existing approaches that require full training cycles. The authors provide both theoretical justification and extensive empirical validations, showing that DUAL matches the state-of-the-art accuracy under various conditions—including label noise, image corruption, and cross-architecture generalization.  While the core ideas are rooted in extensions of earlier work such as Dyn-Unc, the integration and practical improvements are notable. I recommend it for acceptance.
Decision: Accept (poster)