Medium-Difficulty Samples Constitute Smoothed Decision Boundary for Knowledge Distillation on Pruned Datasets
Authors: Yudong Chen
Xuwei Xu
Frank de Hoog
Jiajun Liu
Sen Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Knowledge distillation
dataset pruning
image recognition
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper proposes a static dataset pruning method for efficient knowledge distillation.

Abstract: This paper tackles a new problem of dataset pruning for Knowledge Distillation (KD), from a fresh perspective of Decision Boundary (DB) preservation and drifts. Existing dataset pruning methods generally assume that the post-pruning DB formed by the selected samples can be well-captured by future networks that use those samples for training. Therefore, they tend to preserve hard samples since hard samples are closer to the DB and better characterize the nuances in the distribution of the entire dataset. However, in KD, the limited learning capacity from the student network leads to imperfect preservation of the teacher's feature distribution, resulting in the drift of DB in the student space. Specifically, hard samples worsen such drifts as they are difficult for the student to learn, creating a situation where the student's DB can drift deeper into other classes and make incorrect classifications. Motivated by these findings, our method selects medium-difficulty samples for KD-based dataset pruning. We show that these samples constitute a smoothed version of the teacher's DB and are easier for the student to learn, obtaining a general feature distribution preservation for a class of samples and reasonable DB between different classes for the student. In addition, to reduce the distributional shift due to dataset pruning, we leverage the class-wise distributional information of the teacher's outputs to reshape the logits of the preserved samples. Experiments show that the proposed static pruning method can even perform better than the state-of-the-art dynamic pruning method which needs access to the entire dataset. In addition, our method halves the training times of KD and improves the student's accuracy by 0.4% on ImageNet with a 50% keep ratio. When the ratio further increases to 70%, our method achieves higher accuracy over the vanilla KD while reducing the training times by 30%. Code is available at https://github.com/chenyd7/MDSLR.

Published as a conference paper at ICLR 2025
MEDIUM-DIFFICULTY SAMPLES CONSTITUTE SMOOTHED DECISION BOUNDARY FOR KNOWLEDGE DISTILLATION ON PRUNED DATASETS
Yudong Chen Xuwei Xu Frank de Hoog" Jiajun Liu
Sen Wang"*
The University of Queensland CCIRO Data61
{yudong. chen, xuwei... xu, sen. w edu. au {jiajun. 1iu, frank. dehoog)@csiro..e
ABSTRACT
This paper tackles new problem of dataset pruning for Knowledge Distilla- tion (KD), from fresh perspective of Decision Boundary (DB) preservation and drifts. Existing dataset pruning methods generally assume that the post-pruning DB formed by the selected samples can be well-captured by future networks that use those samples for training. Therefore, they tend to preserve hard samples since hard samples are closer the DB and better characterize the nuances in the dis- tribution of the entire dataset. However, in KD, the limited learning capacity from the student network leads to imperfect preservation of the teacher's 's feature distri- bution, resulting in the drift of DB in the student space. Specifically, hard samples worsen such drifts as they are difficult for the student to learn, creating a situa- tion where the student's 's DB can drift deeper into other classes and make incorrect classifications. Motivated by these findings, our method selects medium-difficulty samples for KD-based dataset pruning. We show that these samples constitute a smoothed version of the teacher's DB and are easier for the student to learn, obtaining general feature distribution preservation for class of samples and reasonable DB between different classes for the student. In addition, to reduce the distributional shift due to dataset pruning, we leverage the class distribu- tional information of the teacher's outputs to reshape the logits of the preserved samples. Experiments show that the proposed static pruning method can even per- form better than the state-of-the-art dynamic pruning method which needs access to the entire dataset. addition, our method halves the training times of KD and improves the student's accuracy by 0.4% on ImageNet with a 50% keep ratio. When the ratio further increases to 70%, our method achieves higher accuracy over the vanilla KD while reducing the training times by 30%. Code is available at htttps://githhb com/ cwtncch /MDSLR
INTRODUCTION
Deep Neural Networks (DNNs) (Krizhevsky et al., 2012: Szegedy et al. 2015 He et al. 2016 Liu et al., 2022) have dominated the computer vision field in the past decade as an effective feature extraction tool. One of the most important factors the success of DNNs the large number of parameters. For instance, the early GoogLeNet (Szegedy et al.) 2015) utilizes approximately 6.6 million parameters to obtain 69.77% top-1 accuracy on ImageNet (Russakovsky et al. 20155) classification, and the recently proposed ConvNeXt (Liu et al. (022) with 197.8 million parameters increases the accuracy to 84.41%. Although large networks are powerful, they also significantly increase computation costs for inference, which limits their deployments in real-world applications.
To mitigate this problem, Knowledge Distillation (KD) (Hinton et al. 2015) is proposed to en- hance the performance of lightweight networks for efficient inference. The mechanism of KD is to utilize the knowledge hidden the teachers (large networks) to supervise the training of students (lightweight networks). Inspired by the promising performance of KD, series of extensions have
"Corresponding Author

---

Published as a conference paper at ICLR 2025
been proposed by exploiting knowledge in different layers of the teacher network (Ji et al. 2021
Chen et al. 2022 or designing more effective distillation loss (Tian et al. 2020 Zhao et al. 2022).
Although distillation-ben based training can greatly improve lightweight networks performance, the training complexity is high due to the addition of the teacher and the usage of large-scale datasets.
There are several methods designed reduce the training cost of KD. For instance, self-distillation proposes to replace the teacher's 8 knowledge with the latent knowledge hidden in the student itself (Sun et al.] 2019 Zhang et al. 2019 Liang et al., 2022; Yang al. 2023). Although the training times are reduced by discarding the cumbersome teacher network, the improvement of the student's accuracy is limited due to the intrinsic performance gap between the student and the teacher. An- other solution for KD acceleration is to reduce the frequency of forward propagation (Shen & Xing 2022 Beyer et al., 2022). A typical way is to store the knowledge generated by the pre-trained teacher in advance. However, since the technique of data augmentation plays an important role in KD, the fixed knowledge tends to be less effective as shown in (Beyer et al.) 2022). Recently, dy- namic dataset pruning methods have shown great potential in speeding up training (Li et al. 2022 Truong et al, 2023 Qin et al. 2024). The main idea of dynamic dataset pruning is to skip the less informative samples for forward propagation during training based on the loss values from the pre- vious epoch. By focusing more on learning those hard samples, the training cost can be reduced and the distillation performance can be maintained a large extent (Li et al. 2022). However, dynamic data pruning methods still need to access the entire dataset. As the sizes of training datasets con- stantly increase in the field of computer vision, the I/O operations for large-scale datasets will stress resource-constranng devices (Dryden et al. 2021 Nguyen et al. 20022).
To reduce the training cost and the size of datasets simultaneously, this paper explores how to per- form static dataset pruning (Welling 2009; TTneva et al., 2019) Paul et al. 202T) given a pre-trained teacher network. Specifically, static pruning decides which samples should be preserved at the be- ginning of training and does not need to store the entire dataset. However, existing static pruning methods are less effective in KD scenarios as illustrated in the recent research (Ben-Baruch et al., 2024) and Table 5 This is because these methods are not tailored for distillation and neglect the capacity gap between the student and the teacher. In general, static pruning methods tend to pre- serve hard samples for training
they are more informative and closer to the decision boundary. However, we observe the drift of decision boundary in distillation since the student is unable to fully mimic the teacher's feature distribution of hard samples, leading the overlap of features of different classes. Based on this observation, we propose preserving medium- -iffiuuudiffiuutty samples for distillation they form a smoothed decision boundary of the original distribution and are easier to learn. In addition, to mitigate the distributional shift due to the lack of a large portion of samples, we record the average distribution information of the teacher's predictions for subsequent distilla- tion. By reshaping the logits of the preserved samples with the recorded information, the student's s performance can be further improved. In summary, the contributions of this paper include:
We investigate the effects of using different types of samples for distillation from the per- spective of decision boundary. To avoid the drift of decision boundary, we propose pre- serving the
samples for training.
To mitigate the distributional shift due to dataset pruning, we use the average logit distri- bution information of the teacher to reshape the logits of preserved samples and modify the distillation loss for the training on a pruned dataset.
Experimental results demonstrate that our method can even perform better than the state- of-the-art dynamic pruning method. In addition, when using 70% of samples, our method achieves higher accuracy over the vanilla KD while redueing the training times by 30%.
2 RELATED WORK
Efficient knowledge distillation. The original KD and its extensions generally require a pre-trai teacher network to generate soft labels (Hinton et al., 2015 Zhao et al. (022) or intermediate fea- tures (Ji et al. 2021 Chen et al. 2022 to guide the student's 's training, which is computationally intensive due to the forward propagation of the teacher. In contrast, self-distillation aims to obtain knowledge without using large teacher network. For instance, some methods add multiple classi- fiers to the student network to generate soft labels from the shallow features (Sun et al. 2019) Zhang et al. 2019). Since the additional branches inevitably introduce more parameters, the computation
2

---

Published as a conference paper at ICLR 2025
costs are still high. Recent self- methods have been proposed to discover the knowledge hidden in the student itself. Zipf's LS (Liang et al. 2022) reuses the classifier of the student to per- the feature maps before the pooling layer and obtains soft labels by ranking the frequency of the predicted class in the dense prediction. Universal Self-Knowledge Distillation (USKD) (Yang et al., (203) further divides the distillation loss into non-target and target objectives based on Zipf's LS. Although Zipf"s LS and USKD significantly reduce the training complexity of self-distillation, the distilled students are less discriminative than those using the knowledge from the large teacher. Since the teacher's 's knowledge is indispensable, another line of efficient KD is to reduce the times of forward propagation. An intuitive idea is store the knowledge (e.g., soft labels) of different samples generated by the teacher before distillation as the weights of the teacher are fixed during training. However, this not only increases the costs for storing the fixed knowledge but also degrades the effectiveness of KD due to the lack of data augmentation and the inconsistency between the inputs of the student and the teacher at different epochs (Beyer et al. 2022). Therefore, this paper focuses on pruning the less important training samples for efficient distillation.
form dense prediction
Dataset pruning. Most recently, dynamic dataset pruning methods have shown promising perfor- mance by selectively feeding the samples to the networks for optimization based on the loss from the previous epoch (Truong et al. 2023] Qin et al. 2024). The rationale behind dynamic pruning meth- ods is that samples with larger losses
more informative for the student learn. By eliminating the less important samples for forward propagation, the computation costs can be greatly reduced and the overall performance is surprisingly competitive (Qin et al.) (022). A drawback of dynamic pruning methods is that they need to access the entire dataset. As the sizes of datasets constantly increase in the trend of Scaling Laws (Alabdulmohsin et al. (022). the I/O overhead of accessing data is likely to be a bottleneck in training a model (Dryden et al. 2022 Nguyen et al. (022). There- fore, it is desirable to develop static pruning methods for efficient training. The difference between static and dynamic pruning is that the static methods prune samples in advance and only access the pruned dataset during training. Similarly, existing static pruning methods are prone to preserve hard samples. For example, Forgetting (Toneva et al., 2019) selects samples that are harder to memorize and Error L2-Norm (EL2N) (Paul et al.. (0221) preserves samples that have larger discrepancies with the corresponding ground-truth labels. Since these static methods do not consider the capacity gap between the student and the teacher, their effectiveness in KD is degraded.
METHODOLOGY
In this paper, we aim to provide insights for practitioners that directly use the pre-trained models (e.g., downloading model weights from the repository of Pytoreh") to distil a student and try to prune the dataset using the prior knowledge from the pre-trained teacher without introducing too much computation overhead. The study on static dataset pruning for KD is limited. For instance, Ye| et al. (2024) conducts few-shot classification experiment based on the KD framework. However, the focus of (Ye et al., 2024) is to train an appropriate teacher for the student's distillation, which is orthogonal to the goal in this paper. The recent research (Ben- (Ben-Baruch et al.) 2024) proposes to adaptively adjust the sppee-pprameters of distillation and classification losses according to the size of the pruned dataset. However, BBen-Baruch et al. (2024) suggests randomly pruning samples in KD scenarios. On the contrary, we propose to utilize medium-difficuly samples for distillation and achieve better performance. Our pruning strategy is similar to Moderate Dataset Selection (MoDS) (Xia et al. 2023), which aims at improving the robustness and generalization of dataset pruning by preserving the samples that are closer the median distances the corresponding class centres. Different from MoDS, we investigate the effects of using medium samples from the perspective of distillation- based training. In addition, we design a logit reshaping method for the improvement of distillation on pruned data as illustrated in the following subsections.
3.1 PRELIMINARIES
The vanilla KD uses the teacher's logits to guide the student's training. In this paper, the teacher's logit and the student's logit of a certain sample are denoted by p Rex1 and E Rex1, respectively, where e is the number of classes. After obtaining the logits, the distillation loss used to train the
3

---

Published as a conference paper at ICLR 2025
(a) Full
(b) Hard
(c) Easy
Figure 1: Decision boundaries of (a) full samples, (b) hard samples, and (c) easy samples selected by the pre-trained network ResNet50 on CIFAR-100. Different shapes denote samples of different classes and the selected samples are visualized with darker colours. Note that hard samples better preserve the original decision boundary, covering the general distribution of the entire dataset.
student as follows:
exp(ps/u) log exp(z//u) E, cxp(p//p) E, exp(2;/1))
LKL =
where p: and 2; are the i-th element of p and 2, respectively, and μ is the temperature coefficient. By combining the distillation loss with the cross-entropy loss LoR, the total loss for the student's optimization as follows:
LKD aLCE + (11-a)a aLLKL,
(2)
where is hyper-parameter used to trade off the two losses. As shown in Equation 2 the distillation-base training requires pre-trained teacher network to generate logits for supervision, which will significantly increase computation costs. To mitigate this problem, a feasible way is to reduce the number training samples. In KD scenarios, networks downloaded from different open-source repositories for distillation. Therefore, this paper aims to directly prune the dataset according to the predictions of the pre-trained teacher without re- training (Toneva al., (019) introducing an additional proxy network (Coleman al. 2020) that the computation overhead for dataset pruning can be reduced. Existing dataset pruning methods score the difficulties of training samples by using different measurements (Paul et al. 2021 Qin et al. 2024). For example, EL2N obtains the score of training sample by computing the L2 dis- tance between the network's predictions and the one-hot label vectors. Similarly, this paper scores the samples by measuring the discrepancy between the network's predictions and the one-hot label vectors with the cross-entropy loss since we conduct experiments on the image classification task, which generally adopts the cross-entropy loss for optimization. Given pre-trained teacher network, the difficulties of training samples are defined as follows:
common to use the pre-trained teacher
exp(p:)
MDIF
Yi log Σ, exp((p;)
a one-hot vector constructed from the ground-truth label. We rank the difficulty of training samples based the values of MDIF and obtain index vector d = {di d2..... di....,dy d....... E 1 where n is the total number of training samples and d, is the index of the training sample with the i-th largest loss. Samples with larger losses, i.e., d (d). d2,..., di,.., drn) E rRr are regarded as hard samples and samples with smaller losses, i.e., d {dn-rn+1, dn-rn+2 ..., da-rn+i- d,} grax1 are considered as easy samples, where 0 < r < 1 is the ratio of the preserved samples and we assume rn is an integer for simplicity.
where y E Rex1
3.2 SMOOTHED DECISION BOUNDARY FROM MEDIUM-DIFFICLLYY SAMPLES
Most the existing dataset pruning methods tend to preserve hard samples Toneva et al. 2019
Paul al. 2021) for training hard samples are informative while few methods find the easy
4

---

Published as a conference paper at ICLR 2025
(a) Hard (CKL 2.42)
(b) Easy (Cxr. 1.37)
(c) Medium (CxI. 1.84)
Decision boundaries of (a) hard samples, (b) easy samples, and (c) medium samples selected by the teacher ResNet50 on the feature space of the distilled student MobileNetV2 CIFAR- 100. is shown that the decision boundary of hard samples drifts into other classes and the decision boundary of easy samples is too small to cover the general distribution of the dataset. Medium-difficulty samples obtain a balance between these situations.
Figure 2:
samples beneficial (Welling 2009). To compare the difference between hard and easy samples selected by the teacher network, we visualize their decision boundaries as shown in Figure [ In these visualizations, we first map the deep features generated by the pre-trained ResNet50 (He al. 2016) into a two-dimensional space by using t-SNE (Van der Maaten & Hinton 2008) and then train classifier with the selected samples to generate the decision boundary. Figure demonstrates that the decision boundary obtained by hard samples is closer to that generated by the original full data. Since the feature distribution of easy samples is relatively concentrated, its decision boundary retreats too much from the original, which may affect the network's generalization ability. Figure [ suggests that preserving hard samples is beneficial to dataset pruning. However, we find that the performance of existing static pruning methods is far from satisfactory in distillation-based training. We hypothesize this because student network is unable to learn the exact feature distribution of the teacher due the large capacity gap, leading to the deviation of the decision boundary.
To verify this hypothesis, we first compare the training ac- curacy of the distilled student MobileNetV2 (Sandler et al. 2018) and the pre-trained teacher ResNet50. In Higure 3 select three subsets from the training set based the
Top TTrnning Arcurery (%)
teacher's predictions, including 30% of hard samples, 30%
80-
of medium samples, and 30% of easy samples. is shown
that there is a large gap between the student and the teacher regarding the training accuracies of hard samples. This is because hard samples are close to the decision bound- ary of the corresponding class and tend to be misclassified. Since the student is unable to mimic the teacher exactly, the hard samples selected by the teacher may not be suit- able for the student's 's training. As shown in Figure 2(a) some of the hard samples belonging to the "circle" class indistinguishable from the "triangle" class. As sult, the decision boundary formed by the hard samples of the "circle" class involves several samples belonging the "triangle" class, leading to incorrect classification. On the other hand, although easy samples are casier learn ac- cording to the lower distillation loss, the range of the deci- sion boundary of easy samples is small shown in Fig- ure 2(b) Therefore, a large portion of data points are not included, which will affect the net- work's generalization ability. The above analysis indicates that neither hard nor easy samples are suitable for distillation to obtain accurate and generalizable decision boundaries for different classes. Motivated by these observations, we propose preserving the mediun-difficulty samples, ie., d (dr "..., d[ d denim 1 -1) E grax1 for distillation on pruned dataset and em-
70-
Hard
Medium
60-
Basy
Hard
Medium
Easy
40
120
160
200
240
Epochs
Figure 3: Top-1 training accuracy of different types of samples CIFAR- 100. The red lines represent the results obtained by the pre-trained ResNer50 and the blue lines represent the results obtained by the distilled MobileNetV2.
5

---

Published as a conference paper at ICLR 2025
Reehsping Reahaping
wi Hesisaping Reahapisa
20
20-
20-
15
10
10-
Valus
10-
20
40 Callgorrees
100
60
20
80
100
Categories
Chttggottes
(a) Under-confident logits
(b) Over-confident logits
Mean logits
Figure 4: Reshaping logits of the preserved samples using the class-wise distribution information of the entire dataset. Given (a) under-confident logit vector (b) an over-confident logit vector of the preserved sample, we reshape it to better fit the distribution of (c) the mean logit vector of the corresponding class on the full dataset.
pirically find that such dataset pruning strategy work well in different teacher-student combinations. Similarly, we visualize the decision boundary learned by the medium-difficulty samples in Figure 2(c) It is shown that the medium-difficulty samples constitute a smoothed decision boundary and are easier to learn than hard samples. In addition, the -iffuumdiificully samples obtain trade-off' between underfitting and overfitting on the training set compared to the hard and easy samples as illustrated Figure Therefore, the student's generalizability on the testing set is better by using medium-difficulty samples for distillation as shown in Section 4.1
3.3
MITIGATING BIAS OF THE PRUNED DATA
Although the distillation on a pruned dataset can be improved by using medium-difficulty samples, it is still challenging achieve lossless performance due the information loss of large portion of pruned samples. To mitigate this problem, we record the global distribution information of the teacher's ' predictions to reshape the logits of the selected samples. Specifically, we obtain the class- wise logit information of the i-th class as follows:
Σ my MSTD(P'>)
K
(4)
mi
where m, is the number of samples of the i-th class, MsTD(.) is used to compute the standard deviation of a logit vector and p² is the logit vector of the j-th samples in i-th class. Since the recorded information & is a e-dimensional vector, it barely increases the storage overhead. After obtaining the average logit information, we use it to reshape the logits of the preserved samples and rewrite the distillation loss as follows:
ExT. - Σ explp./u)
log exp(z1/u)
(5)
E, exp(5/p) Σ, exp(2,/1)
and kip the average logit information of the class that p belongs to. The idea of KD) is to use the soft labels generated by the teacher to characterize the latent between- class information of the dataset. Since the pruned dataset only preserves a portion of samples, the teacher may be biased regarding the relationships between classes. By using the global distribution information to reshape the logits of the preserved samples, the distributional shift of an incomplete dataset can be mitigated. We Figure to illustrate the effect of logit reshaping.
where i = t-pkp//sSDDD
4 EXPERIMENTS
This section compares the effectiveness of different dataset pruning methods for distillation. Several sencher-student combinations are introduced to verify the generalization abilities of these methods on different datasets. Experimental details are listed follows.

---

Published as a conference paper at ICLR 2025
Baselines. Existing static dataset pruning methods such as Herding (Welling 2009), Forgetting (Toneva et al. 2019), EL2N (Paul et al. 2021) and MoDS (Xia et al. 2023) are inc ludedde for compar- ison. Different from these methods that not designed for KD, Ben Ben-Barccc Baruch et al. (2024) proposes
to randomly prune samples for distillation and adaptively tune the hyper- parameearseeters of the distilla-
tion loss according the pruning ratio. We compare the method in BarucBa et al. (024) in
experiments, which is denoted by Adaptive. Following Adaptive, we randomly prune samples from the dataset in class-balanced manner form baseline method Random. For fair comparison, the proposed method also selects the medium-diiie difficulty samples evenly from each class. In addition, a recently proposed dynamic pruning method InfoBatch (Qin et al.) (024) is also included.
Datasets. The main experiments are conducted on two benchmark datasets, i.c., CIFAR-100
Krrzzevskyy al. 2009) and ImageNet (Russakovsky et al. (015). CIFAR-100 contains 60,000 images in total belonging to 100 classes. The dataset is divided into training set with 50,000 im- ages and a testing set with 10,000 images. ImageNet a larger dataset collecting images from 1,000 classes. It contains 1,281, 167 training images and 50,000 testing images. The images on CIFAR-100 and ImageNet are resized to 32 x 32 and 224 x 224 pixels, respectively.
Teacher-student combinations. We evaluate the generalization ability of different pruning methods by using different network combinations. The architectures of teacher networks include ResNet34 (He et al. 2 ResNet50 (He et al. 2016), VGG13 (Simonyan & Zisserman 2014). WRN-40-2 Zagoruyko ko & Komodakis) 2016) and ResNet32x4 (He et al., 2016). The architectures of student
networks include MobileNet (Howard et al.) 2017). MobileNetV2 (Sandler et al., (2088), VGG8
(Simonyan & Zisserman
2014). ResNet8x4 (He et al. 2016) and WRN-16-2 (Zagoruyko & Ko-
modakis 2016).
Training setup. We follow the training scheme of a popular open-source distillation repository (Tin al. 2020) to set up the learning rate, learning rate decay rate, training epochs, batch size, weight decay rate and optimizer. On CIFAR-100, 100, the hyper-parametee a is set to 0.1. For Adaptive, we set a 0.1, a 0.15, a 0.2 and a 0.5 when the keep ratio r is 30%, 40%, 50% and 70%, respectively. On ImageNet, we set the hyper-parameters to 0.5 and 0.9 for the cross-entropy loss and the distillation loss, respectively. experiments. The results are obtained by using an NVIDIA V100 GPU.
addition, the temperature coefficient is set to 4 in all
4.1 ABLATION STUDIES
We first investigate the distillation perfor- mance with samples selected from d" d".
Table 1: Top-1 accuracy-sstandard deviation (%) on CIFAR-100 with ResNet50-MobileNett using dif- ferent data pruning strategies. Reshape denotes re- shaping logits with the average information.
d Table lists the top-1 accuracy of dif- ferent students distilled with different types of samples. The results on CIFAR- 100 are averaged over 3 trials and the standard de- viation is also reported. As shown in the ta- ble, since hard samples are difficult for the student to learn, the student distilled with hard samples underperforms those distilled with random samples in a low keep ratio. However, as the ratio increases, the situa- tion is different. This is because the student has more training iterations learn the teacher's feature distribution with a higher keep ratio. On the other hand, the teacher's knowledge of easy samples is easy to learn and helps the student obtain better performance in a low keep ratio compared to using random samples. However, as shown in Figures aand 2 the decision boundary formed by easy samples is limited and can only cover a small portion of features, which degrades the generalization ability of the student. By selecting medium- difficulty samples, the proposed method obtains better trade-off between sample difficulty and the network's 's generalization ability achieving the best performance with different keep ratios of data. In addition, by using the logit reshaping technique, the student's accuracy can be further increased.
Strategy Random
r == 40%
== 70%
55.95
63.89+1.03
Hard
54.17=0.67
64.9910.29
Easy
56.20+1..56
63.5710.27
Medium
57.59+00.66
65.59+0.40
Medium+Reshape
58.0440074
66.18-0.11
7

---

Published as a conference paper at ICLR 2025
4.2 MORE JUSTIFICATION OF USING MEDIUM-DIFHICUME SAMPLES
Apart from comparing the classification accuracy, we investigate the effects of using different types of samples for distillation from the perspectives of gradient variation and mutual information. Specifically, we compute the 1.2-norm of the gradient of the last layer by using different samples for training and obtain the standard deviation of the gradient magnitudes over different training epochs. In this way, we can analyze how different samples affect the network's learning. In addition, we measure the alignment between decision boundaries formed by the student and the teacher by using mutual information.
As shown in Table 2 the standard deviation of gra- dients of hard samples is lower, leading to more stable training process. However, the mutual infor- mation of hard samples is also lower, which indi- cates the student's s decision boundaries of hard sam- ples do not well align with that of the teacher. Al- though the mutual information is high in the case of easy samples, the training process is unstable in terms of the standard deviation of gradient magni-
Table 2: Standard Deviation (SD) of gradi- ents and Mutual Information (MI) of decision boundaries on CIFAR- 100 with ResNet50- MobileNetV2 using different strategies.
Strategy SD
MI
Hard
0.018
0.046
Easy
0.030
0.054
Medium
0.022
0.051
tudes, which may result in overfitting and affect the generalizability of the student. Overall, medium- difficulty samples obtain a trade-off between standard deviation and mutual information.
4.3 COMPARISON OF DIFFERENT LOGIT RESHAPING METHODS
The technique of logit reshaping has been well-studied in the existing KD methods. For instance, Spherical Knowledge Distillation (SKD) (Guo et al.] 2020) proposes to align the magnitude of the student logit with that of the teacher logit. NormKD (Chi et al.) 2023) rescale student logits and teacher log- its with their corresponding standard devia- tions. Logit Standardization in Knowledge Dis- tillation (LSKD) (Sun et al. 2024) further re- shape the logit to zero mean vector based NormKD. The technical details of existing logit
Table 3: Top-1 uccuracy=standarr deviation (%) on CIFAR-I00 with ResNec50-MobbleNett2 us- ing different logit reshaping methods.
Method r 40%
T= 70%
Medium
57.59-0.06
65.5910.40
SKD
56.40=0.41
64.94:10.27
NormKD
55.99=0.93 65.55+1031
LSKD
56.25=0.91
65.06=10.50
Ours
58.04 0.74
66.18
reshaping methods are described in Appendix(A.3 Different from these logit reshaping methods that focus on aligning the smoothness of student and teacher logits, we aim to align the distribution the preserved teacher logits with the global distribution information for distillation on pruned dataset. As shown in Table [3 existing logit reshaping methods degrade the distillation performance with different keep ratios. The proposed method improves the student's accuracy by clear margin.
4.4 DISCUSSIONS ON CIFAR-100
Logit distillation. Figure compares the proposed method with the existing static dataset pruning methods on CIFAR- 100 with different keep ratios. In these experiments, CesNet50-MooileNettt and ResNet50- VGG8 are used to test the teacher and the student that have different network archi- tectures and VGGI3-VGG8 is used to evaluate the teacher and the student that has similar network architectures. More combinations are given in Appendix A.4 Experimental results show that ex- isting methods, such as Herding, Forgetting, and EL2N underperform the other methods in most cases since they tend to select easy or hard samples for training. Random and Adaptive obtain bet- ter performance by approximately evenly selecting easy, medium and hard samples from datasets. By comparison, MoDS and the proposed method focus more on the medium -iifuuddiffiiultyy samples and achieve consistent improvements for different keep ratios. The differences between our method and MoDS are two-fold. Firstly, we introduce a logit reshaping method to convey the global information of the teacher's 's predictions to mitigate the gradient bias of pruned dataset. Secondly, we select medium-difficul samples in a class-balance manner to avoid classification bias in the teacher. Therefore, our method consistently performs better than MoDS as shown Figure
8

---

Published as a conference paper at ICLR 2025
74
65
72
Top Accurary (6)
70
60
67.5
688
Herding
Random
05.0
Random
Herding
Herdin
Random
E12N
Forgetting
82.5
BL2N
Fongetting
Tupe1
Forgetting
EL2N
MoDS
MoD5
MaDS
Adagttvn
57.5
Adagatov
60
40
70%
50%
40%
30%
55.0
100%
30%
100%
70%
50%
40%
30%
Keep Natto
Keep laato
Keep Ratio
(a) ResNecS0-MobilNeeW2
(b) ResNet50-VGGS
(c) VGGI3-VGGS
Figure 5: Top-1 accuracy of different static data pruning methods on CIFAR-100 datasets with different keep ratios. The proposed method consistently outperforms the existing pruning methods under different teacher-student combinations and network architectures.
Table Top-1 accuracy (%) on ImageNet with different teacher-student pairs using different data pruning strategies. InfoBatch" is dynamic pruning method, which needs to store the entire training set during distillation.
Pair
ResNaa34-MobileNet
ResNei5O-MobileNa
Keep Ratio
30%
50%
r 70%
30%
50%
70%
Random
67.25
69.39
70.20
66.84
68.9611.69
69.91
Herding
61.9618.76
66.58
68.88
60.7619.89
66.30
68.82
EL.2N
64.7016.02
69. 19.14
10.57.0.15
63.4777.18
68.4772.18
70. 70.1640.49
Moderate
66.79
69.05 11.67
69.93
66.4614.19
68.7241.93
69.75
InfoBatch"
67. 67.43
69.91 10.81
70.47
66.95 (3.70
69.83 jo.82
70.46
Ours
67.91
69.98 1o.74
n0.20
67.67 12.98
69.97 10.68
70.93 10.28
Full Dataset w/o KD
69.57
69.57
Full Dataset w/ KD
70.72
70.65
Feature distillation. As representative extension the vanilla KD, feature dis- tillation methods aim to distil the knowl- edge hidden in the intermediate layers of the teacher network. Table 4 evaluates the performance of different dataset prun- ing methods for feature distillation. In this experiment, we use the method in (Chen) al.) (022) to perform feature distilla- tion. Similar to the results of logit distilla- tion, selecting medium- -diffucuuit samples beneficial to the feature distillation pruned dataset. The comparison between MoDS and
Table 4: Top-1 accuracy (%) CIFAR-100 with
DesNet50-Mobileeett using feature distillation.
Strategy
40%
70%
Random
50.98
66.04:1..0
Herding
55.42-10.40
65.03
Forgetting
50,4110.33
1.33
EL2N
54.41=1.22
65.95=0.554
MoDS
57.61
66.0610.36
Ours (w/o Reshape)
58.1140.611
66.46
the proposed method further demonstrates the importance class-balanced sampling in distillation.
4.5 DISCUSSIONS ON IMAGENET
Performance comparisons. We further verify the effectiveness of the proposed method on larger dataset ImageNet. Table 5 shows the top-1 accuracies of different methods with teacher-student pairs ResNer34-MobileNerr and ResNettO--MobileNet, Experimental results show that the proposed method consistently outperforms the other static pruning methods under different ratios of preserved samples. addition, the proposed method achieves lossless accuracy by solely using 70% of the training data. This is because the large-scale dataset contains more redundant samples. Therefore, is necessary to develop effective dataset pruning methods to identify useful samples for training efficiency improvement. Apart from static pruning methods, recently proposed dynamic pruning method InfoBatch also included for comparison in Table 5 Since InfoBatch prunes different numbers of samples different epochs, in the original paper, report the top-1 accuracies of

---

Published as a conference paper at ICLR 2025
different keep ratios by matching the number of forward propagation during training. In addition, InfoBatch tends to preserve hard samples for training, which will also lead to the drift of the decision boundary. Although the proposed method does not access the entire dataset during training, it still obtains competitive performance compared to InfoBatch.
Training complexity. Since the proposed method di- rectly utilizes the pre-trained teacher to rank the diffi- culties of training samples, the overhead for data selec- tion can be significantly reduced. Therefore, we in- vestigate how the pruned dataset affects the training time and accuracy of distillation. Figure g displays the curves of testing accuracy versus training time us- ing the teacher-student pair ResNec34-MobileNh using 50% of training samples, the proposed method outperforms the student without using distillation by 0.41% in terms of top-1 accuracy. Besides, the train- ing time of distillation can be halved by using dataset pruning. On the other hand, the student trained on a full dataset requires one and half times of training time compared to the distillation using 50% of train- ing data. In addition, by using the proposed method to facilitate logit distillation on the pruned datasets, the distilled student converges faster with different keep ratios compared to the training on a full dataset.
TIE
Top-1 Accuracy (%)
10
By
x0. 50
20% (Quz))
20
10
Training Time
Figure 6: Top-1 accuracy and training time on ImageNet with different keep ratios.
Comparisons with self-knowledge distillation. Self-knowledge distillation is another technical route for acceler- ating the training of KD. Therefore, we compare our method with several recently proposed self-knowledge dis- tillation methods in Table 6 The top-1 accuracies of Zipf's i LS and USKD are cited from (Yang et al.) 2023). Since
Table Top-1 accuracy and training time on ImageNet of self-knowledee distillation and the proposed method with ResNet34-MobileNet using 70% of samples.
Baseline
Zipf s LS
USKD
Ours
Ace (%)
69.57
69.59
70.38
70.92
Time (h)
39.86
>39.86
>39.86
36.98
the self-knowledge distillation methods need to use the entire dataset and additional distillation losses for training. their computation complexities will be slightly higher than the standard training procedure (i.e., Baseline). However, the proposed method can significantly reduce the training cost by using 70% of training data. In terms of the student's accuracy, the improvement of self-knowledge distillation methods is restricted due to the lack of the teacher's knowledge. By eliminating less important samples, the proposed method can speed up the training process and benefit from the supervision of the teacher. In addition, the performance of our method can be further improved by equipping with a more advanced distillation loss as shown in Appendix A.7
CONCLUSION
This paper proposes dataset pruning method with logit reshaping accelerate the training of KD. Specifically, we observe that the teacher's 's knowledge of hard samples is difficult for the student to learn, which will lead deviation of the decision boundary. Therefore, we suggest preserving medium-difficulty samples for dataset pruning so that students can achieve smooth and discrimina- tive decision boundaries between classes. addition, to mitigate the gradient bias caused by dataset pruming. an improved distillation loss is proposed by utilizing the global information of the teacher's s predictions to reshape the logits of the preserved samples. Experiments on different datasets with multiple teacher-student pairs verify the effectiveness and generalizability of the proposed method.
6 ACKNOWLEDGMENTS
This work is supported by Australian Research Council DP230101753 and CSIRO's Research Plus Science Leader Project R-91559.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Ibrahim Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. In NeurlPS, 2022.
Emanuel Ben-Baruch, Adam Botach, Igor Kviatkovsky. Manoj Aggarwal, and Gérard Medioni. Distilling the knowledge data pruning. arXiv preprint arXiv:2403. 077544, 2024.
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher patient and consistent. In CVPR, pp. 10925-10934, 2022.
Yudong Chen, Sen Wang, Jiajun Liu, Xuwei Xu, Frank de Hoog, and Zi Huang. Improved feature distillation via projector ensemble. In NeurIPS, pp. 12084-12095, 2022.
Zhihao Chi, Tu Zheng, Hengjia Li, Zheng Yang, Boxi Wu, Binbin Lin, and Deng Cai. Normkd: Normalized logits for knowledge distillation. arXiv preprint 20Xi::2308.00500, 2023.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In ICLR, 2020.
Nikoli Dryden, Roman Böhringer, Tal Ben-Nun, and Torsten Hoefler. Clairvoyant prefetching for distributed machine learning i/o. In Proceedings of the International Conference for High Perfor- mance Computing, Networking, Storage and Analysis, pp. 1-15, 2021.
Jia Guo, Minghao Chen, Yao Hu, Chen Zhu, Xiaofei He, and Deng Cai. Reducing the teacher- student gap via spherical knowledge disitllation. arXiv preprint arXi::2010.07483, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint wwXiv:1503.0253: 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 2017.
Mingi Ji, Byeongho Heo, and Sungrae Park. Show, attend and distill: Knowledge distillation via attention-based feature matching. In AAAI, pp. 7945-7952, 2021.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine-grained categorization. In ICCVW, pp. 554-561, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurIPS, volume 25, 2012.
Chenxin Li, Mingbao Lin, Zhiyuan Ding, Nie Lin, Yihong Zhuang, Yue Huang, Xinghao Ding, and Liujuan Cao. Knowledge condensation distillation. In ECCV, pp. 19-35, 2022.
Jiajun Liang, Linze Li, Zhaodong Bing, Borui Zhao, Yao Tang, Bo Lin, and Haoqiang Fan. Efficient one pass self-distillation with zipf's ' label smoothing. In ECCV, pp. 104-119. Springer, 2022.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer. Trevor Darrell, and Saining Xie. A convnet for the 2020s. arXiv preprint 2tti:2200003545, 2022.
Truong Thao Nguyen, François Trahay, Jens Domke, Aleksandr Drozd, Emil Vatai, Jianwei Liao, Mohamed Wahib, and Balazs Gerofi. Why globally re-shuffle? revisiting data shuffling in large scale deep learning. In IPDPS, pp. 1085-1096, 2022.

---

Published as a conference paper at ICLR 2025
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning a data diet: Finding important examples early in training. In NewrIPS, pp. 20596-206600 2021.
Ziheng Qin, Kai Wang, Zangwei Zheng, Jianyang Gu, Xiangyu Peng, Zhao Pan, Daquan Zhou, Lei Shang. Baigui Sun, Xuansong Xie, and Yang You. Infobatch: Lossless training speed up by unbiased dynamic data pruning. In ICLR, 2024.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-255 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, pp. 4510 45200,4500440 2018.
Zhiqiang Shen and Eric Xing. fast knowledge distillation framework for visual recognition. In ECCV, pp. 673-690, 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXivv1409.155 2014.
Dawei Sun, Anbang Yao, Aojun Zhou, and Hao Zhao. Deeply-supervised knowledge synergy. In CVPR. pp. 6997-7006, 2019.
Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang. and Xiaochun Cao. Logit standardization in knowledge distillation. In CVPR, pp- 15731-15740, 2024.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, pp. 1-9, 2015.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. ICLR, 2019.
Thao Nguyen Truong, Balazs Gerofi, Edgar Josafat Martinez-Noriega, François Trahay, and Mo- hamed Wahib. KAKURENBO: Adaptively hiding samples in deep neural network training. In NeurlPS, 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 911), 2008.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-201 dataset. Technical Report CNS-TR 7RR20011-00) 2011.
Max Welling. Herding dynamical weights to learn. In ICML., pp- 1121-1128. 2009.
Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In ICLR, 2023.
Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. From knowledge distillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels. In ICCV, pp. 17185-17194, 2023.
Linfeng Ye, Shayan Mohajer Hamidi, Renhao Tan, and EN-HUI YANG. Bayes conditional distri- bution estimation for knowledge distillation based on conditional mutual information. In ICLR, 2024.
Sergey Zagoruyko and Nikos Komodakis. 1rXiv:1605.07116. 2016.
Wide residual networks.
arXiv
preprint
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In ICCV, pp. 3713-3722, 2019.
Borui Zhao, Quan Cui, Renjie Song. Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In CVPR, pp. 11953-11 11962, 2022.
12

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 IMPLEMENTATION DETAILS
Following CRD Tian et al. 2020), we set the batch size, training epochs, and weight decay rate to 64/256, 240/100, and 0.0005/0.0001 on CIFAR 10/mmeee/// The initial learning rate is 0.01 for MobileNetV2 and 0.05 for the other students and multiplied by 0.1 at 150, 180, and 210 epochs on CIFAR-100. On ImageNet, the initial learning rate is 0.1 and is multiplied by 0.1 at 30, 60, and 90 epochs. In addition, the commonly used data augmentation techniques, e.g., random crop and horizontal flip utilized for training. The optimizer is Stochastic Gradient Descent (SGD) with momentum of 0.9. The feature distillation loss we use in Table 4 is as follows:
f(s) -FD= - - [[) 114|22
LFD
where and & the student and teacher feature vectors of certain sample, respectively, and (s) is a function used to align the feature dimensions of the student and teacher with the projector ensemble mechanism. The hyper-parameters of the cross-entropy loss, the distillation loss and the number of projectors are set to 1, 25 and 3, respectively, as in the code released by the authors"
A.2 DETAILS OF REPRODUCING BASELINES OF DATASET PRUNING
In the experiments, we reproduce existing dataset pruning methods by using the codes released by the authors or implement the codes according the descriptions the corresponding paper. The technical details of different baselines are as follows:
Random (Ben-Baruch Baruch et al. 2024): We evenly prune samples from different classes to achieve class-balanced sampling. Ben-Baruch al. (2024) demonstrates such dataset pruning strategy can outperform existing methods in KD.
Herding (Welling) 2009): We preserve samples with deeper features generated by the pre- trained teacher that are closer to the corresponding class centres.
Forgetting (Toneva et al. 2019): We use the code released by the author to re-train the teacher network and monitor the forgetting events of samples. Samples that are harder to memorize (i.e., the number of forgetting events is higher) during training will be preserved.
EL2N Paul et al. 2021): The original EL2N compares the discrepancy between the ground-truth labels and the predictions of networks trained with a few epochs. In our experiments, we use the pre-trained teacher to compute the EL2N scores. Samples with larger scores are preserved for distillation.
MoDS (Xia et al., 2023): We the code released by the authors to select samples. Specifically, samples with deeper features generated by the pre-trained teacher that are closer to the median distance of the corresponding class centres are preserved.
Adaptive (Ben- et al. 2024): as Random.
This baseline adopts the same dataset pruning strategy
InfoBatch (Qin et al. 2024): We use the code released by the author dynamically prune data during distillation. Samples that have larger LKD values will be preserved at the next training epochs. In addition, InfoBatch re-scales the loss according to the data pruning ratio for further improvement.
A.3 TECHNICAL DETAILS OF EXISTING LOGIT RESHAPING METHODS
Let and denote the teacher and the student logit vectors, the logit reshaping methods normalize the logit vectors follows:
hhuts:/ggthab.coo//cheey//// forgetting
13

---

Published as a conference paper at ICLR 2025
p p, (P) p
MMSTD(z)
NormKD: p= aMSTD aMSTD(2)
(8)
p
2-2
LSKD: p= MMSD MSTD(Z)
Ours: p - MMST((p) ==z
(10)
where p is the mean of p. is hyper-parameter. Unlike the previous methods that aim to align the scale of the student logits with those of the teacher logits, the proposed method focuses on aligning the teacher logits with their corresponding class-wise information, mitigating the distribution shift due to dataset pruning. Therefore, the proposed method is significantly different from the previous logit roshappinng methods. In addition, we can integrate these methods one framework, i.e., p 3.p and 2 = 222, where 31 and 32 are different coefficients for scaling. To apply different logit reshap- ing methods, we just need to set the corresponding coefficients (e.g., 33 kip (P), 1).
A.4 MORE COMBINATIONS ON CIFAR-100
In this subsection, we introduce more teacher-student combinations for evaluation shown in Tables to 10 Combinations WRN-40-2WRRN-16-2 and ResNet32x4-ResNetBxx share similar network architectures while combinations WVRN4402-VGGG) and ResNer32x4-VGGr have differ- these experiments, we perform logit distillation between the stu-
ent network architectures. dent and the teacher with the keep ratios of samples ranging from [30%, 40%, 50%, 70%]. Ex- perimental results in these tables demonstrate that the proposed method performs better than the other static dataset pruning methods in most ferent methods with about nine different teacher-student pairs. We use the ratio of parameters of teacher-student pairs roughly quantify their learning capability discrepancy (i.e., parame- ters of student network/parameters of teacher network). The ratios of these pairs range from [0.02, 0.08, 0.16, 0.2, ..., 0.87]. involving teacher and student networks with different learning pabilities. The diversity of the teacher-student pairs and the corresponding results demonstrate the generalizability of the proposed method.
Overall, evaluate the effectiveness of dif-
Table Top-1 accuracy-tstandart deviation (%) on CIFAR-100 dataset using teacher-student pair with similar network architectures.
Teacher-Student Pair
WRN-40-2-RRN-162
Strategy / Ratio
30%
40%
50%
r= 70%
Full Dataset
Random
67.7310.07
69.6911.00
71.13:0.37
73.2910.25
Herding
61.93++0.25
65.52+4..2
68.3740.26
71.99+0.15
EL.2N
64.79+40.59
68.23+0.39
70.53+0.05
73.1240.12
74.82+0021
Moderate
67.8710.23
70.4010.07
71.59:10.31
73.6310.11
Adaptive 67.73:10.07
70.0410.19
71.11:10.30
72.9
Ours
0.55
7.056
72.33+0020
73.81+00.00
Table Top-1 standac deviation (%) on CIFAR-100 dataset using teacher-student pair with similar network architectures.
Pair
Sirategy Ratio
30%
r 4040%
r 50%
r= 70%
Full Dataset
Random
63.6410.36
66.47-0.07
68.4510.05
71.0810.34
Herding
59.69+40.16
63.46+0.21
66.58++0.04
70.57+0.21
E1.2N
61.71+0.27
65.65+0.37
68.48++0.17
71.37+-0.13
73.24+10.23
Moderate
65.73+1..20
68.04-10.31
69.80++0.0
71.67+0.33
Adaptive
63.64
66.46-10.88
68.2440.22
70.92+0.15
Ours
65.46400.18
68.33110.23
69.80-10.14
72.77-0.13
14

---

Published as a conference paper at ICLR 2025
Table Top-1 accuracy deviation (%) on CIFAR- dataset using teacher-studen pair with dissimilar network architectures.
Peacher-Sttdent Pair
RGsN
Strategy Ratio
r 30%
T= 40%
50%
r 70%
Full Dataset
Random
61.40:10.14
64.50=0.22
66.8120.18
69.8410.06
Herding
57.91=0.225
62.13=0.40
65.42:10.40
69.6811.12
F1.2N
58.74-+0.48
63.10=0.17
66.044+0.48
69.75++0.22
72.50+0.19
Moderate
62.79++0.32
65.66+0029
67.69+00118
70.51+0.18
Adaptive
61.40-t0.14
64.15=0.17
66.30++0.32
69.54410.33
Ours
63.07:0.222
66.0911.49
68.1810.08
70.97=0.24
Table 10: Top-1 accuracy standard deviation (%) on CIFAR-100 dataset using a teacher-student pair with dissimilar network architectures.
Teacher- Studeet--tuentttt Pair
WRN-40-2-VGG8
Strategy Ratio
r 30%
r 40%
50%
r r= 70%
Full Dataset
Random
64.06::0.18
66.63=0.21
68.49:10.26
71.30=0.20
Herding
59.52:t0.08
63.55=0.24
66.70=10.08
70.67=0.23
EL2N
60.08:t0.34
64.49=0.23
67.28=
70.56=0.42
73.06:10.43
Moderate
14.10:::2
66.80=0.13
69.08=0044
71.31
Adaptive
64.06::0.18
66.61=0.33
68.29:0.32
70.63=0.08
Ours 65.37--0.14
67.97-0.22
69.62+0.14
72.10-
A.5 EVOLUTION OF STUDENT FEATURE SPACE
In Figure 2 we visualize the decision boundaries formed by different types of samples in the student feature space. In this subsection, we further investigate the change of decision boundaries during the distillation process. Figures 7 g and Dillustrate the evolution of decision boundaries formed by hard, easy, and medium samples at 120, 160 and 240 epochs, respectively. For hard samples, is shown that the problem of the drift of decision boundaries is more severe at the early training stage. This because the teacher's knowledge of hard samples too complex for the student to learn. On the other hand, the training for easy samples converges rapidly and the constructed decision boundaries avoid overlap between different classes. However, these decision boundaries are too small cover the overall feature distribution. The medium-difficuly samples constitute smoothed decision boundaries during the distillation process, leading to a discriminative and generalizable student network.
A.6 RESULTS ON A HIGHLY-IMBALANCER DATASET
To verify if the proposed logit reshaping method works in highly imbalanced dataset, we sam- ple the CIFAR-100 dataset an imbalanced manner. To be specific, we set the sampling rates to be [100%, 99%, 98%, ..., 1%] for class [0. 1, 2, .99), respectively, to create highly imbalanced dataset and use this imbalanced dataset for the student's distillation. Experimental results in Table II demonstrate that even if the dataset is highly imbalanced, the proposed logit reshaping method still improves the student's performance in terms of results in the second and third columns. This further verifies the generalizability of the proposed method. In addition, there is potential way to optimize the proposed method further. Specifically, instead of aligning the teacher's logits with the class-specific distributions, we align the teacher's logits with the average distribution information of the entire dataset (i.c., the mean of class-specific distributions). In this way, we can balance the logit variance between the minority and the majority classes. Results in the last column demonstrate the effectiveness of such reshaping technique.
15

---

Published as a conference paper at ICLR 2025
(a) Hard
(b) Easy
(c) Medium
Figure 7: Decision boundaries of (a) hard samples, (b) easy samples, and (c) medium samples selected by the teacher ResNet50 on the feature space of the distilled student MobileNetV2 on CIFAR-100 100 120 epochs.
(a) Hard
(b) Easy
(c) Medium
Figure 8: Decision boundaries of (a) hard samples, (b) easy samples, and (c) medium samples selected by the teacher ResNet50 the feature space of the distilled student MobileNetV2 on CIFAR-100 1-0 at 160 epochs.
(a) Hard
(b) Easy
(c) Medium
Figure 9; Decision boundaries of (a) hard samples, (b) easy samples, and (c) medium samples selected by the teacher ResNet50 on the feature space of the distilled student MobileNetV2 on CIFAR-100 240 epochs.
A.7 ADVANCED DISTILLATION Loss
As mentioned in the Introduction, various extensions of the vanilla KD have been proposed recent years. Since this paper focuses the acceleration of distillation, mainly adopt the vanilla KD for illustration in the experiments. This subsection demonstrates that the student's 's accuracy can be further increased by combining the proposed method with advanced logit distillation loss. Specif- ically, the recently proposed Decoupled Knowledge Distillation (DKD) Zhao et al. (2022) firstly decomposes the vanilla logit distillation loss into Target Class Knowledge Distillation (TCKD)
16

---

Published as a conference paper at ICLR 2025
Table 11: Experiments CIFAR-100 with imbalanced sampling by using ResNer50-MohileNe12
Method
w/o Reshaping
w/ Reshaping
w/ Improved Reshaping
Top-1 Testing Accuracy (%)
56.64
57.46
58.09
Table 12:
Top-1 accuracy (%) ImageNet by using different logit distillation losses with the pro- posed dataset pruning and logit reshaping method. The student network MobileNet.
Teacher
ResNet34
ResNet50
Method
r 50%
== 70%
r
50%
r r= 70%
KD
69.98
70.92
69.97
70.93
DKD
70.17
71.27
70.22
71.61
loss and a Non-target Class Knowledge Distillation (NCKD) loss and the applies different hyper- parameters for these two losses. Following the code released by the author set the hyper- parameters of TCKD and NCKD to be I and 0.5 on ImageNet, respectively. As shown in Table [2] the student's performance can be consistently improved by using DKD under different keep ratios, which demonstrates the generalizability of the proposed method.
A.8 TRANSFER LEARNING
We further validate the performance of the distilled students from the view transfer learning. In these experiments, we introduce two image datasets CUB-200 (Wah et al. (01T)) and Cars-196 (Krause et al. 2013) for evaluation. For classification on different dataset, we freeze the parameters of the backbones of the students distilled on ImageNet and re-train the last classification layer. Table
13
lists the top-5 accuracy of the students distilled with different static dataset pruning methods. As
shown in the table, the proposed method outperforms the existing dataset pruning methods by clear margin on different datasets.
Table 13:
Top-5 accuracy (%) on CUB-200-2011 and Cars- 196 datasets using the pre-trained student MobileNet distilled by different static dataset pruning strategies with keep ratio r 30%.
Teacher
ResNet34
ResNet50
Strategy
CUB200
Cars
CUB200
Cars 196
Random
88.59
76.65
88.02
74.41
Herding
87.78
75.35
87.02
72.35
B1.2N
87.19
74.75
85.76
71.54
Moderate
89.04
75.11
87.81
73.62
Ours
89.48
78.10
88.31
75.97
A.9 VISUALISATION OF SELECTED SAMPLES
To illustrate the differences between the preserved images, we visualize some images on ImageNet selected by the pre-trained teacher ResNet50. Figure [0 displays the images of hard samples from three different classes. It is shown that images of hard samples are more difficult to learn since these images contain more uncorrelated objects in the image of "Ballon" or fewer features of the object as in the image of "Red wolf". addition, hard samples may contain noise as the image of "Hamster". On the contrary, images of casy samples easy to distinguish as shown in Figure
From these visualizations, medium-difficulyy samples achieve a trade-off between easy and hard samples shown in Figure [2] that the student can learn accurate and generalizable decision boundaries between classes.
17

---

Published as a conference paper at ICLR 2025
(a) Balloon
(h) Hamster
(c) Red wolf
Figure 10: Images of hard samples selected by the pre-trained ResNet50 on ImageNet.
(a) Balloon
(b) Hamster
(c) Red wolf
Figure 11: Images of easy samples selected by the pre-trained ResNet50 on ImageNet.
(a) Balloon
(b) Hamster
(c) Red wolf
Figure 12: Images of medium samples selected by the pre-trained ResNet50 on ImageNet.
18


Reviews:
[{"review_id": "rr8GNZPvlH", "reviewer": "Reviewer_FjLn", "summary": "This manuscript introduces a dataset pruning method to enhance the efficiency of knowledge distillation (KD) by retaining a strategically selected subset of samples and reshaping logits to better align with the teacher’s knowledge. \n\nThe authors propose that selecting medium-difficulty samples strikes a good balance and helps in preserving samples that are neither too easy nor too difficult. This in turn helps to increase the efficiency and accuracy of models.\n\nFurther, they introduce a logit reshaping technique that uses global distribution information to counteract biases that dataset pruning might introduce.\n\nThe proposed method is evaluated on two benchmark datasets CIFAR-100 and ImageNet with multiple teacher-student architectures, demonstrating consistent improvements over existing pruning techniques.", "questions": "I have some queries regarding the article----\n\n1. How does the proposed logit reshaping method handle potential conflicts between preserving class-specific distributions and mitigating the variance of logits in a highly imbalanced dataset? Could the method be further optimized for such cases? Can I get some explanations for this?\n\n2. In the context of medium-difficulty sample selection, how does the method ensure that the pruned dataset retains enough diversity to avoid overfitting, especially when dealing with high-dimensional feature spaces in complex models?\n\n3. Given that the pruned dataset relies on teacher feedback to rank sample difficulty, could this process introduce potential biases in class relationships, and how might this be addressed if the teacher's knowledge is imperfect or misaligned with the pruned subset?\n\n4. Can the authors provide some tSNE figures for better understandings.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a compelling approach to dataset pruning. This method looks interesting focusing on selecting medium-difficulty samples for distillation, which strikes a balance between easy and hard examples and leads to better generalization in the student model.\n\n2. The proposed logit reshaping method effectively mitigates the distributional shift caused by pruning. This can be a key to enhancing distillation performance on pruned datasets.\n\n3. The technique is thoroughly evaluated across multiple benchmark datasets (CIFAR-100, ImageNet) and different teacher-student architectures.\n\n4. The proposed pruning method significantly reduces training time without sacrificing performance. This can give us with some really good solutions for real world use and even we can use this method for time consuming computationally large datasets.\n\n5. The proposed method outperforms existing static pruning techniques (e.g., Herding, Forgetting, MoDS) and achieves comparable or superior performance to dynamic methods like InfoBatch.", "weaknesses": "1. The article does not provide a detailed analysis of how sensitive the method is to hyperparameter tuning. The authors should try out with different keep ratios or distillation losses.\n\n2. This article compares with static pruning techniques but a more in-depth comparison with other knowledge distillation methods, especially self-knowledge distillation, would strengthen its position and give us a more in-depth analysis.\n\n3. The approach is evaluated on CIFAR-100 and ImageNet, but it remains unclear how well it would scale with more complex models like ResNet family and Wide ResNets (WRN).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WsKIIRDUV8", "reviewer": "Reviewer_RWDY", "summary": "In the paper, the authors addresses the challenge of efficient knowledge distillation (KD) on pruned datasets. The authors propose a static pruning method that focuses on selecting medium-difficulty samples, as opposed to traditional methods that prefer hard or easy samples. The method aims to mitigate decision boundary drift caused by the student's limited capacity to mimic the teacher's feature distribution. Additionally, the paper introduces a logit reshaping technique to reduce distributional shift in the pruned dataset by leveraging the teacher's global logit information. Experimental results demonstrate that the proposed method outperforms existing dynamic and static pruning techniques across various benchmarks, achieving notable efficiency and performance gains.", "questions": "1. **Logit Reshaping Justification**: How sensitive is the logit reshaping method to variations in the teacher’s predictions? Could this technique inadvertently introduce bias in scenarios with highly imbalanced classes?\n2. **Generalizability Across Domains**: What challenges do the authors anticipate in adapting the propsoed method to other domains of say NLP and time-series?\n3. **Efficiency vs. Performance Trade-off**: Can you elaborate on the practical trade-offs between training time and accuracy? Are there specific scenarios where maximizing efficiency might not justify the associated performance loss?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. **Clear Motivation**: The authors clearly articulate the problem of decision boundary drift and how hard or easy samples influence KD differently. This sets a strong foundation for their proposed solution.\n2. **Methodical Innovation**: The selection of medium-difficulty samples for KD and the use of global logit reshaping are novel concepts. The visualizations of decision boundaries and empirical evidence support the validity of these innovations.\n3. **Comprehensive Evaluation**: The paper presents results from multiple datasets (CIFAR-100, ImageNet) and various teacher-student configurations. Ablation studies and comparisons with other pruning methods provide a holistic understanding of the approach’s effectiveness.\n4. **Efficiency Gains**: By achieving better student performance while significantly reducing training times, the proposed method demonstrates practical utility, especially for resource-constrained settings.", "weaknesses": "1. **Theoretical Justification of Medium-Difficulty Samples**: \nDespite the novelty of using medium-difficulty samples, the overall methodological advances feel incremental. The authors should need more rigorous theoretical analysis to make the proposed methodology convince to the reader of the underlying mechanisms driving performance improvements.\n\n   - **Explicit Analysis**: The authors should provide a rigorous mathematical analysis that formalizes how medium-difficulty samples contribute to smoother decision boundaries. This could involve deriving relationships between sample difficulty, gradient stability, and boundary smoothness.\n   - **Statistical Modeling**: Conduct an empirical risk analysis or use statistical measures to model the impact of medium-difficulty samples on decision boundary drift. For example, the authors could analyze the variance of gradients during training for different difficulty levels and quantify how medium-difficulty samples stabilize learning.\n   - **Geometric Insights**: The authors can use tools from computational geometry to offer insights into how medium-difficulty samples influence class separability. Visualizing the feature space evolution with respect to class overlap during training could also strengthen their argument.\n\n2. **Deeper Analysis of Logit Reshaping**:\n   - **Impact Study on Logit Distribution**: Perform a sensitivity analysis on the logit reshaping process. Specifically, the authors can quantify how reshaping logits using the teacher’s global information affects the student’s performance across classes, especially in cases of class imbalance or noisy labels.\n   - **Ablation of Reshaping Components**: Break down the logit reshaping mechanism and assess the impact of each component (e.g., the standard deviation adjustment and the mean logit alignment) to isolate what contributes most to performance gains.\n   - **Bias and Overfitting Analysis**: Evaluate whether the logit reshaping method introduces bias or overfitting. This could involve testing the method on datasets with varying distributions and comparing generalization performance on unseen data.\n\n3. **Quantitative Analysis of Decision Boundary Dynamics**:\n   - **Boundary Drift Measurement**: The authors can introduce or provide a metric or framework for quantifying decision boundary drift in the student network. This could involve computing the alignment of decision boundaries between the teacher and student models using tools like mutual information or boundary discrepancy measures.\n   - **Difficulty Spectrum Analysis**: Assess how the performance changes as samples are divided across a spectrum of difficulties, rather than just hard, medium, or easy categories. This can help establish a more granular understanding of why medium-difficulty samples are optimal.\n\n4. **Generalizability and Scalability Experiments to Other KD frameworks on other modalities?**:\nThe method’s effectiveness on more diverse datasets or in non-vision tasks remains unexplored. The authors should test out the method on a range of datasets beyond vision (e.g., NLP or time-series data) to demonstrate the versatility of the approach. This would help clarify if the medium-difficulty sample selection has domain-specific benefits or if it generalizes well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bSZIpEftxf", "reviewer": "Reviewer_s3Pu", "summary": "The paper aims to investigate experimentally how to select a balanced subset of a training set, based on a pretrained teacher, so as to reduce the training time needed to train students in the knowledge distillation (KD) framework while controlling the performance degradation of distilled students as much as possible, where the distilled students are trained on the balanced set. The proposed method consists of two phases. In phase 1, all samples in the original whole training set are sorted in the decreasing order of their corresponding cross entropy losses given by the pretrained teacher, those located in the middle range are selected, and the average of the standard deviation of a logit vector over all logit vectors within each class is computed and recorded. In phase 2 (corresponding to the training stage in KD), the logit vector of each selected sample from the pretrained teacher is first reshaped by the recorded average standard deviation of the class that the selected sample belongs to and  then passed to the KD objective function. To maintain balance, the sorting and sample selection can be carried out on a per class basis, as implied in the first paragraph of Section 4.3. Experiments were carried out on CIFAR 100 and ImageNet for several teacher-student pairs to verify the effectiveness and generalizability of the proposed method.", "questions": "1. How are the shaping methods in the literature applied in the setup for Table 2? The comparison in Table 2 seems  like apple vs orange, and may be not fair.\n\n2. More experiments are needed to show the value of the proposed method. \n\n3. The few shot cases considered in the paper ``BAYES CONDITIONAL DISTRIBUTION ESTIMATION FOR KNOWLEDGE DISTILLATION BASED ON CONDITIONAL MUTUAL INFORMATION'' by L. Ye (ICLR 2024) can also be used to reduce the KD training time. Please compare the performance of the proposed method with the few shot case in that paper in terms of accuracy vs training time to see if the proposed method in this paper is orthogonal.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is generally well written. \n\n2. The methodology is easy to follow and simple to implement.", "weaknesses": "1. The value proposition in terms of the accuracy vs time savings is kind of weak in this line of research. First, the intended time savings is for the KD training phase, not for the inference phase. Time savings for the inference phase is more important. Second, there is always accuracy degradation when the training dataset is pruned. When the ratio of kept sample is high, the training time savings is small. To have a meaningful training time savings, the ratio of kept sample has to be much smaller, which, however, results in significant, unacceptable accuracy performance degradation. \n\n2. In comparison with Moderate Dataset Selection (MoDS) and reshaping techniques proposed in the literature, the novelty of the proposed method is kind of limited. \n\n3. Experiments are insufficient to demonstrate the universality of the proposed method with respect to different students with variety of learning capability. The proposed method is mainly motivated by the learning gap between the pretrained teacher and students. What would happen if the learning capability of the student is close to, or very far away from that of the pretrained teacher? If the pruned training dataset works only for certain students, but not for others, the method won't fly since finding which student fits by experiments also takes significant amounts of training time.\n\n4. Experiments are insufficient to demonstrate its competitiveness with another technique called data distillation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9NeauhL27W", "reviewer": "Reviewer_EGEk", "summary": "The paper deals in data pruning for knowledge distillation. While previous observations show that random pruning is superior over other pruning methods, the authors proposed to retain medium difficulty samples for distillation. The motivation is that these samples are closer to the median distances between class centers.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The explanation the authors suggest for the question of why a student model is sub-optimal when performing hard-sample based pruning methods is interesting and looks like it is the main novelty of the paper.  Also, The visualization in Figure 3 is clear and insightful.\n\n- Experimental results are encouraging and look promising.", "weaknesses": "- In Figure  2 the authors show “a large accuracy gap” between the teacher and the student. The figure which indeed shows “Large accuracy gap” does not clearly demonstrate the specific degradation of the student’s accuracy due to the dataset pruning used. An accuracy gap between teacher and student is what we typically see in knowledge distillation in general. What does it mean “large accuracy gap”? Maybe the authors better show accuracy plots for easy, medium and hard pruning. Currently, Figure 2 is out of context.   \n\n\n- Equation (3), the M_DIF is computed to score hard samples. It is not quit clear why the M_DIF is defined here and what is the context? Is it another form of data pruning algorithm? (in figure 5, various data pruning are compared by the authors). Or it severs only the discussion in section 3.2? I would clarify the motivation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)