Distilling the Knowledge in Data Pruning
Authors: Emanuel Ben Baruch
Adam Botach
Igor Kviatkovsky
Manoj Aggarwal
Gerard Medioni
Venue: ICML 2025 poster (2025)
Keywords: Data pruning
knowledge distillation
Primary area: deep_learning->algorithms

Abstract: With the increasing size of datasets used for training neural networks, data pruning has gained traction in recent years. However, most current data pruning algorithms are limited in their ability to preserve accuracy compared to models trained on the full data, especially in high pruning regimes. In this paper we explore the application of data pruning while incorporating knowledge distillation (KD) when training on a pruned subset. That is, rather than relying solely on ground-truth labels, we also use the soft predictions from a teacher network pre-trained on the complete data. By integrating KD into training, we demonstrate significant improvement across datasets, pruning methods, and on all pruning fractions. We first establish a theoretical motivation for employing self-distillation to improve training on pruned data. Then, we empirically make a compelling and highly practical observation: using KD, simple random pruning is comparable or superior to sophisticated pruning methods across all pruning regimes. On ImageNet for example, we achieve superior accuracy despite training on a random subset of only 50% of the data. Additionally, we demonstrate a crucial connection between the pruning factor and the optimal knowledge distillation weight. This helps mitigate the impact of samples with noisy labels and low-quality images retained by typical pruning algorithms. Finally, we make an intriguing observation: when using lower pruning fractions, larger teachers lead to accuracy degradation, while surprisingly, employing teachers with a smaller capacity than the student's may improve results. Our code will be made available.

Distilling the Knowledge in Data Pruning
Emanuel Ben Baruch' Adam Botach
Igor Kviatkovsky
Manoj Aggarwal Gérard Medioni
Abstract
retain the most representative samples of given dataset and enable the conservation of memory and reduction of computational costs by allowing training on compact and small subset of the original data. For instance, data pruning can be useful for accelerating hyper-parameter optimization or neural architecture search (NAS) efforts. It may also be used in continual learning or active learning applications.
With the increasing size of datasets used for train- ing neural networks, data pruning has gained trac- tion in recent years. However, most current data pruning algorithms are limited in their ability to preserve accuracy compared to models trained on the full data, especially in high pruning regimes. In this paper we explore the application of data pruning while incorporating knowledge distilla- tion (KD) when training on a pruned subset. That is, rather than relying solely on ground-truth la- bels, we also use the soft predictions from a teacher network pre-trained the complete data. We first establish theoretical motivation for em- ploying self-distillation to improve training on pruned data. Then, we empirically make a com- pelling and highly practical observation: using KD, simple random pruning is comparable or su- perior sophisticated pruning methods across all pruning regimes. On ImageNet for example, we achieve superior accuracy despite training random subset of only 50% of the data. Addition- ally, we demonstrate a crucial connection between the pruning factor and the optimal knowledge dis- tillation weight. This helps mitigate the impact samples with noisy labels and low-quality images retained by typical pruning algorithms. Finally, make an intriguing observation: when using lower pruning fractions, larger teachers lead to ac- curacy degradation, while surprisingly, employing teachers with a smaller capacity than the student's may improve results.
Existing methods for data pruning have shown remarkable success in achieving good accuracy while retaining only fraction, I < 1, of the original data; see for example (Toneva et al., 2018; Paul et al., 2021; Feldman & Zhang, 2020; Meding et al., 2021) and the overview in (Guo et al., 2022). However, those approaches are still limited their ability to match the accuracy levels obtained by models trained on the complete dataset, especially in high compres- sion regimes (low f)-
Score-based data pruning algorithms typically rely on the en- tire data to train neural networks for selecting the most rep- resentative samples. For example, the 'forgetting' method (Toneva et al., 2018) counts for each sample the number of instances during training where the network's prediction for that sample shifts from "correct" to "misclassified". Sam- ples with high rates of forgetting events are assigned higher scores as they are considered harder and more valuable for the training. Other methods use the gradient norm, as in GraNd and EL2N (Paul et al., 2021), or measure changes in the optimal empirical risk, as employed by MoSo (Tan et al., 2023), to score the samples. Typically, once the sample scores are calculated, the models trained on the full dataset are discarded and are longer in use.
In this paper, we explore the benefit of using a model trained on complete dataset to enhance training on a pruned subset of the data using knowledge distillation (KD). The motiva- tion behind this approach is that a teacher model trained on the complete dataset captures essential information and core statistics about the entire data. This knowledge can then be utilized when training on pruned subset. While KD has been extensively studied and demonstrated signif- icant improvements in tasks such as model compression, herein we aim to investigate its impact in the context of data pruning and propose innovative findings for practical us- age. Note that, in contrast to traditional model compression techniques, here we focus on self-distillation (SD), where
1. Introduction
Recently, data pruning has gained increased interest in the literature due the growing size of datasets used for train- ing neural networks. Algorithms for data pruning aim to
I Amazon. Correspondence to: Emanuel Ben Baruch <cmanbb@ samanbb@amazon.com amazon.com>.............
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Distilling the Knowledge in Data Pruning
fx(x)
D
Accurecy F2
KD
Randomr Random Forgetting
Teacher:
ResNer20
p
(x)x)
Applecd32
Teelac20
Seelee52
dataser (teachrr
0.1
0.2
0.5
0.4
U.S
0.7
088
0.5
2.0
Data fraction
Data fraction (f)
(a) KD for data pruning
(b) Acc. vs. pruning methods (CIFAR-100)
(c) Impact of teacher size (CIFAR-100)
Figure 1. Knowledge distillation for data pruning. (a) We investigate the usage of a teacher model, pre-trained on a full dataset, to guide student model during training pruned subset of the same data. (b) We find that by integrating KD into the training, simple random pruning outperforms other sophisticated pruning algorithms across all pruming regimes. (c) Interestingly, we observe that when using small data fractions, training with large teachers degrades accuracy, while smaller teachers are favored. This suggests that in high pruning regimes (low f), the training is more sensitive to the capacity gap between the teacher and the student.
the teacher and student have identical architectures. The training scheme is illustrated in Figure la.
propose to adapt the weight of the KD loss according the pruning factor. That is, for low pruning factors, we should increase the contribution of the KD term as the teacher's 's soft predictions reflect possible label ambiguity embedded in the class confidences. On the other hand, when the pruning factor is high, we can decrease the contribution of the KD term to rely more on the ground-truth labels.
We experimentally demonstrate that incorporating the (soft) predictions provided by the teacher throughout the training process on the pruned data significantly and consistently improves accuracy across multiple datasets, various pruning algorithms, and all pruning fractions (see Figure lb for ex- ample). In particular, using KD, can achieve comparable or even higher accuracy with only small portion of the data (e.g., retaining 50% and 10% of the data for CIFAR-IA 100 and SVHN, respectively). Moreover, a dramatic improvement is achieved especially for small pruning fractions (low f). For example, on CIFAR- 100 with pruning factor ʃ 0.1, accuracy improves by 17% (from 39.8% to 56.8%) using random pruning. On ImageNet with f 0.1, the Top-5 accuracy increases by 5% (from 82.37% to 87. 87.19%) using random pruning, and by 20% (from 62.47% to 82.47%) using EL2N. To explain these improvements, we provide theoretical motivation for integrating SD when training on pruned data. Specifically, we show that using a teacher trained on the entire data reduces the bias of the student's estimation error.
Finally, observe a striking phenomenon when training with KD using larger teachers: in high pruning regimes (low S) the optimization becomes more sensitive to the capacity gap between the teacher and the student model. This relates to the well known capacity gap problem (Mirzadeh et al., 2019). Interestingly, we find that for small pruning fractions, the student benefits more from teachers with equal or even smaller capacities than its own, see Figure Ic.
The contributions of the paper can be summarized as fol- lows:
Utilizing KD in data pruning, we find that training is robust to the choice of pruning mechanism at high pruming fractions. Notably, random pruning with KD achieves comparable or superior accuracy compared to other sophisticated methods across all pruning regimes.
In addition, we present several empirical key observations. First, our results demonstrate that simple random pruning outperforms other sophisticated pruning algorithms in high pruning regimes (low f) both with and without knowledge distillation. Notably, prior research demonstrated this phe- nomenon in the absence of KD (Sorscher et al., 2022: Zheng al., 2022). Second, we demonstrate useful connection between the pruning factor f and the optimal weight of the KD loss. Generally, utilizing data pruning algorithms to select high-scoring samples amplifies sensitivity to samples with noisy labels or low quality. This is because keeping the hardest samples increases the portion of these samples as we retain a smaller data fraction. Based on this observation, we
We theoretically show, for the case of linear regression, that using teacher trained on the entire data reduces the bias of the student's 's estimation error.
We demonstrate that by appropriately choosing the KD weight, one can mitigate the impact of label noise and low-quality samples that are retained by common pruming algorithms.
We make the striking observation that, for small prun- ing fractions, increasing the teacher size degrades ac- curacy, while, intriguingly, using teachers with smaller capacities than the student's improves results.
2

---

Distilling the Knowledge in Data Pruning
2. Related work
vacy concerns. Common approaches for DFKD involve generating synthetic samples suitable for KD (Luo et al., 2020; Yoo et al., 2019) or inverting the teacher's 's information to reconstruct synthetic imputs (Nayak et al., 2019; Yin et al., 2019). Recently, the works in (Cui et al., 2022; Yin et al., 2023), utilized pscudo labels in training with dataset distil- lation. Unlike dataset distillation and DFKD, which include synthetic data generation, our work focuses on enhancing models trained on pruned datasets created through sample selection, using KD. Moreover, this paper presents practical and innovative findings for applying KD in data pruning.
Data pruning. Data pruning, also known as coreset se- lection (Mirzasoleiman et al., 2019; Huggins et al., 2016; Tolochinsky & Feldman, 2018), refers to methods aiming to reduce the dataset size for training neural networks. Recent approaches have shown significant progress in retaining less data while maintaining high classification accuracy (Toneva et al., 2018; Paul et al., 2021; Feldman & Zhang, 2020; Med- ing et al., 2021; Chitta et al., 2019; Sorscher et al., 2022). In (Sorscher et al., 2022), the authors showed theoretically and empirically that data pruning can improve the power law scaling of the dataset size by choosing an optimal pruning fraction as a function of the initial dataset size. Additionally, studies in (Sorscher et al., 2022; Ayed & Hayou, 2023) have demonstrated that existing pruning algorithms often under- perform when compared to random pruning methods, espe- cially in high pruning regimes. In (Zheng et al., 2022), the authors suggested a theoretical explanation to this accuracy drop, and proposed a coverage-centric pruning approach which better handles the data coverage. Also, in (Yang et al., 2022), the authors proposed to model the sample selection procedure as a constrained discrete optimization problem.
Knowledge distillation. Knowledge distillation is a popular method aiming at distilling the knowledge from one network to another. It is often used to improve the accuracy of small model using the guidance of a large teacher network (Bucila et al., 2006; Hinton et al., 2015). In recent years, numerous variants and extensions of KD have been developed. For example, (Zagoruyko & Komodakis, 2016; Romero et al., 2014) utilized feature activations from intermediate layers to transfer knowledge across different representation lev- els. Other methods have proposed variants of KD criteria (Yim al., 2017; Huang & Wang, 2017; Kim al., 2018; Ahn al., 2019), as well as designing objectives for repre- sentation distillation, as demonstrated in (Tian et al., 2019; Chen et al., 2020). Self-distillation (SD) refers to the case where the teacher and student have identical architectures. It has been demonstrated that accuracy improvement can be achieved using SD (Furlanello et al., 2018). Recently, theoretical findings were introduced for self-distillation in the presence of label noise (Das & Sanghavi, 2023).
Recently, several pruning methods have been introduced to address specific limitations of earlier approaches. (Tan et al., 2023) introduced an alternative pruning technique to the costly leave-one-out procedure, leveraging first-order approximation. This approach assigns higher scores to sam- ples whose gradients consistently align with the gradient expectations across all training stages. D2 (Maharana et al., 2024) proposes a graph-based formulation to represent the data distribution, enabling the selection of a coreset that favors both diverse and difficult regions of the data space. DUAL (Cho et al., 2025) identifies influential training exam- ples early in the learning process, leveraging early signals to guide pruning. In contrast, (Xia et al., 2023) introduce Mod- crate Coreset, which-given any scoring function-retainn the samples whose scores lie near the median, aiming to ob- tain a lightweight subset that remains robust across diverse data scenarios.
In our paper, we explore the process of distilling knowledge from a model trained on a large dataset to a model trained on a pruned subset of the original data. We focus on self- distillation and present several striking observations that emerge when integrating SD for data pruning.
3. Method
i= Given dataset DD with N labeled samples Vi (1) data pruning algorithm A aims at selecting a subset P C D of the most representative samples for training. We denote by f the pruning factor, which represents the fraction of data to retain, calculated as f = N₂/N where N₂ is the size of the pruned dataset. Note that < f < 1. Score- based algorithms assign score to each sample, representing its importance in the learning process. Let 8, be the score corresponding to a sample Ij, sorting them in a descending order 8k, > 8kg. ..., > 8ky following the sorting indices [k1, ky we obtain the pruned dataset by retaining the highest scoring samples, P {**, ...., Tkx, }. Usually, basre-based algorithms retain hard samples while excluding the easy ones. Note that in random pruning, we simply sample the indices ky ..., ky uniformly. In this paper, given
Data pruning proves valuable reducing memory and com- putational cost in various applications, including tasks such as
search (Coleman et al., 2019), NAS (Dai et al., 2020), continual and incremental learning (Lange et al., 2019), as well as active learning (Mirzasoleiman et al., 2019; Chitta al., 2019).
Other related fields are dataset distillation and data-free knowledge distillation (DFKD). Dataset distillation ap- proaches (Wang et al., 2018; Zhao et al., 2020; Yu et al., 2023) aim to compress a given dataset by synthesizing a small number of samples from the original data. The goal of DFKD is to employ model compression in scenarios where the original dataset is inaccessible, for example, due to pri-
3

---

Distilling the Knowledge in Data Pruning
pruning algorithm A, our objective is to train model on the pruned dataset P while maximizing accuracy.
(a) CIFAR-100 highest score pruning samples
3.1. Training the pruned dataset using KD
Typically, score-based pruning methods involve training multiple models the full dataset D to compute the scores (Toneva et al., 2018; Paul et al., 2021; Feldman & Zhang, 2020; Meding et al., 2021). These models are discarded and are not utilized further after the scores are computed. We argue that a model trained on the full dataset encap- sulates valuable information about the entire distribution of the data and its classification boundaries, which can be leveraged when training the pruned data P. In this work, we investigate training scheme which incorporates the soft predictions of a teacher network, rre-trained on the full dataset, throughout training on the pruned data.
(b) SVHN highest score pruning samples
Figure 2. Highest scoring samples. Top 10 highest scoring sam- ples selected by the 'forgetting' pruning method for CIFAR-100 and SVHN datasets. The labels of the majority of the images are ambiguous due to class complexity low image quality.
distillation. We note that prior work has demonstrated this phenomenon in the absence of KD (Sorscher et al., 2022). Interestingly, we also observe that training with KD is robust to the choice of the data pruning method, including simple random pruning, for sufficiently high pruning fractions.
Let f(( be the teacher backbone pre-trained on D. The teacher outputs logits {=:)-= where C' is the number of classes. The teacher's soft predictions are computed by,
These observations on the effectiveness of random prun- ing in the presence of KD are compelling, especially in scenarios where data pruning occurs unintentionally product of the system, such as cases where the full dataset is no longer accessible due to privacy concerns. However, using knowledge distillation we can train student model on the remaining available data while maintaining a high level of accuracy.
exp(zz/T) 4: = exp(2//7) i 1=.....,
(I)
by-
where is the temperature syper-parameter. Similarly, we denote the student model trained on the dataset P as fz; 0), where 0 represents the student's parameters. The student's i-th soft prediction denoted by p; (f). We opti- mize the student model using the following loss function,
3.2. Mitigating noisy samples in pruned datasets
(()) (1- (0) (LLKD()),
(2)
In general, hard samples are essential for the optimization process as they are located close to the classification bound- aries. However, retaining the hardest samples while ex- cluding moderate and easy ones increases the proportion of samples with noisy and ambiguous labels, images with poor quality. For example, in Figure 2, we present the highest scoring images selected by the 'forgetting' pruning algorithm for CIFAR-100 and SVHN. As can be seen, in the majority of the images determining the class is non-trivial due to the complexity of the category (e.g., fine- fine-grained classes) or due poor quality. By using knowledge distilla- tion, the student can learn such label ambiguity and mitigate noisy labels.
where the classification loss (0) measures the cross- entropy between the ground-truth labels and the student's predictions, represented as: E; yi log (o)). For the KD term (0). common choice is the Kullback-Leibler (KL) divergence between the soft predictions of the teacher and the student. The hyper-parameter a controls the weight of the KD term relative to the classification loss.
Integrating the KD loss into the training process allows us to leverage the valuable knowledge embedded in the teacher's soft predictions 9:- These predictions may encap- sulate potential relationships between categories and class hierarchies, accumulated by the teacher during its training on the entire dataset. Intuitively, reliable data and class dis- tributions can be effectively learned from large datasets, but are harder to infer from small datasets.
In recent work (Das & Sanghavi, 2023) it was demon- strated that the benefit of using a teacher's predictions in- creases with the degree of label noise. Consequently, was found that more weight should be assigned to the KD term as the noise variance increases. Similarly, in our work we empiricaly demonstrate that the pruning factor becomes lower, we should rely more on the teacher's predic- tions by increasing a in Eq. 2. Conversely, as the pruning factor is increased, we may rely more on the ground labels by decreasing a. We find that setting a properly is crucial when applying pruning methods that retain hard sam-
In Section 1. we empirically demonstrate that integrating knowledge distillation into the optimization process of the student model, trained on pruned data, leads to significant improvements across all pruning factors and various pruning methods. In addition, we show that simple random pruning outperforms other sophisticated pruning methods for low pruning fractions (low f), both with and without knowledge
4

---

Distilling the Knowledge in Data Pruning
ples. Formally, the objective should be aware of the pruning fraction f as follows,
-1- +LJ) 1 x + '
(4)
f) (1 (1- (Id aff)
(3)
=X,X; + (L) x, (11 a)y/ axf
For example, as can be seen from Figure 6, when the prun- ing fraction is low (f 0.1), training with = is su- perior, achieving more than 8% higher accuracy compared to a = 0.5. Conversely, for high pruning fractions (e.g. f f= 0.7), using a = 0.5 outperforms a = 1 by more than 1% accuracy. We further explore the relationship between a and/ f in Section 4.3.
where y xX the teacher's predictions of the student's samples Xy. Note that in regular self- self-distillation (without pruning), we have ft 1, and > 0. Also, in a regular training on pruned data (without KD), f < 1, and a = 0. In our scenario we utilize self- seff-ditilllation for data pruning, i.e... < Jt < 1, and a > 0.
We denote the student estimation error as E (a f.)) = (a f. ft) - - In (Das & Sanghavi, 2023), the authors show that employing self-distillation (a > 0) reduces the variance of the student estimation, but on the other hand, increases its bias. In the following, we show that distilling the knowledge from teacher trained on a larger data subset w.r.f the student, decreases the error estimation bias.
3.3. Theoretical motivation
In this section we provide a theoretical motivation for the success of self-distillation in enhancing training on pruned data. We base our analysis on the recent results reported in (Das & Sanghavi, 2023) for the case of regularized linear regression. Note that while we use logistic regression in practice, we anchor our theoretical results in linear regres- sion for the sake of simplicity. Also, it often allows for a re- liable emulation of outcomes observed in processes applied to logistic regression (see e.g. in (Das & Sanghavi, 2023)). In particular, we show that employing self-distillation using teacher model trained larger dataset reduces the error bias of the student estimation.
Theorem 3.1. Let X E d and y E RN be the full observation matrix and label vector, respectively. Let x + +7f. where O is the ground-truth projection vector and n E RNN is Gaussian uncorrelated noise independent on X. Let c,( ft) (a J. fe) - 00" be the student estimation error. Also, assume that d S N₂ S N, and S Jv. Then, for any a,
We are given data matrix, X = XX.....XNY E Rd and a corresponding label vector y = [yi,.... .Nv] E R. where N and d are the number of samples and their dimen- sion, respectively. Let 0* E Rd be the ground-trutt model parameters. The labels are assumed to be random variables, linearly modeled by y g + 17, where 7) E R is as- sumed to be Gaussian noise, uncorrelated and independent on the observations. In data pruning, we select N columns from X and their corresponding labels: X, E TRdxN,, y, E 1 Thus, y,- = X + Ty. We also assume that S N S N which true in most practical scenarios. Solving linear regularized regression using pruned dataset with fraction f, the parameters of the trained model are obtained by:
||E,|es(a, S, f+lll S (aB,
We include the proof for Theorem 3.1 in the supplementary. As data pruming is susceptible to label noise due to retaining the hardest samples, this finding demonstrates the utility of the proposed method. It suggests that employing self- distillation with teacher trained on the entire dataset (Je 1) enables the reduction of estimation bias in a student trained on a pruned subset. In Section 4.2 we analyze the impact of different Se values on the student's 's accuracy, with the corresponding results illustrated in Figure 5.
4. Experimental results
In this section we provide empirical evidence for our method through extensive experimentation over a variety of datasets, an assortment of data pruning methods and across wide range of pruning levels. Then, we also investigate how the KD weight, the teacher size and the KD method affect student performance under different pruning regimes.
(f) = argmin ||y, -
xfo +
1100
where X > 0 is the regularization pareer-araaeter,, and L E Rdxd is the identity matrix. Note that a teacher trained on the full data is given by: θ. ((1) [XXT
Datasets. We perform experiments on four classification datasets: CIFAR- 10 (Krizhevsky et al., a) with 10 classes, consists of 50,000 training samples and 10,000 testing sam- ples; SVHN (Netzer et al., 201 1) with 10 classes, consists of 73,257 training samples and 26,032 testing samples; CIFAR- 100 (Krizhevsky et al., b) with 100 classes, con- sists of 50,000 training samples and 10,000 testing samples;
Here, we look at the more general case where the student is trained on a pruned subset with factor f. and the teacher model is trained on a larger subset of the data, ft > f. Following (Das& & Sanghavi, 2023), the model learned by the student given by,
5

---

Distilling the Knowledge in Data Pruning
Fandan,
Bandom, with K7
No gottingg wn KD
fargetingg.. with
Accuretty (h)
Accurcy (NO
Rardom, WD RD
Randon, Mi
Rardom, with
Ransom,
KD
Parpdiing
Pargating mib.tD
Fergeting
Fargeting,
Podertt 25,
DUAL-S wel KD
Geahid with
Grlts with LD
DUAL 68 with 40
5.2% 1D
20
11.%% cD
ditacet besched
6-2%, with
fL.% with KD
Pal iiteeet
FuBl detenut teacher)
20
Fut oacase (reacher)
0.3
0.2
03
04
05
0.7
08
09
10
0.0
D00
0.00
9.33
14.33
0.50
0.50
0.40
00
0.4
Data fraction:1 (1)
Data
Data Fraction
(a) CIFAR-100
(b) SVHN
(c) CIFAR-10
Figure 3. Data pruning results with knowledge distillation. Accuracy results across different pruning factors f. and various pruning approaches on the CIFAR-I00. SVHIN, and CIFAR-10 10 datasets. We equalized weight in the loss (i.e., 0.5). Using KD, significant improvement is achieved across all pruning regimes and all pruning methods. Random pruning outperforms other pruning methods for low pruning factors. For sufficiently high f. the accuracy is robust to the choice of the pruning approach in the presence knowledge distillation.
70
20
Top-5 10-
Reicur, nb KD Rancom, wis KD
Random, wit K3
Randon, with KD
t0p
Frrgeting win KD Forgettig with KD Memorizaton. we 40 Remorizatin, with su EL2h. wio
Fergeting wth XD Forgettng w KD
Kemorizat wi ID
Nemorizee with
EL2N, KD
BL2N, with k2
EL2N with KD
detsoal (twacher)
Full detaset (leache)
fraction ())
(a) ImageNet, Top-1 accuracy
(b) ImageNet, Top-5 accuracy
Figure 4. Data pruning results with KD on ImageNet. Accuracy results across different pruning factors f. and various pruning methods on the ImageNet dataset. We equalized weight (a 0.5) in Eq. 2.
and ImageNet (Russakovsky et al., 2015) with 1,000 classes, consists of 1. 1.2M training samples and 50K testing samples.
that in the presence of KD the respective teachers that are utilized are trained on the full datasets.
Pruning Methods. We utilize several data-pruning algo- rithms: 'orgeetting' (Toneva al., 2018), Gradient Norm (GraNd), Error I.2-Norm (EL2N) (Paul al., 2021), 'mem- orization" (Feldman & Zhang. 2020), D2 (Maharana et al., 2024), DUAL (Cho et al., 2025), and Moderate-coress (Xia et al., 2023). We also utilize a class-balancee random prun- ing scheme, which, given a pruning budget, randomly and equally draws samples from each class.
As can be observed Figures 3 and 4, the incorporation of KD into the training process consistently enhances model accuracy across all of the tested scenarios, regardless of the tested dataset, pruning method pruning level. For example, compared to baseline models trained on the full datasets without KD, utilizing KD can lead to comparable accuracy levels by retaining only small portions of the orig- inal datasets (e.g., 10%, 30%, 50% on SVHN, CIFAR-10. and CIFAR-I00 respectively, using 'forgetting'). In fact, even large scale dataset as ImageNet, comparable ac- curacy be achieved by randomly retaining just 30% of the data, while training larger subsets remarkably results in superior accuracy to the baseline (e.g., +1.6% using a random subset of 70%).
4.1. Training on pruned data with KD
To demonstrate the advantage of incorporating KD-based supervision when training on pruned data, we utilize the aforementioned data pruming methods on each dataset using wide range of pruning factors. Then, train models the produced data subsets with and without KD. We note
As shown in Figure 3, at low pruning fractions, random prun- ing combined with KD consistently outperforms all other methods, with the exception of D2 KD, which achieves comparable performance. However, unlike D2 12-chich re-
'ee that while the authors of memorization did origi- nally utilize method for data pruning, efficacy on ImageNet was later demonstrated by (Sorscher et al., 2022).
6

---

Distilling the Knowledge in Data Pruning
70-
Accuracy (%6
Accuracy (%)
Studant's fraction fww.1
Studenc's
Tem2.3
bio
sccurscy
accuracy
Teacher Data Fraction (6)
Teacher Data Fraction (6)
(a) CIFAR-100
(b) SVHN
Figure 5. Accuracy versus teacher data fraction (f)) The parameters f. and fi represent the fractions of data used to train the student and teacher models, respectively. The red circles emphasize the self-distillation (SD) accuracy; while the dashed black line depicts the teacher's 's accuracy. This figure highlights two insights: (1) increasing f: consistently improves accuracy on top of self-distillation; (2) in all scenarios, outperforms standard training without knowledge distillation, as indicated by the circles being positioned above the dashed purple curve. These results support the theoretical motivation presented in Section 3.3.
quires careful tuning of multiple hyperparameters (ki B, 7>) for each pruning fraction and dataset- our approach, based on simple random pruning with KD, involves no such tun- ing. This makes it significantly more practical and easier to apply in real-world scenarios.
renders the student model robust to the pruning technique used. This finding is significant as suggests that it may be possible forgo state-of-the-an pruning techniques in favor of basic random pruning in the presence of KD.
4.2. Impact of Teacher's Training Data Fraction
Moreover, we note that the accuracy gains due to KD are most significant in cigh-compesssion scenarios. For in-
Up to this point, employed teacher trained on the full dataset, i.e., teacher on smaller data fractions (0 < Sa < 1) affects the student's 's accuracy. Figure 5 presents the student's 's accu- racy on CIFAR-100 and SVHN across different data frac- tions used to train the teacher and the student, The results highlight two key findings: (1) increasing Se consistently enhances accuracy beyond SD; (2) in every scenario, SD surpasses standard training without KD. These observations align with the theoretical insights discussed in Section 3.3.
stance, on CIFAR-I00 with f = 0.1, KD contributes to absolute accuracy improvements of 17%, 22.4%, 21%, and 19.7% across the random, 'forgetting', GraNd, and EL2N pruming methods, respectively. Similarly, SVHN, which permits even stronger compression, improvements of the same order of magnitude can be observed at a lower pruning factor (f = 0.01).
1. We now explore how training the
These findings support the idea that the soft-predictions produced by a well-informed teacher contain rich and valu- able information that greatly benefit student in limited-data setting. This 'dark knowledge', notably ab- sent in conventional one-hot labels, allows the student to deduce stronger generalizations from each available data sample, which in turn translates to better performance given the same training data.
4.3. Adapting the KD weight vs. the pruning factor
We wish investigate how varying the KD weight affects the performance of the student under different pruning levels of a given dataset. To explore this we conduct experiments on CIFAR- with 'forgetting" as the pruning method and present the results in Figure As can be observed, lower pruning fractions favor higher values of a, while higher pruning fractions advocate for lower ones. As explained earlier, aggressive pruning via score-based methods tends to result in subsets with greater proportions of label noise and low quality samples. Hence, for lower pruning factors, increasing the KD weight seems to help the student mitigate the extra noise by relying more on the teacher's predictions. Conversely, as the pruning factor increases and the propor- tions of noise in the pruned subset gradually diminish, it appears be beneficial for the student balance the contri- butions of KD and the bround-truth labels. Similar results
Finally, two additional interesting patterns emerge from experiments. First, in high-compression scenarios (e.g.. S 0.4 in CIFAR-100, < 0.08 in SVHN), it is evident that random pruning surpasses all other methods in effective- ness, both with and without KD. This aligns with the notion that aggressive pruning via score-based techniques retains larger concentrations of low quality or noisy samples due to mistaking them for challenging cases. This phenomenon was previously noted without KD in (Sorscher et al., 2022). Second, under low-compression conditions (e.g.. f 2 0.5 in CIFAR-100, f 2 0.2 in SVHN), we observe that KD
7

---

Distilling the Knowledge in Data Pruning
0=0.
/= 0.3
[= 0.5
/= 0.7
0.9.9
660
72.0
54
73.
01-
110
Accurety [%)
Accurocy [%6]
663-
71.0
17.5
Accuracy
16.0
264
150
833
7L.
D.F 2.1 03
1.
640
00
58
1/0
01.0
03
09
10
05
08
OF
13.0
0.6
27
04
53
KD walant (2)
<D weight (o)
KD waicht (a)
KD waight
KD waight
Figure 6. Optimal KD weight versus pruning factor. Accuracy is presented for CIFAR-100 while varying the KD weight for different pruning factors. We utilize 'forgetting" as the pruning method. For low pruning fractions (low f), accuracy generally increases when increasing the KD weight rely the teacher's soft predictions. As higher pruning fractions (high f)), it is usually better lower a in order to increase the contribution of the ground labels.
0=1.1
0=
0.3
f=
f=
f=1
Accuracy 180
5D
me
50
Accuracy (96
50
Accurecy [S]
SD
Accurscy (%)
oza
restrio
(a) CIFAR-100. Student architecture: ResNet-32 (RN32).
Accurity (3))
623-
Accuratt (8]
Accuraty [%)
Accuracy [sg 5
Accurasy [8]
Accures
mere
2012
(b) CIFAR- 100. Student architecture: 2e (RN20).
0=1
[= 0.2
/= 0.3
/=0
/= 0.9
Accuracy [8]
MD.
Accurssy (%g
3D
Accur (%)
5D
Sccurscy [tg]
Wil
S0
partin
arte
4014
terie untilel
(c) CIFAR- 10. Student architecture: ResNet-32 32 (RN32).
Figure Exploring the effect the teacher's capacity. Accuracy results a student with (a) ResNet-32 and (b) ResNet-20 architectures while using teacher models with increasing capacities along the horizontal axes. In each instance, denote the teacher whose architecture matches that the student by 'SD' (self-distillation). We use random pruning with different fractions. Interestingly, under low pruning factors, inereasing the teacher's capacity results lower student accuracy.
on SVHN can be found in the supplementary.
pacities: (1) ResNet- 14 with 89.9% accuracy, (2) ResNet-20
with 70.23% accuracy, (3) ResNet-32 with 71.6% accuracy,
4.4. Using teachers of different capacities
(4) ResNet-56 with 72.7% accuracy, (5) ResNet- with
74.4% accuracy, and (6) WRN-40-2 with 75.9% accuracy. Also, note that for each teacher architecture experiment with five different temperature values in the range 2 7. We show the impact of the temperature selection in the sup- plementary. Similarly, in Figure 7b we present results for the same experiment using a ResNet-20 student, while Fig- ure 7c depicts results of similar experiment on CIFAR-10 for the ResNet-32 student. As observed, at low pruning factors, increasing the teacher's 's capacity harms the accuracy
Until now, we have focused on the case where both the student and teacher share the same architecture (i.c.. self- distillation). In this section, we explore how the capacity of the teacher affects the student's performance across dif- ferent pruning regimes. In Figure 7a, present accuracy results across various pruning factors for the case of ran- domly pruning CIFAR-100 100 and training with ResNet-33 32 student. We employ 6 teacher architectures of increasing ca-
8

---

Distilling the Knowledge in Data Pruning
70
Accuracy (%)
Accuracy (%)
80
75
Student, ne-CD subset)
Shudent, no-KD (30% subsat)
Student, KD (30% subset)
Studient KD 130% subseti
Teacher accuracy
(70% accuracy
0.2
0.4
0.8
1.0
0.025
0.050
0.075
0.100
0.125
0.150
0.175 0.200
Pruning Data Fraction
Pruning Data Fraction
(a) CIFAR-100
(b) SVHN
Figure 8. Accuracy when the student and teacher are trained on disjoint subsets. Notably, combining knowledge distillation with data pruning yields significant performance gains, even when the student is trained on a pruned dataset that differs from the teacher's 's training data. For instance, in CIFAR-100 with random pruning f 50%, observe 14.5-point accuracy improvement when the teacher model was trained on a different subset.
of the student. This trend is consistently observed across various student architectures and datasets, and is robust to the selection of the KD temperature. Additional results are provided in the supplementary.
5. Conclusion
In this paper, we investigated the application of knowledge distillation for training models on pruned data. We demon- strated the significant benefits of incorporating the teacher's soft predictions into the training of the student across all pruning fractions, various pruning algorithms and multi- ple datasets. We empirically found that incorporating KD while using simple random pruning can achieve compara- ble or superior accuracy compared to sophisticated pruning approaches. We also demonstrated useful connection be- tween the pruning factor and the KD weight, and propose to adapt a accordingly. Finally, for small pruning fractions, we made the surprising observation that the student benefits more from teachers with equal or even smaller capacities than that of its own, over teachers with larger capacities.
This observation highlights striking phenomenon: the capacity gap problem, which denotes the disparity in archi- tecture size between the teacher and student, becomes more pronounced when applying knowledge distillation during training on pruned data.
4.5. Data Pruning in Disjoint Datasets
In this section, we evaluate a practical setting where the pruned dataset is not a subset of the data originally used to train the teacher model. This scenario highly relevant in real-wond weal-world applications, particularly cases where ac- cess to the full dataset is restricted, such due privacy regulations, discussed at the end of Section 3.1.
Impact Statement
Let P be a pruned dataset sampled from D to train the student model, and let S be the training data for the teacher. In the following experiments, and S and are disjoint i.c.,
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
P n 6. For the empirical study, we used 70% of the training data to train the teacher and the remaining 30% to train the student with different pruning ratios. Specifically, compared the performance with and without KD for CIFAR-100 and SVHN datasets.
References
Ahn, S., Hu. S. X., Damianou, A. C., Lawrence, N. D., and Dai, Z. Variational information distillation for knowledge transfer. 2019 IEEE/CVF Conference Computer Vi- sion and Pattern Recognition (CVPR), pp. 9155-9163, 2019. URL https: / //ppp/app./aa//// wwwant azg/CorpusID:1186...
The experimental results are shown in Figure Notably, combining knowledge distillation with data pruning yields significant performance gains, even when the student is trained on pruned dataset that differs from the teacher's training data. For instance, in CIFAR-100 with random pruning at f = 50%, we observe a 14.5-point accuracy improvement when the teacher model was trained on dif- ferent subset.
Ayed, F. and Hayou, S. Data pruning and neural scaling
laws: fundamental limitations of score- ccore-based algo- rithms. ArXiv, (e4.2302.06660) 2023. URL https:

---

Distilling the Knowledge in Data Pruning
//api semaantisccoolar.ooggoooooo 256846521.
http://ccccpappe///////// o html.
Bucila, C., Caruana, R., and Niculescu-Mizil, A. Model compression. In Knowledge Discovery and Data Mining, 2006. URL https:/ /appi http://ap...aaaa..ee.eeee. semant ierannicscchn icscholar. org/ / http/ooppuII::112525
Furlanello, T., Lipton, Z. C., Tschannen, M., Itti, L., and Anandkumar, A. Born again neural networks. In International Conference on Machine Learning, 2018. URL https: http://app.sematiiccccsoo org/ htg//orrpusD::411000
Chen, I.,, Gan, Z., Wang, D., Liu, J., Henao, R., and Carin, L. Wasserstein contrastive representation distil-
lation. 2021 IEEE/CVF Conference on Computer Vi- sion and Puttern Recognition (CVPR), pp. 16291-16300, 2020. URL https: api. org/Corppp CorpusID 229220499.
Guo, C., Zhao, B., and Bai, Y. Deepcore: A compre- hensive library for coreset selection in deep learning. International Conference on Database and Ex-
pert Systems Applications, 2022. URL https:
/ //a //api.semannee www.nt www.ollarr.rg/oopusosi.... 248239610.
Chitta, K., Álvarez, J. M., Haussmann, E., and Fara- bet, C. Training data subset search with en- semble active learning. IEEE Transactions on Intelligent Transportation Systems, 23:14741-14752, 2019. URL https: / https://ai..emmattccchoo... semanticscholar. org/ Co worpusrrppascc : 226282535.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual leam- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Cho, Y. Shin, B., Kang, C., and Yun, C. Lightweight dataset pruning without full training via example difficulty and prediction uncertainty, 2025. URL https://arxix: org/ org/abs/ abs/ 250.. 06905.
Heo, B., Lee, M., Yun, S., and Choi, J. Y. Knowl- edge transfer via distillation of activation boundaries formed by hidden neurons. In AAAI Conference on Artificial Intelligence, 2018. URL https::/api. semanticscholar. wwwp:laa..rrr//ooooooooooooo hat.aaar....roooossSD5552222222.
Coleman, C. A., Yeh, C., Mussmann, S., Mirzasoleiman, B., Bailis, P. D., Liang, P., Leskovec, J., and Za- haria, M. A. Selection via proxy: Efficient data selection for deep learning. ArXiv, abs/1906.11888 2019. URL attpss//aii.semtttiicc..t. semanticscoola.. org/Corpus 119.5506119955222
Hinton, G. E., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. ArXiv, abs/1503.02533 2015. URL https: apii semanticscholar: org /tt/CCoouIID::200034
Cui, J., Wang, R., Si, S., and Hsich, C.-. Scaling dataset distillation to imagenet-lk with constant mem- ory. In International Conference on Machine Learning, 2022. URL https: http:///app.s.aa.i.....omm www.nticscholl... org//orrppaooooooooooo Corp//ooppii:iia. 253735319.
Horn, R. A. and Johnson, C. Matrix analysis. Cambridge university press, 2012.
Huang, Z. and Wang, N. Like what you like: Knowledge distill via neuron selectivity transfer, 2017.
Dai, X., Chen, D., Liu, M., Chen, Y., and Yuan, L. Da-nas: Data adapted pruning for efficient neural architecture search. In European Conference on Computer Vision 2020. URL https: /tpi.ap...// htttp:://api.ssman.... www.ntiisccolll......... org/Corpus ID: 214693401.
Huggins, J., Campbell, T., and Broderick. T. Coresets for scalable bayesian logistic regression. In Neural Informa- tion Processing Systems, 2016. URL https: /api. semant cscholar org cttp:/CrrpsSS:22122.
Kim, J., Park, S., and Kwak, N. Paraphrasing com- plex network: Network compression via factor transfer.
Das, R. and Sanghavi, S. Understanding self- oeff-istilllatin in the presence of label noise, 2023.
ArXiv, abs/1802.04977, 2018. URL https: /api.
wwwp:echhhoaa..ooooooooooooo semanticscholo or :rt.:ooppssIII:3300033.3.
Feldman. V. and Zhang, C. What neural networks memorize and why: Discovering the long tail via influence estimation. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Krizhevsky, A., Nair, V., and Hinton, G. Cifar-10 (canadian institute for advanced research). a. URL cs cs.toronto. wwt.oonto.edu//oeee edu/ cifar. html.
Krizhevsky, A., Nair, V., and Hinton, G. Cifar- (canadian institute for advanced research). b. URL http: /ww. cs torotto...... ceattto.dduueee wwtpeedu//////e html.
URL. https: httss///pccceeeieee
10

---

Distilling the Knowledge in Data Pruning
Lange, M. D., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G. G., and Tuyte- laars, T. A continual learning survey: Defying forget- ting in classification tasks. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 44:3366-3385, 2019. URL https: http:/pp...aaiiiseeeeeeeeee sewanticschheteeeeeeee org/CorpusID: 20 8889122.
//ufldl. stanford. wdw.eedu/uuooooo http://uuuuuubbb////// n/ps2011_housemanee www.hhuuusuuusss..mmm pdf.
Park, W. Kim, D., Lu, Y., and Cho, M. Relational knowledge distillation. 2019 IEEF/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 3962-3971, 2019. URL https:
sepplissematiissee wtt.oolaa..rrgooousssss... 131765296.
Loshchilov, I. and Hutter, F. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017. URL https:// openreview net/ forum? ? 98cxx.
Passalis, N. and Tefas, A. Learning deep represen-
tations with probabilistic knowledge transfer. In
European Conference Computer Vision, 2018. URL https://apl httpss//api.semanttccccctt semanticscholar.cooooo CorpusID:52012991
Luo, L., Sandler, M., Lin, Z. Zhmoginov, A., and Howard, A. G. Large-scale generative data-free distillation. ArXiv, abs/2012.05578,, 2020. URL https: semanteettttttme www.::seeetnnccccoooooieeoeieoce icscholar. Crr/Coopp 228083866.
Paul, M., Ganguli, S., and Dziugaite, G. K. Deep learning on data diet: Finding important examples early in training. CoRR, abs/2107.07017 2021. URL https: //apxivv////vv.. org/ abs//ab//// /tt.//11..00000 07075.
Maharana, A., Yadav, P., and Bansal, M. D2 pruning: Message passing for balancing diver- sity difficulty in data pruning. In Inter- national Conference on Learning Representations, 2024. URL https:/ /api. http//pppp.smaamteeeeeeeeeeeeeee semanticscholarr TTTp/CorpuSTD::777466000
Romero, A., Ballas, N., Kahou, E., Chassang, A., Gatta, C., and Bengio, Y. Fitnets: Hints for thin deep nets. CoRR, abs/1412.6550, 2014. URL https: /aap:. semanticscholar wttp/oorposS::22211...
Meding, K., Buschoff, L. M. S., Geirhos, R., and Wichmann, Trivial or impossible dichoto- mous data difficulty masks model differences (on imagenet and beyond). ArXiv, abs/2110.05922, 2021. URL https://api.set http://pp...emaaiccccccooooeeeee org/CoopuuSD:22863411
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3)2111-52, 2015. doi: 10. 10.1007/s11263-00 015-0816-y
Mirzadeh, S. I., Farajtabar, M., Li, A., Levine, N., Matsukawa, A., and Ghasemzadeh, H. Improved knowledge distillation via teacher assistant. In AAAI Conference on Artificial Intelligence, 2019. URL https://app semanticscholar. CorpusID:21290877
Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Morcos, A. S. Beyond neural scaling laws: beating power law scaling via data pruning. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https: /poeereeveww net www.foooruuuuiao Serrum?id--nnSSe--py.
Mirzasoleiman, B., Bilmes, J. A., and Leskovec, J. Core- sets for data-efficient training of machine learning mod- els. In International Conference on Machine Learning, wwwppp.seatttscccceeo.eeoomo 2019. URL 211259075.
Tan, H., Wu, S., Du, F., Chen, Y., Wang, 7., Wang, F,, and Qi, X. Data pruning via moving-one-sample out. ArXiv, Abs/2310.14660 2023. URL https: //api pepi.semman semanticscholees Seaecholar..rg/Coopusstt.. 264426070.
Nayak, G. K., Mopuri, K. R., Shaj, V., Babu, R. V., and Chakraborty, A. Zero-shot knowledge distil- lation in deep networks. ArXiv, abs/1905.08 14, 2019. URL org/CoorpsTDD:15000
Tian, Y., Krishnan, D., and Isola, P. Contrastive representa- tion distillation. In International Conference on Learning Representations, 2019.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsuper- vised Feature Learning 2011, 2011. URL http:
Tolochinsky, E. and Feldman, D. Coresets for mono- tonic functions with applications to deep learn- ing. ArXiv, abs/1802.07382. 2018. URL https: //api. www.stiisccoooll.o....o htt.://scccoaa...oooorpe...... 125549990.

---

Distilling the Knowledge in Data Pruning
Toneva, M., Sordoni, A., des Combes, R. T., Trischler, A., Bengio, Y., and Gordon, G. J. An empirical study of example forgetting during deep neural network learning. ArXiv,
Yu, R., Liu, S., and Wang, X. Dataset distilla-
tion: A comprehensive review. IEEE transactions pattern analysis and machine intelligence, PP. 2023. URL https: www.eeeeaaaatnnaeeeeeeeeeee arg/CorpusTD:2255422
2018. URL https:/ /api. semant www.ciccchhll..ooo ar. .rg. www.Cororrroooo.o.oooooo 55481 903.
Tung, F. and Mori, G. Aimilarity-preserving knowl- edge distillation. 2019 IEEE/CVF International Con- ference on Computer Vision (ICCV), pp. 1365-1374, 2019. URL https: wtt.pp..mtttttccoooo...o.ooooooe hts/CorppusD::19111
Zagoruyko, S. and Komodakis, N. Paying
tention to attention: Improving the performance of convolutional neural networks via attention transfer. ArXiv, abs/1612.033288, 2016. URL https:////// /api. semanticscholas Wwgg/oorpusID.. 829159.
Zhao, B., Mopuri, K. R., and Bilen, H. Dataset condensa- tion with gradient matching. ArXiv, (at//2006.09229 2020. URL https: wwtppiiseemattissssiooosoo a2g/Corpus1D::21555575
Wang, T. Zhu, J.-Y., Toralla,, A., and Efros, A. A. Dataset distillation. ArXiv, abs/1811.10959,
2018. URL bttpss://api semanitcccmmiicccoo...... or htg/CorpusID:5377333
Zheng, H.,, Liu, R., Lai, F., and Prakash, A. Coverage- centric coreset selection for high pruning rates. ArXiv, abs/2210.15809, 2022. URL https: / 253224188.
Xia, X., Liu, J., Yu, J., Shen, X., Han, B., and Liu, T. Moderate coreset: A universal method of data selec- tion for real-world data-efficient deep learning. In In- ternational Conference on Learning Representations, http:/app...aaiiic.cccccomo semanticscholee
2023. URL https::/appp org/ :
Yang, S., Xie, Z., Peng, H., Xu, M., Sun, M., and Li, P. Dataset pruning: Reducing training data by exam- ining generalization influence. ArXiv, 099229 2022. URL org/ Cttp/orppsID:::: 2488872255
Yim, J., Joo, D., Bae, J.-H. and Kim, J. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. 2017 IEEE Conference Computer Vision and Pattern Recog- nition (CVPR), pp. 7130-7138, 2017. URL https: //api. www.steettttt.e commaniccccoola.. semaatticcccoll....oorr/oooooop 206596723.
Yin, H., Molchanov, P., Li, Z., Álvarez, J. M., Mallya, A., Hoiem, D., Jha, N. K., and Kautz, J. Dreaming to distill: Data-frea free knowledge transfer via deepinver- sion. 2020 IEEE/CVF Conference on Computer Vi- sion and Putlern Recognition (CVPR), pp. 8712-8721, 2019. URL https: app..pii...... ort/Co:pusID:2294C222
http:///appp.saaaiiii.ee.mem.eee waappsssmaanne semanticscholas
Yin, Z., Xing, E., and Shen, Z. Squeeze, recover and rela- bel: Dataset condensation at imagenet scale from a new perspective, 2023.
Yoo, J., Cho, M., Kim, T., and Kang, U. Knowledge extraction with no observable data. In Neural In- formation Processing Systems, 2019. URL https: //api. semanticacholas wttp:/ora..rg//rouu..... 202774028.
12

---

Distilling the Knowledge in Data Pruning
Appendix
A. Implementation Details
For computational efficiency we conduct our self-distillation experiments on all datasets using the ResNet-32 (He et al., 2016) architecture, except for ImageNet for which we utilize the larger ResNet-50. Our training and distillation recipes are simple. We utilize SGD with Momentum optimize the models and incorporate basic data-augmentations during training. Additional implementation details can be found in the supplementary.
A.1. Obtaining the pruning scores
We utilize the default pruning recipes offered by the DeepCore framework (Guo et al., 2022) in order to compute most of the pruning scores used in our experiments. For SVHN (Netzer et al., 2011), CIFAR-I0 (Krizhevsky et al., a) and CIFAR-100 (Krizhevsky et al., b) we compute the scores using the ResNet-34 34 (He et al., 2016) architecture. For ImageNet (Russakovsky et al., 2015) we compute the scores for the 'forgetting' pruning method (Toneva et al., 2018) using ResNet-50, while for the 'memorization" (Feldman & Zhang, 2020) and EL2N (Paul et al., 2021) methods we directly utilize the scores released by (Sorscher et al., 2022). Specifically, we note that for EL2N on ImageNet we adopt the released variant of the scores which was averaged over 20 models.
A.2. Conducting the distillation experiments
We conduct our knowledge distillation experiments on the pruned SVHN, CIFAR-10 10 and CIFAR-100 datasets using a modified version of the RepDistiller framework (Tian al., 2019). For the most part we adopt the default training and distillation recipes offered by the framework. The models are trained for 240 epochs with batch size of 64. For the optimization process we use SGD with learning rate 0.05, momentum value of 0.9 and weight decay of 5e-4, The learning rate is decreased by factor of 10 on the 150th, 180th and 210th epochs. To conduct the distillation experiments on ImageNet we expand the DeepCore (Guo et al., 2022) framework to support knowledge distillation on pruned datasets. Apart from this change we mostly rely on the default training recipe offered by the framework. The models are trained for 240 epochs with a batch size of 128. We utilize SGD with learning rate 0.1, momentum value of 0.9 and weight decay of -4, The learning rate is gradually decayed during training using cosine-annealing scheduler (Loshchilov & Hutter, 2017). In all of our distillation experiments
4 as the temperature for the KD's soft predictions computation in Equation (1).
f= 0.01
f= 0.07
f= 0.2
f= 0.3
1= 0.7
37.50
903
70.35
3/0
57.00
Accuracy [%6]
90.10
Accuracy [%%)
Accuracy [%)]
Accuracy
30-
Accuracy
06.55
20.5
38.09
2032
99.0
19:99
4..
w0-0
9.15
25
07
00.00
96.6
$2.00
KD weight (a)
waight (or)
<D weight (o)
walght (9))
KD welght (6)
Figure 9. Optimal KD weight versus pruning factor. Accuracy is presented on SVHN while varying the KD weight a across different pruning factors. We utilize 'forgetting' the pruning method. For low pruning fractions (low f), accuracy generally increases when increasing the KD weight to rely more on the teacher's soft predictions. However, as we use higher pruning fractions (high f), it is usually better to use lower values order to increase the contribution the ground-tron truth labels.
B. Adapting the KD weight vs. the pruning factor
Following Section 4.3, in Figure 9 we present additional accuracy results which show the effect of varying the KD weight across different pruning factors f. this time on the SVHN dataset. We utilize 'forgetting' as the pruning method. Here, a similar trend to the one previously observed on CIFAR-100 100 can be seen: for low pruning fractions, accuracy improves as we increase the KD weight, while for higher pruning fractions it usually better to use lower a values.

---

Distilling the Knowledge in Data Pruning
0.01
0.0.55
f= 0.1
f= 0.5
f=
0,9
f= 1.0
0+5D
SD
Accuracy
Mio KD
W/o 1D
We x
Accuracd
Wil RD
8
353
1972
1920
1822
19210
1972
1912
18210
Pigs
1972
repp
18210
Redd
rory
rapo
1812 R0ss
18110
Piss
ror1
rotan
ras32
RM110
1011
tapo
(a) SVHN dataset. Student architecture: ResNet-8 (RN8).
0.01
0.5
0.9
1.0
22
24
Accuracy (%)
SD
(%6
5D
Accuracy (%6
Accurary [9%]
56.0
SD
20-
Accuracy
260
rase
faun
racse
PRI10
nwe
1912
l850
noer
roc2
ress
rorto
(b) SVHN dataset. Student architecture: ResNet-32 32 (RN32).
0=1
F= 0.2
f= 0.3
0.5
f= 0.9
Accuracy [%)]
5D
Accuracy [%)
SD
Accuracy [%]
72-
SD
Accuracy [%)
SD
Recuraa [%]]
Accuracy
30
wit RD
N/o ED
will KD
eelo KD
00.9
17.5
26.
66
Ru14
R014
R014
mizo
Hozz
19110
R9714
1922
n920
19110
WRito
not
repo
1o
1997440
(c) CIFAR-100 datasct. Student architecture: ResNet-56 (RN56).
f=
f= 0.2
f= 0.3
f= 0.9
f=1
5D
SD
80.
SD
Accuracy [%)
Arcuracy Beuraayy [%)]
Accurac acy [5g
Arcuracy [%)]
20.0
Accuracy
20.8
21.
80.
maz
AM310
1924
mipd
RMZL
mao
R0130
role
N0110
moom
19110
1924
mizz
(d) CIFAR- 10 dataset. Student architecture: ResNet-20 (RN20).
different pruning fractions using teacher models with increasing capacities for: (a) a ResNet-8 student on SVHN, (b) a ResNet-32 student on SVHN, (c) a ResNet-56 student on CIFAR-100, for (d) a ResNet-20 student CIFAR-10 Random pruning is utilized. These results further corroborate our observation that teachers with smaller capacities lead to higher student accuracy when utilizing low pruning fractions.
Figure 10. Exploring the effect of the teacher's capacity. Accuracy results
C. Using teachers of different capacities
In Section 4.4 have made the observation that teachers with smaller capacities lead to higher student accuracy when utilizing low pruning fractions. Here we provide additional results which demonstrate the consistency of this observation. In Figures 10a and 10b we present student accuracy results on SVHN using different teachers and various pruning fractions, where the utilized student architectures are ResNet-8 and ResNet-32, respectively. Similarly, Figure 10c depicts results on CIFAR- 100 with a ResNet-56 student, and Figure 10d shows the same on CIFAR-10 with a ResNet-20 student. Random pruning utilized in all experiments.
2

---

Distilling the Knowledge in Data Pruning
F-
F-
0.5
0.7
0.9
24-
00.3
69 1%6)
Accuracy ayy (96)
Accuracy [96]
57.5
04.3
Accuracy
00.0
Accura
Accuracy
RNJJ
613
RN20
Rz20
Rh20
RN20
Rh20
RN32
RN32
Rh32
RX32
RN32
RM56
RV5
RMM
RV110
PN:10
RX110
RV110
BN110
10
16
temperature (t)
KD temperature (r)
temperature
temperature
temperature
Figure 1I. Impact of the KD temperature on the student's accuracy using teachers with different capacities. We present accuracy results across different pruning fractions CIFAR-100 for ResNet-20 student. Random pruning is utilized. As be seen, for lower pruning fractions (e.g. f = 0.1 and f = 0.3), teachers with lower capacities outperform teachers with higher capacities.
Method
5%
10%
30% %
50 %
w/o KD
14.46
22.21
49.41
67.47
KD (Hinton et al., 2015)
28.62
46.27
66.82
70.95
FitNets (Romero et al., 2014)
25.66
44.84
65.7
70.77
AB (Heo et al., 2018)
30.5
47.68
66.15
71.22
AT (Zagoruyko Komodakis, 2016)
28.26
42.59
65.75
70.45
FT (Kim et al., 2018)
28.34
44.01
64.95
70.75
FSP (Yim al., 2017)
27.62
37.16
62.79
69.72
NST (Huang & Wang, 2017)
26.2
44.5
64.93
70.97
PKT (Passalis & Tefas, 2018)
27.3
44.09
65.22
70.7
RKD (Park et al., 2019)
21.69
43.03
65.43
70.36
SP (Tung & Mori, 2019)
29.09
42.53
65.62
70.72
VID (Ahn et al., 2019)
32.5
49.46
67.38
71.16
Table 1. Comparison of different KD approaches several pruning levels of CIFAR-100. We add various KD loss terms to Eq. 2. in addition to vanilla KD term. 'Forgetting" utilized pruning method. observed. integrating VID (Ahn et al., 2019) further improves training on the pruned dataset.
D. Impact of KD temperature
In Section 4.4 we have made the observation that for low pruning fractions, employing KD using smaller teachers results in higher student accuracy. To demonstrate the consistency of this observation across different KD temperatures, in Figure 11 we present the impact of the KD temperature on the student's accuracy when utilizing teachers with different capacities, and across various pruning fractions. The experiment was conducted on CIFAR- with random pruning using a ResNet-20 student. As can observed, the benefit of smaller teachers in high pruning regimes (lower f values) is evident over a wide range of temperature values.
E. Comparing different KD approaches
So far, we have utilized solely vanilla KD during training. Next we explore integrating additional KD approaches to the loss. In particular, we add an additional KD loss term Lg as follows: L()) = Les (0) + (x + (0). where /3 is a hyper-parameter. this experiments, we simply set & and /3 to In Table we compare the performance of different KD methods on CIFAR-IE 100 under low average compression regimes. For fair comparison, the vanilla KD, we set 2, and 0. As can be observed, integrating the Variational Information Distillation (VID) loss (Ahn al., 2019) improves results considerably for the tested cases. These results suggest that further improvement be achieved by incorporating additional approaches to extract knowledge from the teacher.
the case employing only
F. Impact of pruning levels
In this section, we present results comparing 'casy.' 'moderate' and 'hard' pruning levels when integrating knowledge distillation (KD) into the loss function. Figure 12 illustrates the accuracy achieved on CIFAR-100 across the three pruning
3

---

Distilling the Knowledge in Data Pruning
Accuracy (%6
90
easy - with KD easy without_KK KD
moderate with KD
moderate without
hard with with_KD
hard without KD
0.1 0. 03
0.4
0.5
0.6
0.7
68
0. 10
(Fraction of Data)
Figure 12. Pruning levels (easy, moderate, and hard pruning). easy (hard) pruning, select the f-percentile of lowest (highest) scores. Moderate pruning refers to selecting the middle f /-percntilee. This figure reveals multiple insights: (1) easy and moderate pruning produce higher results compared to hard pruning for low pruning fractions (hoth with and without KD); (2) using KD, moderate pruning leads to top performance compared to "casy" and 'hard' pruning levels; and (3) using KD, the variance between pruning levels reduced. These results obtained CIFAR-100.
levels. Specifically, we employed the forgetting approach to compute a score for each training sample. For "easy" pruning, we selected the f-percentile samples with the lowest scores, while for 'hard' pruning, we selected the f-percentile of samples with the highest scores. 'Moderate' pruning involved selecting samples within the middle f f-percentile. The results highlight several key insights: (1) both "easy' and 'moderate' pruning outperform 'hard' pruning terms of accuracy (with and without KD) in low pruning fractions; (2) incorporating KID, 'moderate' pruning achieves the highest performance compared to 'easy' and 'hard' pruning; and (3) KD reduces the variance in performance across the different pruning levels.
G. Theoretical Motivation
Lemma G.1. Given data matrix X dRdxN and sub-matrix X, E RRXN,, while d S Ny S N,
oy (x) 2 ox(X)), 1....,..
where (X) is the k's largest singular value of X.
Proof. Let Z denote the remaining sub-matrix after excluding the X, columns from X, i.e., X [X, Thus,
Xx + zz
All three matrices are positive semidefinite and therefore based on Weyl's inequality (Horn & Johnson, 2012)(T 4.3.1), A (XX") 2 X?7) where A (A) is the k's largest eigenvalue A. This also implies that o (X) 2 0 (X,) for k = I,....d.
Theorem G.2. Let X E R and y E R. denote the observations matrix and ground-trute -ttth label vector, respectively. Let J. denote the student model obtained using Eq. 4 using pruning factor f ft and distilled from the teacher model (())) using KD weight o. Then, the following holds,
ff < (. (a.
Proof. Similarly to (Das & Sanghavi, 2023) we base our proof on the Singular Value Decomposition (SVD) of both the pruned and the full data matrices used to train the student and the teacher, respectively. Thus, X,, UEE'V' and

---

Distilling the Knowledge in Data Pruning
x UEVI. We also assume that N > N, 2 d. which is practical assumption in machine learning and therefore the rank of both the full and the pruned data matrices is d. Thus the estimator SVD has the following form in terms of the SVD) of the full and pruned data matrices,
O, (a. fi) =
AL) (110
-U Aaa)
Σ AL)) -
-E + ) Σ (11-a) V n,)+
+ + (Σ + +>aa) -
3' O + vmTn))
of
2
(1-a) ii) +
00
u/>a/, ui) ust
of +X
0
2 + x
|=1
d
o'j
(1- + 00;
(n (u),,, un) u,
2
+1
+ X
1=
of
d
(1-a):
Σ ==1
00 u/Ju/, u,) + a
10" uأ}) (u', ui)
+Y
j=1
d
d'
+
(1)0)(7).33) + 00; Σ
(m/e v5y (u/j ui) u;
of +1
x
1=
d Σ
o7
d
(1-a) Σ 0
Σ
00 u))( un) u,
+ a
+X
2
j=1
j=1
d Σ
djj
(1-a)())v + 00;
(ns:
(1's un) u,
0. +1
o
2
+ x
j=1
?
0', , (u) u:) (1-0
x +1
u,+
2
07² 12
d
dj
(1-a)(n + +o0 00;
Σ
(1/5 v2) (ul, un-
o
+ A
2
+ X
/=1
1
The estimation error is therefore,
(a J.S) = O,((,, - -0- = fb)) - Σ 10' u.)u:
Σ
0. uuj u.)uu
i=1 j=l
d d
df
d d
0 u uu) up)
1 1-a
u, -
-1 10 .u)) u.),
+ x
o,2 +1
1=1 j=1
d/
(1- (m5, v1) + a0,
(1
Σ
(N v3) (u/, un) u
o. +
0
2 +
1=1
d d
Σ
0 u1)(ul. un)
oी +
1
a
0j² x -1) n
1
5

---

Distilling the Knowledge in Data Pruning
d
d E
o
0i
(1 (1-a) v.) + +0;
(sr
Va)( ui) in
o
+ x
0
2
+
x
j=1
d
d Σ
A
2 of
0 uf}Kuu. un} o? + x
1+
u.
0 +
i=1 j=1 j=1
d
Σ o2 (ns V3) (u/y ui) uz.
Σ
(1 (1-a) vi) a0;
+
j=1
The expectation of the bias term over the noise parameter 7) which is uncorrelated and independent of X is,
( (1 of u.) of +
X
(a,
0 Xu
u.
of +>
Therefore the bias error term of the estimation process is,
[E, f.
Σ
d
Σior 0
uz) (u's us)
(1 a oj 03 + (1)))
of
+
Note that given that the student and the teacher are trained using the same dataset x,. i.e., , 0; = of and
= u for
i = 1,...,d d, the bias error term reduces to what is reported in (Das & Sanghavi, 2023) (Fq. 24):
2
2
of
[eyles f. 2
Σ 0
U1)
(uz u1)
of
+
d Σ
of of
0" u,) 1=1
1 a
Now, let us consider the impact of a minimal augmentation of the dataset used to train the teacher w.r.t. that used to train the student. In other words, we assume that a single data sample is added, i.e., St = I + N. where N is the total number of available samples. Given that adding single sample to a significantly larger set of Ny samples not sufficient to change its distribution, we can assume that u' x u, for i 1,...,dd. d. Thus, the derivative of the error bias term with respect to of is,
o (a, f, + 2
d
A
2 d
Σ
0
u') uz).
00%
+
1=1
j=1
12 2
20
1
1
a (0 + A)²
(0", uu()) (u'e u:)
X
ot
0f
R 4a
1
(0" n112
oh +
0,
+
(0 2 + 2
0.
According to Lemma G.1, (x( 2 ob (X,)), = I....,d. d. Since we have shown that a| (. S 80 f1 +
i.e., the derivative of the error bias term w.r.t a singular value of of the teacher data matrix X,, is non-positive, and the pruned data matrix used to train the student necessarily has smaller corresponding singular values, it necessarily implies that (E,, f.f + 1 ]]] < (a, D) Applying the same logic iteratively over the process of adding more and more data samples, implies that [[,, f. f2] < [|E, (a, f)lll for any ft > f.
6


Reviews:
[{"review_id": "XW7xmzuIRd", "reviewer": "Reviewer_Kpwz", "summary": "This article explores the use of knowledge distillation (KD) to improve model performance when training on pruned datasets. The authors investigate how transferring soft predictions from a teacher model can eliminate the accuracy loss caused by aggressive data pruning.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comprehensive and constructive feedback and for highlighting the strengths of our paper. We appreciate the insightful questions and will make efforts to address the reviewer’s concerns.\n\n&nbsp;  \n**Incorporating Recent Pruning Methods**\n\nFollowing the reviewer’s suggestion, we have included three recent data pruning methods:\n\n* Moderate Coreset: A Universal Method of Data Selection for Real-world Data-efficient Deep Learning (ICLR, 2023)\n\n* D² Pruning: Message Passing for Balancing Diversity & Difficulty in Data Pruning (ICLR, 2024)\n\n* Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty (ArXiv, 2025)\n\n\nAccuracy results are available at the following link:\n\n[Recent_pruning_methods](https://ibb.co/xKB0F9hq)\n\n&nbsp;  \nAs observed, incorporating KD during training with data pruning also improves accuracy in these three pruning approaches. Also, please note that GraNd and EL2N were published in 2021, while memorization was [first applied](https://arxiv.org/abs/2206.14486) to data pruning in 2022.\n\nWe are happy to include these three additional baselines in the main paper (please see our response to Reviewer RfVw).\n\n&nbsp;  \n**On the Use of Simplified Assumptions in Theoretical Analysis**\n\nWe acknowledge that the assumptions made in our theoretical section (Gaussian noise, independent samples) may appear limiting when considering complex real-world scenarios. However, our goal was to provide a clear and tractable analysis that offers foundational insights into the behavior of self-distillation in pruned data settings. Employing simplified assumptions, such as linear regression frameworks, is a common practice in theoretical investigations within machine learning, enabling the derivation of interpretable results. For instance:\n\n* Understanding the Gains from Repeated Self-Distillation (NeurIPS 2024)\n* A Theoretical Analysis of Fine-tuning with Linear Teachers (NeurIPS 2021)\n* Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regression (NeurIPS 2023).\n\nFollowing the reviewer's suggestion, we will include a discussion on these methodological limitations and potential extensions in a dedicated section on limitations and future work.\nAlso, please note that the proof for Theorem 3.1 was moved into the supplementary due to lack of space.\n\n&nbsp;  \n**On Why Larger Teachers Degrade Performance under Extreme Pruning and Impact of KD Temperature**\n\nWe thank the reviewer for the opportunity to clarify this observation. Below, we provide our intuition based on our current understanding.\nWe hypothesize that learning complex decision boundaries becomes increasingly challenging when the number of samples is significantly reduced. Consequently, when training a student model on a pruned dataset with a high pruning ratio, smaller teachers tend to be more beneficial. Larger teachers, while capable of handling hard samples, can cause the student to focus on these difficult instances, which are inherently hard to resolve given the limited data. In contrast, smaller teachers are more tolerant to errors from hard samples and guide the student toward more manageable patterns.\nWhile increasing the temperature can soften the teacher's output, the softmax temperature is a global transformation that affects all logits simultaneously, and thus may not sufficiently reduce the impact of complex decision boundaries imposed by larger teachers.\n\nWe illustrate this in the following figure:\n\n[Teacher_capactity_accumulated_predictions.png](https://ibb.co/WNGK191R)\n\n&nbsp;  \n**Details on Tuning KD Temperature and Sensitivity Analysis**\n\nFollowing the reviewer's suggestion to analyze the KD temperature, we kindly refer to Figure 10 in the appendix, where we present accuracy results for various temperature values across different pruning fractions and architectures.\n\n&nbsp;  \n**Additional Questions**\n\n1. Pruning methods can retain hard or noisy samples, leading to poor student performance (see Section 3.2). However, training a teacher on a (noisy) pruned dataset is not recommended, as it can result in a poorly trained teacher that degrades student’s performance.\n2. We expect our approach to generalize to other domains. Data pruning with KD is promising for NLP tasks, and while not tested on regression, we believe it could be suitable. As for RL, further investigation is needed.\n3. Data pruning reduces computational burden, e.g., in HPO or Active Learning, training many models on a small portion of the data with soft predictions from a full-data-trained teacher maintains high accuracy while lowering computational costs.\n\n4. As suggested, we will include a section on the method's limitations and future work.\n\nPlease let us know if you'd like us to elaborate on any point, as this grants 5,000 more characters.\n\n&nbsp;  \nWe hope our response clarifies your concerns, and we would appreciate it if you could consider increasing your score.", "claims_and_evidence": "Yes, the article's claims are well-supported by theoretical analysis and extensive empirical results. Below, I assess key claims and their corresponding evidence:\n\n1. Self-Distillation Reduces Bias in the Student Model\nClaim: Training a student with a teacher model trained on a larger dataset reduces the bias in the student’s estimation error.\n\nEvidence: The authors provide a mathematical derivation (Theorem 3.1) showing that self-distillation decreases bias.\nEmpirical results (Figure 5) demonstrate that increasing the teacher’s data fraction (𝑓𝑡) consistently improves student accuracy, supporting the claim.\n\n2. Knowledge Distillation Improves Accuracy Across Different Pruning Methods and Levels\nClaim: Incorporating KD consistently improves accuracy, even when training on heavily pruned datasets.\n\nEvidence: Figures 3 & 4 show that models trained with KD outperform those without KD across all pruning methods and datasets.\nResults indicate that models trained on only 10%-50% of the data with KD achieve accuracy comparable to full-data training.\nIn high-pruning scenarios, KD leads to 17-22% accuracy improvements over standard training (especially on CIFAR-100 and SVHN).\n\n3. Random Pruning + KD Matches or Outperforms Sophisticated Pruning Methods\nClaim: Using KD with simple random pruning can achieve accuracy comparable to or better than advanced score-based pruning methods.\n\nEvidence: Figure 4 shows that, at high pruning levels, random pruning with KD outperforms sophisticated methods like GraNd, EL2N, and Forgetting. The authors argue that aggressive pruning can accidentally retain noisy samples, making structured pruning less effective than random selection when combined with KD.\n\n4. The Optimal KD Weight (𝛼) depends on the Pruning Level\nClaim: The effectiveness of KD depends on the balance between the pruning factor (𝑓) and KD weight (𝛼).\n\nEvidence: Figure 6 shows that at lower pruning fractions (𝑓≤0.1), higher 𝛼 values yield better accuracy.\nThe authors explain that strong pruning increases label noise, and relying more on teacher predictions (higher 𝛼) helps reducing this.\nAs pruning becomes less aggressive, lower 𝛼 values work better since the dataset becomes cleaner.", "methods_and_evaluation_criteria": "The methods and evaluation criteria used in the paper are appropriate for the problem of training models on pruned datasets with knowledge distillation (KD).", "theoretical_claims": "The article presents a theoretical foundation for self-distillation in pruned data settings, but some claims lack direct empirical validation and broader generalization. While Theorem 3.1 suggests that training a teacher on a larger dataset reduces estimation bias, its proof is moved to the supplementary. \n\nAdditionally, the assumption of Gaussian noise and independent samples may not hold in real-world data distributions. The claim that KD significantly enhances generalization in low-data regimes is well-supported by empirical results but lacks a novel theoretical derivation beyond prior work. Providing explicit bias-variance decomposition experiments and relaxing assumptions in the theoretical analysis would be needed to verify the paper’s contributions.", "experimental_designs_or_analyses": "The experimental design is comprehensive, evaluating knowledge distillation (KD) under different pruning levels, datasets, and teacher-student configurations. The use of multiple pruning methods (e.g., forgetting, GraNd, EL2N, and random pruning) and four datasets (CIFAR-10, CIFAR-100, SVHN, and ImageNet) strengthens the validity of the results. \n\nHowever, some concerns remain:\n\n1. The study shows that random pruning can outperform more sophisticated score-based methods at high pruning levels, but it does not deeply analyze why this occurs. Additional analysis on the types of samples retained by each method is required.\n\n2. The study tests multiple temperature values but it does not thoroughly analyze the trade-offs between different temperatures in varying pruning conditions.\n\n3. The claim that high-capacity teachers harm students at low pruning factors is interesting but would benefit from an ablation study isolating architecture size from other factors (e.g., regularization or training dynamics).\n\n4. Table 1 does not contain any recent SOTA pruning method. The recent-most method compared with is from 2019.", "relation_to_broader_scientific_literature": "The paper builds on prior work in knowledge distillation (KD) and data pruning, particularly studies on model compression and sample selection.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\nThe paper presents a well-motivated exploration of knowledge distillation (KD) in pruned data settings, providing both theoretical and empirical support for its claims. The experimental results are thorough, spanning multiple datasets, pruning strategies, and teacher-student configurations.\n\nWeaknesses:\n\n1. Theoretical claims, while sound, rely on assumptions (e.g., Gaussian noise, linear regression framework) that may not generalize to more complex deep learning models. \n\n2. I can infer that some empirical findings, particularly regarding teacher capacity, lack deeper analysis on why larger teachers degrade performance under extreme pruning. \n\n3. The article could improve clarity by providing more intuition behind key mathematical results, and certain methodological details (e.g., hyperparameter tuning of KD temperature) remain underexplored.", "other_comments_or_suggestions": "1. The diagram in fig. 1 is over simplistic for an ICML submission.\n\n2. The capacity gap problem is an interesting finding, but further analysis (e.g., on feature representations or optimization dynamics) could provide deeper insights into why larger teachers degrade performance in extreme pruning scenarios.\n\n3. The choice of KD temperature significantly impacts results, yet details on tuning and sensitivity analysis are limited.\n\n4. Suggest experimenting with some DNNs beyond the ResNet and Wide ResNet family.", "questions_for_authors": "1. Given that pruning methods tend to retain hard (and potentially mislabeled) samples, have you analyzed how knowledge distillation interacts with label noise? Could a poorly trained teacher amplify errors instead of eliminating them?\n\n2. Your experiments focus on image classification tasks. Do you expect similar improvements in other domains (e.g., NLP, structured data)? Have you tested the method on regression or reinforcement learning tasks?\n\n3. Given the computational cost of training multiple models (teacher and student) on different dataset fractions, how does this approach compare in terms of efficiency versus direct training on full or pruned datasets? Have you considered any strategies to reduce the added computational burden in real-world applications?\n\n4. Discuss about the limitations of this method and future works to be done.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "XQd6yNyT38", "reviewer": "Reviewer_RfVw", "summary": "The paper explores the use of knowledge distillation (KD) for enhancing training on pruned datasets, demonstrating that simple random pruning with KD can achieve superior accuracy compared to recent data pruning methods. The work also reveals that, when using teachers with smaller capacities, the student can be more beneficial in low pruning fractions. The study provides theoretical motivation and empirical evidence, showing that KD helps mitigate the impact of label noise and improve accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank **Reviewer RfVw** for their careful reading, thoughtful remarks and the positive feedback. In addition, we appreciate their acknowledgement of the novelty, simplicity and effectiveness of our proposed method. \n\nBelow, we kindly address each of the reviewer's questions and concerns:\n\n&nbsp;  \n\n**Comparison with More Recent Approaches**\n\nWe appreciate the reviewer's suggestion to compare our proposed method with more recent data pruning approaches. Following their suggestion and given the time constraints, we have recently conducted additional experiments to compare our method on CIFAR-100 with 3 out of the 4 methods proposed by the reviewer, namely:\n- **Moderate-DS [1]**\n- **D2 [2]**\n- **DUAL [3]**\n\nFor a fair comparison of these methods with our method, we have utilized the official implementation of each one to generate the pruning scores using a common ResNet34 architecture. In addition, all of the necessary hyper-parameters were taken from the respective paper and/or supplementary material of each method.\n\nThe results of this experiment can be viewed in the following link:\n\n[experiments_recent_pruning_methods.png](https://ibb.co/xKB0F9hq).\n\nAs can observed, all pruning methods (including these recent ones) benefit from the incorporation of KD in the training process. In addition, it can be seen that on low pruning fractions random pruning + KD maintains its edge over all other methods, except for D2 + KD which achieves a similar performance. However, contrary to D2 which requires careful tuning of several hyperparameters (k,​$\\gamma_{r}$,β) for each pruning fraction and each dataset, our method (simple random pruning + KD) has no such requirements and is hence more practical and easier to adopt for real-world use cases.\n\nWe note that the last method the reviewer has referred to **[4]** is focused on the problem of *data pruning with re-labeling*. This sub-task of data pruning attempts to re-label noisy samples in a given dataset, and then select the subset of the data with the most accurate re-labeling of erroneous labels. Hence, all experiments in that work are strictly conducted on noisy variants of traditional datasets (e.g., CIFAR-10N, CIFAR-100N), which makes comparisons of that method with traditional data pruning approaches non-trivial.\n\nFinally, we note that we have incorporated all pruning methods suggested by the reviewer into the Related Works section.\n\n&nbsp;  \n\n\n**Discussion: Application of Our Method to LLMs and VLMs**\n\nWe thank the reviewer for this keen observation. For simplicity, all experiments in our paper were conducted on the image classification task in the vision domain. However, we agree that our proposed method (utilizing KD with data pruning) has great potential to be utilized in other domains, like NLP or multimodal domains. This is especially true nowadays since with the recent rise of AI-based methods over the last few years, and the high training costs that came along with them, developing more efficient training algorithms have become more important than ever. For example, we believe that it's worth exploring whether one can reduce the training costs of a given LLM (while maintaining a certain level of accuracy), by training it on a carefully pruned corpus of data with additional guidance from the logits of an informed teacher (another LLM trained on a larger corpus). We hope our work here, both empirical and theoretical, will incentivize other researchers to explore such intriguing directions in the future.\n\n\n&nbsp;  \n\n\n**[1]** Moderate Coreset: A Universal Method of Data Selection for Real-world Data-efficient Deep Learning, ICLR, 2023.\n\n**[2]** D2 Pruning: Message Passing for Balancing Diversity & Difficulty in Data Pruning, ICLR, 2024.\n\n**[3]** Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty, ArXiv, 2025.\n\n**[4]** Robust Data Pruning under Label Noise via Maximizing Re-labeling Accuracy, NeurIPS, 2023.", "claims_and_evidence": "The main claim of this paper is that a simple random pruning method, when combined with knowledge distillation (KD), outperforms all standard data pruning algorithms that rely on hard labels, and this claim is supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed method is technically sound, and the extensive experiments on multiple datasets including ImageNet demonstrate its superiority.", "theoretical_claims": "The theoretical motivation in Section 3.3 explains the benefits of using self-distillation for enhancing training on pruned data. The authors show that self-distillation using a teacher trained on a larger dataset can reduce the bias of the student's estimation error using the context of regularized linear regression.", "experimental_designs_or_analyses": "The experiment setup is extensive and well-constructed.", "relation_to_broader_scientific_literature": "The paper successfully extends the data pruning framework with KD, and its empirical strength promises to offer new perspectives and directions in data pruning research. Most importantly, a major strength lies in its ease of implementation.", "essential_references_not_discussed": "While the paper introduces a simple yet effective approach in data pruning literature, several of recent works are not covered, including:\n- Moderate Coreset: A Universal Method of Data Selection for Real-world Data-efficient Deep Learning, ICLR, 2022\n- Robust Data Pruning under Label Noise via Maximizing Re-labeling Accuracy, NeurIPS, 2023\n- D^2  Pruning: Message Passing for Balancing Diversity & Difficulty in Data Pruning, ICLR, 2024\n- Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty, ArXiv, 2025", "other_strengths_and_weaknesses": "Strength\n- The proposed method is novel, simple, and effective\n- Extensive experiments support the empirical strength of the proposed algorithm\n- The paper is well-written and easy-to-follow\n\nWeakness\n- The reviewer's main concern is the missing of the several important works (in review and experiments as baselines)", "other_comments_or_suggestions": "Can the idea of using knowledge distillation for data pruning also be applied to data pruning for vision-language models (VLM) [1] or large language models (LLM) [2]?\n\n[1] Too Large; Data Reduction for Vision-Language Pre-Training, ICCV, 2023\n\n[2] Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models, ICLR, 2025", "questions_for_authors": "See other review sections above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Ah0nf8pGx", "reviewer": "Reviewer_J77D", "summary": "The key contribution of this paper is to show that using soft predictions from teachers trained on complete data combined with prune data improves the accuracy of students trained on pruned data. The authors conduct a series of experiments with ResNet variations on small datasets, including CIFAR-10, CIFAR-100, and SVHN, to demonstrate this. The results clearly support the authors' claims.\n\nThe paper is easy to follow. The experiments support the claim that using knowledge distillation (KD) improves the performance of students trained on pruned datasets. The authors also provide theoretical motivation for why KD of teacher trained on complete data is helpful.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their valuable feedback and for highlighting the strengths of our work. We appreciate the constructive suggestions and are committed to addressing the concerns raised.\n\n&nbsp;  \n\n**Experimental Results with Disjoint Datasets**\n\nFollowing the reviewer's suggestion, we conducted experiments to evaluate the case where the pruned data is not a subset of the complete dataset used to train the teacher.\n\n\nLet $\\mathcal{P}$  be a pruned dataset sampled from $\\mathcal{D}$ to train the student model, and let $\\mathcal{S}$ be the training data for the teacher. In the following experiments, $\\mathcal{D}$ and $\\mathcal{S}$ are disjoint i.e.,  $\\mathcal{P} \\cap \\mathcal{S} = \\emptyset$.\n\nFor the empirical study, we used 70% of the training data to train the teacher and the remaining 30% to train the student with different pruning ratios. Specifically, we compared the performance with and without knowledge distillation (KD) for CIFAR-100 and SVHN datasets.\n\nThe experimental results can be viewed in the following link:\n\n[Experimental_Results_with_Disjoint_datasets.png](https://ibb.co/dwP4fDyf)\n\nNotably, combining knowledge distillation (KD) with data pruning yields significant performance gains, even when the student is trained on a pruned dataset that differs from the teacher's training data. For instance, in CIFAR-100 with random pruning at $f=50$%, we observe a 14.5-point accuracy improvement when the teacher model was trained on a different subset.\n\n&nbsp;  \n\nWe are happy to include these experimental results in the main paper. We believe that these findings further support our proposed approach, particularly in the context discussed at the end of Section 3.1, namely use cases where the full dataset is no longer accessible (e.g. due to privacy concerns).\n\n&nbsp;  \n\n**Ablation Studies on Temperature**\n\nFollowing the reviewer's suggestion to include ablation studies on temperature, we kindly refer to Figure 10 in the appendix, where we present accuracy results for various temperature values across different pruning fractions and architectures.\n\n&nbsp;  \n\n\nWe hope our response addresses your concerns, and we would greatly appreciate it if you could consider raising your rating of our work.", "claims_and_evidence": "The claim in the paper was supported with experiments, however I have one main concern as below.", "methods_and_evaluation_criteria": "While the paper presents some observations that I have not seen before, my main concern is the practicality of the setup. If the teacher has already been trained on complete data, what is the value of using pruned data to train the student, given that most of the knowledge can come from the teacher? It seems to me that this approach is only useful if the pruned data is not a subset of the complete dataset or if the authors can demonstrate the value of using pruned data in this setup. Could the authors provide either comparison as follows?\n\n1. Conduct similar experiments to show improvements when the pruned data is not a subset of the complete dataset used to train the teacher.\n\n2. Or provide results where the student is trained solely using the teacher’s soft predictions, without pruned data (I assume this corresponds to f = 0?), as a baseline. This would help demonstrate the value of using pruned data to train the student.\n\nThe temperature parameter is quite important when training with knowledge distillation (KD). Therefore, I suggest that the authors include ablation studies on temperatures in these experiments to ensure the best temperature is selected for fair comparisons.\n\nBecause of this, I tend to rate it as weak reject but happy to increase the rating if my concern is addressed.", "theoretical_claims": "The theoretical proof sounds and seems supports the the claim, but I have not check the correctness of the proofs.", "experimental_designs_or_analyses": "See above.", "relation_to_broader_scientific_literature": "The paper might have the small impacts", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "see above", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cFQPoMZHvF", "reviewer": "Reviewer_xePE", "summary": "Dataset pruning is the task of reducing the number of samples within a dataset without impairing accuracy. This article combines dataset pruning with methods inspired by the knowledge distillation (KD) literature by augmenting the training process with soft labels from a teacher network that was trained on the full dataset. As a result, the approach can be used to improve existing pruning methods (higher accuracy with the same number of samples / matched accuracy with fewer samples).\n\n## Update After Rebuttal:\nThe authors were responsive and addressed my concerns, thus I'm in favor of acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank **Reviewer xePE** for the thoughtful reading, positive feedback and appreciation for the novelty and simplicity of our proposed approach. Below, we kindly address each of the reviewer's questions and concerns:\n\n**1. Comparison with a Stronger Baseline (100% Data, KD)**\n\nWe thank the reviewer for this question, and kindly note that the baseline they refer to (100% data, KD) is included already in the paper. As part of our CIFAR-100 and ImageNet experiments (figures 3a, 4a, 4b), this baseline is represented as the joint plot point for all experiments which utilize $f=1$ with KD (and regardless of the pruning method). We emphasize that this plot point is joint for all KD experiments regardless of the pruning method, since when the entire dataset is utilized ($f=1$) no pruning is being conducted, and hence the selected pruning approach has no effect. Following the reviewer’s suggestion, we will include the exact accuracy numbers in the main paper.\n\nIn addition, as can be observed in the aforementioned figures, performance can be mostly preserved w.r.t to this strong baseline across several pruning factors when using several pruning methods with KD. For example, on CIFAR-100, performance is mostly preserved when combining most pruning methods with KD and retaining only 60% of the data.\nFinally, we note that this stronger baseline is not shown for the SVHN and CIFAR-10 experiments (figures 3b, 3c), since, as can be observed in the aforementioned figures, model performance gets saturated very quickly on these datasets. Hence, performance on higher pruning factors was omitted for these datasets for visualization purposes.\n\n**2. Comparison to the SSL Prototypes Method [1]**\n\nWhile we haven’t compared our approach to this specific method, we have recently compared it with other (more recent) approaches, namely: **Moderate-DS [2]**, **D2 [3]**, and **DUAL [4]**. The comparison results can be viewed in this figure:\n\n[experiments_recent_pruning_methods.png](https://ibb.co/xKB0F9hq)\n\nAs depicted in the figure, all pruning methods (including the recent ones) benefit from the incorporation of KD in the training process. In addition, it can be seen that on low pruning fractions, random pruning + KD maintains its edge over all other methods, except for D2 + KD which achieves a similar performance. However, contrary to D2 which requires careful tuning of several hyperparameters (k,$\\gamma_{r}$​,β) for each pruning fraction and for each dataset, our method (simple random pruning + KD) has no such requirements and is hence more practical and easier to adopt for real-world use cases.\n\n**3. Citation Formatting Issues**\n\nWe thank the reviewer for bringing these issues to our attention. These issues will be corrected in the final version of the paper.\n\n**4. Figure 5 Style**\n\nFollowing the reviewer’s suggestion, we have revised Figure 5 by using a sequential color palette and adjusting the figure style to match the other figures in the paper. The revised figure can be observed here:\n\n[updated_figure_5_accuracy_vs_teacher_data_fraction.png](https://ibb.co/cXCVqZnK)\n\nWe thank the reviewer for this comment. \n\n$~$\n\n**[1]** Beyond neural scaling laws: beating power law scaling via data pruning, NeurIPS, 2022. \n\n**[2]** Moderate Coreset: A Universal Method of Data Selection for Real-world Data-efficient Deep Learning, ICLR, 2023. \n\n**[3]** D2 Pruning: Message Passing for Balancing Diversity & Difficulty in Data Pruning, ICLR, 2024. \n\n**[4]** Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty, ArXiv, 2025.", "claims_and_evidence": "To my knowledge, the first paper that combines KD with dataset pruning. Most of the empirical results are done on small-scale datasets (CIFAR, SVHN) which is a bit of a drawback. At the same time, pruning experiments are expensive, and not all academic labs have the resources to run dozens / hundreds of ImageNet training runs, thus I don't think this should be held against the paper.\n\nThe empirical results are strong; KD leads to strongly improved accuracies and can be combined with existing pruning methods, which is nice. KD of course comes with the limitation that one needs to train a full model on the entire dataset first; this is definitely a limiting factor for some scenarios but nonetheless, if one would like to run lots of ablations / variants one could train the full model once, and then run the ablations on a smaller subset of the dataset, thus there are certain cases where KD can be helpful and still save compute.\n\nExperimentally, **my biggest concern is that results are compared against the baseline of \"100% data, no KD\". If KD is proposed as part of the training procedure, shouldn't a stronger baseline be the case of \"100% data, KD\"?** If so, which methods / pruning metrics still maintain accuracy w.r.t. this stronger baseline?", "methods_and_evaluation_criteria": "The benchmark datasets and comparison metrics are standard in the literature (accuracy with varying pruning fractions). The core idea - combining KD with dataset pruning - is compellingly simple and straightforward.", "theoretical_claims": "I did not carefully assess the theory since dataset pruning is very much an empirical field; while a theoretical motivation is great the ultimate test that decides whether a dataset pruning metric will be useful are experimental results.", "experimental_designs_or_analyses": "I checked the soundness and validity of the experimental design. Overall it is sound; the two downsides to the employed approach are that (1.) a teacher [trained on the full dataset] is needed; (2.) the approach introduces an additional hyperparameter $\\alpha$ controlling the loss weight (cf. Figure 6). However, none of those two downsides invalidate the approach.", "relation_to_broader_scientific_literature": "To my knowledge, this is the first paper that combines KD with dataset pruning. The focus is on pruning, while the approach is based on KD. I appreciate the transfer / bridging of those two related subfields.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "The paper is well written overall, accessible, and clear. Figure 1 is a great overview; I found in particular subfigure 1c intriguing since it's a very systematic exploration, with a clear (and at first sight, counter-intuitive) result.", "other_comments_or_suggestions": "- Would recommend a pass checking for \\cite{}, \\citet{}, \\citep{} differences - e.g. when talking about a specific paper by Authors (YYYY) this should not be cited as (Authors, YYYY) in the text.\n- Nit: Figure 5 has a different plotting style compared to other figures (e.g. grid). Furthermore, I would suggest to use a sequential color palette for Figure 5 due to the nature of the sequential data.", "questions_for_authors": "Major:\n- [copied from above] Experimentally, my biggest concern is that results are compared against the baseline of \"100% data, no KD\". If KD is proposed as part of the training procedure, shouldn't a stronger baseline be the case of \"100% data, KD\"? If so, which methods / pruning metrics still maintain accuracy w.r.t. this stronger baseline?\n\nMinor:\n- Given that the authors mentioned that they used several metrics from https://github.com/rgeirhos/dataset-pruning-metrics, why not the SSL prototypes metric (which is one of the best-performing ones)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Three of the four reviewers rated the paper positively (1 accept, 2 weak accept, one weak reject).

The paper presents a novel approach to dataset pruning, which augments the training process with soft labels from a teacher network that was trained on the full dataset.

The paper's claims are well-supported by theoretical analysis and extensive empirical results. In particular, the main claim of this paper - that a simple random pruning method, when combined with knowledge distillation (KD), outperforms all standard data pruning algorithms that rely on hard labels - is supported by clear and convincing evidence.

There was a concern about the novelty of combining knowledge distillation with data pruning. However the review discussion clarified that while there is some recent work on dataset distillation, the novelty of the paper lies in utilizing knowledge distillation within data pruning. 
The paper demonstrates that in the presence of KD, model accuracy remains robust regardless of the data pruning method employed, suggesting that simple random pruning, when combined with KD, is a viable alternative to more sophisticated pruning methods.

The assumption of Gaussian noise and independent samples may not hold in real-world data distributions. The claim that KD significantly enhances generalization in low-data regimes is well-supported by empirical results but lacks a novel theoretical derivation beyond prior work. Explicit bias-variance decomposition experiments and relaxing assumptions in the theoretical analysis would strengthen the paper.

Two of the reviewers suggest examining more closely the role of temperature. The study tests multiple temperature values but it does not thoroughly analyze the trade-offs between different temperatures in varying pruning conditions. The choice of KD temperature significantly impacts results, yet details on tuning and sensitivity analysis are limited. However, as pointed out by the authors, this issue is covered already to some extent in appendix D, figure 10, which should be merged with the main paper.
Decision: Accept (poster)