Teach Less, Learn More: On the Undistillable Classes in Knowledge Distillation
Authors: Yichen Zhu
Ning Liu
Zhiyuan Xu
Xin Liu
Weibin Meng
Louis Wang
Zhicai Ou
Jian Tang
Venue: NeurIPS 2022 Accept (2022)

Abstract: Knowledge distillation (KD) can effectively compress neural networks by training a smaller network (student) to simulate the behavior of a larger one (teacher). A counter-intuitive observation is that a more expansive teacher does not make a better student, but the reasons for this phenomenon remain unclear. In this paper, we demonstrate that this is directly attributed to the presence of  \textit{undistillable classes}: when trained with distillation, the teacher's knowledge of some classes is incomprehensible to the student model. We observe that while KD improves the overall accuracy, it is at the cost of the model becoming inaccurate in these undistillable classes. After establishing their widespread existence in state-of-the-art distillation methods, we illustrate their correlation with the capacity gap between teacher and student models. Finally, we present a simple Teach Less Learn More (TLLM) framework to identify and discard the undistillable classes during training. We validate the effectiveness of our approach on multiple datasets with varying network architectures. In all settings, our proposed method is able to exceed the performance of competitive state-of-the-art techniques. 

Teach Less, Learn More: On the Undistillable Classes in Knowledge Distillation
Yichen Zhu, Midea Group
Ning Liu Midea Group
Zhiyuan Xu Midea Group
Xin Liu East China Normal University
Weibing Meng
Yi Wang Midea Group
Zhicai Ou Midea Group
Jian Tang" Midea Group
Tsinghua University
Abstract
Knowledge distillation (KD) can effectively compress neural networks by training a smaller network (student) to simulate the behavior of larger one (teacher). A counter-intuitive observation is that a more expansive teacher does not make better student, but the reasons for this phenomenon remain unclear. In this paper, we demonstrate that this is directly attributed to the presence of undistillable classes: when trained with distillation, the teacher's knowledge of some classes is incomprehensible to the student model. We observe that while KD improves the overall accuracy, it is at the cost of the model becoming inaccurate in these undistillable classes. After establishing their widespread existence in state-of- the-art distillation methods, between teacher and student models. Finally, we present a simple "Teach Less Learn More" (TLLM) framework identify and discard the undistillable classes during training. We validate the effectiveness of our approach on multiple datasets with varying network architectures. In all settings, our proposed method is able exceed the performance of competitive saate-of-the-eee techniques.
illustrate their correlation with the capacity gap
Introduction
More accurate machine learning models often demand more computation and memory at test time, making them difficult to deploy on computational-conste alleviates this burden by transferring the knowledge from expensive teacher model to cheap student model to improve the performance of this more efficient network. Over the past several years, have witnessed huge success in knowledge distillation and a surge of related literature on designing better distillation techniques.
devices. Knowledge distillation (KD)
An intuitive thought of knowledge distillation is that better teachers are supposed to make better students since a larger model can learn more visually meaningful and taskerelated information, and a student model can presumably benefit from learning with more informative knowledge. However, recent works show that such intuition is incorrect. In fact, prior works found that students distilled from bigger teacher, with more parameters and higher accuracy, can perform worse than the same students distilled from a smaller teacher [23, 9, 36, 25, 33). The situation is getting more severe when training large-scale, challenging dataset such ImageNet [5]. More surprisingly, the model with the same architecture [49] or even a model with lower performance [18, 46, 50] can be used as a teacher network to perform knowledge distillation, which plays role of regularization [32, 24, 52]. Indeed, a large body of researches [55, 35, 53, 17, 34] devoted to solving this problem, but few minimal efforts have been put into understanding this phenomenon.
"ooresponniig Author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In this paper, we propose new data-centric perspective on the phenomenon of "larger teacher, worse student".. In contrast the previous works, we cast the inefficacy of large teachers as a fundamental consequence of the class-dependent bias in KD. Specifically, we claim that:
The undistillable classes are direct result of the inefficacy of large teachers in distillation.
Recall that prior works attribute the failure of large teachers in KD to the capacity mismatch between the teacher and student model. In other words, the student model cannot learn the teacher's knowledge due to their insufficient expressive power. Indeed, we find that their claim is true for some classes that posses bad distillability. We posit that these classes cause seemingly mismatched capacity over all samples in the dataset.
The empirical study supports our hypothesis. For samples from the undistillable classes, the feature representation similarity between the teacher and student model is extremely low compared to the regular classes. Furthermore, perhaps less surprisingly, the distilled model could obtain lower aceuracy on these undistillable classes than their maaiill-ttrinee counterpart. To further corroborate our theory, we show that the number of undistillable classes increases when the capacity gap between the two models increases.
After verifying our hypothesis, natural question would be raised: to what extent do these undistillable classes exist? When discussing the property of knowledge distillation, previous literature typically evaluates vanilla knowledge distillation [14]. However, considering the evolution of distillation techniques over the past several years, we believe it necessary to expand our study. To this end, we investigate the existence of the undistillable classes over three standard datasets (CIFAR 10 [21], ImageNet1 K [6]. CUB-200 [42]) with more than 20 modern distillation techniques. We also include the evaluation of advanced architectures, i.e., vision transformer [39]. Our study shows that the undistillable classes generally exist, usually taking over from 10% to 50% of the total classes, depending on the distillation methods, datasets, and teacher-student pairs.
To resolve this issue, we present a novel distillation framework called "Teach Less, Learn More" distillation (TLLM). It is motivated by the successful teaching strategy in pedagogy(37], where teachers are encouraged to do less rote learning and provide more spaces for students to explore and discover their talents. Our methodology is conceptually similar: we aim to identify the undistillable classes and remove them entirely during distillation, allowing the student model to explore the representations of these classes with their own capacity. Specifically, we record the per-class teaching curve (distillation losses) and calculate its change over a moving window of K epochs. If the teaching curve for some particular classes is stagnated or upward, we consider the knowledge transfer process for these classes no longer necessary. Then, we remove these classes in distillation (if it is an output- based KD, we replace them with the label smoothing technique to compensate for the regularization effect). Our approach, despite its simplicity, is proven to be effective in preventing the adverse effect brought by the undistillable classes and improving the overall accuracy.
Overall, we emphasize our contributions are the following: 1) we analytically show that the undistill- able classes are the cause of inefficacy of the large teacher model; 2) we demonstrate that the accuracy of these undistillable classes drops after distillation, and our extensive experiments showing this is a general phenomenon, irrespective of distillation methods, datasets, or teacher-student pairs; 3) we propose a simple yet effective framework to identify and discard undistillable classes and illustrate the effectiveness our approach with extensive experiments.
2 On the Undistillable Classes in Knowledge Distillation
2.1 Formal Definition
In this section, we formally define the undistillable classes. Considering an multi-class predictor
F ddd + & as student and another predictor G idd + R* as teacher. Giving a traing dataset S (1., y4 1 } ~ 1m for distribution P over set of instance X' and label 3' = [L] [1, 2, ... L]. In the standard training, the goal is to approximate the risk R(f; S) of functional class f E F via the empirical risk R((; S) = 1/2 En ((,,f(x), f()). for any bounded loss. For knowledge distillation, we can rewrite the distilled risk as R)(F; = E.. (g((x), f()),, where the functional class g E G.
The conventional setting consider classes equally. Without loss of generality, we will consider the S == S. U., ~ P", where we split the data over distribution IP into two categories, each category
2

---

Class-level Distillability
Samples From Undistilable
Samples From Regular
chan
dee
chasc seprrtua
clax bridgs
ddplin
clbss otter
darc drow
cbsc reaks
dlace
das: spáker
dlsc whole
Figure I: The representation agreement measured CKA between teacher model and student model. For student network, the samples from undistillable classes clearly fail to match the representation from teacher model.
contains non-overlapping instances S, = {(,, 9/) ( Ln+2. L}. S, S Sn, |- le be appropriate distillability measurement. In our context, define the distillability function le as the difference in performance between the distilled student model with the train- from-scratch counterpart under certain evaluation metrics. Let E(-) be an evaluation metric that measures the performance of a particular multi-class predictor on S. Then, formally, we define le = E(Faa) F(Foanilla). Finally, we define that S,, are set of samples from the undistillable classes distillation iff f;; Sule < S,
2.2 Mismatched Representation for Undistillable Classes
The previous section formally defines the undistillable classes as classes with bad distillability. In this section, we develop criterion to measure the distillability of class quantitatively. Recall that the student is typically encouraged to mimic the behavior the teacher model. Therefore, a converged student should impeccably match the teacher's 's behavior in an ideal setting. It implies that their feature representations should be the same as the teachers', given the same input on both models. As result, harness the similarity of feature representations between teacher and student to reflect the level of distillability of the student model. Inspired by Zhu et al. [55], we use the Center Kernel Alignment (CKA) [20] calculate the representation similarity between two neural networks. To create a typical case of capacity mismatch, we follow the previous works [5] to use ResNet56 as our pre-trained teacher model and modified ResNet24 the student model (see Appendix for details). We perform standard feature knowledge distillation [31] on stage four, which contains nine consecutive convolutional blocks. We choose compare the network representation at convergence. For the CKA score figure, we primarily focus the score on the diagonal, which indicates the representation similarity of the convolutional layer the same position in the network. order to compare the representation similarity of samples from different types of classes, we deliberately choose 12 test samples from distinct classes. Note that we use samples from the test set ensure that the observed behavior is not due to ad-hoc memorization.
In the middle of Figure the feature similarity between teacher and student on each corresponding layer is presented. We can observe that the CKA scores for most layers are less than 0.6, and for some layers are even less than 0.5. These results indicate that despite the student indeed learning to match the teacher's representations, their features are far from perfect alignment. Notably, the representation similarity score calculated based on the average score over a batch. Therefore, we decouple the matrices and check the CKA score for each sample individually. On the left side and right side of Figure 1, we give the CKA matrices the last three layers for 12 samples. Each side contains six samples. We notice a clear trend: the CKA scores for samples from the left side are obviously much lower than those on the right side, indicating that the mismatched representations are not uniformly distributed over all samples. Therefore, we name the classes on the left side as undistillable classes. Our observation contradicts the conclusion from the prior work, which attributes the mismatched capacity general issue for all data [55, 5, 23]. We later show that at the left top of Figure 4, the accuracy for the undistillable classes drops after training with KD compared their train- coan-from-soee scratch counterpart.
3

---

ResNedS2xt
ResNer322d ShuileeetVV1
Perach's
APA
APA
APA
Figure 2: The APA versus AAce for three teacher-student pairs on CIFAR100. The Pearson's is higher than 0.65 for all figures.
2.3 Bad Distillability hurts Accuracy
As aforementioned, a model would contain some classes with bad distillability. But what is the consequence of distilling knowledge to these classes? In this section, show that the quality of KD (i.e., accuracy) highly correlated to their distillability. Here we record the difference in accuracy between the vanilla trained model (training without distillation) and the model trained with knowledge distillation. Particularly, we adopt per-class accuracy:
AACC ACC ACCka
where E Y. The motivation behind this metric is straightforward: if the AACC > 0. then for mh class, the effect of KD is positive; otherwise, the KD hurts the student's s performance on mh class. To demonstrate the correlation between per-class accuracy and per-class distillability, a naive approach is to compare the AACC with the CKA scores, as shown in the previous section. However, CKA measures per-layer feature representation similarity; simply averaging over the total number of layers may be inaccurate, not to mention that the teacher model usually contains more layers than students. Therefore, we present new metric based on average prediction agreement:
PA 1 Σ n2
T
argmax (24,)) argmax (2g,5)
(2)
exp(x;) where (2,) ΣS exp(2;) and (2.) exp(s.) Σ, exp(2.) is the output of softmax function for teacher model and student model, respectively. Equation 2 is inspired by the fact that if the feature representations between models are exactly the same, these two models ought produce the same top-I I label. Similar to Acc, here we focus on the difference in prediction agreement APA between two models; one is trained from scratch, and another trained with distillation:
APA
PA
PA
Specifically, we put these two metrics in scatter plot chart for three varying teacher-studen pairs on CIFAR 100. Figure 2 reveals that for most of the classes with AACC < 0, its corresponding APA < as well, indicating that if the student become disagrees with the teacher's prediction, the teacher's supervision can hurt student's performance on this particular class. In addition, we can observe that the level of disagreement is linearly correlated to the level of performance change: when measured with the Pearson's r, it obtained over 0.65 with a p-value less than 0.0005 all three teacher-student combination, showing a statistically significant linear correlation between these two metries. In conclusion, for any class in the student model, if it is undistillable. it certainly has AACOM 0.
2.4 The Larger the Capacity Gap, the More the Undistillable Classes
Now, we redirect our attention to the correlation between the level of capacity gap and the presence of the undistillable classes. We propose to leverage multiple teacher networks with varying capacity to distilled single, low-capacity student. Specifically, select the ResNet20 as our student model,

---

Capacity Gap versus Per Class Accuracy
10st not
Proportion of Undistillable Classes
MM
MM M M
MMM
ResNet66
A0X (N0
ResNuc110
undatdiable
repular cheses
43
63
150
Teadhen
Resiieedd
Modhet303
Stadent: BeaNie220
Figure 3: Left: Per-class ACC and per-class AACC for three teacher-student pairs CIFAR100.
Right: Proportion of Undistillable Classes distilling ResNet20 with three teacher models.
and pick ResNet44, ResNet56, and ResNetl 10 as three individual teacher model. For each set of experiments, adopt one teacher model from the above train student model via vanilla KD [14]. This procedure mirrors that of mismatched capacity experiments in Cho et al. [5], except that our goal investigate the relationship of per-class accuracy with the capacity gap. As shown in Figure we observe a clear relation between increase of the number undistillable classes, as increase in the capacity gap between models. Specifically, the number total undistillable classes for the smallest student model (ResNet44) is almost tripled compare to the largest largest model (ResNet101).
2.5 On the Existence of the Undistillable Classes
The knowledge distillation is well-known for its "free" gain over compact models on average on an i.i.d test set. Modern distillation techniques only focus improving the overall accuracy of the student model, with an intuition that the distillation is beneficial for all classes. Indeed, the overall accuracy paramount, but the per-class accuracy also important in many cases. Nevertheless, the distillation method inevitably brings class- class-dependent bias into the student model. As briefly shown in Figure the distilled student model shows obvious class-dependent bias all datasets. It is also worth noting that the accuracy drops dramatically for some undistillable classes. For instance, on ImageNetIK, K, there are 127 classes that lose over 4% accuracy compared to their vanilla trained counterpart, and more than nine classes lose over 10% on the test set.
In recent years, the techniques of distillation have evolved dramatically, i.e., feature-based distillation and self-supervision-bassee distillation [38]. Therefore, to ensure the undistillable classes are not unique byproduct of the vanilla KD, extend our experiments include 26 state-of-the-at distillation frameworks, each of them evaluated by five different teacher-student pairs on CIFAR 100.
We categorize them into two groups (output-base) KD and feature-based KD). Table I give a list of all methods that we used for evaluation. We only present the results for 13 of them due to limited space; the complete analysis and more evaluation of ImageNet-IK can be found in Appendix. Figure shows the number of undistillable classes. It surprising to show that all of them contain undistillable classes. For instance, the Resmet32x4:Resneee the half of the total classes. Despite so many undistillable classes, all distillation methods improve vanilla student model, some by large margin. Such a phenomenon indicates that the effect of knowledge distillation is non-uniform at the class level. Moreover, the proportion of the undistillable classes taking over from 10% to 50% over the total number classes. Therefore, is critical and necessary to control and reduce the number of undistillable classes in the student network.
with vanilla KD has 52 undistillable classes, more
5

---

Table 1: A summary of all distillation methods that used for evaluation in Section 2.5
[D[14]
[3]
(47
(P14
CC]2
RKD (27]
[25]
[B
[T
TSP [15]
[16] CRD]38]
Pub
Arsiv 14
KCLR'15
Ausiv
1CCV"19
JCCV 19
CVPR
CVPR
ECCV'S 1s
AAAFI9 Neuris
CVPR'17
Arxiv
ICLR'19
Ouipul
x
X
V
Feature
Mothod
SSSD (45] PAD
WSKD [54] LST (33)
(12)
TA [25]
ESKD [5] SCKD [55]
KR [3]
SPTN [26
CI (7)
FRD [1]]
DKD [51]
Puh
ICIRR'0
ICLR/20
IOCV' 19
20
19
ICCV'21
ICCV"21
Neurips
Neurigs
ICLR
Ourput
Feature
X
bowl
crocodinn
Ion
ote
shre
snkkn
nastrull
IoAhne
groebesk
belled
kilbwake
Re4 S: Re
ResNet50 S: ResNet18
20
10
10
& A.ccc (%)
-10
12
CIFAR10O
CUB200 Class
RegNetY DelT-Small
-3
-10
50
100
150
200
250
300
352
400
450
ImageNet Class
Figure 4: The per-class AACC of knowledge distillation on three datasets (CI- FARI00 121//mmaeNettKK [6]/CBB-200 [42]) with vanilla KD [14]. We only show the first 500 class in ImageNet1K and the first 100 classes in CUB-200. The dashed line denotes the improvement of average accuracy using KD.
Teach Less, Learn More (TLLM): Identifying and Discarding the Undistillable Classes
We have stressed the adverse effect and characterized the properties of the undistillable classes from previous section. this end, we give a simple framework, named "Teach Less, Learn More" distillation (TLLM), which analogy to successful teaching principal in the pedagogy, to build better KD algorithms. Our idea is conceptually similar to the TLLM in the pedagogy, where the student encouraged to have more autonomy and devise their own solutions to problems. In practice, the student model supervised by both teacher's knowledge and the ground truth labels. Training with the model's task-oriented objectives is critical to the eventual performance [14], and provably to correct the wrong gradient directions that are brought by the teacher's s mistaken knowledge [18]. Therefore, what we do is simple: identify the undistillable classes and then remove them from the distillation process. As result, the student can learn these classes directly from the ground truth labels without interference.
The critical challenge of TL.LM how to find the undistillable classes during training. naive yet computational expansive approach is first to train the model to convergence with distillation and then

---

Reinet20ck ResnetBx4
WRN-4U-2: W8N-16-2
Resnet50: MobileNetV2
VG613: VGGB
KD
FaNa
AT
SP
VID
RED
PRT
AB
FSP
1
Knowledge distillation
Iepar byees
tyyess
Curgert loger
Student
Teacher
Number of Undisillable Classes
data
Distill
abeer
befoer
efter
Reguker
Figure Top: the number undistillable classes for diverse teacher-student pair evaluated by various distillation methods. More can be found in the Appendix. Bottom left: Three types of distillation approaches. Bottom middle: Definition of undistillable classes. Bottom right: Each size of circle represent range of undistillable class number.
retrain by manually eliminating the undistillable classes according to the performance on the test set. Instead, we propose to monitor the teaching curves (distillation losses) and set up criterion to find the undistillable classes during training that are more computationally efficient.
Specifically, we pay attention to the per-class teaching curve, i.e., teaching curve (( f(x)) on th class, instead of the averaged distillation loss over the entire datasets. The teaching curve is calculated on the validation set. Then,
moving window over K epochs, which is started at 5th epoch and end at (S K)* th epoch. During the distillation process, we record and update the per-class teaching curve Its [1m: f(r)), Is+ (ym. f f()))) If Is (5 f()) Is+ K (ym f(x)) S S 17, meaning that the loss is moving upward the epochs moving window. Figure 3 gives an example of the teaching curve for an undistillable class. After training with approximately 50 epochs, the loss for this particular class suddenly jumps. Therefore, our method can capture such behavior, then identify and remove these undistillable classes during training. It is worth noting that these classes remain the task-triented objective for the student, making sure that the student can still learn from these data. Our approach is very efficient since the extra computational cost is linearly correlated with the number classes, usually not a big number.
Experiment
4.1
Main Results
In this section, we evaluate our proposed method on CIFARI00 [21] and ImageNet [6] under various distillation settings. We refer reader the Appendix for the implementation details.
CIFAR100. As aforementioned, our approach is orthogonal other distillation techniques. Therefore, we investigate the performance of TLLM when combined with other methods. We report experimental results on CIFAR 100, which average over five trials. Specifically, we compare six distillation methods, and for each of them, we add our method to theirs. As shown in Table 2, our method consistently improve the overall accuracy for baseline. For instance, we improve the vanilla KD [14] by 2.54%. The performance gain mostly comes from the increased accuracy undistillable

---

Table 2: Experimental results on CIFARI00 with CNN architectures. We observe that our approach improves overall accuracy and significantly reduces the number of undistillable classes.
T-S Pair
ResNad22x4/ReNNetxxx (%)
ResNer50MMobllNNa12 (%)
ResNed22x4/ShulleeNee (%)
Method
Vanilla
TLLM
A
Vanilla
TLLM
A
Vanilla
TLLM
A
Teacher
79:42
79.34
79.42
Student
72.50
64.60
70.50
KD [14]
73.08
75.53
+2.54
67.28
69.54
+2.26
74.07
76.67
+2.60
FitNet [31]
73.50
75.52
+2.02
63.06
66.33
+3.27
73.59
75.82
+2.23
AT [47]
73.44
75.21
+1.77
58.58
61.70
+3.12
71.73
75.10
+3,37
OH [12]
74.98
76.82
+1.84
67.69
69.21
+1.52
77.43
79.05
+1.62
KR [3]
75.63
76.84
+1.21
69.89
71.84
+1.95
77.14
78.87
+1.73
SFTN [26]
76.64
77.83
+1.19
70.01
70.94
+0.93
79.58
80.04
+0.46
classes. Later, we will show that our approach also reduces the number of undistillable classes. It also surprising to find that some methods that were once inferior to other approaches, i.e., the accuracy of OH [12] is 0.65% lower than KR [3] on ResNer32x4/ResNetkks obtain higher accuracy after applying TLLM, showing that many "outdated" algorithms have the potential to be quality methods. Moreover, our approach practical for both output-based (i.e., KD [14])and feature-based distillation [31, 3, 47] algorithms.
ImageNet. ImageNet is a large-scale classifica- tion dataset that contains 1.28 million images for training and 50k images for validation. Unlike conventional papers that only report the perfor- mance of CNN models, transformer (ViT), an advanced network archi- tecture that has drawn increasing attention lately.
Rogular
Undistilablee Glass
also consider vision
For CNN-based architectures, we evaluate two teacher-student pairs: 1) ResNet32 as pre- trained teacher and ResNer18 as a student model, widely adopted setting follows CRD [38], 2) ResNet50 as a pretrained teacher, and Mo- bileNetV1 as student model, less conven- tional setting that some papers have used. The experimental results reported in Table 7. Our TLLM, which is built based on KR [10], achieves a considerable improvement over its baseline. is also worth mentioning that the performance of TLLM outperforms other state-of-the distillation methods.
Figure Teaching curve for regular class and undistillable class.
For ViT architectures, we evaluate two types of teacher-student pairs: 1) the RegNetY [30] and ViT-Tiny/ViT-Smie [8] pairs are the popular setting in DeiT [39] which apply a CNN model as a teacher network to distill vision transformer, 2) the CaiT-S23 [40] and ViT-Tiny pairs adopt an advanced VIT model as teacher, whose performance is slightly inferior to RegNetY. We compare with DeiT [39], benchmark that uses teachers' hard labels dark knowledge supervise the student network. We notice that Table 4, we achieve higher accuracy than DeiT on all teacher-student pairs. Particularly, when the capacity gap between teacher and student large, i.e., RegNetY/ViT Tiny and HaiT-S24/ViT-Small the improvement in the validation set is more notable.
4.2 Ablation Study
Number of the Undistillable classes. Our method specifically identified the undistillable classes and then discarded them during distillation. Therefore, we conduct experiments to check whether our approach reduces the number of undistillable classes. In the Appendix, we show that our proposed TLLMM significantly reduces the number of undistillable classes in the student models, allowing balanced performance improvement.

---

Table 3: Experimental results on ImageNetIK with CNN architectures.
Baseline
Primework (%)
RosNer32
ResNet18
KD114
AT (T(47]
OPD [12]
CRD [38]
SKRI, [44]
KR [3]
SHTN [26]
CID [12]
DRD [83]
TLLM
73.3
69.8
70.7
20.7
71.2
71.2
71.7
71.6
71.1
71.9
71.7
72.6
ResNet50
MobileNetVi
KD
AT
OFD
CRD
SRRL
KR
SETN
CID
DKD
TLLM
76.2
68.9
70.5
69.6
71.3
71.4
71.7
71.5
72.3
72.1
73.2
Table 4: Experimental results on I with vision transformer architectures.
Pair
RegNett/ViT-Tiny (%)
RegNetY/ViT- (%)
CaiT--24/ViT-Tnng (%)
CaiT-S24/ViT- (%)
Method
Vanilaa
TLLM
Vanilla
TLLM
Vanilla
TLLM
Vanilla
TLLM
Teacher
82.9
82.7
82.7
82.7
Student
72.2
79.9
KD/DeTT [39]
74.5
16.3
81.2
81.
74.4
76.1
81.3
82.0
Does TLLM allow larger teacher to make better student? Our approach is motivated to resolve the "larger teacher, worse student problem"; thus, it natural to verify whether our approach solve this issue. In the Appendix, show that approach allows the larger teacher to distill students with higher accuracy.
Analysis of the Undistillable Classes
This section will thoroughly analyze the undistillable classes observed in the previous empirical study.
5.1 Hard Classes of Student f Undistillable Classes
One may wonder whether the undistillable classes be represented by the hard classes in the student model (classes with relatively low accuracy). This a reasonable hypothesis since challenging classes of student networks (trained without distillation) can be hard to gain more knowledge from the teacher. Nevertheless, we show that undistillable classes are not the hard class in the student model. Specifically, we consider three varying teacher-student pairs: ResNet32x4/ResNe111 WRN- 40-2/WRN-16-11 and ResNet566RRsNet20 on CIFAR 100 dataset that is trained with vanilla KD [14]. We compare the test accuracy of the student model with the improvement in test accuracy with the student model train with and without KD. Figure 7 present the results.
The Tondd50taaaee2
Raker
Act GludeeeOD Badertto)
Acc (DuuersoODD Seccerr
Are DiudeeeDDDDDD Secemn//)
Figure 7: Hard classes of student model are not undistillable classes
We can observe that the correlation between students' accuracy and the improvement of KD is almost random, showing that the hard classes student are not equivalent to undistillable classes.
5.2 Hard Classes of Teacher f Undistillable Classes
Another reasonable hypothesis is that the challenging classes in teacher model can be undistillable classes because teacher model is normally imperfect. For instance, ResNet-50 can only achieve 76.2% on ImageNet dataset. Can we find undistillable classes by looking at the hard classes of the teacher model? To answer this question, we conduct similar experiments as in the previous section. The only difference is that we record the teacher's 's test accuracy instead of recording the student's test

---

Re Mochertie
WWW. 31WAN
Teache Pec
Figure 8: Hard classes of teacher model are not undistillable classes
accuracy. Also, we conduct experiments on three teacher-student pairs. Figure 8 presents the results, which lead to the same conclusion that hard classes of teacher model are not undistillable classes.
6
Related Work
The most correlated works are the research discussing the relationship between teacher and student. Many works observe that the performance of teacher networks does not necessarily improve the performance of its distilled student. Cho et al. [5] is the first work to present this particular phe- nomenon and argue that the capacity mismatch between teacher and student gives rise to this behavior. They propose applying the early stopping technique to train the teacher, and such an carly-stopped teacher can surprisingly improve students' accuracy. It was later proved that early-stopped teachers improve data efficiency during distillation [18). Mirzadeh et al. [23], inspired by BAN [9), show that using single or multiple intermediate networks, which are larger than the student yet smaller than the teacher, can also narrow the capacity gap. Zhu et al. [55] propose
adaptive distillation method based on observation that the status the capacity gap change constantly during training. Lukas et al. [22] find out that teachers trained with label smooth severely affect the student's accuracy if vanilla KD is used. Although this opinion is disputed by Shen et al. [33], following work demonstrate that the label-smoothed model can be effective teacher if and only if some strict rules applied, ie., assign a large number to temperature term [2]. Other works see that even a bad teacher can teach good student because self-distillation is provably regularization technique [52, 24, 49, 15]. Nevertheless, none of these works investigate the capacity mismatch problem from the class level.
Conclusion
This paper studies the "larger teacher, worse student" phenomena in knowledge distillation. Unlike prior works, which attribute the inefficacy of large teachers as capacity mismatch issue on samples, we consider the presence of undistillable classes the direct cause of mismatched capacity. Our paper verified that these classes lower the ability of the student model to match the teacher's 's feature representation and predictive distribution, making the accuracy drops considerably in some arbitrary classes, therefore, hurting the overall performance. We further demonstrate that the existence of undistillable classes universal, irrespective of distillation methods, datasets, and student-teacher configurations. Finally, we propose a "teacher less, learn more" (TLLM) method to alleviate this issue by targeting and removing samples from the undistillable classes. Our experiments have shown that our TLLM can greatly eliminate the adverse effect of undistillable classes to improve the overall performance significantly. Overall, we believe that our analysis and proposed method provides an interesting and practical new perspective on designing distillation algorithms.
10

---

References
[1] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Vari- ational information distillation for knowledge transfer. In Proceedings the IEEE/CVF 2019.
Conference on Computer Vision and Pattern Recognition, pages
[2] Keshigeyan Chandrasegaran, Ngoe-Trung Tran, Yunqing ZHAO, and Ngai man Cheung. To smooth or not to smooth? on compatibility between label smoothing and knowledge distillation, 2022. URL https: http:///pppeeeiiieeee aet/forumidd=Vanee (rmujzzzU_23
[3] Pengguang Chen, Shu Liu, Hengshuang 7hao, and Jiaya Jia. Distilling knowledge via knowl- edge review. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5008-5017, 2021.
[4] Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv: 151.011274, 2015.
[5] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4794-4802, 2019.
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 leee, 2009.
[7] Xiang Deng and Zhongfei Zhang. Comprehensive knowledge distillation with causal interven- tion. Advances in Neural Information Processing Systems, 34, 2021.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[9] Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference Machine Learning, pages 1607- 1616. PMLR, 2018
[10] Jianping Gou, Baosheng Yu, Stephen J Mayhank, and Dacheng Tao. Knowledge distillation: A survey, International Journal of Computer Vision, 1796)11789-1819, 2021.
[11] Bobby and Mete Ozay. Feature kernel distillation. In International Conference on Learning Representations, 2022. URI. https://openrrwiiee
[12] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1921-1930, 2019.
[13] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3779-3787, 2019.
[14] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, al. Distilling the knowledge a neural network. arXiv preprint
2(7), 2015.
[15] Yaomin Huang, Xinmei Liu, Yichen Zhu, Zhiyuan Xu, Chaomin Shen, Zhengping Che, Guixu Zhang, Yaxin Peng, Feifei Feng, and Jian Tang. Label-guided auxiliary training improves 3d object detector. arXiv preprint 22X. 2022.
[16] Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint arXiv: 2017.
[17] Aref Jafari, Mehdi Rezagholizadeh, Pranav Sharma, and Ali Ghodsi. Annealing knowledge distillation. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages
2021.

---

[18] Guangda Ji and Zhanxing Zhu. Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher. Advances Neural Information Processing Systems, 33: 20823-20833, 2020.
[19] Jangho Kim, Sk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. Advances in neural information processing systems, 31, 2018.
[20] Simon Korblithh, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton, Similarity of neural network representations revisited. International Conference on Machine Learning, pages 1519.3529. PMLR, 2019.
[21] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[22] Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In International Conference on Machine Learning, pages 6448-6458. PMLR, 2020.
[23] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5191-5198, 2020.
[24] Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regular- ization in hilbert space. Advances in Neural Information Processing Systems, 33:3351-3361, 2020.
[25] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
[26] Dae Young Park, Moon-Hyun Cha, Daesin Kim, Bohyung Han, et al. Learning student-friendly teacher networks for knowledge distillation. Advances Neural Information Processing Systems, 34, 2021.
[27] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3967-3976, 2019.
[28] Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference Computer Vision (ECCV), pages 268-284, 2018
[29] Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu. Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5007-5016, 2019.
[30] Iliaa Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10428-10436. 2020.
[31] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.660 2014.
[32] Zhiqiang Shen, Zhankui He, and Xiangyang Xue. Meal: Multi-model ensemble via adversarial learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4886-4891 2019.
[33] Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang- Ting Cheng, and Marios Savvides. Is label smoothing truly incompatible with knowledge distillation: An empirical study, arXiv preprint arXiv:2104.00666 104.00676, 2021.
[34] Wenxian Shi, Yuxuan Song, Hao Zhou, Bohan Li, and Lei Li. Follow your path: progressive method for knowledge distillation. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 9. Springer, 2021.
12

---

[35] Wonchul Son, Jaemin Na, Junyong Choi, and Wonjun Hwang. Densely guided knowledge distillation using multiple teacher assistants. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9395-9404, 2021.
[36] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances Neural Information Processing Systems,
34, 2021.
[37] KH Tan, Charlene Tan, and IS Chua. The "teach less, learn more" initiative in singapore schools. Innovation in education, pages 153-171, 2008.
[38] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In International Conference on Learning Representations, 2019.
[39] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficien image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[40] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Go- ing deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 32 32-42,22 2021.
[41] Frederick Tung and Greg Mori. the IEEE/CVF International Conference on Computer Vision, pages 1365-1374, 2019.
knowledge distillation. In Proceedings of
[42] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010.
[43] Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self-supervision. In European Conference on Computer Vision, pages
Springer, 2020.
[44] Jing Yang, Brais Martinez, Adrian Bulat, Georgios Tzimiropoulos, et al. Knowledge distil- lation via softmax regression representation learning. International Conference on Learning Representations (ICLR), 2021.
[45] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4133-4141, 2017.
[46] Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3903-3911, 2020.
[47] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:16/2.03922 2016.
[48] Chang-Bin Zhang, Peng-Tao Jiang, Qibin Hou, Yunchao Wei, Qi Han, Zhen Li, and Ming-Ming Cheng. Delving deep into label smoothing. IEEE Transactions Image Processing, 30: 5984-5996, 2021.
[49] Linfeng Zhang and Kaisheng Ma. Improve object detection with feature- knowledge distillation: Towards accurate and efficient detectors. In International Conference Learning Representations, 2020.
[50] Linfeng Zhang, Zhanhong Tan, Jiebo Song, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Scan: A scalable neural networks framework towards compact and efficient models. Advances in Neural Information Processing Systems, 32, 2019.
[51] Youcai Zhang, Zhonghao Lan, Yuchen Dai. Fangao Zeng. Yan Bai, Jie Chang, and Yichen Wei. Prime-awar adaptive distillation. In European Conference on Computer Vision, pages 658. Springer, 2020.
[52] Zhilu Zhang and Mert Sabuneu. Self-distillation as instance-specific label smoothing. Advances in Neural Information Processing Systems, 33:2184-2199 2020.
13

---

[53] Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. 2022.
[54] Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. arXiv preprint (tt::200200650 2021.
[55] Yichen Zhu and Yi Wang. Student customized knowledge distillation: Bridging the gap between student and teacher. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5057-5066, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(C) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No] The code is proprietary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] error bars are not reported for some experiments because would be too computationally expensive
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [Yes]
(c) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [No] We use standard benchmark in all our experiments.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "jszy-HpoXIL", "reviewer": "Reviewer_jTkD", "summary": "This paper introduced a new data-centric perspective on the phenomenon of 'larger teacher, worse student’.  They search for the classes that disrupt the distillation process, they defined them as the undistillable classes, and then exclude them to improve the performance of the small student network when distillation; this framework is called 'Teach Less Learn More (TLLM)’. Specifically, to find the undistillable classes, they suggest two criteria; per class accuracy and agreement prediction agreement. Various ablation studies support their observation and experimental results on multiple datasets with varying networks validate the effectiveness of TLLM. ", "questions": "I was wondering if there is a practical way to apply TLLM over the new dataset. ", "limitations": "Please check the weaknesses and questions; in terms of practicality and analysis. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The observations are interesting and well supported experimentally. \n\n- The method is conceptually simple and seems to work well across the board. \n\n- Experiments are conducted with several neural network architectures and various datasets. \n\n### Weaknesses\n- The concept and approach of the proposed method are different from ESKD[3] but somewhat similar in methodology for improving the KD performance; ESKD also halts distillation from teacher to student at the last phase. A direct comparative experiment with ESKD will enhance the completeness of this paper. \n\n- The proposed method, TLLM, is potentially impactful, but it should use significantly more computational resources and time for finding undistillable classes. Thus, this point is not useful from a practical point of view. It would be great if they could suggest an efficient way to find undistillable classes. \n\n- They claim that unstillable classes are universes, but there is no basis for that. It is necessary to analyze the characteristics of the classes and why they interfere with distillation. \n\n- Minor point \n> Line127, replace j with M  \n> Line163-5, please check the suitability of examples  \n> Line166, what is the meaning of crystal  \n> Figure 4, explain the meaning of the dashed line  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cWh8Hv4KR4", "reviewer": "Reviewer_WWqA", "summary": "This paper propose that some classes is not distillable to students, and by removing these classes, the distillation performance would be improved.", "questions": "Would the author analyze why some undistillable classes are consistent across multiple experiments, while others are not? And in those consistent classes, is the same group of instances undistillable?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n  1. The paper is well written and easy to follow.\n  2. The idea is simple and effective in practice. The observation that some classes are not distillable to the student is novel.\n  3. Experiment result is competitive against some mainstream KD methods.\n\nCons:\n  1. The paper lacks deep insight into these bad distillable classes. The method of this paper is similar to \"On the efficacy of knowledge distillation\". Both learn teachers at the beginning, and then gradually start to learn one-hot labels. The difference is that this paper argues that certain classes are the cause of the problem. So why do hard examples come in the form of categories, are all samples of these certain classes hard to distill?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XvvhBJP9XJ8", "reviewer": "Reviewer_NdAQ", "summary": "The major contributions of this paper are 2-fold:\n1) The thesis of this paper is to study the “larger teacher, worse student” phenomenon in knowledge distillation (KD) from a data-centric approach. The authors show that the above phenomenon happens due to the presence of undistillable classes in the teacher. This observation is shown across different KD methods, datasets and teacher-student architectures.\n\n2) The authors propose a simple, yet effective fix -- Teach Less, Learn More (TLLM) -- to identify and discard undistillable classes during student training, thereby improving the final accuracy of students.\n", "questions": "Please see Weaknesses section above for a list of questions. Further please consider answering the following questions:\n\n1) What is the final optimization / loss function (preferably in equation form)?\n2) What is the mixture parameter ($\\alpha$) used for label smoothing in undistillable classes?\n", "limitations": "Limitations / potential societal impacts discussed in Supplementary Section A.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1) This paper is written / presented clearly (especially Fig 5, bottom Middle). It is easy to follow.\n2) The proposed TLLM framework shows noticeable improvements on CIFAR-100 (across different KD methods, teacher-student architectures). \n\n\n**Weaknesses:**\n1) Although the work is technically sound, do you think the explanation / definition of undistillable classes is a bit overkilled? Can the following be a more simpler explanation (I will use ImageNet experiment as example): The teacher network is not perfect. That is the top1 accuracy of ResNet-50 teacher is 76.2% (Table 3). Therefore, the teacher tends to make a noticeable amount of errors. If class-wise analysis is performed on the teacher based on misclassified samples, it corresponds to a subset of classes which can be treated as undistillable classes. Therefore, the proposed TLLM avoids such situations by eliminating classes where the teacher's predictions are largely incorrect and replacing them with ground-truth labels (one-hot vectors). Some analysis I’d like to see are:\n- What is the average accuracy of ResNet-50 teacher (Table 3) for regular and undistillable classes?\n- Can you decompose the improvements obtained ($\\Delta$) in Tables 2, 3 for regular and undistillable classes?\n\n2) ImageNet results are insufficient to assess the efficacy of proposed TLLM: What is the reason for using only KR (Chen et. al) as the baseline for ImageNet experiments? Can the authors produce a table similar to Table 2 for ImageNet to show the improvements of the proposed TLLM scheme?\n\n3) The Baseline performance of MobileNetV2 in Table 3 (ImageNet) seems to be very low. top1 accuracy should be ~72.154% whereas the reported accuracy is only 68.900%. Please clarify. Link to public pytorch models with top1 accuracies: https://pytorch.org/vision/stable/models.html\n\n4) If the authors use label smoothing for undistillable classes (Lines 66-68) in the student, it is not reasonable to compare with reported Baselines as label smoothing will further improve accuracy (by alleviating models’ overconfidence [1, 2, 3]). A more reasonable comparison will be to include the accuracies of students trained with label smoothing. Can you report these results for Tables 2, 3. I suspect that a noticeable amount of improvement can be obtained by using label smoothing, especially if undistillable classes are semantically similar classes (See [2, 3]). \n\n5) CUB200 KD results are missing (Not available in Supplementary as well). \n\nThis is an interesting paper. In my assessment, I feel that the weaknesses of this paper outweigh the strengths. But I’m happy to change my opinion based on the rebuttal. \n\n\n====================\n\n**Post-Rebuttal:**\n\nThank you authors for the great effort on the rebuttal. \n\nI have increased my recommendation. Authors have addressed most of my concerns although concrete understanding of undistillable classes still remains unclear. \n\nAs discussed, please include details on the effect of label smoothing on the proposed TLLM framework.\n\n==================\n\n[1] Müller, Rafael, Simon Kornblith, and Geoffrey E. Hinton. \"When does label smoothing help?.\" Advances in neural information processing systems 32 (2019).\n\n[2] Shen, Z., Liu, Z., Xu, D., Chen, Z., Cheng, K. T., & Savvides, M. (2021). Is label smoothing truly incompatible with knowledge distillation: An empirical study. In ICLR\n\n[3] Chandrasegaran, K., Tran, N. T., Zhao, Y., & Cheung, N. M. (2022). Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing?. ICML\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Xcpf0lKynj", "reviewer": "Reviewer_6Hiv", "summary": "This paper aims to study the problem of “larger teacher, worse student” in knowledge distillation (KD). \nFrom a data-centric perspective, this paper claims that undistillable classes are the cause of the inefficacy of large teachers. Furthermore, this paper proposes a new KD framework called Teach Less, Learn More (TLLM) to address the issue. In detail, TLLM tries to identify the undistillable classes during training with a moving window and lets the student learn these classes directly from the ground truth labels. ", "questions": "The explanation from per-class accuracy in general novel. Nevertheless, the above weaknesses refrain me to give a better score. I would be happy to improve my rating as long as the authors address the above weaknesses in the rebuttal.   ", "limitations": "Yes, this paper clearly discusses the limitations and potential negative social impacts in the supplementary. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The data-centric perspective is novel and interesting in the study of “larger teacher, worse student”, which understands the problem much better than the previous capacity mismatch explanation. Moreover, To the best of my knowledge, the per-class accuracy has not been explored in KD before. It deserves to be studied more in the future since the undistillable classes are universal. \n\n- Experiments are comprehensive and the results are convincing. This paper considers many teacher-student pairs and even some advanced architectures such as vision transformers (ViTs). Moreover, this paper considers both output-based KD and feature-based KD. \n\n\nWeaknesses\n\n- A clearly mathematical definition of the “undistillable class” is missing. \nFrom section 2.1, we know that the “undistillable classes” are classes with bad distillability. However, a specific mathematical distillability measurement ||c is not given, which makes it difficult to understand the detailed implementation in this paper.  As in Figure 5, we know that the undistillable class is a class whose accuracy after KD is lower than that before KD. However, it is unclear whether this is a necessity and sufficiency of undistillable class. It is also unclear about what specific criterion this paper uses to select samples in Figure 1. To this end, this paper should give a clear definition of it in the rebuttal letter. \n\n\n- More studies of the undistillable classes should be discussed. \nThis paper mainly studies the proportion of undistillable classes, which is insufficient. More studies about the distribution of these classes should be explored. For example, whether these undistillable classes are challenging classes (i.e., classes with relatively low accuracy before KD), and whether these classes are network-independent.   \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept