Can Students Outperform Teachers in Knowledge Distillation based Model Compression?
Authors: Xiang Deng
Zhongfei Zhang
Venue:  (2021)
Keywords: Knowledge Distillation
Deep Learning
Supervised Learning
Model Compression

Abstract: Knowledge distillation (KD) is an effective technique to compress a large model (teacher) to a compact one (student) by knowledge transfer. The ideal case is that the teacher is compressed to the small student without any performance dropping. However, even for the state-of-the-art (SOTA) distillation approaches, there is still an obvious performance gap between the student and the teacher. The existing literature usually attributes this to model capacity differences between them. However, model capacity differences are unavoidable in model compression. In this work, we systematically study this question. By designing exploratory experiments, we find that model capacity differences are not necessarily the root reason, and the distillation data matters when the student capacity is greater than a threshold. In light of this, we propose to go beyond in-distribution distillation and accordingly develop KD+. KD+ is superior to the original KD as it outperforms KD and the other SOTA approaches substantially and is more compatible with the existing approaches to further improve their performances significantly.

Under review as a conference paper at ICLR 2021
CAN STUDENTS OUTPERFORM TEACHERS IN KNOWL- EDGE DISTILLATION BASED MODEL COMPRESSION?
Anonymous authors Paper under double-blind review
ABSTRACT
Knowledge distillation (KD) is an effective technique to compress a large model (teacher) to a compact one (student) by knowledge transfer. The ideal case is that the teacher is compressed to the small student without any performance dropping. However, even for the state-of- ah-tthe-rt (SOTA) distillation approaches, there is still an obvious performance gap between the student and the teacher. The existing liter- ature usually attributes this to model capacity differences between them. However, model capacity differences are unavoidable in model compression. In this work, we systematically study this question. By designing exploratory experiments, we find that model capacity differences are not necessarily the root reason, and the distillation data matters when the student capacity is greater than threshold. In light of this, we propose to go beyond in in-distribution distillation and accordingly develop KD+. KD+ is superior to the original KD as it outperforms KD and the other SOTA approaches substantially and is more compatible with the existing approaches further improve their performances significantly [
INTRODUCTION
Deep neural networks (DNNs) have achieved remarkable performances in various domains, but they require large amounts of computation and memory. This seriously limits their deployment with limited resources or a strict latency requirement. One solution to this problem is knowledge distillation which transfers the knowledge from a large network (teacher) to a small one (student).
Hinton et al. (2015) proposed the original knowledge distillation (KD) which uses softened logits of a teacher as supervision to train a student. To make the student better capture the knowledge from the teacher, the existing studies focus aligning their representations by using different criteria. However, there is still a significant performance gap between the teacher and the student. Figuring out the reason for this gap is essential for further improving the student performance.
Mirzadeh et al. (220) argue that the model capacity difference causes the failure for transferring the knowledge from large teacher a small student, thus leading a large performance gap. Similarly, Cho & Hariharan (219)) point out that as the teacher grows in capacity and accuracy, it is difficult for the student emulate the teacher. In this paper, we systematically study why students underperform teachers and how students can match or outperform teachers. We find that in most experimental settings of the existing literature, the root reason for the performance gap is not necessarily the capacity differenc as the student is powerful enough to memorize the teacher's outputs. The reason lies in the distillation dataset on which the knowledge is transferred.
As an old proverb says, indigo comes from blue, but it is bluer than blue. In reality, it is not rare for human students to do better than their teachers. These excellent human students not only well capture the knowledge from their teachers but also learn more related knowledge on their own. This gives an insight for students in KD to match or outperform their teachers. We find that currently the students in KD have not well captured the knowledge in their teachers they only mimic the behavior of the teachers on sparse training data points. In light of this, we propose KD+ which goes beyond in-distribution distillation to substantially reduce the performance gap between students and teachers.
Our main contributions are summarized as follows:
'The code will be released online.
In this paper, we KD denote the original knowledge distillation algorithm
Hinton al. 2015

---

Under review as a conference paper at ICLR 2021
Different from the common belief that model capacity differences result in the performance gap between students and teachers, we find that capacity differences are not necessarily the greater
root reason and instead the distillation data matters when students' capacities than a threshold. To our best knowledge, this is the first work that systematically explores why small students underperform teachers and how students can outperform large teachers.
By designing exploratory experiments, we find the following: (1) only fitting teachers' outputs at sparse training data points cannot make students well capture the local, in- distribution shapes of the teacher functions; (2) different from the case on standard supervised learning, -f--o---bittttiiionnn data (but not all) can be beneficial to knowledge distillation.
Different from the existing work focusing on using different criteria to align representations or logits between teachers and students, we address knowledge distillation from novel (data) perspective by going beyond in-distribution distillation and accordingly develop KD+,
Extensive experiments demonstrate that KD+ largely reduces the performance gap between students and teachers, and even enables students to match or outperform their teachers. KD+ is superior to KD it outperforms KD and more than 10 SOTA methods substantially and shows a better compatibility with the existing methods and superiority in few-shot scenario.
2 RELATED WORK
The objective function of knowledge distillation can be simply expressed as a combination of the regular cross-entropy objective and distillation objective. According to the distillation objective, the existing literature can be divided into logit-based approaches (Hinton et al. (0055) and representation- based approaches (Romero et al. 2015). Logit- approaches construct the distillation objective based on output logits. Hinton et al. (2015) propose KD which penalizes the softened logit differences between a teacher and a student. Park et al. 2019) propose to transfer data sample relations from a teacher to student by aligning their logit- based structures. On the other hand, representation -based approaches design the distillation objective based on feature maps. FitNet (Romero et al.) 2015)
aligns the features of teacher and student through regressions. AT (Zagoruyko & Komodakis 2017) distills feature attention from a teacher into a student. CRD (Tian et al. 2020) maximizes
the mutual information between student and teacher representations. Other representation-bate
methods YYm et al., 2017:; Huang & Wang 2017 Kim et al.) 2018: Liu et al. 2019; Srinivas &
Fleuret 2018: Wang et al.) 2018 Heo et al., [219a]] Cho & Hariharan [2099 Ahn et al. [019]]
et al. [2199] [Shen & Savvides] 2020) use different criteria to align feature representations. SSKD (Xu al. 2020p introduces extra suff-supeoiision tasks to assist KD.
Koratana et al. 2
Online knowledge distillation (Zhang et al., 2018b] Chen et al.) 2020: Anil et al. 2018 Chung et al. 2020; Zhu et al. 2018) trains multiple students simultancously. Self-distillation (Furlanello et al. 2018 Yuan et al. 2020) approaches train a DNN by using itself the teacher. It is observed that the existing studies focus on designing different criteria to align teacher- student representations or logits on in-distribution data. In this work, we address knowledge distillation from a data perspective by embedding out-of-distribution distillation into a regularizer.
Mirzadeh al. (2020) observe that the model capacity gap results in the failure for transferring knowledge from a large teacher to a small student, thus causing a performance gap. To reduce this gap. gap,they propose a multi-step knowledge distillation framework by using several intermediate-size networks (teacher assistants). However, the students still underperform the teachers substantially. Cho & Hariharan (019) argue that as the teacher grows in capacity and accuracy, it is difficult for the student to emulate the teacher. To reduce the influence of the large capacity gap, they regularize both the teacher and the knowledge distillation by early stopping. We find that capacity differences are not necessarily the root reason when student capacities are greater than a threshold.
On the other hand, KD+ goes beyond in-distribution distillation by exploring the knowledge between two training samples. Similar techniques have been used in many applications with different goals and mechanisms. Mixup (Zhang et al. 2018a) enforces local linearity of a DNN by linearly interpo- lating random pair of training samples and their one-hot labels simultancously. However, simply interpolating two labels may not match the generated sample as pointed out in (Guo et al., 2019). KD+ does not have the above issue as it teaches student mimic the local shape of a powerful teacher. MixMatch (Berthelot et al. 2019b) linearly interpolates labeled and unlabeled data to improve the semi-supervised learning performances. MeMixMatch (Berthelot et al. 2019a) improves MixMatch
2

---

Under review as a conference paper at ICLR 2021
by introducing distribution alignment and augmentation anchoring. DivideMix (Li al. 2020 aims to learn with noisy labels by modifying MixMatch with label co-refinement and label co-guessing on labeled and unlabeled samples, respectively. AugMix (Hendrycks al. 2019) linearly interpolates original training samples and augmented training samples to improves the robustness and uncertainty estimates of DNNs.
REFORMULATING KD
Hinton al. 2015) propose KD which minimizes the softened logit differences between student and a teacher over training data D, = (X,, Y,) where X, and Y, are the training samples and the ground truth, respectively. The complete objective is:
LKD
Σ ((X.))(()))))))
(aLcE(fs,It.b) (t, + (f..
(1)
where a and are balancing weights and Lce the regular cross-entropy objective:
LCElJs, zt, Wb) H H (ff((z))))
where H()) is the cross-entropy and a is softmax. LK in
I is the distillation objective:
CKL (fs: x1) 7 (o
(0(
fr (xt)
(2+)
fs (II)
(3)
where 7 is a temperature to generate soft labels and KL represents KL-divergence. KD can be considered using one function (fs) to fit the outputs of another function (fr).
We notice that (T). Lcg requires both data samples X, and the corresponding ground truth Y, while L only needs data samples X, for distilling the teacher knowledge. In light of the difference, we consider KD from semi-supervised perspective and reformulate (1 more general form:
C
Σ
at.
Σ (dE(XA)
[Lri
(BL.)E((X.)))
where we introduce a new concept: distillation dataset Xa set of samples on which the knowledge is transferred from teacher to student. The first term in the right hand side of (4) is supervised while the second term is unsupervised. obvious that the widely used objective i8 special case of 4 when Xa is set to X
WHY SMALL STUDENTS UNDERPERFORN LARGE TEACHERS?
In this part, systematically analyze the reason for the performance gap between students and teachers in KD based model compression. We first introduce several definitions.
Definition 4.1 Memorization Error (ME): For a given task with data distribution P(X,Y) ME measures the degree of student Is fitting the outputs of teacher Jr over the data distribution:
Elfs.fr.P x~P(X)
E
M(fr (x), (x))
(5)
where M denotes a distance metric such as KL-divergence or mean square error. When ME is 0, it means that the student can completely memorize the outputs of the teacher over the data distribution. In this paper, we take KL-ddieegeenee as M.
Definition 4.2 Capable Students (CSTs) and Incapable Students (ISTs): network fs with parameters Os a CST of teacher fr if there exists Og such that Elfs, fT, P)=0, otherwise, an IST.
Obviously, a CST is able to fully fit the teacher outputs over data distribution P(X, Y). In contrast, an IST does not have the capacity to fit the teacher. For ISTs, the common belief holds that the student-teacher capacity gap causes the performance gap. For example, we cannot expect a two-layer neural network with 1000 parameters fit the outputs of ResNet-101 with 1.7M parameters
3

---

Under review as a conference paper at ICLR 2021
Table I: ME of different networks on 10, CIFAR-100, and Tiny ImageNet
Teacher
WRN-40-2
VGG-13
ResNet3224
ResNet-110 VGG-8
ResNet32x4 ShuffleNerV2
VGG-13
VGG-13
Student
WRN-16-2
VGG-8
ResNet8x4
SN2
SN3
CIFAR-10
0.0
0.0
1.7
0.1
CIFAR-I0 100
0.0
0.0
0.0
0.0
0.0
2.4
0.3
Tiny ImageNet
0.0
0.0
0.0
0.0
4.2
1.9
Table 2: Simulation results CIFAR-100 in terms of test accuracy (%)
Teacher Student
ResNet32x4 ResNet8
WRN-40-2
VGG-13 VGG-8
ResNet32x4 ShuffleNet22
VGG-13
VGG-13
WRN-16-2
SN2
SN3
Teacher
79.52
75.81
74.97
79.52
74.97
74.97
Vanilla Student
72.50
73.26
70.36
71.82
26.29
55.31
Student Type
CST
CST
CST
CST
IST
IST
KD
73.33
74.92
72.98
72.14
26.04
55.32
Simulation KD
79.91
78.46
77.99
81.64
25.58
57.50
CIFAR-100. However, in the current SOTA approaches and applications, the commonly used students modern neural network architectures, such as ResNet-20,, ResNer-8x4, VGG-8, and WRN-40-1. We empirically show that these models are CSTs on commonly used benchmark datasets.
To check whether student fs is CST of teacher fr on a task, minimize ME check whether E(ss fr. P) can achieve O. However, in practice, it is impossible to calculate E(ss, Jr, P) as the data distribution P is typically unknown. Fortunately, we have the access set of training data (Xe, Yi). With the training data, we approximate ME E(fs, fr, P) with the empirical error:
Frem (Sa, JT, Xi)
1 XX,
Σ 8.EX.
M(Jr (1t) (xx))
For comparison, we also evaluate two small neural networks which are expected to be ISTs, i.e., SN-2 and SN-3 with two and three layers, respectively. We report the ME in Table where we adopt the students and the teachers that share the same architectures (c.g., WRN-40-2 and WRN-16-2) or use different architectures (e.g., ResNet- 10 and VGG-8). As expected, the widely used students achieve ME 0.0 these benekmark datasets, i.e., CIFAR-10, CIFAR-100, and Tiny ImageNet while the small networks (i.e., SN2 and SN3) have large ME (e.g., 2.4 and 4.2), which demonstrates that the widely used students are CSTs. However, as observed in the existing literature, these CSTs underperform the teachers by a significant margin on the test data. This suggests that these students have well captured the knowledge on sparse training data points but have not well captured the local shapes of the teachers within the data distribution.
Corollary 4.1 In KD, for CSTs, fitting the outputs of teachers on sparse training data points cannot enable them to well capture the local, in-distribution shapes of the teachers, thus leading to performance gap. For ISTs, capacity differences cause the performance gap.
Proof: We empirically show this by comparing the student performances in the following two settings: (a) setting the distillation dataset training data points; (b) setting the distillation dataset real data distribution P(X). As P(X) is typically unknown in practice, we conduct a simulation experiment on CIFAR-100 100. We suppose that the union of the training dataset and the test dataset in CIFAR- can accurately represent the real data distribution for this task. Then randomly draw data samples from the vicinity around the training data and the test data as the distillation dataset, i.e., X, in (4). Consequently, the distillation dataset that in the experiments, we never spy the ground truth of the test samples, since the distillation dataset does not use ground truth as shown in 4 This means that the students are trained without any additional supervision compared with the teachers as training datset (X,, Y)) in (4) does not change.
sufficiently represent the real data sample distribution. Note
As CSTs able to fully memorize the outputs of the teachers, expect them to achieve the same
"The ME values in Table are accurate to decimal place.
4

---

Under review as a conference paper at ICLR 2021
69
dxl
861
5 P. Ps P9-151" Ag5
Figure 1: KD with training data
Figure 2: KD with augmentation
Figure 3: KD+
accuracies as higher accuracies than those of the teachers. In contrast, expect ISTs achieve lower accuracies than those of the teachers. Table shows the simulation results. As expected, all the CSTs outperform the teachers in the simulation experiments (i.e., Simulation KD). This is due to the following facts: first, by using the simulated distillation dataset, the distillation objective in makes the CSTs fully capture the knowledge of the teachers within the data distribution: second, the cross-entropy objective in 4 enables the CSTs to learn their own knowledge. Consequently, CSTs contain both the teacher knowledge and the knowledge learned on their own, which results in better performances than those of the teachers. SN2 and SN3 still underperform the teachers in the empiricall
simulation experiments due to their limited capacities. These results empirically prove Corollary 4.1.
The simulation experiments also suggest way for CSTs to outperform teachers. That is sufficiently distill the knowledge in the teachers with a well representative distillation dataset. Unfortunately, is impossible to have such distillation datset as the real data sample distribution P(X) is typically unknown in practice. Motivated by this, we propose to go beyond in-distribution distillation.
GOING BEYOND IN-DISTRIBUTION DISTILLATION
5.1 DIFFERENCES BETWEEN SUPERVISED LEARNING AND KNOWLEDGE DISTILLATION
As analyzed above, the reason for the performance gap lies in distillation datasets. Distillation datasets are different from training datsets. As shown in (4), training datsets contain both samples and their ground truth, which are used for standard supervised learning. In contrast, distillation datsets only contain samples without ground truth, which are used for knowledge distillation. Knowledge distillation and standard supervised learning differ substantially. Standard supervised learning is to learn a function (e.g., DNN) for mapping to where (æ,, y) follows real data distribution P(X,Y) The quality of J is constrained by the training data (X,, Y.) that we have. In contrast, knowledge distillation is to learn a function (i.e., student fs) for mapping x to where (ær
(x))))). Q(X, Z) different from P(X, Y) unless teacher fr is perfect. The advantage of knowledge distillation is that Q(X, Z) is more tractable than P X, Y). since for given any sample x, fr can always give output fr(xx) and (x, (r(=)) follows QX,, Z). Even for out-of-distriben -irriiitiitttonn samples, fr can still output soft labels although these soft labels are semantically meaningless. However, the existing literature ignores this advantage as they only distill the knowledge on sparse training data points.
follows a teacher-definer distribution Q(X, Z) and
5.2 IMPROVING KD BY FURTHER EXPLORING TEACHER-DEFNEE DISTRIBUTION
The simulation experiments demonstrate that only fitting sparse individual data points cannot neces- sarily enable students to well capture the local shapes of the teacher functions. As shown in Figure where the yellow regions denote real data sample distribution P(X). ). even if student I's perfectly fits teacher fT at each training data point, i.e., 3T, T2, and æ3, their local shapes near these samples can still be highly different. To mitigate this issue, the typically used strategy is data augmentation (Simard] al. (1998) formalized by the Vicinal Risk Minimization (VRM) Chapelle al. (200I) principle. In VRM, human knowledge is necessary to define a vicinity or neighborhood around each training data point. Then, additional new data points can be drawn from the vicinity distribution of the training data. For example, in image classification, it is common to define the vicinity of an image as the set of its random crops after mildly padding and flipping. Nevertheless, data augmentation has its own limitation that newly generated data point is very close the original training data point, since they contain almost the identical objective only with different backgrounds caused by padding or cropping. Due this limitation, as shown in Figure 2 even if the student fs fits the teacher fr at all the training data points (i.c., 31, I2, and 23)) and the augmented data points (i.e., x x and ag aug), their local shapes still differ substantially.
aug aug
5

---

Under review as a conference paper at ICLR 2021
Table 3: Ablation study on CIFAR-100 in terms of test accuracy (%)
Teacher: WRN-10-2, Student: WRN-10-1
Teacher: ResNet32x4, Student: ShuffleNerV2
KD
73,54+00.20
74.45++0.27
p=2
p=3
p=5
p=10
p=2
p=3
P=5
p=10
1
75.10=0..13
74.79=0.17
74.72=00.19
74.68=0.24
75.72+00.2
76.03=00155
75.77=0.12
76.03=0.17
1=1:1
75.21+0.14
75.35+0.16
75.13++0.17
74.72+0.31
76.80++0.08
77.22=0.21
76.42+0.27
76.44+0.29
1:2
75.00:1.111
75.25=0.12
74.76:0.32
74.76=0.41
76.50++0.07
77.21=0.14
76.57=0.17
76.01:0.38
1:5
74,40+0.18
75.15=0.14
74.16=0.42
74.25=0.34
75.35++0.34
76.02=0.37
75.17=0.35
75.50=0.32
To address the above issue. we propose KD+ that regularizes KD by enforcing students to mimic the behavior of teachers on the region between two training (or augmented) samples. As shown in Figure 3 KD+ first defines p 1 points (i.e., Pt. P2, .... Pp-1) that evenly divide the region between two training samples (i.e., 21 and 22) into p pieces. We denote the set of Pt. P2, and Pp-1 by P contains in-distribution points and iutoof-distribution points. KD+ enforces the students to mimic the behavior of the teachers on P. which serves as data-driven regularizer. KD+ goes beyond in-distribution distillation as it also uses -ittr--distribtionn points in the regulaizer. As seen from Figure 3 the regularizer can make the student better explore and capture the local shape of the teacher. Consequently, the complete objective of KD+ is written as:
LKD+ = LKD Σ CkIlfs,
P.EP
where À is balancing weight and simply setting À to 1 works pretty well. KDD+ is very concise ap- proach without requiring complex hyperparameter tuning and can sufficiently explore the knowledge in the teacher by using freely obtained in-distribution and out-of-distribttion points as a regularizer.
6 EXPERIMENTS FOR EVALUATING KD+
In this section, We first conduct ablation study. Then we show that KD+ is superior to KD by (1) comparing KD+ with KD and other SOTA approaches. (2) showing that KD+ is more compatible with these approaches, (3) showing the superiority of KD+ under few-shot setting.
6.1 DATASETS, ARCHITECTURES, COMPETITORS, AND
Experiments are conducted on three benchmark datasets. CIFAR- 100 (Kriheessky Hinton 2009) has 100 classes with 50k training images and 10k test images. Tiny ImageNet has 200 classes with 100k training images and 10k test images. ImageNet (Deng et al. 2009) has 1000 classes with 1.28M training images and 50k validation images. We the standard data augmentation strategy for each dataset. We adopt various modern architectures, i.e., ResNet (He et al. 2016). WRN Zagoruyko & Komodakis 2016). VGG (Simonyan & Zisserman 2015). MobileNet (Sandler et al.
2018), and
ShuffleNet (Ma et al.) 2018). We compare KD+ with KD and several SOTA methods, i.e., FitNet (Romero al. 2015)) AT (Zagoruyko & Komodakis 2017). SP (Tung & Mori 2019). FT Kim et al.
2018). NST (Huang & Wang 2017). CC (Peng et al. 2019), FSP (Yim et 2017). PKT Passalis &
Tefas 2018). AB (Heo et al. 2019b). VID (Ahn et al. 2019). RKD (Park et al., 2019).
CRD (Tian
et al. 2020), and SSKD (Xu et al. 2020). For these SOTA approaches, we report the author-reported results, or use author- provided codes and the optimal hyper- perer-parameters if they are publicly available. Otherwise, we use the implementation of Tian et al. (2020).
We follow KD and set a, f3, and T to 0.1, 0.9, 1, and 4, respectively, on all the datasets except on ImageNet where we follow the existing literature to set a = and 7 = 3. We have trained all the networks for 240, 100, and 120 epochs with SGD with momentum 0.9 on CIFAR-100, Tiny ImageNet, and ImageNet, respectively. We set the initial learning rate to 0.05 for for ResNet, WRN, and VGG, and 0.01 for MobileNet and ShuffleNet. On CIFAR, the learning rate is divided by 10 every 30 epochs after the first 150 epochs. On Tiny ImageNet, the learning rate is divided by 5 every 30 epochs. On ImageNet, the learning rate is initilized to 0.1 and is divided by 10 every 30 epochs. More implementation details reported in Appendix A.

---

Under review as a conference paper at ICLR 2021
Table 4: Test accuracy on CIFAR-100. Underline denotes that students match outperform teachers.
Teacher Student
VGG-13
ResNet32x4
WRN-40-2
ResNet-110
WRN-40-2
ResNer32
VGG-8
ResNet8x4
WRN-40-1
ResNet-32
VGG-8
ShuffleNetV2
Teacher Vanilla Student
74.64
79.42
75.61
74.31
75.61
79.42
70.36
72.50
71.98
71.14
70.36
71.86
KD
72.98+0.11
73.33+0.22
73.54+0.20
73.08+0..1
73.51+0.17
74.45++0.27
KD+
75.05.10.24
76.1910.119
75.35
74.22.10.21
75.4710.13
77.22:10.21
FitNet
71.02++0.31
73.50=0.288
72.24++0.24
71.06+0.13
71.14:0.17
73.5440.22
AT
71.43-10.09
73.44-1.119
72.77:10.10
72.3110.08
70.30:10.21
72.73-L0.09
SP
72.68-10.19
72.94-1.223
72.43-1..27
72.6940.41
73.1240.118
74.56:10.22
CC
70.71
72.97=0.17
72.21=0.25
71.8
70.64+0.20
71.29:+0.38
VID
71.23+00.06
73.09=0.21
73.30+0013
72.61+0.28
71.86+00.23
73.40++0.17
RKD
7..4
71.90
72.2211..00
71822:0.34
71.00:0.19
73.21=:0.28
PKT
72.88+0.09
73.6410.18
73.45:1.119
72.61+0.17
72.74::0.22
74.69:10.34
AB
70.94+0.18
73.17+0.31
72.38+0.31
70.98+0.39
72.21+0.41
74.31+0.11
FT
70.58100008
72.8610.12
71.5911..15
72.3710.31
68.3310.22
72.5010.15
CRD
73.94+0.22
75.51=0.18
74.14+0.22
+3.4
74.08
75.65++0.10
NST
71.53+0..3
73.30=0.225
72.24+0:22
71.96+0.07
69.56+0022
74.68++0.26
Table 5: Test accuracies (%)
Tiny ImageNet.
Teacher
Student
KD
KD+
FitNet
AT
CC
SP
VID
RKD
CRD
PKT
AB
VGG-13
VGG-8
60.21
62.20
55.26
56.82
54.14
56.99
54.57
56.60
59.95
56.36
$5.41
(61.62)
(55.46)
10.19
20.11
+0,20
#0.46
40.19
=0.2
10.26
10.13
#0.23
10.17
10.36
WRN-40-2
WRN-40-1
56.25
57.65
55.41
55.84
55.37
56.75
56.31
55.76
(61.84)
(55.39)
+0.15
00.25
+0.31
#0.4
+0.43
-026
#0.23
+0.29
#0.33
+0.22
+0.26
Table 6: Comparison results on ImageNet
Teacher
Student
KD
KD+
AT
CRD
SP
CC
TOP-1 TOP-5
73.31 91.42
69.75
70.66
71.81
70.70
71.17
70.22
69.96
89.07
89.88
90.72
90.00
90.13
89.80
89.17
6.2 ABLATION STUDY AND OUT-OF-DISTRIBUTION DISTILLATION
We investigate how the performance varies with the values of p. We also check how the performance varies with the number of points used the regularizer of KD+ as P contains much more samples than the training dataset. We use to denote the ratio of the number of training samples to the number samples used the regularizer. Table 3 reports the results of KD+ with different and r. The value of p determines what points included in the regularzier of KD+ When p=2, it means that we only use the middle points between two training (or augmented) samples in the regularizer. These middle points have high probability of being out-of-distribution
they do not belong to any predefined classes. As seen from Table 3 by distilling on these middle points (i.e., p=2) as a regularizer, KD+ outperforms KD significantly (e.g., from 73.54 to 75.21 WRN-40-1), which demonstrates that the distir-dittiitttinn samples can be beneficial to knowledge distillation. Note that all outtt--distribution samples are useful(e.g, (.. randomly generated samples from normal distribution are harmful as shown in Appendix D). The reason for the usefulness of these middle points may be that these points are not far from the real data distribution as they share some statisties with the training data (e.g., the mean, the variance, and the relation among data dimensions). We also notice that the performance KD+ is not sensitive the values and p. The best performance achieved when r=1:1 and 3=3. Thus, we simply set r=1:1 and p=3 in the rest of the experiments.
6.3 COMPARISON WITH KD AND SOTA APPROACHES
Table summarizes the comparison results on CIFAR-I00, We have the following observations. First, there is obvious performance gap between the students and the teachers for the existing approaches (e.g., KD, FitNet, and AT). Second, with a simple regularizer, KD+ substantially reduces the performance gap all the six teacher-student pairs, and even matches outperforms the teachers on four pairs (that are denoted by underline). On the other two teacher-student pairs, although the students still underperform the teachers, the performance gap largely reduced by KD+ Note that
7

---

Under review as a conference paper at ICLR 2021
Table 7: Compatibility performances CIFAR-100
Teacher Student
x4
VGG-13 VGG-8
ResNet-50 VGG-8
ResNet32x4
ResNet-110
WRN-40-2
ResNet8 x4
ShuffleNetV2
ResNet-20
WRN-16-2
KD
74.66:10.26
73.22:10.11
73.2410.27
75.15=0.19
70,67:10.11
75.12:10.33
Ki
76.22:10.23
74.68:10.16
75.48:10.22
77.90-0.28
71.27:10.26
75.89=00155
AT+KD
7..53:10.18
73.48:10.19
74.01=0.225
75.39 0.29
10.97:10.17
75.32:10.15
AT+KD+
75.41:0..17
74.08:10.25
74.93:10.13
75.97=0.31
71.23:10.23
75.87=0018
SP+KD
74.02+0.22
73.49+0.19
73.52+0022
74.88
0.16
71.02+0.22
74.98-0.22
SP+KD+
75.34+0022
74.77+00.16
7..22++0.3
76.24
0.18
71,21+0024
75.23+00188
CC+KD
74.21::0.26
73.04:10.15
73.48:t0.29
74.71=0221
70,88-10.20
05.293
CC+KD+
7..28:0.23
74.45:10222
75.33:1..27
77.35-0.15 0.19
71.32:10.27
75.88
VID+KD
74.56+0.10
73.19+0.20
73.46+0.25
74.85
74.85-0.28
71.10+0.18
05.14+0.15
VID+KD+
7..07+0.19
7..08
75.63++0117
77.00 0.29
0.14
75.70-0.22
RKD+KD
73.79=0.18
72.97=0.08
73.51+0.33
74.55-0.23
70.77=0.16
74.89=0.20
RKD+KD+
75.81=00.28
74.49+0.12
75.58+00311
76.47-0.16
71.32+0.17
75.20=0.22
PKT+KD
74.23+0.13
73.25+0.21
73.61+0.28
74.66 0.30
70.72+0.24
75.33++0118
PKT+KD+
7..12+0.22
74.80+00.19
75.70++0.15
76.63-0.15 0.12
71.36+0..0
7,,65+0029
CRD+KD
75.46=00225
74.29=00.12
74.58+0.27
76.05=0.09
71.56+0..16
75.64=0.21
CRD+KI+
76.84=00.19
74.74+0.21
7..82+0.17
76.89-0.12
72.00=0.23
76.08=0.200
AB+KD
74.40=0.227
73.35+10.20
73.65:10.41
74.99-0.35
70.97=10.19
70.27=10.17
AB+KD+
7..95
74.92:10.21
76.11::0.18
0.18
71.75-10.27
71.35=00229
NST+KD
74.28+0022
73.33+0.15
71.74#0.29
75.24-0.40
71,01+0.24
76.67=0.26
NST+KD+
75.60+00119
74.53+0.23
73.85+0.17
77.67-0.27
71.13+0.29
75.68+0.32
SSKD
76.20=10.36
75.33+0.27
75.76::0.40
78.61-0.33
71.38:40.26
76.04=0.21
SSKD+
76.59+0.11
75.60:10.21
76.011:0.25
78175-0.18
71.54+10.00
76.34=0027
there is no guarantee for KD+ make students match or outperform teachers as the regularizer in KD+ cannot fully compensate for the unknown data sample distribution. Third, KD+ consistently outperforms KD and the other SOTA approaches by a large margin across different architectures, which demonstrates the superiority of KD+ Fourth, the pair of WRN-40-2 and VGG-8, almost all representation-baat
approaches (e-g., FitNet and AT) fail to transfer knowledge from the teacher to the student, even underperform the vanilla student. The reason is that WRN-40-2 and VGG-8 have extremely different architectures. Aligning their feature maps hurts the student performance. In contrast, KD+ shows its robustness and superiority in this case, and even enables student VGG-8 to match the performance of teacher 40--40-2.
Table reports the comparison results on Tiny ImageNet. KD+ beats KD and the other approaches significantly, and even outperforms teacher VGG-13, which demonstrates the effectiveness of KD+.
We further evaluate KD+ large scale dataset ImageNet. Limited by computation resources, only adopt one teacher-student pair on ImageNet. We follow CRD and use ResNer-34 and ResNet-18 as the teacher and the student, respectively. As shown in Table 6 KD+ improves the accuracy over KD and the other approaches significantly, which demonstrates the applicability and usefulness of KD+ on large scale datasets. We also notice that there is still an obvious performance gap between the teacher and the student on ImageNet. The reason can be the model capacity difference as we find that ResNet- 18 an IST of ResNet-3d 344 on the large and complex dataset ImageNet.
6.4 COMPATIBILITY WITH SOTA APPROACHES
The existing SOTA approaches can be combined with KD to obtain further performance gain. We show that these approaches combined with KD+ are able to obtain more performance gain. As shown in Table 7 the existing approaches when combined with KD+ consistently achieve much better performances than when combined with KD in all the settings where the teachers and the students use similar or different architectures. This demonstrates that KD+ has a better compatibility than KD and the regularizer of going beyond in-distribution distillation also benefits the existing approaches.
8

---

Under review as a conference paper at ICLR 2021
Table 8: Test accuracies CIFAR-100 under few-shot scenario
60% Training Data
40% Training Data
20% Training Data
10% Training Data
Teacher
R4
VGG-13
ResNe132x4 4
VGG-13
ResNet32x4
VGG-13
ResNet32x4
VGG-13
Student
R
VGG-8
ResNet8xx4
VGG-8
ResNet8x4
VGG-8
ResNet8
VGG-8
Teacher
79.42
74.64
79.42
74.64
79.42
74.64
79.42
74.64
Vanilla Student
68.54
65.57
64.35
61.45
54.70
$2.50
42.76
39.30
KD
69.12
69 69.90
66.44
66.89
58.23
59 14
47.95
49.00
KD+
73.65
72.10
70.75
70.53
65.68
64.94
57.00
$6.50
FüNNtt
69.61
66.60
66.97
62.06
60.18
53.57
51.46
39.89
AT
69.35
67.36
66.19
65.12
57.72
58, 16
44.70
47.
69.62
69.76
58.57
45.44
40.27
CC
68.37
65.37
64.26
60.60
54.68
51.27
42.74
39. 19.16
67.29
65.87
62.58
54,86
44,41
41.07
RKD
67.71
66.18
63.51
62.32
$2.29
52.13
39.19
39.70
PKT
70.48
69.21
66.41
65.97
59.06
58.08
43,50
41.15
71.29
70.46
46.33
AB 69.25
65.98
65.30
63.07
58.48
56.35
48.61
48.27
FT 67.05
64.88
63.38
60.37
53.85
50.42
39.55
38.10
NST 69.87
67.12
66.24
63.56
60.27
56.63
51.91
47.44
6.5
FEW-SHOT SCENARIO
In reality, it happen that powerful model is released, but only few data samples publicly accessible due to the privacy or confidentiality issues. We evaluate KD+ under few-shot scenario where knowledge is transferred from powerful teacher to student with limited data. Table presents the comparison results. We observe that KD+ outperforms KD and the other approaches by large margin all the of KD+ becomes more obvious under few-shot scenario, e.g., 9.05% accuracy improvement over KD on ResNet8 x with 10% training data. The reason that under few-shot scenario, the training data becomes extremely sparse. Corollary 4.1 holds strongly that only fitting sparse data points cannot enable the students to well capture local shapes of the teachers. KD+ substantially mitigates this issue by using regularizer to go beyond the sparse in-distribution distillation.
with 60%, 40%, 20%, and 10% training data available. The superiority
CONCLUSION
In this paper, systematically study why students underperform teachers and how students can outperform teachers under KD based model compression. Through designing exploratory experiments, we find that model capacity differences matters when the student capacity is greater than a threshold. Inspired by this, we propose KD+ which goes beyond in-distribution distillation. Extensive experiments demonstrate that KD+ superior to KD as it outperforms KD and the other SOTA approaches substantially, is more compatible with the existing approaches, and shows obvious superiority few-shot scenario.
not necessarily the root reason and the distillation data
REFERENCES
Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao, Xing Fan, and Edward Guo. Knowledge distillation from internal representations. arXiv preprint arXiv: 191000372, 2019.
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition, pp. 9163-91711 2019.
Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E Dahl, and Geoffrey E Hinton. Large scale distributed neural network training through online distillation. arXiv preprint orXiv:/804.0320 2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXhv:1701.07875 2017.
Wasserstein
gan.
arXiv preprint
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv: 191 0,09755, 2019a.

---

Under review as a conference paper at ICLR 2021
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Information Processing Systems, pp. 5049-5059, 2019b.
Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. In Advances in neural information processing systems, pp. 416-422, 2001.
Defang Chen, Jian-Ping Mei, Can Wang, Yan Feng, and Chun Chen. Online knowledge distillation with diverse peers. In Thirty-Forth AAAI Conference on Artificial Intelligence, 2020.
Jang Hyun Cho and Bharath Hariharan. On the efficacy knowledge distillation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4794 4802, 2019.
Inseop Chung, SkongUk Park, Jangho Kim, and Nojun Kwak. Feature-map-leve online adversarial knowledge distillation. In Proceedings of International Conference on Machine Learning, 2020.
J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. arXiv preprint arXiv: 047700, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2672-2680, 2014.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of-manifold regulariza- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 17-2222,, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint 1912.0279 2019.
Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A com- prehensive overhaul of feature distillation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1921-1930. 2019a.
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 3779-3787, 2019b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint T0X3v:/503.02555 2015.
Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint arXiv: 177. 2017.
Jangho Kim, ScongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. In Advances in Neural Information Processing Systems, pp. 2760-2769, 2018
Animesh Koratana, Daniel Kang, Peter Bailis, and Matei Zaharia. Lit: Learned intermediate representation training for model compression. In International Conference on Machine Learning, pp. 3509-3518, 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. arXiv preprint arXiv:2002.07394 2020.
10

---

Under review as a conference paper at ICLR 2021
Ruishan Liu, Nicolo Fusi, and Lester Mackey, Teacher-student compression with generative adversar- ial networks. arXiv preprint arXiv:1812.02277 2018.
Yufan Liu, Jiajiong Cao, Bing Li, Chunfeng Yuan, Weiming Hu, Yangxi Li, and Yunqiang Duan. Knowledge distillation via instance relationship graph. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 7096-7104, 2019.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European Conference Computer Vision (ECCV), pp. 116-131, 2018.
Seyed- Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant, AAAI Conference on Artificial Intelligence, 2020.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3967-3976. 2019.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 268-2844 284, 2018.
Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li. Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5007-5016, 2019.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In International Conference on Learning Representations, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 4510-4520, 2018
Zhiqiang Shen and Marios Savvides. Meal v2: Boosting vanilla resnet-50 80%+ top-1 accuracy on imagenet without tricks. arXiv preprint WwXiv:2009.88533 2020.
Patrice Y Simard, Yann A LeCun, John S Denker, and Bernard Victorri. Transformation invariance in pattern recognition troenitton-tangeent distance and tangent propagation. In Neural networks: tricks of the trade, pp. 239-274. Springer, 1998.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference Learning Representations, 2015.
Suraj Srinivas and Francois Fleuret. Knowledge transfer with Jacobian matching. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4723-4731, Stockholmsmässan. Stockholm Sweden, 10-15 Jul 2018. PMLR.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Interna- tional Conference on Learning Representations, 2020.
Frederick Tung and Greg Mori. IEEE International Conference on Computer Vision, pp. 1365-1374, 2019.
preeerrviy-- knowledge distillation. In Proceedings of the
Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Kdgan: Knowledge distillation with generative adversarial networks. In Advances in Neural Information Processing Systems, pp. 775-786, 2018.
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self- supervision. In European Conference Computer Vision, pp. 588-604 Springer, 2020.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4133 2017.
11

---

Under review as a conference paper at ICLR 2021
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattemn Recognition, pp. 3903-3911, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In International Conference on Learning Representations, 2017.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical
risk minimization. International Conference Learning Representations, 2018: URL https
//openreview.... net for
Ying Zhang, Tao Xiang. Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4320-4328, 2018b.
Xiatian Zhu, Shaogang Gong, et al. Knowledge distillation by on-the-fly native ensemble. In Advances in neural information processing systems, pp. 7517-7527, 2018.
A MORE IMPLEMENTATION DETAILS
The code for this work will be released online. Besides the hyper-parameters reported in the paper, below we report more implementation details.
We adopt the standard preprocessing and data augmentation strategies for each dataset. Each image is preprocessed by subtracting the mean of the whole training set and dividing it by the standard deviation. We use the standard data augmentation strategy, i.e., randomly flipping horizontally, padding 4 pixels for CIFAR (8 pixels for Tiny ImageNet), and then cropping to x2x for CIFAR 644x64 for Tiny ImageNet). On ImageNet, we use the widely used scale and aspect ratio augmentation strategy.
On exploratory experiments, the architectures of SN2 and SN3 are Conv(128) Conv(128)-BN-Avgeee BN-AvgPooling(3)> FC and
respectively.
Following KD, we set a, 3, and r to 0.1, 0.9, 1, and 4, respectively. On CIFAR-100, we have trained all the networks for 240 epochs with SGD with momentum 0.9 and batch size 64. On Tiny ImageNet, all the networks are trained with SGD with momentum 0.9 for 100 epochs with batch size 64. On ImageNet, we have the network for 120 epochs with SGD with momentum 0.9 and batch size 256.
For the SOTA approaches, their objective is combination of the regular cross-entropy loss and distillation loss:
L LoB CLdistal!
where e is a weight for balancing the two terms. We report the author-reported results, or use author-provided codes and the optimal hyper-parameteee from the original papers if they are publicly available. Otherwise, we use the implementation of Tian et al. (020)) Specifically, the hyper- parameters for each method are: (1) FitNet:
100; (2) AT:
1000; (3) SP:
3000; (4)
CC: c = 0.02: (5) VID: c = 1; (6) RKD: c1 = 25 for the distance metric and 09 = 50 for the angle metrie; both terms combined following the original paper; (7) PKT: 30000: (8) AB:
c = 0: distillation happens in the pre-training stage where only distillation objective is used; (9) FT: 500: (10) CRD: 0.8; (11) FSP: O: distillation happens in the pre-training stage where only distillation objective is used; (12) NST: c = 50; (13) the KD objective is [ :): a, B and T is set to 0. 0.9, and 4, respectively.
For compatibility experiments, KD+ is combined with the existing SOTA approaches. The objective written as:
LCompKD+ LXD+ cLasstil
(9)
The values of have been reported above.
12

---

Under review as a conference paper at ICLR 2021
When the state-of-te oeee-of-the-rr approaches combined with KD, the objective is:
D LKD + chdistill
(10)
For all the experiments, we report the last epoch test accuracy over 3 runs.
B SEACHER-STUDEN SHAPE DIFFERENCES
As stated in Corollary 4.1, only fitting the teacher outputs at sparse data points cannot enable students to well capture the local, in-distribution shapes of teachers. In this part, we show that the students trained with KD+ can better capture the local shapes of the teachers than those trained with KD. The local shape a function can be repre- sented by set of pairs (z, where x is the input and y is the output of the function. To measure the
Table 9: S-T DIFs (Shape differences) on CIFAR-100
Teacher Student
ResNet32 ResNet8
WRN-40-2 WRN-16-2
WRN-40-2
VGG-13
shape difference, we report the av- erage mean square student-teacher output logit differences (S-T DIFs) by using test data as inputs. As shown in Table S-T DIFs of KD+ are consistently smaller than those of KD, which demonstrates that the student shapes of KD+ are closer to the teacher shapes and indicates that the regularizer benefits the students in capturing the local shpaes of the teachers.
WRN-40-1
VGG-8
2.81
2.74
2.94
1.77
KD+
1.45
2.02
2.10
1.11
C COMPARISON WITH THE REGULARIZER OF INJECTING NOISE TO INPUTS
KD+ goes beyond in-distribution distillation by using data-driven regularizer. We compare the regularizer in KD+ with the regularizer of injecting small noise to inputs. Intuitively, distilling on noise-injected samples can also explore more knowledge in the teacher. We call this method NoiseKD). We compare KD+ with NoiseKD. We grid search the best hyperparameter for Noise by using different levels of Gaussian noise, i.e., (0, 0.1,, (00, 0.05). (O, 0.01), and (O, 0.005). Table ?? reports the comparison results. It is observed that when the noise in NoiseKD is large (e.g..(()) 0.1) and (0, 0.05)), NoiseKD even underperforms KD, which indicates that large noise is harmful for knowledge distillation. When noise is relatively small (e.g., (e.g,,((0., 0.01)), NoiseKD slightly improves the performances over KD, which indicates that small noise is useful for knowledge distillation. We also see that KD+ consistently outperforms NoiseKD with different levels of noise as regularzers, which demonstrates the superiority of the proposed regularizer.
D NOT ALL OUT-OF- SAMPLES ARE USEFUL
In KD+, when p = 2, the regularizer almost only uses -iditiidiitiiitiionn samples as the middle points of two samples do not belong any predefined class. The experimental results in Table h shown that by distillation on these out-of-disttibuutin samples as a regularizer, the student performance improved substantially. Here, we show that not all duttrf-distribution samples knowledge distillation. We randomly draw image-size noise from a normal distribution. And then we distill on these randomly generated noisy samples as a regularizer for KD (we denote this method by NoiseRegKD). The results are reported in Table I0 It is not surprising that the performances drop significantly, e.g., from 72.98 6.59 VGG-8. This indicates that the out-of-distribution samples far from the real data distribution are harmful for knowledge distillation.
useful for
E LEARNING DATA DISTRIBUTION WITH GENERATIVE MODELS
As stated in Corollary 4.1, only fitting the teacher outputs at sparse data points cannot enable students to well capture the local, in-distribution shapes of teachers. One natural idea is to use generative adversarial networks (GAN) Goodfellow et al. (2014); Arjovsky et al. (2017); Liu et al. (2018) to learn the data distribution and then use the generator to generate fake data for knowledge distillation. However, there two issues: first, training GAN is computationally expensive especially for large
13

---

Under review as a conference paper at ICLR 2021
Table 10: Out-of-distribution distillation regularizes on CIFAR-100
Teacher
VGG-13
WRN-40-2
ResNet32x4
ResNet32
Student
VGG-8
WRN-16-2
ShuffleNerV2
ResNet8x4
Teacher
74.64
75.61
75.61
79.42
Vanilla Student
70.36
73.26
71.98
72.50
KD
72.98
74.92
73.54
73.33
NoiseRegKD
6.59
6.32
16.35
3.26
Table 11: KD with GAN regularize on CIFAR-10
Teacher Student
WRN-40-2 WRN-16-1
VGG-13
MobileNetV2
Teacher 91.32
94.80
93.65
Vanilla Student
89.19
KD
91.77
89.21
KD-GAN
92.23
89.25
KD+
93.15
90.51
Table 12: Compatibility performances on CIFAR-100 under few-shot scenario
60% Training Data
40% Training Data
10% Training Data
Teacher Student
ResNet32x4 ResNet8 x
VGG-13
ResNe132x4
VGG-13 VGG-8
ResNet32 ResNet8 x4
VGG-13
VGG-8
ResNet8 x4
VGG-8
Teacher
79.42
74.64
79.42
74.64
79.42
74.64
Vanilla Student
68.54
65.57
64.35
61.45
42.76
39.30
KD
72.09
69.21
69.64
65.94
54.72
46.30
FitNet+KD+
74.57
72.23
72.34
69.94
62.43
56.43
AT+KD
71.54
70.61
68.01
68.22
50.22
53.91
AT+KD+
73.15
71.88
70.47
69.58
56.20
57.08
SP+KD
70.45
69.70
67.22
66.73
49.94
45.62
SP+KD+
72.42
72.03
70.42
70.28
55.82
54.42
CC+KD
70.67
69.38
66.81
66.27
48.54
48.37
CC+KD+
73.71
72.70
70.72
70.53
57.00
57.32
VID+KD
70.00
69.54
67.65
66.42
47.84
46.81
VID+KD+
73.86
72.59
71.25
70.76
57.38
56.98
RKD+KD
70.33
69.74
66.63
66.15
46.43
48,20
RKD+KD+
72.85
72.49
70.47
69.79
55,58
56.12
PKT+KD
70.98
70.18
67.43
66.13
48.78
47.88
PKT+KD+
73.21
72.19
70.83
70.56
56.20
55.63
CRD+KD
71.97
70.74
68.83
66.84
48.58
47.94
CRD+KD+
73.65
72.51
70.29
69.14
54.89
55.84
AB+KD
70.04
69.95
67.75
67.47
53.76
60.69
AB+KD+
73.76
73.67
71.48
71.77
60.36
65.09
NST+KD
71.15
69.36
67.92
66.87
55.05
52.49
NST+KD+
72.57
72.20
71.37
71.00
60.28
59.12
datasets (e.g., ImageNet) while KD+ freely obtained in-distribution and out-of-distributie points; second, the diversity and quality of the generated fake data from GAN are highly limited by sparse training data, which means that it cannot accurately learn the real data sample distribution,
14

---

Under review as a conference paper at ICLR 2021
Table 13: Training time of KD and KD+
Teacher
VGG-13
ResNet32x4
WRN-40-2
Student
VGG-8
ResNet8x4
WRN-40-1
KD KD+
7.32s/epoch
17.21s/epoch
15. 15.62s/epoch
13.32s/epoch
32. 12./8//epoch
28.40s/epoch
Table 14: Comparison between baselines and baselines+
Teacher Student
VGG-13
WRN-40-2
VGG-8
WRN-16-2
FitNet 71.02
72.24
FitNet+
74.15
75.09
AT
71.43
72.77
+T+
73.55
74.81
SP
72.68
72.43
SP+
73.81
74.59
CC
70.71
72.21
CC+
74.21
74.99
PKT
72.88
73.45
PKT+
74.39
75.01
just like we cannot obtain 100% test accuracy by training a deep neural network on the training data samples and their ground truth of CIFAR-100.
We conduct an exploratory experiment by using GAN to learn the data sample distribution on CIFAR- 10 Krizhheevyky & Hinton (2009) as GAN can easily converge CIFAR-10. And then distill the generated fake data as regularizer for KD. The Results are reported in Table [1 It is obsrved that the GAN regularizer (i.e., KD-GAN) improves the performances over KD, but it underperforms KD+ substantially. This indicates that GAN can generate some useful fake samples for knowledge distillation, but the diversity and usefulness of these samples are highly constrained by the training data. As it is almost impossible to learn the real data sample distribution from sparse training data points, KD+ compensates this by going beyond in-distribution distillation and thus beats KD and the other approaches by a large margin.
F COMPATIBILITY WITH SOTA APPROACHES UNDER FEW-SHOP SCENARIO
In this part, report the compatibility of KD+ with the existing approaches under few-shot scenario as this case can happen in reality where only a few samples are available due the privacy or confidentiality issues. The comparison results are reported in Table 12, It is observed that the existing approaches when combined with KD+ obtain much better performances than when combined with KD. Moreover, the overall accuracy improvement becomes larger when less training data samples available. The reason that when the training data become extremely sparse, Corollary 4.1 holds strongly that only fitting sparse data points cannot enable the students well capture the local shapes of the teachers. KD+ substantially mitigates this issue by using a regularizer to go beyond the sparse in-distribution distillation.
G TRAINING TIME OF KD AND KD+
As KD+ explores more knowledge in the teacher by going beyond in-distribution distillation, it is more computationally expensive than KD. We report the training data on CIFAR-100 with GPU RTX 2080Ti. Both KD and KD+ trained for 240 epochs. The training time is reported in Table T3
15

---

Under review as a conference paper at ICLR 2021
H COMPARISON RESULTS BETWEEN BASELINES AND BASELINES+
To further explore the performances of the proposed approaches on different distillation methods, we compare baselines with baselines+. Baselinest are obtained by using the points assist the baselines (note that the KD+ objective is not included). The comparison results are reported in Table
14 We observe that the beselines+ consistently outperform the baselines by large margin (e.g.. 13% accuracy improvement from FitNet to FitNet+), which demonstrates the generalization and effectiveness of the proposed strategy across different distillation approaches.
16


Reviews:
[{"review_id": "yP57gYnSTCB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates why conventional student network is not outperformed to teacher network. Based on several experiments design, this paper argues this is due to the student only fit the local, in-distribution shapes of the network rather than model compacity.  Besides, this paper design a go beyond in-distribution distillation approach to overcome this issue. Overall, I think this paper has some merits, but also some concern need to be addressed. \n\nPros:\n1）This paper is well-written. The motivation and the organization of the paper are easy to follow. \n2)   The experiments design to prove the assumption part is good to read and make sense, e.g. Tabel1, Tabel2 CST/ IST\n\nCons:\n1）The first concern is the result of Table 1, I was very supervised the loss of teacher and student is equal to 0. For my experience, the distillation loss in very small, but it still has the same value, which means the student output can not total the same as teacher output.\n\n2）The second concern is about the novelty of KD+, the selection of Out of distribution sample is similar to the technique applied in the current state of art semi-supervised learning methods: 1) mixmatch[1] 2) remixmatch[2]. I suggest the author add these reference and discuss. They also use KD and use middle points.  \n\n3) I also concern about the fairness of Table 4, since using the middle points is a regularization technique, like, mixup[3]. Does it fair to compare with KD without mixup.\n\n4) The last one is the student can outperform to teacher has been studied some literature? I remember some of the paper point out this. \n\nReference:  \n[1]mixmatch: a holistic approach to semi-supervised learning \n[2]remixmatch: semi-supervised learning with distribution alignment and augmentation anchoring\n[3]mixup: Beyond Empirical Risk Minimization\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "woZr9_dadW1", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an argument that model capacity differences are not necessarily the root reason for the performance gap between the student and the teacher, and the distillation data matters when the student capacity is greater than a threshold. Based on this, the authors develop KD+ to reduce the performance gap between them and enable students to match or outperform their teachers. In addition, this paper designs experiments to confirm the proposed arguments. \nHowever, this paper should be rejected because:\n(1)the results of Table 1 cannot confirm the authors’ argument that the widely used students are CSTs. It also depends on the model capacity differences whether a CST is able to fully fit the teacher outputs or not. \n(2) some arguments in this paper are unclear and lack verification. For example, the paper states“This suggests that these students have well captured the knowledge on sparse training data points but have not well captured the local shapes of the teachers within the data distribution.” in Definition 4.2. How did you come to this conclusion?\n(3) the most important or relevant references are not cited, for example [1] Xu G, Liu Z, Li X, et al. Knowledge Distillation Meets Self-Supervision, 2020. And the proposed KD+ is not superior to the approach of [1], for example, some results in Table 4 and Table 8.\n(4) This paper lacks experimental settings and details.\nPlease refer to the above comments and answer these questions.", "extra_scores": {}, "extra_text": {}}, {"review_id": "CJFzlLHfjF", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores why the large gap existing from the viewpoint of data distribution rather than student model capacity. Base on the exploratory experiments and some analyses, the paper proposes KD+ by sampling out-of-distribution data points and achieve better results than current KD methods on the experiment.\n\nStrengths: \n1. The exploratory experiments on out-of-distribution data is interesting, which is meaningful to discuss why the gap between the student and teacher model exists from a new viewpoint of data distribution rather than the student capacity.\n2. KD+ achieved promising results.\n\n\nSome concerns:\n1. A recent KD work [3] also achieved promising results on ImageNet by adopting a discriminator to judge the output distribution is from the teacher model or student model, I think it shares a similar idea with this work, but it uses GAN-style learning to make student's output be more like teachers. This paper fails to compare their results with [3] or discuss the difference by using a GAN style to learn the output distribution of the teacher. \n2.  KD+ will sample data on the original training data, so it would result in more training data than KD or other methods. How does the training time change? The paper fails to discuss training time changes on more training data.\n3. The key point of KD+ is to sample from training data $x$, while it only gives an illustration figure (Figure3) on one dimension data $x$ rather than on real images. Namely, the paper doesn't discuss how to sample $p$ images from two images $x_1$ and $x_2$, by interpolating from $x_1$ and $x_2$ or some other methods? \n4. In Table 2, the simulation results of students (CSTs) are better than the teacher models. I want to know what if use the teacher model to teach itself on the same simulation experiments, as the work in Born Again Network [1] or Tf_self [2].  The author would like to consider this improvement as regularisation [3] or some dark knowledge from out-of-distribution? \n\n\n[1] \"Meal v2: Boosting vanilla resnet-50 to 80%+ top-1 accuracy on imagenet without tricks.\" Shen, et al. arXiv:2009.08453 (2020).\n\n[2] \"Born again neural networks.\" Furlanello et al, ICML 2018.\n\n[3] \"Revisiting knowledge distillation via label smoothing regularization.\", Yuan et al, CVPR 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3QP7GLaIuoo", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies knowledge distillation. In particular, it tries to disentangle the effect of student model capacity and distillation dataset on the performance of the student. The paper goes on to present KD+, a knowledge distillation approach that goes beyond in-distribution data. Experiments on multiple image recognition models and datasets show that KD+ outperforms KD consistently.\n\nThe study is very interesting in how it tries to characterize the impact of  student capacity and distillation data on the performance of  the distilled student. The ablations studies are interesting and shed light on several angels including few-shot learning scenarios, different algorithms for distillation, etc.\n\nThere are some important questions though that arise:\n\n- Is the effect of improved student performance coming from having out-of-distribution data or simply more data for distillation? This also relates to the bigger gains with few-shot learning settings. \n\n- When using p=2 for data augmentation (adding a middle point between two samples), are classes considered in any way? In other words, could it be that the model is benefiting from adding harder examples to the distillation set? Similar observations were made in self-training where there has been a lot of work on leveraging how hard an example is, or how  confident a model is to select samples for self-training?\n\n- The gains seem to vary given the distillation algorithm, with bigger gain when using basic KD. Could other distillation algorithms that try to align the representations of the student and teacher by using different criteria be accomplishing similar generalization effect to the proposed approach\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies Knowledge Distillation (KD) to better understand the reasons behind the performance gap between student and teacher models. The analysis is done by conducting exploratory experiments. The paper establishes that the distillation data used for training a student can play a critical role in the performance gap apart from the model capacity. Building on this idea, the authors propose a new approach to distillation, KD+, utilizing out-of-distribution data when training a student. Extensive experiments are performed to demonstrate the efficiency of KD+. Overall, the paper studies an interesting problem. The results provide a more in-depth explanation of how the distillation data and model capacity play a role in the performance gap between student and teacher models in KD.

I want to thank the authors for providing the rebuttal and sharing their concerns about the quality of one of the reviews.  The reviewers appreciated the paper's ideas; however, all the reviewers were on the fence with borderline scores. In summary, this is a borderline paper, and unfortunately, the final decision is a rejection. The reviewers have provided detailed and constructive feedback for improving the paper. In particular, the authors should incorporate the reviewers' feedback to better position the work w.r.t. the existing literature and provide clear reasoning behind the gains for KD+ in experiments. This is exciting and potentially impactful work, and we encourage the authors to incorporate the reviewers' feedback when preparing future revisions of the paper.
Decision: Reject