Title: REVISITING KNOWLEDGE DISTILLATION: A TEACHER-FREE FRAMEWORK

Abstract: Knowledge Distillation (KD) aims to distill the knowledge of a cumbersome teacher model into a lightweight student model. Its success is generally attributed to the privileged information on similarities among categories provided by the teacher model, and in this sense, only strong teacher models are deployed to teach weaker students in practice. In this work, we challenge this common belief by following experimental observations: 1) beyond the acknowledgment that the teacher can improve the student, the student can also enhance the teacher significantly by reversing the KD procedure; 2) a poorly-trained teacher with much lower accuracy than the student can still improve the latter significantly. To explain these observations, we provide a theoretical analysis of the relationships between KD and label smoothing regularization. We prove that 1) KD is a type of learned label smoothing regularization and 2) label smoothing regularization provides a virtual teacher model for KD. From these results, we argue that the success of KD is not fully due to the similarity information between categories, but also to the regularization of soft targets, which is equally or even more important. Based on these analyses, we further propose a novel Teacher-free Knowledge Distillation (Tf-KD) framework, where a student model learns from itself or manually-designed regularization distribution. The Tf-KD achieves comparable performance with normal KD from a superior teacher, which is well applied when teacher model is unavailable. Meanwhile, Tf-KD is generic and can be directly deployed for training deep neural networks. Without any extra computation cost, Tf-KD achieves up to 0.65% improvement on ImageNet over well-established baseline models, which is superior to label smoothing regularization.

1. INTRODUCTION:

proposed Knowledge Distillation (KD) to transfer knowledge from one neural network (teacher) to another (student). Usually, the teacher model has strong learning capacity with higher performance, which teaches a lower-capacity student model through providing "soft targets". It is commonly believed that the soft targets of a teacher model can transfer "dark knowledge" containing privileged information on similarity among different categories (Hinton et al., 2015) to enhance the student model.

In this work, we first examine such a common belief through following exploratory experiments: 1) let student models teach teacher models by transferring soft targets of the students; (2) let poorlytrained teacher models with worse performance teach students. Based on the common belief, it is expected that the teacher model would not be enhanced significantly via training from the students and poorly-trained teachers would not enhance the students, as the weak student and poorly-trained teacher models cannot provide reliable similarity information between categories. However, after extensive experiments on various models and datasets, we observe contradictory results: the weak student can improve the teacher and the poorly-trained teacher can also enhance the student remarkably. Such intriguing results motivate us to interpret KD as a regularization term, and we re-examine knowledge distillation from the perspective of Label Smoothing Regularization (LSR) (Szegedy et al., 2016) that regularizes model training by replacing the one-hot labels with smoothed ones.

We then analyze theoretically the relationships between KD and LSR. For LSR, by splitting the smoothed label into two parts and examining the corresponding losses, we find the first part is the ordinary cross-entropy for ground-truth distribution (one-hot label) and outputs of model, and the second part, surprisingly, corresponds to a virtual teacher model which provides a uniform distribution to teach the model. For KD, by combining the teacher's soft targets with the one-hot groundtruth label, we find that KD is a learned LSR where the smoothing distribution of KD is from a teacher model but the smoothing distribution of LSR is manually designed. In a nutshell, we find KD is a learned LSR and LSR is an ad-hoc KD. Such relationships can explain the above counterintuitive results-the soft targets from weak student and poorly-trained teacher models can effectively regularize the model training, even though they lack strong similarity information between categories. We therefore argue that the similarity information between categories cannot fully explain the dark knowledge in KD, and the soft targets from the teacher model indeed provide effective regularization for the student model, which are equally or even more important.

Based on the analyses, we conjecture that with non-reliable or even zero similarity information between categories from the teacher model, KD may still well improve the student models. We thus propose a novel Teacher-free Knowledge Distillation (Tf-KD) framework with two implementations. The first one is to train the student model by itself (i.e., self-training), and the second is to manually design a targets distribution as a virtual teacher model which has 100% accuracy. The first method is motivated by replacing the dark knowledge with predictions from the model itself, and the second method is inspired by the relationships between KD and LSR. We validate through extensive experiments that the two implementations of Tf-KD are both simple yet effective. Particularly, in the second implementation without similarity information in the virtual teacher, Tf-KD still achieves comparable performance with normal KD, which clearly justifies: dark knowledge does not just include the similarity between categories, but also imposes regularization on the student training.

Tf-KD well applies to scenarios where the student model is too strong to find teacher models or computational resource is limited for training teacher models. For example, if we take a cumbersome single model ResNeXt101-32×8d (Xie et al., 2017) as the student model (with 88.79M parameters and 16.51G FLOPs on ImageNet), it is hard or computationally expensive to train a stronger teacher model. We deploy our virtual teacher to teach this powerful student and achieve 0.48% improvement on ImageNet without any extra computation cost. Similarly, when taking a powerful single model ResNeXt29-8×64d with 34.53M parameters as a student model, our self-training implementation achieves more than 1.0% improvement on CIFAR100 (from 81.03% to 82.08%).

Our contributions are summarized as follows:

• By designing two exploratory experiments, we find knowledge distillation can be interpreted as a regularization method. We then provide theoretical analysis to reveal the relation between knowledge distillation and label smoothing regularization.

• We propose Teacher-free Knowledge Distillation (Tf-KD) by self-training or using a welldesigned virtual teacher model. Tf-KD achieves comparable performance with normal knowledge distillation that has a superior teacher.

• Tf-KD can be deployed as a generic regularization to train deep neural networks, which achieves superior performance to label smoothing regularization on ImageNet-2012.

2. EXPLORATORY EXPERIMENTS AND COUNTERINTUITIVE OBSERVATIONS:

To examine the common belief on dark knowledge in KD, we conduct two exploratory experiments:

1) The standard knowledge distillation is to adopt a teacher to teach a weaker student. What if we reverse the operation? Based on the common belief, the teacher should not be improved significantly because the student is too weak to transfer effective knowledge.

2) If we use a poorly-trained teacher which has much worse performance than the student to teach the student, it is assumed to bring no improvement to the latter. For example, if a poorly-trained teacher with only 10% accuracy is adopted in an image classification task, the student would learn from its soft targets with 90% error, thus the student should not be improved or even suffer worse performance. We name the "student teach teacher" as Reversed Knowledge Distillation (Re-KD), and the "poorlytrained teacher teach student" as Defective Knowledge Distillation (De-KD) (Fig. 1 ). We conduct Re-KD and De-KD experiments on CIFAR10, CIFAR100 and Tiny-ImageNet datasets with a variety of neural networks. For fair comparisons, all experiments are conducted with the same settings. Detailed implementation and experiment settings are given in Appendix A.1.

2.1. REVERSED KNOWLEDGE DISTILLATION:

We conduct Re-KD experiments on the three datasets respectively. CIFAR10 and CI-FAR100 (Krizhevsky et al., 2009) contain natural RGB images of 32x32 pixels with 10 and 100 classes, respectively, and Tiny-ImageNet1 is a subset of ImageNet (Deng et al., 2009) with 200 classes, where each image is down-sized to 64x64 pixels. For generality of the experiments, we adopt 5-layer plain CNN, MobilenetV2 (Sandler et al., 2018) and ShufflenetV2 (Ma et al., 2018) as student models and ResNet18, ResNet50 (He et al., 2016) , DenseNet121 (Huang et al., 2017) and ResNeXt29-8×64d as teachers. The results of Re-KD on the three datasets are given in Tabs. 1 to 3.

In Tab. 1, the teacher models are improved significantly by learning from students, especially for teacher models ResNet18 and ResNet50. The two teachers obtain more than 1.1% improvement when taught by MobileNetV2 and ShuffleNetV2. We can also observe similar results on CIFAR10 and Tiny-ImageNet. When comparing Re-KD (S→T) with Normal KD (T→S), we can see in most cases, Normal KD achieves better results. It should be noted that Re-KD takes the teacher' accuracy as the baseline accuracy, which is much higher than that of Normal KD. However, in some cases, we can find Re-KD outperforms Normal KD. For instance, in Tab. 2 (3rd row), the student model (plain CNN) can only be improved by 0.31% when taught by MobileNetV2, but the teacher (Mo-bileNetV2) can be improved by 0.92% by learning from the student. We have similar observations for ResNeXt29 and ResNet18 (4th row in Tab. 2).

Table 1 : Normal KD and Re-KD experiment results on CIFAR100. We report mean±std (in %) over 3 runs. The number in parenthesis means increased accuracy over baseline (T: teacher, S: student). We claim that while the standard knowledge distillation can improve the performance of students on all datasets, the superior teacher can also be enhanced significantly by learning from a weak student, as suggested through the Re-KD experiments.

2.2. DEFECTIVE KNOWLEDGE DISTILLATION:

We conduct De-KD on CIFAR100 and Tiny-ImageNet. We adopt MobileNetV2 and ShuffleNetV2 as student models and ResNet18, ResNet50 and ResNeXt29 (8×64d) as teacher models. The poorlytrained teachers are trained by 1 epoch (ResNet18) or 50 epochs (ResNet50 and ResNeXt29), with very poor performance. For example, ResNet18 only obtains 15.48% accuracy on CIFAR100 and 9.41% accuracy on Tiny-ImageNet after trained with 1 epoch, and ResNet50 obtains 45.82% and 31.01% on CIFAR100 and Tiny-ImageNet, after trained with 50 epochs (200 epochs in total).

From De-KD experiment results on CIFAR100 in Tab. 4, we observe that the student can be greatly promoted even when distilled by a poorly-trained teacher. For instance, the MobileNetV2 and Shuf-fleNetV2 can be promoted by 2.27% and 1.48% when taught by the one-epoch-trained ResNet18 with only 15.48% accuracy (2nd row). For poorly-trained ResNeXt29 with 51.94% accuracy (4th row), we find ResNet18 can still be improved by 1.41%, and MobileNetV2 obtains 3.14% improvement. From the De-KD experiment results on Tiny-ImageNet in Tab. 4, we find ResNet18 with 9.14% accuracy can still enhance the teacher model MobileNetV2 by 1.16%. Other poorly-trained teachers are all able to enhance the students to some degree.

To better demonstrate the distillation accuracy of a student when taught by poorly-trained teachers with different levels of accuracy, we save 9 checkpoints of ResNet18 and ResNeXt29 in the normal training process. Taking these checkpoints as teacher models to teach MobileNetV2, we observe that MobileNetV2 can always be improved by poorly-trained ResNet18 or poorly-trained ResNeXt29 with different levels of accuracy (Fig. 2 ). So we can say while a poorly-trained teacher provides much more noisy logits to the student, the student can still be enhanced. The De-KD experiment results are also conflicted with the common belief.

The counterintuitive results of Re-KD and De-KD make us rethink the "dark knowledge" in KD, and we argue that it does not just contain the similarity information. Lacking enough similarity information, a model can still provide "dark knowledge" to enhance other models. To explain this, we make a reasonable assumption and view knowledge distillation as a model regularization, and investigate what is the additional information in the "dark knowledge" of a model. In the next, we will analyze the relationships between knowledge distillation and label smoothing regularization to explain the experiment results of Re-KD and De-KD. The final point of two blue lines is the result taught by "fully-trained teacher".

3. KNOWLEDGE DISTILLATION AND LABEL SMOOTHING REGULARIZATION:

We mathematically analyze the relationships between Knowledge Distillation (KD) and Label Smoothing Regularization (LSR), hoping to explain the intriguing results of exploratory experiments in Sec. 2. Given a neural network S to train, we first give loss function of LSR for S. For each training example x, S outputs the probability of each label k ∈ {1...K} :

p(k|x) = softmax(z k ) = exp(z k ) K i=1 exp(zi)

, where z i is the logit of the neural network S. The ground truth distribution over the labels is q(k|x). We write p(k|x) as p(k) and q(k|x) as q(k) for simplicity. The model S can be trained by minimizing the cross-entropy loss: H(q, p) = -K k=1 q(k) log(p(k)). For a single ground-truth label y, the q(y|x) = 1 and q(k|x) = 0 for all k = y.

In LSR, it minimizes the cross-entropy between modified label distribution q (k) and the network output p(k), where q (k) is the smoothed label distribution formulated as

EQUATION

which is a mixture of q(k) and a fixed distribution u(k), with weight α. Usually, the u(k) is uniform distribution as u(k) = 1/K. The cross-entropy loss H(q , p) defined over the smoothed labels is

EQUATION

where D KL is the Kullback-Leibler divergence (KL divergence) and H(u) denotes the entropy of u and is a constant for the fixed uniform distribution u(k). Thus, the loss function of label smoothing to model S can be written as

EQUATION

For knowledge distillation, the teacher-student learning mechanism is applied to improve the performance of the student. We assume the student is the model S with output prediction p(k), and the output prediction of the teacher network is

p t τ (k) = softmax(z t k ) = exp(z t k /τ ) K i=1 exp(z t i /τ )

, where z t is the output logits of the teacher network and τ is the temperature to soften p t (k) (written as p t τ (k) after softened). The idea behind knowledge distillation is to let the student (the model S) mimic the teacher by minimizing the cross-entropy loss and KL divergence between the predictions of student and teacher as

EQUATION

Comparing Eq. ( 3) and Eq. ( 4), we find the two loss functions have a similar form. The only difference is that the p t τ (k) in D KL (p t τ , p τ ) is a distribution from a teacher model and u(k) in D KL (u, p) is the pre-defined uniform distribution. From this view, we can consider KD as a special case of LSR where the smoothing distribution is learned but not pre-defined. On the other hand, if we view the regularization term D KL (u, p) as a virtual teacher model of knowledge distillation, this teacher model will give a uniform probability to all classes, meaning it has a random accuracy (1% accuracy for CIFAR100, 0.1% accuracy for ImageNet).

Since D KL (p t τ , p τ ) = H(p t τ , p τ ) -H(p t τ )

, where the entropy H(p t τ ) is constant for a fixed teacher model, we can reformulate Eq. ( 4) to

EQUATION

If we set the temperature τ = 1, we have L KD = H(q t , p), where qt is

EQUATION

If we compare Eq. ( 6) with Eq. ( 1), it is more clearly seen that KD is a special case of LSR. Moreover, the distribution p t (k) is a learned distribution (from a trained teacher) instead of a uniform distribution u(k). We visualize the output probability p t (k) of a teacher and compare it with label smoothing in Appendix A.2, and find with higher temperature τ , the p t (k) is more similar with the uniform distribution u(k) of label smoothing.

Based on the comparison of the two loss functions, we summarize the relationships between knowledge distillation and label smoothing regularization as follows:

• Knowledge distillation is a learned label smoothing regularization, which has a similar function with the latter, i.e. regularizing the classifier layer of the model.

• Label smoothing is an ad-hoc knowledge distillation, which can be revisited as a teacher model with random accuracy and temperature τ = 1.

• With higher temperature, the distribution of teacher's soft targets in knowledge distillation is more similar to the uniform distribution of label smoothing.

Therefore, the experiment results of Re-KD and De-KD can be explained as the soft targets of the model in high temperature are closer to a uniform distribution of label smoothing, where the learned soft targets can provide model regularization for the teacher model. That is why a student can enhance the teacher and a poorly-trained teacher can still improve the student model.

4. TEACHER-FREE KNOWLEDGE DISTILLATION:

As we above analyzed, the "dark knowledge" in a teacher model is more of a regularization term than the similarity information between categories. Intuitively, we consider replacing the output distribution of the teacher model with a simple one. We therefore propose a novel Teacher-free Knowledge Distillation (Tf-KD) framework, which uses no teacher model. Tf-KD is especially applicable to cases where the teacher model is not available, or only limited computation resources are provided. We propose two implementations of our Tf-KD.

The first Tf-KD method is self-training knowledge distillation, denoted as Tf-KD self . As aforementioned, the teacher can be taught by a student and a poorly-trained teacher can also enhance the student. Hence when the teacher model is not available, we propose to deploy "self-training". Specifically, we first train the student model in the normal way to obtain a pre-trained model, which is then used as the teacher to train itself by transferring soft targets as in Eq. ( 4). Formally, given a model S, we denote its pre-trained model as teacher S t ; then we try to minimize the KL divergence of the logits between S and S t by Tf-KD self . The loss function of Tf-KD self to train model S is

EQUATION

where p, p t τ are the output probability of S and S t respectively, τ is the temperature and α is the weight to balance the two terms.

The second implementation of our Tf-KD method is to manually design a teacher with 100% accuracy. In Sec. 3, we reveal LSR is a virtual teacher model with random accuracy. So, if we design a teacher with higher accuracy, we can assume it would bring more improvement to the student. We propose to combine KD and LSR to build a simple teacher model which will output distribution for classes as the following:

EQUATION

)

where K is the total number of classes, c is the correct label and a is the correct probability for the correct class. We always set a ≥ 0.9, so the probability of a correct class is much higher than that of an incorrect one, and the manually-designed teacher model has 100% accuracy for any dataset. We name this method as Teacher-free KD by manually-designed regularization, denoted as Tf-KD reg . The loss function is

EQUATION

where τ is the temperature to soften the manually-designed distribution p d (as p d τ after softening). We set a high temperature τ ≥ 20 to make this virtual teacher output a soft probability, in which way it gains the smoothing property as LSR. We visualize the distribution of the manually designed teacher in Fig. 3 . As Fig. 3 shows, this manually designed teacher model outputs soft targets with 100% classification accuracy, and also has the smoothing property of label smoothing.

The two Teacher-free methods, Tf-KD self and Tf-KD reg , are very simple yet effective, as validated via extensive experiments in the next section.

5. EXPERIMENTS ON TEACHER-FREE KNOWLEDGE DISTILLATION:

In this section, we conduct experiments to evaluate Tf-KD self and Tf-KD reg on three datasets for image classification: CIFAR100, Tiny-ImageNet and ImageNet. For fair comparisons, all experiments are conducted with the same setting. Codes will be publicly available.

5.1. EXPERIMENTS FOR SELF-TRAINING KNOWLEDGE DISTILLATION:

The hyper-parameters (temperature τ and α) for Tf-KD self are given in Appendix A.3. CIFAR100. On CIFAR100, we use baseline models including MobileNetV2, ShuffleNetV2, GoogLeNet, ResNet18, DenseNet121 and ResNeXt29(8×64d). The baselines are trained for 200 epochs, with batch size 128. The initial learning rate is 0.1 and then divided by 5 at the 60th, 120th, 160th epoch. We use SGD optimizer with momentum of 0.9, and weight decay is set to 5e-4. For hyper-parameters, we use grid search to find the best value. Fig. 4 (a) shows the test accuracy of the six models. It can be seen that our Tf-KD self consistently outperforms the baselines. For example, as a powerful model with 34.52M parameters, ResNeXt29 improves itself by 1.05% with self-training (Fig. 4(b) ). Even when compared to Normal KD with a superior teacher in Fig. 5 (a) and Fig. 5 (b), our method achieves comparable performance (experiment settings for Tf-KD and Normal KD are the same). For example, with ResNet50 to teach ReseNet18, the student has a 1.19% improvement, but our method achieves 1.23% improvement without using any stronger teacher model. 128 and then divided by 10 at the 60th, 120th, 160th epoch. We use SGD optimizer with momentum of 0.9, and weight decay is set to 5e-4. Tab. 5 shows the results of Tf-KD self on Tiny-ImageNet. It can be seen that Tf-KD self consistently improves the baseline models and achieves comparable improvement with Normal KD. ImageNet. ImageNet-2012 is one of the largest datasets for object classification, with over 1.3m hand-annotated images. The baseline models we use on this dataset include ResNet18, ResNet50, DenseNet121, RexNeXt101 (32x8d), and we adopt official implementation2 of Pytorch to train them. We set batch size bn = 512 for ResNet18, ResNet50, DenseNet121, and bn = 256 for RexNeXt101. Following common experiment settings (Goyal et al., 2017) , the initial learning rate Comparing our two methods Tf-KD self and Tf-KD reg , we observe that Tf-KD self works better in small dataset (CIFAR100) while Tf-KD reg performs slightly better in large dataset (ImageNet).

5.3. TEACHER-FREE REGULARIZATION:

For Tf-KD reg , the manually designed teacher is a regularization term, meaning no extra computation added. So it can serve as a generic regularization method to normally train neural networks. We compare our Teacher-free regularization with label smoothing on CIFAR100, Tiny-ImageNet and ImageNet. For fair comparisons, experiment settings for Tf-KD reg and LSR are the same. The results are shown in Tab. 8, Tab. 9 and Figure 6 (a). It can be seen that our Tf-KD reg regularization consistently outperforms label smoothing regularization, which is a better choice for model regularization compared with label smoothing.

6. RELATED WORK:

Knowledge Distillation Since Hinton et al. (2015) proposed knowledge distillation based on prior work Ba & Caruana (2014) , KD has been widely adopted or modified (Romero et al., 2014; Yim et al., 2017; Yu et al., 2017; Furlanello et al., 2018; Anil et al., 2018; Mirzadeh et al., 2019; Wang et al., 2019) . Different from existing works, our work challenges the common belief of knowledge distillation based on our designed exploratory experiments. A related work is deep mutual learning (Zhang et al., 2018) , which proposes to let an ensemble of student models to learn with each other by minimizing the KL Divergence of predictions. Comparatively, our work reveals the relationship between KD and label smoothing, and our proposed Teacher-free regularization can serve as a general method for neural network training. Another related work is Born-again networks (Furlanello et al., 2018) , which use similar method as self-training. The difference is that Born-again networks utilize an ensemble of students to trian itself in the final step. Our work can also explain why Born-again networks works well. Label Smoothing Szegedy et al. (2016) proposed label smoothing regularization to replace the "hard labels" with smoothed labels, boosting performance of many tasks like image classification, language translation and speech recognition (Pereyra et al., 2017) . Recently, Müller et al. (2019) empirically showed label smoothing can also help improve model calibration. In our work, we adopt label smoothing regularization to understand the regularization function of knowledge distillation.

7. CONCLUSION AND FUTURE WORK:

In this work, we find through experiments and analyses that the "dark knowledge" of a teacher model is more of a regularization term than similarity information of categories. Based on the relationship between KD and LSR, we propose Teacher-free KD. Experiment results show our Tf-KD can achieve comparable results with Normal KD in image classification. One limitation of this work is that we only test the proposed methods in image classification tasks. In the future, we will explore its application to video classification and natural language understanding. Our work also suggests that, when it is hard to find a good teacher for a powerful model or computation resource is limited for obtaining teacher models, the targeted model can still get enhanced by self-training or a manually-designed regularization term.

:

https://tiny-imagenet.herokuapp.com/

https://github.com/pytorch/examples/tree/master/imagenet

Figures and Tables:

Figure 1: (a) Normal KD framework. (b)(c) Diagrams of exploratory experiments we conduct.

Figure 2: MobileNetV2 taught by ResNet18 and ResNeXt29 with different accuracy on CIFAR100. MobileNetV2 is enhanced by different poorly-trained teachers compared with baseline (the red line). The final point of two blue lines is the result taught by "fully-trained teacher".

Figure 3: Distribution of manually designed teacher (softened by τ = 20) on 10-class dataset. C6 is the correct label. As a comparison, the orange bar is the uniform distribution of LSR.

Figure 4: Classification accuracy by using Tf-KD self . (a) Tf-KD self improves all six models. (b) Improvement on ResNeXt29 trained by itself (Tf-KD self ) is better than the normal training.

Figure 5: Comparison of our Tf-KD self with Normal KD. (a) Tf-KD self achieves comparable results to Normal KD. (b) MobileNetV2 obtains similar improvement by self-training or by ResNet18. Tiny-ImageNet. On Tiny-ImageNet, we use baseline models including MobileNetV2, Shuf-fleNetV2, ResNet50, DenseNet121. They are trained for 200 epochs with batch size bn = 128 for MobileNetV2, ShuffleNetV2 and bn = 64 for ResNet50, DenseNet121. The initial learning rate is η = 0.1 * bn128 and then divided by 10 at the 60th, 120th, 160th epoch. We use SGD optimizer with momentum of 0.9, and weight decay is set to 5e-4. Tab. 5 shows the results of Tf-KD self on Tiny-ImageNet. It can be seen that Tf-KD self consistently improves the baseline models and achieves comparable improvement with Normal KD.

(b) ResNet50 on ImageNet.

Figure 6: Classification accuracy by using Tf-KD reg on ImageNet. (a) Top1 test accuracy on Ima-geNet. (b) Without extra computation, ResNet50 outperforms baseline model by 0.65%

Re-KD experiment results (accuracy, mean±std over 3 runs in %) on CIFAR10.

Table content: <table><tr><td>Teacher: baseline</td><td>Student: baseline</td><td>Normal KD (T→S)</td><td>Re-KD (S→T)</td></tr><tr><td>ResNet18: 95.12</td><td colspan="3">Plain CNN: 87.14 MobileNetV2: 90.98 91.69±0.14 (+0.71) 95.71±0.11 (+0.59) 87.67±0.17 (+0.53) 95.33±0.12 (+0.21)</td></tr><tr><td colspan="2">MobileNetV2: 90.98 Plain CNN: 87.14</td><td colspan="2">87.45±0.18 (+0.31) 91.81±0.23 (+0.92)</td></tr><tr><td>ResNeXt29: 95.76</td><td>ResNet18: 95.12</td><td colspan="2">95.80±0.13 (+0.68) 96.49±0.15 (+0.73)</td></tr></table>

Re-KD experiment results (accuracy, in %) on Tiny-ImageNet.

Table content: <table><tr><td colspan="3">Teacher: baseline Student: baseline</td><td colspan="2">Normal KD (T→S) Re-KD (S→T)</td></tr><tr><td>ResNet18: 63.44</td><td colspan="2">MobileNetV2: 55.06 ShuffleNetV2: 60.51</td><td>56.70 (+1.64) 61.19 (+0.68)</td><td>64.12 (+0.68) 64.35 (+0.91)</td></tr><tr><td/><td colspan="2">MobileNetV2: 55.06</td><td>56.02 (+0.96)</td><td>67.68 (+0.21)</td></tr><tr><td>ResNet50: 67.47</td><td colspan="2">ShuffleNetV2: 60.51</td><td>60.79 (+0.28)</td><td>67.62 (+0.15)</td></tr><tr><td/><td>ResNet18:</td><td>63.44</td><td>64.23 (+0.79)</td><td>67.89 (+0.42)</td></tr></table>

De-KD accuracy (in %)  on two datasets. Pt-Teacher is "Poorly-trained Teacher". Refer to the "Normal KD" in Tabs. 1 to 3 for the accuracy of students taught by "fully-trained teacher".

Table content: <table><tr><td>Dataset</td><td>Pt-Teacher: baseline Student: baseline</td><td>De-KD</td></tr></table>

Test accuracy (in %) on CIFAR100. We run 3 times and report the mean of the best results.

Table content: <table><tr><td>Model</td><td colspan="2">Baseline Tf-KD self</td></tr><tr><td>MobileNetV2</td><td>68.38</td><td>70.96±0.24 (+2.58)</td></tr><tr><td>ShuffleNetV2</td><td>70.34</td><td>72.23±0.16 (+1.89)</td></tr><tr><td>ResNet18</td><td>75.87</td><td>77.10±0.20 (+1.23)</td></tr><tr><td>GoogLeNet</td><td>78.72</td><td>80.17±0.15 (+1.45)</td></tr><tr><td>DenseNet121</td><td>79.04</td><td>80.26±0.19 (+1.22)</td></tr><tr><td colspan="2">ResNeXt29 (8x64d) 81.03</td><td>82.08±0.37 (+1.05)</td></tr><tr><td colspan="3">(a) (b) ResNeXt29 on CIFAR100.</td></tr></table>

Tf-KD self experiment results on Tiny-ImageNet (in %).

Table content: <table><tr><td>Model</td><td colspan="2">Baseline Tf-KD self</td><td>Normal KD [Teacher]</td></tr><tr><td colspan="2">MobileNetV2 55.06</td><td colspan="2">56.77 (+1.71) 56.70 (+1.64) [ResNet18]</td></tr><tr><td colspan="2">ShuffleNetV2 60.51</td><td colspan="2">61.36 (+0.85) 61.19 (+0.68) [ResNet18]</td></tr><tr><td>ResNet50</td><td>67.47</td><td colspan="2">68.18 (+0.71) 68.23 (+0.76) [DenseNet121]</td></tr><tr><td colspan="2">DenseNet121 68.15</td><td colspan="2">68.29 (+0.14) 68.31 (+0.16) [ResNeXt29]</td></tr></table>

Peer review (iclr2020):
This paper connects knowledge distillation and label smoothing regularization. With two sets of well designed experiments, train large network with small teacher or poorly trained teacher, the authors claim knowledge distillation has a strong regularization effect. The authors then show that both knowledge distillation and label smoothing loss can be written as linear combination of supervised learning loss and KL divergence loss. Finally, two “teacher free” methods are proposed, use the same network as student and teacher, and a variant of label smoothing. I am generally on the fence for this paper. I think it makes sense to view knowledge distillation as a regularizer. It is not a surprising finding but I have not seen it in previous papers. On the other hand, there are multiple ways to interpret knowledge distillation, including categorical relationship, loss smoothing, and regularizer. It is hard to convince me regularization is the main drive because in original KD, even without the supervised part H(q, p), it still works reasonably well. It looks to me this paper is more about label smoothing/regularization than knowledge distillation. But since label smoothing has been shown to be an effective regularizer in previous works, it is difficult to say how much practical value is there. About the two proposed teacher free methods, the first one looks like Born Again Network, the second looks like can be achieved by tuning the label smoothing parameter alpha. A relatively minor issue is how the hyper-parameters (alpha and there should be a temperature parameter T) are decided for knowledge distillation.