Can Students Outperform Teachers in Knowledge Distillation based Model Compression?

1 INTRODUCTION . Deep neural networks ( DNNs ) have achieved remarkable performances in various domains , but they require large amounts of computation and memory . This seriously limits their deployment with limited resources or a strict latency requirement . One solution to this problem is knowledge distillation which transfers the knowledge from a large network ( teacher ) to a small one ( student ) . Hinton et al . ( 2015 ) proposed the original knowledge distillation 2 ( KD ) which uses softened logits of a teacher as supervision to train a student . To make the student better capture the knowledge from the teacher , the existing studies focus on aligning their representations by using different criteria . However , there is still a significant performance gap between the teacher and the student . Figuring out the reason for this gap is essential for further improving the student performance . Mirzadeh et al . ( 2020 ) argue that the model capacity difference causes the failure for transferring the knowledge from a large teacher to a small student , thus leading to a large performance gap . Similarly , Cho & Hariharan ( 2019 ) point out that as the teacher grows in capacity and accuracy , it is difficult for the student to emulate the teacher . In this paper , we systematically study why students underperform teachers and how students can match or outperform teachers . We find that in most experimental settings of the existing literature , the root reason for the performance gap is not necessarily the capacity differenc as the student is powerful enough to memorize the teacher ’ s outputs . The reason lies in the distillation dataset on which the knowledge is transferred . As an old proverb says , indigo comes from blue , but it is bluer than blue . In reality , it is not rare for human students to do better than their teachers . These excellent human students not only well capture the knowledge from their teachers but also learn more related knowledge on their own . This gives an insight for students in KD to match or outperform their teachers . We find that currently the students in KD have not well captured the knowledge in their teachers as they only mimic the behavior of the teachers on sparse training data points . In light of this , we propose KD+ which goes beyond in-distribution distillation to substantially reduce the performance gap between students and teachers . Our main contributions are summarized as follows : 1The code will be released online . 2In this paper , we use KD to denote the original knowledge distillation algorithm Hinton et al . ( 2015 ) . • Different from the common belief that model capacity differences result in the performance gap between students and teachers , we find that capacity differences are not necessarily the root reason and instead the distillation data matters when students ’ capacities are greater than a threshold . To our best knowledge , this is the first work that systematically explores why small students underperform teachers and how students can outperform large teachers . • By designing exploratory experiments , we find the following : ( 1 ) only fitting teachers ’ outputs at sparse training data points can not make students well capture the local , indistribution shapes of the teacher functions ; ( 2 ) different from the case on standard supervised learning , out-of-distribution data ( but not all ) can be beneficial to knowledge distillation . • Different from the existing work focusing on using different criteria to align representations or logits between teachers and students , we address knowledge distillation from a novel ( data ) perspective by going beyond in-distribution distillation and accordingly develop KD+ . • Extensive experiments demonstrate that KD+ largely reduces the performance gap between students and teachers , and even enables students to match or outperform their teachers . KD+ is superior to KD as it outperforms KD and more than 10 SOTA methods substantially and shows a better compatibility with the existing methods and superiority in few-shot scenario . 2 RELATED WORK . The objective function of knowledge distillation can be simply expressed as a combination of the regular cross-entropy objective and a distillation objective . According to the distillation objective , the existing literature can be divided into logit-based approaches ( Hinton et al. , 2015 ) and representationbased approaches ( Romero et al. , 2015 ) . Logit-based approaches construct the distillation objective based on output logits . Hinton et al . ( 2015 ) propose KD which penalizes the softened logit differences between a teacher and a student . Park et al . ( 2019 ) propose to transfer data sample relations from a teacher to a student by aligning their logit-based structures . On the other hand , representation-based approaches design the distillation objective based on feature maps . FitNet ( Romero et al. , 2015 ) aligns the features of a teacher and a student through regressions . AT ( Zagoruyko & Komodakis , 2017 ) distills feature attention from a teacher into a student . CRD ( Tian et al. , 2020 ) maximizes the mutual information between student and teacher representations . Other representation-based methods ( Yim et al. , 2017 ; Huang & Wang , 2017 ; Kim et al. , 2018 ; Liu et al. , 2019 ; Srinivas & Fleuret , 2018 ; Wang et al. , 2018 ; Heo et al. , 2019a ; Cho & Hariharan , 2019 ; Ahn et al. , 2019 ; Koratana et al. , 2019 ; Aguilar et al. , 2019 ; Shen & Savvides , 2020 ) use different criteria to align feature representations . SSKD ( Xu et al. , 2020 ) introduces extra self-supervision tasks to assist KD . Online knowledge distillation ( Zhang et al. , 2018b ; Chen et al. , 2020 ; Anil et al. , 2018 ; Chung et al. , 2020 ; Zhu et al. , 2018 ) trains multiple students simultaneously . Self-distillation ( Furlanello et al. , 2018 ; Yuan et al. , 2020 ) approaches train a DNN by using itself as the teacher . It is observed that the existing studies focus on designing different criteria to align teacher-student representations or logits on in-distribution data . In this work , we address knowledge distillation from a data perspective by embedding out-of-distribution distillation into a regularizer . Mirzadeh et al . ( 2020 ) observe that the model capacity gap results in the failure for transferring knowledge from a large teacher to a small student , thus causing a performance gap . To reduce this gap , they propose a multi-step knowledge distillation framework by using several intermediate-size networks ( teacher assistants ) . However , the students still underperform the teachers substantially . Cho & Hariharan ( 2019 ) argue that as the teacher grows in capacity and accuracy , it is difficult for the student to emulate the teacher . To reduce the influence of the large capacity gap , they regularize both the teacher and the knowledge distillation by early stopping . We find that capacity differences are not necessarily the root reason when student capacities are greater than a threshold . On the other hand , KD+ goes beyond in-distribution distillation by exploring the knowledge between two training samples . Similar techniques have been used in many applications with different goals and mechanisms . Mixup ( Zhang et al. , 2018a ) enforces local linearity of a DNN by linearly interpolating a random pair of training samples and their one-hot labels simultaneously . However , simply interpolating two labels may not match the generated sample as pointed out in ( Guo et al. , 2019 ) . KD+ does not have the above issue as it teaches a student to mimic the local shape of a powerful teacher . MixMatch ( Berthelot et al. , 2019b ) linearly interpolates labeled and unlabeled data to improve the semi-supervised learning performances . ReMixMatch ( Berthelot et al. , 2019a ) improves MixMatch by introducing distribution alignment and augmentation anchoring . DivideMix ( Li et al. , 2020 ) aims to learn with noisy labels by modifying MixMatch with label co-refinement and label co-guessing on labeled and unlabeled samples , respectively . AugMix ( Hendrycks et al. , 2019 ) linearly interpolates original training samples and augmented training samples to improves the robustness and uncertainty estimates of DNNs . 3 REFORMULATING KD . Hinton et al . ( 2015 ) propose KD which minimizes the softened logit differences between a student and a teacher over training data Dt = ( Xt , Yt ) where Xt and Yt are the training samples and the ground truth , respectively . The complete objective is : LKD = ∑ ( xt , yt ) ∈ ( Xt , Yt ) [ αLCE ( fS , xt , yt ) + βLKL ( fS , fT , xt ) ] ( 1 ) where α and β are balancing weights and LCE is the regular cross-entropy objective : LCE ( fS , xt , yt ) = H ( yt , σ ( fS ( xt ) ) ) ( 2 ) where H ( . ) is the cross-entropy and σ is softmax . LKL in ( 1 ) is the distillation objective : LKL ( fS , fT , xt ) = τ2KL ( σ ( fT ( xt ) τ ) , σ ( fS ( xt ) τ ) ) ( 3 ) where τ is a temperature to generate soft labels and KL represents KL-divergence . KD can be considered as using one function ( fS ) to fit the outputs of another function ( fT ) . We notice that in ( 1 ) , LCE requires both data samples Xt and the corresponding ground truth Yt while LKL only needs data samples Xt for distilling the teacher knowledge . In light of the difference , we consider KD from semi-supervised perspective and reformulate ( 1 ) in a more general form : L = ∑ ( xt , yt ) ∈ ( Xt , Yt ) αLCE ( fS , xt , yt ) + ∑ xd∈ ( Xd ) βLKL ( fS , fT , xd ) ( 4 ) where we introduce a new concept : distillation dataset Xd is a set of samples on which the knowledge is transferred from a teacher to a student . The first term in the right hand side of ( 4 ) is supervised while the second term is unsupervised . It is obvious that the widely used objective ( 1 ) is a special case of ( 4 ) when Xd is set to Xt . 4 WHY SMALL STUDENTS UNDERPERFORM LARGE TEACHERS ? . In this part , we systematically analyze the reason for the performance gap between students and teachers in KD based model compression . We first introduce several definitions . Definition 4.1 Memorization Error ( ME ) : For a given task with data distribution P ( X , Y ) , ME measures the degree of a student fS fitting the outputs of a teacher fT over the data distribution : E ( fS , fT , P ) = E x∼P ( X ) M ( fT ( x ) , fS ( x ) ) ( 5 ) where M denotes a distance metric such as KL-divergence or mean square error . When ME is 0 , it means that the student can completely memorize the outputs of the teacher over the data distribution . In this paper , we take KL-divergence as M . Definition 4.2 Capable Students ( CSTs ) and Incapable Students ( ISTs ) : network fS with parameters ΘS is a CST of teacher fT if there exists ΘS such that E ( fS , fT , P ) =0 , otherwise , it is an IST . Obviously , a CST is able to fully fit the teacher outputs over data distribution P ( X , Y ) . In contrast , an IST does not have the capacity to fit the teacher . For ISTs , the common belief holds that the student-teacher capacity gap causes the performance gap . For example , we can not expect a two-layer neural network with 1000 parameters to fit the outputs of ResNet-101 with 1.7M parameters on CIFAR-100 . However , in the current SOTA approaches and applications , the commonly used students are modern neural network architectures , such as ResNet-20 , ResNet-8×4 , VGG-8 , and WRN-40-1 . We empirically show that these models are CSTs on commonly used benchmark datasets . To check whether student fS is a CST of teacher fT on a task , we minimize ME to check whether E ( fS , fT , P ) can achieve 0 . However , in practice , it is impossible to calculate E ( fS , fT , P ) as the data distribution P is typically unknown . Fortunately , we have the access to a set of training data ( Xt , Yt ) . With the training data , we approximate ME E ( fS , fT , P ) with the empirical error : Eem ( fS , fT , Xt ) = 1 |Xt| ∑ xt∈Xt M ( fT ( xt ) , fS ( xt ) ) ( 6 ) For comparison , we also evaluate two small neural networks which are expected to be ISTs , i.e. , SN-2 and SN-3 with two and three layers , respectively . We report the ME in Table 1 3 , where we adopt the students and the teachers that share the same architectures ( e.g. , WRN-40-2 and WRN-16-2 ) or use different architectures ( e.g. , ResNet-110 and VGG-8 ) . As expected , the widely used students achieve ME 0.0 on these benckmark datasets , i.e. , CIFAR-10 , CIFAR-100 , and Tiny ImageNet while the small networks ( i.e. , SN2 and SN3 ) have large ME ( e.g. , 2.4 and 4.2 ) , which demonstrates that the widely used students are CSTs . However , as observed in the existing literature , these CSTs underperform the teachers by a significant margin on the test data . This suggests that these students have well captured the knowledge on sparse training data points but have not well captured the local shapes of the teachers within the data distribution . Corollary 4.1 In KD , for CSTs , only fitting the outputs of teachers on sparse training data points can not enable them to well capture the local , in-distribution shapes of the teachers , thus leading to a performance gap . For ISTs , capacity differences cause the performance gap . Proof : We empirically show this by comparing the student performances in the following two settings : ( a ) setting the distillation dataset to training data points ; ( b ) setting the distillation dataset to real data distribution P ( X ) . As P ( X ) is typically unknown in practice , we conduct a simulation experiment on CIFAR-100 . We suppose that the union of the training dataset and the test dataset in CIFAR-100 can accurately represent the real data distribution for this task . Then we randomly draw data samples from the vicinity around the training data and the test data as the distillation dataset , i.e. , Xd in ( 4 ) . Consequently , the distillation dataset can sufficiently represent the real data sample distribution . Note that in the experiments , we never spy the ground truth of the test samples , since the distillation dataset does not use ground truth as shown in ( 4 ) . This means that the students are trained without any additional supervision compared with the teachers as training datset ( Xt , Yt ) in ( 4 ) does not change . As CSTs are able to fully memorize the outputs of the teachers , we expect them to achieve the same 3The ME values in Table 1 are accurate to 1 decimal place . accuracies as or higher accuracies than those of the teachers . In contrast , we expect ISTs to achieve lower accuracies than those of the teachers . Table 2 shows the simulation results . As expected , all the CSTs outperform the teachers in the simulation experiments ( i.e. , Simulation KD ) . This is due to the following facts : first , by using the simulated distillation dataset , the distillation objective in ( 4 ) makes the CSTs fully capture the knowledge of the teachers within the data distribution ; second , the cross-entropy objective in ( 4 ) enables the CSTs to learn their own knowledge . Consequently , CSTs contain both the teacher knowledge and the knowledge learned on their own , which results in better performances than those of the teachers . SN2 and SN3 still underperform the teachers in the simulation experiments due to their limited capacities . These results empirically prove Corollary 4.1 . The simulation experiments also suggest a way for CSTs to outperform teachers . That is to sufficiently distill the knowledge in the teachers with a well representative distillation dataset . Unfortunately , it is impossible to have such a distillation datset as the real data sample distribution P ( X ) is typically unknown in practice . Motivated by this , we propose to go beyond in-distribution distillation .

Summary: The paper studies knowledge distillation. In particular, it tries to disentangle the effect of student model capacity and distillation dataset on the performance of the student. The paper goes on to present KD+, a knowledge distillation approach that goes beyond in-distribution data. Experiments on multiple image recognition models and datasets show that KD+ outperforms KD consistently.