Reducing the Teacher-Student Gap via Adaptive Temperatures

1 INTRODUCTION . Deep neural networks have achieved remarkable success in most of computer vision tasks ( He et al. , 2016 ; Deng et al. , 2009 ) , while the increasing network capacity in the current state-of-the-art models also results in severe computational burdens and high inference time ( Tian et al. , 2020 ; Yim et al. , 2017 ) . One direction to reduce the model size is knowledge distillation ( KD ) , which trains small efficient models ( student ) by learning from the output of large models ( teacher ) ( Tang et al. , 2020 ) . In 2015 , Hinton proposed to soften the model output by raising the temperature of the last softmax layer ( Hinton et al. , 2015 ) . Since then , this temperature-based kowledge distillation has drawn the main stream attentions and achieved many successes ( Zagoruyko & Komodakis , 2017 ; Kim et al. , 2020 ; Furlanello et al. , 2018 ; Fu et al. , 2020 ) . However , recent researches found that knowledge distillation suffers from a mysterious performance degradation problem ( Cho & Hariharan , 2019 ; Mirzadeh et al. , 2019 ) . Specifically , since the idea of KD is transferring the knowledge of teacher into student , a nature assumption is larger teachers would train better students . But recent researches refuted this assumption and found that the student often performs worse with oversized teachers ( Cho & Hariharan , 2019 ) . This paper studies this degradation problem by considering the sharpness ( or softness ) of the model output . It is well known that networks with more parameters tend to produce sharper output ( Guo et al. , 2017 ; Lee et al. , 2018 ) , which would create the gap of sharpness between teacher and student , i.e . the teacher is usually much sharper than the student . It should be noted that temperature technology in vanilla knowledge distillation does not reduce this gap . This is because in vanilla knowledge distillation , the teacher and student are softened with the same temperature , which maintains the large gap of sharpness . When the teacher grows larger , this gap will further increases , making it more difficult for the student to learn from the teacher . To solve this problem , let us consider a modified softer teacher model with the same accuracy . In this way , the sharpness gap between the student and the teacher models would be narrowed , making learning more accessible . One way to build this softer teacher is to use higher temperatures for the teacher model , but it would require labor-intensive searches on the validation data set . With this insight , we propose to adaptively change the temperatures of the teacher and the student by their sharpness . Our contribution is three-fold : 1 . We propose a metric to quantify the sharpness of models , allowing us to precisely control the sharpness of model output . Formally , we use the realsoftmax function ( Nielsen & Sun , 2016 ) on the logits vector as the sharpness metric , which approximates a smoother version of the max function . If we denote student logits as z , the teacher logits as v , teacher temperature and student temperature as τT and τS respectively ( τT is equal to τS in the vanilla knowledge distillation ) , the sharpness metric and the gap of sharpness are defined as follows : Ssharpness = log ∑ j ezj/τ Gs gap = log ∑ j evj/τT − log ∑ j ezj/τS ( 1 ) 2 . We propose Adaptive Temperature Knowledge Distillation ( ATKD ) , which automatically changes the temperatures of the teacher and the student , to reduce the sharpness gapGs gap . ATKD relieves the burden of searching for the appropriate temperature on the validation set . Specifically , by Taylor second-order expansion , our method can be implemented easily by normalizing the logits with the standard deviation . 3 . Our Ssharpness metric can be used to explain why two existing methods work . Specifically , the Early Stopped method and the Teacher Assistant method both reduce the sharpness gap between the teacher and the student . We present comprehensive experiments on CIFAR-100 Krizhevsky et al . ( 2009 ) and ImageNet datasets ( Deng et al. , 2009 ) to evaluate our method . The experiment results show that : 1 ) The proposed ATKD method tremendously mitigates the performance degradation problem and trains the best ResNet18 model on ImageNet as we know ( 73.01 % accuracy ) . 2 ) The proposed ATKD method is easy to optimize , exhibits much lower training/test loss with oversized teachers than KD . 2 METHODOLOGY . 2.1 BACKGROUND . Vanillay Knowledge Distillation During training , we minimize the negative log likelihood of the ground truth class to update model parameters . After the model is properly trained , the probability of the ground truth would be close to 1 , while the rest wrong predictions are near zeros . In 2015 , Hinton noticed that these small wrong probabilities are useful to unveil “ dark knowledge ” ( Hinton et al. , 2015 ) . Take a picture of “ cat ” for example , the model are more likely to output higher probability for class “ dog ” than class “ airplane ” . These wrong probabilities imply the relationship between the two classes and unveil how a model tends to generalize . This observation inspired to use the output of large models as soft targets to train efficient small models . However , modern deep networks tend to produce peaky probabilities ( Guo et al. , 2017 ; Lee et al. , 2018 ) , that the numbers of those wrong classes ( near zero values ) would be negligible compared to the ground truth ( near one ) . Thus Hinton proposed to raise the temperature of the last softmax to soften the output probabilities , which can be used as soft targets to train small networks . We denote logits of the teacher as v , and logits of the student as z , temperature as τ , i and j denote the ith and jth value of logits ( i.e . the ith and jth category of K classes ) . the loss of knowledge distillation is as follows : LKD = − ∑ i pTi log p S i pSi = ezi/τ∑ j e zj/τ , pTi = evi/τ∑ j e vj/τ ( 2 ) The final loss for the student is then the weighted sum of the cross entropy loss Lcls and the knowledge distillation loss LKD : L = λLKD + ( 1− λ ) Lcls ( 3 ) The popular choice of the temperature τ is in { 3 , 4 , 5 } and the weight λ = 0.9 ( Hinton et al. , 2015 ; Cho & Hariharan , 2019 ; Tian et al. , 2020 ) . Performance Degradation Problem While knowledge distillation achieved success in many fields , a mysterious performance degradation problem was spotted in 2019 . Since the idea of knowledge distillation is transferring teacher knowledge into students , one natural hypothesis is that a larger and more accurate teacher would capture more knowledge and thus train better students . Unfortunately , previous studies invalidate this hypothesis by showing that the student performance would degenerate unexpectedly with larger teachers . Table 1 shows our experiment result . The student is ResNet14 . With larger teachers , the student accuracy degrades , and the KD loss increase . Cho & Hariharan ( 2019 ) hypothesizes that the mismatch of capacity causes this problem . We will discuss this problem in the rest of the section . 2.2 ADAPTIVE TEMPERATURE KNOWLEDGE DISTILLATION . We investigated the degradation problem and proposed to use adaptive temperatures during training to mitigate this problem : 1 . We found that the mismatch sharpness between the teacher and the student may cause the degradation problem . 2 . We propose a metric to quantify the sharpness of a model . 3 . We propose the adaptive temperature knowledge distillation ( ATKD ) methods . The sharpness of neural networks Two main theories explain why knowledge distillation is effective . The first theory originated from Hinton ’ s first paper , which argued that teachers are more accurate in capturing category similarities . This information of category similarity helps students to generalize better on unseen data . On the other hand , some contend that the soft output of the teacher prevents the student network from overconfidence ( ie . label smoothing technique ) . However , when the teacher gets larger , these two theories will contradict each other . This is because that larger models are usually more accurate and peaky at the same time . On the one hand , a more accurate teacher is beneficial to the student model . On the other hand , a sharper teacher tends to make the student overconfident . Therefore , the larger teacher capacity is a double-edged sword for distillation . This analysis shows that the sharpness of teachers should be investigated carefully , while previous studies have overlooked this critical attribute of teachers . To this end , we propose to use realsoftmax ( Nielsen & Sun , 2016 ) to quantify the sharpness of the model output , which is defined as the logarithm of the sum of the exponentials of the logits : Ssharpness = log ∑ j ezj ( 4 ) Such metric has at least two advantages , therefore reasonably measure the sharpness of the model output : 1 ) This metric is a smooth approximation to the maximum function maxj ( z ) 2 ) It is also differentiable . Meanwhile , we quantify the sharpness gap between the teacher and the student as the difference of the sharpness metric : Gs gap = log ∑ j evj − log ∑ j ezj ( 5 ) If we consider the temperatures , the sharpness gap is : Gs gap = log ∑ j evj/τ − log ∑ j ezj/τ ( 6 ) Table 2 shows sharpness metric value for models of different capacity . It shows that Ssharpness increases with network capacity , and there is a sharpness gap between networks of different sizes . Adaptive Temperature One idea is using higher temperatures for the teacher to reduce the sharpness gap between teachers and students . However , searching on the validation data set requires a lot of computational costs . Instead , we propose adaptive temperature knowledge distillation , which automatically tunes the temperature according to the Ssharpness . By Taylor second expansion : Gs gap = log ∑ j evj/τ − log ∑ j ezj/τ ≈ log ( K + ∑ j vj/τ + 1 2 ∑ j v2j /τ 2 ) − log ( K + ∑ j zj/τ + 1 2 ∑ j z2j /τ 2 ) ( 7 ) We follow the assumption from Hinton ( Hinton et al. , 2015 ) , that the logits have been zero-meaned separately for each training example so that ∑ j zj = ∑ j zj = 0 . We found from experiments that the sum of logits is indeed very small numbers close to zero . The experiment results and further theoretical analysis are provided in the appendix . Given the above assumption ∑ j zj = ∑ j zj = 0 , We can get : Gs gap = log ( K + 1 2 ∑ j ( vj/τ ) 2 ) − log ( K + 1 2 ∑ j ( zj/τ ) 2 ) = log ( 1 + 1 2τ2K ∑ j v2j − log ( 1 + 1 2τ2K ∑ j z2j ) ) ( 8 ) The 1K ∑ j v 2 j in the equation is the variance of logits , which can be represented by the square of the standard deviation std2 : Gs gap = log ( 1 + 1 2 ∗ ( stdT /τ ) 2 ) − log ( 1 + 1 2 ∗ ( stdS/τ ) 2 ) ( 9 ) This result shows the relationship between the temperature and sharpness gap is pretty straightforward , that the standard deviation controls the sharpness gap between the teacher and the student . Therefore , we set the temperature of the teacher and the student as the standard deviation of the logits vector respectively . These adaptive temperatures would automatically change during training , reducing the sharpness gap between the teacher and the student . It is worth noting that there is no need to search for the proper temperature on the validation set like the vanilla KD . Formally , if we denote standard deviation function as Std , we propose Adaptive Temperature Knowledge Distillation as follows : τTi = Std ( vi ) , τ S i = Std ( zi ) LATKD = − ∑ i pTi log p S i pSi = ezi/τ S i∑ j e zj/τSi , pTi = evi/τ T i∑ j e vj/τTi L = λLATKD + ( 1− λ ) Lcls ( 10 )

Summary: This paper is to improve the vanilla knowledge distillation method. Based on the observation in Cho&Hariharan that student degrades by oversized teachers, this paper proposed an adaptive temperature solution. A new metric called sharpness is introduced to quantify the teacher-student gap. Experiments are done on CIFAR100 and ImageNet.