Reducing the Teacher-Student Gap via Adaptive Temperatures

1 INTRODUCTION . Deep neural networks have achieved remarkable success in most of computer vision tasks ( He et al. , 2016 ; Deng et al. , 2009 ) , while the increasing network capacity in the current state-of-the-art models also results in severe computational burdens and high inference time ( Tian et al. , 2020 ; Yim et al. , 2017 ) . One direction to reduce the model size is knowledge distillation ( KD ) , which trains small efficient models ( student ) by learning from the output of large models ( teacher ) ( Tang et al. , 2020 ) . In 2015 , Hinton proposed to soften the model output by raising the temperature of the last softmax layer ( Hinton et al. , 2015 ) . Since then , this temperature-based kowledge distillation has drawn the main stream attentions and achieved many successes ( Zagoruyko & Komodakis , 2017 ; Kim et al. , 2020 ; Furlanello et al. , 2018 ; Fu et al. , 2020 ) . However , recent researches found that knowledge distillation suffers from a mysterious performance degradation problem ( Cho & Hariharan , 2019 ; Mirzadeh et al. , 2019 ) . Specifically , since the idea of KD is transferring the knowledge of teacher into student , a nature assumption is larger teachers would train better students . But recent researches refuted this assumption and found that the student often performs worse with oversized teachers ( Cho & Hariharan , 2019 ) . This paper studies this degradation problem by considering the sharpness ( or softness ) of the model output . It is well known that networks with more parameters tend to produce sharper output ( Guo et al. , 2017 ; Lee et al. , 2018 ) , which would create the gap of sharpness between teacher and student , i.e . the teacher is usually much sharper than the student . It should be noted that temperature technology in vanilla knowledge distillation does not reduce this gap . This is because in vanilla knowledge distillation , the teacher and student are softened with the same temperature , which maintains the large gap of sharpness . When the teacher grows larger , this gap will further increases , making it more difficult for the student to learn from the teacher . To solve this problem , let us consider a modified softer teacher model with the same accuracy . In this way , the sharpness gap between the student and the teacher models would be narrowed , making learning more accessible . One way to build this softer teacher is to use higher temperatures for the teacher model , but it would require labor-intensive searches on the validation data set . With this insight , we propose to adaptively change the temperatures of the teacher and the student by their sharpness . Our contribution is three-fold : 1 . We propose a metric to quantify the sharpness of models , allowing us to precisely control the sharpness of model output . Formally , we use the realsoftmax function ( Nielsen & Sun , 2016 ) on the logits vector as the sharpness metric , which approximates a smoother version of the max function . If we denote student logits as z , the teacher logits as v , teacher temperature and student temperature as τT and τS respectively ( τT is equal to τS in the vanilla knowledge distillation ) , the sharpness metric and the gap of sharpness are defined as follows : Ssharpness = log ∑ j ezj/τ Gs gap = log ∑ j evj/τT − log ∑ j ezj/τS ( 1 ) 2 . We propose Adaptive Temperature Knowledge Distillation ( ATKD ) , which automatically changes the temperatures of the teacher and the student , to reduce the sharpness gapGs gap . ATKD relieves the burden of searching for the appropriate temperature on the validation set . Specifically , by Taylor second-order expansion , our method can be implemented easily by normalizing the logits with the standard deviation . 3 . Our Ssharpness metric can be used to explain why two existing methods work . Specifically , the Early Stopped method and the Teacher Assistant method both reduce the sharpness gap between the teacher and the student . We present comprehensive experiments on CIFAR-100 Krizhevsky et al . ( 2009 ) and ImageNet datasets ( Deng et al. , 2009 ) to evaluate our method . The experiment results show that : 1 ) The proposed ATKD method tremendously mitigates the performance degradation problem and trains the best ResNet18 model on ImageNet as we know ( 73.01 % accuracy ) . 2 ) The proposed ATKD method is easy to optimize , exhibits much lower training/test loss with oversized teachers than KD . 2 METHODOLOGY . 2.1 BACKGROUND . Vanillay Knowledge Distillation During training , we minimize the negative log likelihood of the ground truth class to update model parameters . After the model is properly trained , the probability of the ground truth would be close to 1 , while the rest wrong predictions are near zeros . In 2015 , Hinton noticed that these small wrong probabilities are useful to unveil “ dark knowledge ” ( Hinton et al. , 2015 ) . Take a picture of “ cat ” for example , the model are more likely to output higher probability for class “ dog ” than class “ airplane ” . These wrong probabilities imply the relationship between the two classes and unveil how a model tends to generalize . This observation inspired to use the output of large models as soft targets to train efficient small models . However , modern deep networks tend to produce peaky probabilities ( Guo et al. , 2017 ; Lee et al. , 2018 ) , that the numbers of those wrong classes ( near zero values ) would be negligible compared to the ground truth ( near one ) . Thus Hinton proposed to raise the temperature of the last softmax to soften the output probabilities , which can be used as soft targets to train small networks . We denote logits of the teacher as v , and logits of the student as z , temperature as τ , i and j denote the ith and jth value of logits ( i.e . the ith and jth category of K classes ) . the loss of knowledge distillation is as follows : LKD = − ∑ i pTi log p S i pSi = ezi/τ∑ j e zj/τ , pTi = evi/τ∑ j e vj/τ ( 2 ) The final loss for the student is then the weighted sum of the cross entropy loss Lcls and the knowledge distillation loss LKD : L = λLKD + ( 1− λ ) Lcls ( 3 ) The popular choice of the temperature τ is in { 3 , 4 , 5 } and the weight λ = 0.9 ( Hinton et al. , 2015 ; Cho & Hariharan , 2019 ; Tian et al. , 2020 ) . Performance Degradation Problem While knowledge distillation achieved success in many fields , a mysterious performance degradation problem was spotted in 2019 . Since the idea of knowledge distillation is transferring teacher knowledge into students , one natural hypothesis is that a larger and more accurate teacher would capture more knowledge and thus train better students . Unfortunately , previous studies invalidate this hypothesis by showing that the student performance would degenerate unexpectedly with larger teachers . Table 1 shows our experiment result . The student is ResNet14 . With larger teachers , the student accuracy degrades , and the KD loss increase . Cho & Hariharan ( 2019 ) hypothesizes that the mismatch of capacity causes this problem . We will discuss this problem in the rest of the section . 2.2 ADAPTIVE TEMPERATURE KNOWLEDGE DISTILLATION . We investigated the degradation problem and proposed to use adaptive temperatures during training to mitigate this problem : 1 . We found that the mismatch sharpness between the teacher and the student may cause the degradation problem . 2 . We propose a metric to quantify the sharpness of a model . 3 . We propose the adaptive temperature knowledge distillation ( ATKD ) methods . The sharpness of neural networks Two main theories explain why knowledge distillation is effective . The first theory originated from Hinton ’ s first paper , which argued that teachers are more accurate in capturing category similarities . This information of category similarity helps students to generalize better on unseen data . On the other hand , some contend that the soft output of the teacher prevents the student network from overconfidence ( ie . label smoothing technique ) . However , when the teacher gets larger , these two theories will contradict each other . This is because that larger models are usually more accurate and peaky at the same time . On the one hand , a more accurate teacher is beneficial to the student model . On the other hand , a sharper teacher tends to make the student overconfident . Therefore , the larger teacher capacity is a double-edged sword for distillation . This analysis shows that the sharpness of teachers should be investigated carefully , while previous studies have overlooked this critical attribute of teachers . To this end , we propose to use realsoftmax ( Nielsen & Sun , 2016 ) to quantify the sharpness of the model output , which is defined as the logarithm of the sum of the exponentials of the logits : Ssharpness = log ∑ j ezj ( 4 ) Such metric has at least two advantages , therefore reasonably measure the sharpness of the model output : 1 ) This metric is a smooth approximation to the maximum function maxj ( z ) 2 ) It is also differentiable . Meanwhile , we quantify the sharpness gap between the teacher and the student as the difference of the sharpness metric : Gs gap = log ∑ j evj − log ∑ j ezj ( 5 ) If we consider the temperatures , the sharpness gap is : Gs gap = log ∑ j evj/τ − log ∑ j ezj/τ ( 6 ) Table 2 shows sharpness metric value for models of different capacity . It shows that Ssharpness increases with network capacity , and there is a sharpness gap between networks of different sizes . Adaptive Temperature One idea is using higher temperatures for the teacher to reduce the sharpness gap between teachers and students . However , searching on the validation data set requires a lot of computational costs . Instead , we propose adaptive temperature knowledge distillation , which automatically tunes the temperature according to the Ssharpness . By Taylor second expansion : Gs gap = log ∑ j evj/τ − log ∑ j ezj/τ ≈ log ( K + ∑ j vj/τ + 1 2 ∑ j v2j /τ 2 ) − log ( K + ∑ j zj/τ + 1 2 ∑ j z2j /τ 2 ) ( 7 ) We follow the assumption from Hinton ( Hinton et al. , 2015 ) , that the logits have been zero-meaned separately for each training example so that ∑ j zj = ∑ j zj = 0 . We found from experiments that the sum of logits is indeed very small numbers close to zero . The experiment results and further theoretical analysis are provided in the appendix . Given the above assumption ∑ j zj = ∑ j zj = 0 , We can get : Gs gap = log ( K + 1 2 ∑ j ( vj/τ ) 2 ) − log ( K + 1 2 ∑ j ( zj/τ ) 2 ) = log ( 1 + 1 2τ2K ∑ j v2j − log ( 1 + 1 2τ2K ∑ j z2j ) ) ( 8 ) The 1K ∑ j v 2 j in the equation is the variance of logits , which can be represented by the square of the standard deviation std2 : Gs gap = log ( 1 + 1 2 ∗ ( stdT /τ ) 2 ) − log ( 1 + 1 2 ∗ ( stdS/τ ) 2 ) ( 9 ) This result shows the relationship between the temperature and sharpness gap is pretty straightforward , that the standard deviation controls the sharpness gap between the teacher and the student . Therefore , we set the temperature of the teacher and the student as the standard deviation of the logits vector respectively . These adaptive temperatures would automatically change during training , reducing the sharpness gap between the teacher and the student . It is worth noting that there is no need to search for the proper temperature on the validation set like the vanilla KD . Formally , if we denote standard deviation function as Std , we propose Adaptive Temperature Knowledge Distillation as follows : τTi = Std ( vi ) , τ S i = Std ( zi ) LATKD = − ∑ i pTi log p S i pSi = ezi/τ S i∑ j e zj/τSi , pTi = evi/τ T i∑ j e vj/τTi L = λLATKD + ( 1− λ ) Lcls ( 10 )

Summary: This work aims to alleviate the performance degradation problem of the student model during knowledge distillation. The authors argue that the degradation problem may come from the sharpness gap of the model outputs.  They first demonstrate the relation of the sharpness gap (between teacher and student) and the degradation degree, then introduce the realsoftmax function to measure the sharpness of the model output, and finally propose ATKD to adaptively change the temperatures of the teacher and the student for reducing the sharpness gap.  Besides, the authors also analyze previous methods (e.g., Early Stop and TS) from the perspective of the sharpness of models.