To Smooth or not to Smooth? On Compatibility between Label Smoothing and Knowledge Distillation

1 INTRODUCTION . This paper deeply investigates the compatibility between label smoothing ( Szegedy et al. , 2016 ) and knowledge distillation ( Hinton et al. , 2015 ) . Specifically , we aim to explain and resolve the contradictory standpoints of Müller et al . ( 2019 ) and Shen et al . ( 2021b ) , thereby establishing a foundational understanding on the compatibility between label smoothing ( LS ) and knowledge distillation ( KD ) . Both LS and KD involve training a model ( i.e . : deep neural networks ) with soft-targets . In LS , instead of computing cross entropy loss with the hard-target ( one-hot encoding ) of a training sample , a soft-target is used , which is a weighted mixture of the one-hot encoding and the uniform distribution . A mixture parameter α is used in LS to specify the extent of mixing . On the other hand , KD involves training a teacher model ( usually a powerful model ) and a student model ( usually a compact model ) . The objective of KD is to transfer knowledge from the teacher model to the student model . In the most common form , the student model is trained to match the soft output of the teacher model . The success of KD has been attributed to the transference of logits ’ information about resemblances between instances of different classes ( logits are the inputs to the final softmax which produces the soft targets ) . In KD ( Hinton et al. , 2015 ) , a temperature T is introduced to facilitate the transference : an increased T may produce more suitable soft targets that have more emphasis on the probabilities of incorrect classes ( or equivalently , logits of the incorrect classes ) . To smooth or not to smooth ? Recently , a fair amount of research has been conducted to understand the relationship between LS and KD ( Müller et al. , 2019 ; Shen et al. , 2021b ; Lukasik et al. , 2020 ; Yuan et al. , 2020 ) . One of the most intriguing and controversial discussion is the compatibility between LS and KD . Particularly , in KD , does label smoothing in a teacher network suppress the effectiveness of the distillation ? Müller et al . ( 2019 ) are the first to investigate this topic , and their findings suggest that applying LS to a teacher network impairs the performance of KD . In particular , they visualize the penultimate layer representations in the teacher network to show that LS erases information in the logits about resemblances between instances of different classes . Since this information is essential for KD , they conclude that applying LS for a teacher network can hurt KD . • ‘ If a teacher network is trained with label smoothing , knowledge distillation into a student network is much less effective. ” ( Müller et al. , 2019 ) • “ Label smoothing can hurt distillation ” ( Müller et al. , 2019 ) The conclusion of Müller et al . ( 2019 ) is widely accepted ( Khosla et al. , 2020 ; Arani et al. , 2021 ; Tang et al. , 2021 ; Mghabbar & Ratnamogan , 2020 ; Shen et al. , 2021a ) . However , very recently , this is questioned by Shen et al . ( 2021b ) . In particular , their work discussed a new finding : information erasure in teacher can actually enlarge the central distance between semantically similar classes , allowing the student to learn to classify these categories easily . Shen et al . ( 2021b ) claim that this benefit of using a LS-trained teacher outweighs the detrimental effect due to information erase . Therefore , they conclude that LS in a teacher network does not suppress the effectiveness of KD . • “ Label smoothing will not impair the predictive performance of students. ” ( Shen et al. , 2021b ) • “ Label smoothing is compatible with knowledge distillation ” ( Shen et al. , 2021b ) LS and KD compatibility remains mysterious . We were perplexed by the seemingly contradictory findings by Müller et al . ( 2019 ) and Shen et al . ( 2021b ) . While the latter has shown empirical results to support their own finding , their work does not investigate the opposite standpoint and contradictory results by Müller et al . ( 2019 ) . Critically , there is no effort to understand and resolve the seemingly contradictory arguments and supporting evidences by Müller et al . ( 2019 ) and Shen et al . ( 2021b ) . Consequently , for practitioners , it remains unclear as to under what situations LS can be applied to the teacher network in KD , and under what situations it must be avoided . Our contributions . In this work , we conduct an empirical investigation to establish a foundational understanding on the compatibility between LS and KD . We begin by meticulously scrutinizing the opposing findings of Müller et al . ( 2019 ) and Shen et al . ( 2021b ) . In particular , we discover that in the presence of a LS-trained teacher , KD at higher temperatures systematically diffuses penultimate layer representations learnt by the student towards semantically similar classes . This systematic diffusion essentially curtails the benefits ( as claimed by Shen et al . ( 2021b ) ) obtained by distilling from a LS-trained teacher , thereby rendering KD at increased temperatures ineffective . We perform largescale distillation experiments using ImageNet-1K to comprehensively demonstrate this systematic diffusion in the student qualitatively using penultimate layer visualizations , and quantitatively using our proposed relative distance metric called diffusion index ( η ) . Our finding on systematic diffusion is very critical when distilling from a LS-trained teacher . Particularly , we argue that this diffusion maneuvers the penultimate layer representations learnt by the student of a given class in a systematic way that targets in the direction of semantically similar classes . Therefore , this systematic diffusion directly curtails the distance enlargement ( between semantically similar classes ) benefits obtained by distilling from a LS-trained teacher . Our qualitative and quantitative analysis with our proposed relative distance metric ( η ) in Sec 4 aims to establish not only the existence of this diffusion , but also establish that such diffusion is systematic . We further conduct extensive experiments using fine-grained image classification ( CUB200-2011 ) , neural machine translation ( English to German , English to Russian translation using IWSLT ) and compact student network distillation ( using MobileNetV2 ) tasks to support our key finding on systematic diffusion . Importantly , using systematic diffusion analysis , we explain and resolve the contradictory findings by Müller et al . ( 2019 ) and Shen et al . ( 2021b ) , thereby establishing a foundational understanding on the compatibility between LS and KD . Finally , using our discovery on systematic diffusion , we provide empirical guidelines for practitioners regarding the combined use of LS and KD . We summarize our key findings in Table 1 . The key takeaway from our work is : • In the presence of a LS-trained teacher , KD at higher temperatures systematically diffuses penultimate layer representations learnt by the student towards semantically similar classes . This systematic diffusion essentially curtails the benefits of distilling from a LS-trained teacher , thereby rendering KD at increased temperatures ineffective . Specifically , systematic diffusion was the missing concept that is instrumental in explaining and resolving the contradictory findings of Müller et al . ( 2019 ) and Shen et al . ( 2021b ) , thereby shedding light on whether to smooth or not to smooth a teacher network . Paper organization . In Sec 2 , we review LS and KD . In Sec 3 , we review key findings of Müller et al . ( 2019 ) and Shen et al . ( 2021b ) to to emphasize the research gap . Our main contribution is Sec 4 , where we introduce our discovered systematic diffusion , conduct qualitative , quantitative and analytical studies to verify that the diffusion is not isotopic but systematic towards semanticallysimilar classes , and therefore it directly curtails the benefits of using a LS-trained teacher . In Sec 5 , we perform rich empirical studies to support our main finding on Systematic Diffusion . In Sec 6 , we provide our perspective regarding the combined use of LS and KD as empirical guidelines for practitioners , and finally conclude this study . 2 PREREQUISITES . Label Smoothing ( LS ) ( Szegedy et al. , 2016 ) : LS was formulated as a regularization strategy to alleviate models ’ over-confidence . LS replaces the original hard target distribution with a mixture of original hard target distribution and the uniform distribution characterized by the mixture parameter α . Consider the formulation of LS objective with mixture parameter α as follows : Let pk , wk represent the probability and last layer weights ( including biases ) corresponding to the k-th class . Let x , yk , yLSk represent the penultimate layer activations , true targets and LS-targets where yk = 1 for the correct class and 0 for all the incorrect classes1 . xT is the transpose of x . Then for a classification network trained with LS containing K classes , we minimize the cross entropy loss between LS-targets yLSk and model predictions pk given by LLS ( y , p ) = ∑K k=1 −yLSk log ( pk ) , where pk = exp ( xTwk ) / ∑K l=1 exp ( x Twl ) and yLSk = yk ( 1− α ) + αK . Knowledge distillation ( KD ) Hinton et al . ( 2015 ) : KD uses a larger capacity teacher model ( s ) to transfer the knowledge to a compact student model . The success of KD methods is largely at- 1x is concatenated with 1 at the end to include bias as wk includes biases at the end . tributed to the information about incorrect classes encoded in the output distribution produced by the teacher model ( s ) ( Hinton et al. , 2015 ) . Consider KD for a classification objective . Let T indicate the temperature factor that controls the importance of each soft target . Given the k-th class logit xTwk , let the temperature scaled probability be pk ( T ) . For KD training , let the loss be LKD . For LKD , we replace the cross entropy loss H ( y , p ) with a weighted sum ( parametrized by β ) of H ( y , p ) and H ( pt ( T ) , p ( T ) ) where pt ( T ) , p ( T ) correspond to the temperature-scaled teacher and student output probabilities . That is , pk ( T ) = exp ( x Twk T ) / ∑K l=1 exp ( xTwl T ) and LKD = ( 1 − β ) H ( y , p ) + βT 2H ( pt ( T ) , p ( T ) ) . Following Hinton et al . ( 2015 ) T 2 scaling is used for the soft-target optimization as T will scale the gradients approximately by a factor of T 2 . Following Müller et al . ( 2019 ) ; Shen et al . ( 2021b ) , we set β = 1 for this study since we primarily aim to isolate and study the effects of KD . β = 1 achieves good performance ( Shen et al. , 2021b ) . 3 A CLOSER LOOK AT LS AND KD COMPATIBILITY . In this section , we review the contradictory findings of Müller et al . ( 2019 ) and Shen et al . ( 2021b ) from the perspective of information erase in LS-trained teacher . This discussion is a necessary preamble to discuss our main finding , Systematic Diffusion in the student in Sec 4 . Information Erase in LS-trained teacher . LS objective optimizes the probability of the correct class to be equal to 1 − α + α/K , and incorrect classes to be α/K . This directly encourages the differences between logits of the correct class and incorrect classes to be a constant ( Müller et al. , 2019 ) determined by α . Following Müller et al . ( 2019 ) , the logit xTwk can be approximately measured using the squared Euclidean distance between penultimate layer ’ s activations and the template corresponding to class k. That is , xTwk can be approximately measured by ∥x−wk∥2 . This allows to establish 2 important geometric properties of LS ( Müller et al. , 2019 ) : With LS , penultimate layer activations 1 ) are encouraged to be close to the template of the correct class ( large logit value for the correct class , therefore small distance between the activations and the correct class template ) , and 2 ) are encouraged to be equidistant to the templates of the incorrect classes ( equal logit values for all the incorrect classes ) . This results in penultimate layer activations to tightly cluster around the correct class template compared to the model trained with standard cross entropy objective . We demonstrate this clearly in Figure 1 Observation 1 . With LS applied on the ResNet-50 model , we observe that the penultimate layer representations become much tighter . As a result , substantial information regarding the resemblances of these instances to those of other different classes is lost . This is referred to as the information erase in LS-trained network ( teacher ) ( Müller et al. , 2019 ) . Claim 1 : Information erase in LS-trained teacher cause LS and KD to be Incompatible ( Müller et al. , 2019 ) : Müller et al . ( 2019 ) are the first to investigate this compatibility , and they argue that the information erasure effect due to LS ( shown in Figure 1 Observation 1 ) can impair KD . Given the prominent successes in KD methods being largely attributed to dark knowledge/ inter-class information emerging from the trained-teacher ( Hinton et al. , 2015 ; Tang et al. , 2021 ) , the argument by Müller et al . ( 2019 ) that LS and KD are incompatible due to information loss in the logits is generally convincing and widely accepted ( Khosla et al. , 2020 ; Arani et al. , 2021 ; Tang et al. , 2021 ; Mghabbar & Ratnamogan , 2020 ; Shen et al. , 2021a ) . This is also supported by empirical evidence . Claim 2 : Information erase in LS-trained teacher provides distance enlargement benefits between semantically similar classes , resulting in LS and KD to be Compatible ( Shen et al. , 2021b ) : Recently an interesting finding by Shen et al . ( 2021b ) argue that LS and KD are compatible . Though they agree that information erasure generally happens with LS , their argument focuses more on the effect of LS on semantically similar classes . They argue that information erase in LStrained teacher can promote enlargement of central distance of clusters between semantically similar classes . This allows the student network to easily learn to classify semantically similar classes which are generally difficult to classify in conventional training procedures . We show this increased separation between semantically similar classes with LS in Figure 1 Observation 1 . It can be observed that the central distance between the clusters of standard poodle and miniature poodle increases with using LS on the ResNet-50 teacher . In our work , we further extend to show that this property is inherited by the ResNet-18 student as well in Observation 2 . We remark that this inheritance is not shown by Shen et al . ( 2021b ) . This finding by Shen et al . ( 2021b ) is largely supported by experiments and quantitative results . Though Shen et al . ( 2021b ) claim that the benefit derived from larger separation between semantically similar classes outweigh the drawbacks due to information erase , thereby making LS and KD compatible , their investigation does not address the contradictory findings and empirical results obtained by Müller et al . ( 2019 ) . Research Gap : Studied in isolation , both these contradictory arguments are convincing and are well supported empirically . This has caused serious perplexity among the research community regarding the combined use of LS and KD .

Summary: This paper investigated whether a label smoothing-trained teacher is compatible with knowledge distillation. It started by reproducing the results of two previous studies, i.e., (Muller et al. 2019) claimed that LS erases information in the logits and therefore is harmful to KD, while (Shen et al. 2021b) argued that LS-trained teacher increases the distance between semantically similar classes and therefore is compatible with KD. After that, the authors tried to draw a unified conclusion from the above contradictory claims by introducing experiments with different temperatures: with an LS-trained teacher, a higher temperature introduces systematic diffusion in the representations of the semantically similar classes, which hurts KD. It is the different temperatures used in (Muller et al. 2019) and (Shen et al. 2021b) that resulted in different conclusions. Experiment results supported the main claims.