To Smooth or not to Smooth? On Compatibility between Label Smoothing and Knowledge Distillation

1 INTRODUCTION . This paper deeply investigates the compatibility between label smoothing ( Szegedy et al. , 2016 ) and knowledge distillation ( Hinton et al. , 2015 ) . Specifically , we aim to explain and resolve the contradictory standpoints of Müller et al . ( 2019 ) and Shen et al . ( 2021b ) , thereby establishing a foundational understanding on the compatibility between label smoothing ( LS ) and knowledge distillation ( KD ) . Both LS and KD involve training a model ( i.e . : deep neural networks ) with soft-targets . In LS , instead of computing cross entropy loss with the hard-target ( one-hot encoding ) of a training sample , a soft-target is used , which is a weighted mixture of the one-hot encoding and the uniform distribution . A mixture parameter α is used in LS to specify the extent of mixing . On the other hand , KD involves training a teacher model ( usually a powerful model ) and a student model ( usually a compact model ) . The objective of KD is to transfer knowledge from the teacher model to the student model . In the most common form , the student model is trained to match the soft output of the teacher model . The success of KD has been attributed to the transference of logits ’ information about resemblances between instances of different classes ( logits are the inputs to the final softmax which produces the soft targets ) . In KD ( Hinton et al. , 2015 ) , a temperature T is introduced to facilitate the transference : an increased T may produce more suitable soft targets that have more emphasis on the probabilities of incorrect classes ( or equivalently , logits of the incorrect classes ) . To smooth or not to smooth ? Recently , a fair amount of research has been conducted to understand the relationship between LS and KD ( Müller et al. , 2019 ; Shen et al. , 2021b ; Lukasik et al. , 2020 ; Yuan et al. , 2020 ) . One of the most intriguing and controversial discussion is the compatibility between LS and KD . Particularly , in KD , does label smoothing in a teacher network suppress the effectiveness of the distillation ? Müller et al . ( 2019 ) are the first to investigate this topic , and their findings suggest that applying LS to a teacher network impairs the performance of KD . In particular , they visualize the penultimate layer representations in the teacher network to show that LS erases information in the logits about resemblances between instances of different classes . Since this information is essential for KD , they conclude that applying LS for a teacher network can hurt KD . • ‘ If a teacher network is trained with label smoothing , knowledge distillation into a student network is much less effective. ” ( Müller et al. , 2019 ) • “ Label smoothing can hurt distillation ” ( Müller et al. , 2019 ) The conclusion of Müller et al . ( 2019 ) is widely accepted ( Khosla et al. , 2020 ; Arani et al. , 2021 ; Tang et al. , 2021 ; Mghabbar & Ratnamogan , 2020 ; Shen et al. , 2021a ) . However , very recently , this is questioned by Shen et al . ( 2021b ) . In particular , their work discussed a new finding : information erasure in teacher can actually enlarge the central distance between semantically similar classes , allowing the student to learn to classify these categories easily . Shen et al . ( 2021b ) claim that this benefit of using a LS-trained teacher outweighs the detrimental effect due to information erase . Therefore , they conclude that LS in a teacher network does not suppress the effectiveness of KD . • “ Label smoothing will not impair the predictive performance of students. ” ( Shen et al. , 2021b ) • “ Label smoothing is compatible with knowledge distillation ” ( Shen et al. , 2021b ) LS and KD compatibility remains mysterious . We were perplexed by the seemingly contradictory findings by Müller et al . ( 2019 ) and Shen et al . ( 2021b ) . While the latter has shown empirical results to support their own finding , their work does not investigate the opposite standpoint and contradictory results by Müller et al . ( 2019 ) . Critically , there is no effort to understand and resolve the seemingly contradictory arguments and supporting evidences by Müller et al . ( 2019 ) and Shen et al . ( 2021b ) . Consequently , for practitioners , it remains unclear as to under what situations LS can be applied to the teacher network in KD , and under what situations it must be avoided . Our contributions . In this work , we conduct an empirical investigation to establish a foundational understanding on the compatibility between LS and KD . We begin by meticulously scrutinizing the opposing findings of Müller et al . ( 2019 ) and Shen et al . ( 2021b ) . In particular , we discover that in the presence of a LS-trained teacher , KD at higher temperatures systematically diffuses penultimate layer representations learnt by the student towards semantically similar classes . This systematic diffusion essentially curtails the benefits ( as claimed by Shen et al . ( 2021b ) ) obtained by distilling from a LS-trained teacher , thereby rendering KD at increased temperatures ineffective . We perform largescale distillation experiments using ImageNet-1K to comprehensively demonstrate this systematic diffusion in the student qualitatively using penultimate layer visualizations , and quantitatively using our proposed relative distance metric called diffusion index ( η ) . Our finding on systematic diffusion is very critical when distilling from a LS-trained teacher . Particularly , we argue that this diffusion maneuvers the penultimate layer representations learnt by the student of a given class in a systematic way that targets in the direction of semantically similar classes . Therefore , this systematic diffusion directly curtails the distance enlargement ( between semantically similar classes ) benefits obtained by distilling from a LS-trained teacher . Our qualitative and quantitative analysis with our proposed relative distance metric ( η ) in Sec 4 aims to establish not only the existence of this diffusion , but also establish that such diffusion is systematic . We further conduct extensive experiments using fine-grained image classification ( CUB200-2011 ) , neural machine translation ( English to German , English to Russian translation using IWSLT ) and compact student network distillation ( using MobileNetV2 ) tasks to support our key finding on systematic diffusion . Importantly , using systematic diffusion analysis , we explain and resolve the contradictory findings by Müller et al . ( 2019 ) and Shen et al . ( 2021b ) , thereby establishing a foundational understanding on the compatibility between LS and KD . Finally , using our discovery on systematic diffusion , we provide empirical guidelines for practitioners regarding the combined use of LS and KD . We summarize our key findings in Table 1 . The key takeaway from our work is : • In the presence of a LS-trained teacher , KD at higher temperatures systematically diffuses penultimate layer representations learnt by the student towards semantically similar classes . This systematic diffusion essentially curtails the benefits of distilling from a LS-trained teacher , thereby rendering KD at increased temperatures ineffective . Specifically , systematic diffusion was the missing concept that is instrumental in explaining and resolving the contradictory findings of Müller et al . ( 2019 ) and Shen et al . ( 2021b ) , thereby shedding light on whether to smooth or not to smooth a teacher network . Paper organization . In Sec 2 , we review LS and KD . In Sec 3 , we review key findings of Müller et al . ( 2019 ) and Shen et al . ( 2021b ) to to emphasize the research gap . Our main contribution is Sec 4 , where we introduce our discovered systematic diffusion , conduct qualitative , quantitative and analytical studies to verify that the diffusion is not isotopic but systematic towards semanticallysimilar classes , and therefore it directly curtails the benefits of using a LS-trained teacher . In Sec 5 , we perform rich empirical studies to support our main finding on Systematic Diffusion . In Sec 6 , we provide our perspective regarding the combined use of LS and KD as empirical guidelines for practitioners , and finally conclude this study . 2 PREREQUISITES . Label Smoothing ( LS ) ( Szegedy et al. , 2016 ) : LS was formulated as a regularization strategy to alleviate models ’ over-confidence . LS replaces the original hard target distribution with a mixture of original hard target distribution and the uniform distribution characterized by the mixture parameter α . Consider the formulation of LS objective with mixture parameter α as follows : Let pk , wk represent the probability and last layer weights ( including biases ) corresponding to the k-th class . Let x , yk , yLSk represent the penultimate layer activations , true targets and LS-targets where yk = 1 for the correct class and 0 for all the incorrect classes1 . xT is the transpose of x . Then for a classification network trained with LS containing K classes , we minimize the cross entropy loss between LS-targets yLSk and model predictions pk given by LLS ( y , p ) = ∑K k=1 −yLSk log ( pk ) , where pk = exp ( xTwk ) / ∑K l=1 exp ( x Twl ) and yLSk = yk ( 1− α ) + αK . Knowledge distillation ( KD ) Hinton et al . ( 2015 ) : KD uses a larger capacity teacher model ( s ) to transfer the knowledge to a compact student model . The success of KD methods is largely at- 1x is concatenated with 1 at the end to include bias as wk includes biases at the end . tributed to the information about incorrect classes encoded in the output distribution produced by the teacher model ( s ) ( Hinton et al. , 2015 ) . Consider KD for a classification objective . Let T indicate the temperature factor that controls the importance of each soft target . Given the k-th class logit xTwk , let the temperature scaled probability be pk ( T ) . For KD training , let the loss be LKD . For LKD , we replace the cross entropy loss H ( y , p ) with a weighted sum ( parametrized by β ) of H ( y , p ) and H ( pt ( T ) , p ( T ) ) where pt ( T ) , p ( T ) correspond to the temperature-scaled teacher and student output probabilities . That is , pk ( T ) = exp ( x Twk T ) / ∑K l=1 exp ( xTwl T ) and LKD = ( 1 − β ) H ( y , p ) + βT 2H ( pt ( T ) , p ( T ) ) . Following Hinton et al . ( 2015 ) T 2 scaling is used for the soft-target optimization as T will scale the gradients approximately by a factor of T 2 . Following Müller et al . ( 2019 ) ; Shen et al . ( 2021b ) , we set β = 1 for this study since we primarily aim to isolate and study the effects of KD . β = 1 achieves good performance ( Shen et al. , 2021b ) . 3 A CLOSER LOOK AT LS AND KD COMPATIBILITY . In this section , we review the contradictory findings of Müller et al . ( 2019 ) and Shen et al . ( 2021b ) from the perspective of information erase in LS-trained teacher . This discussion is a necessary preamble to discuss our main finding , Systematic Diffusion in the student in Sec 4 . Information Erase in LS-trained teacher . LS objective optimizes the probability of the correct class to be equal to 1 − α + α/K , and incorrect classes to be α/K . This directly encourages the differences between logits of the correct class and incorrect classes to be a constant ( Müller et al. , 2019 ) determined by α . Following Müller et al . ( 2019 ) , the logit xTwk can be approximately measured using the squared Euclidean distance between penultimate layer ’ s activations and the template corresponding to class k. That is , xTwk can be approximately measured by ∥x−wk∥2 . This allows to establish 2 important geometric properties of LS ( Müller et al. , 2019 ) : With LS , penultimate layer activations 1 ) are encouraged to be close to the template of the correct class ( large logit value for the correct class , therefore small distance between the activations and the correct class template ) , and 2 ) are encouraged to be equidistant to the templates of the incorrect classes ( equal logit values for all the incorrect classes ) . This results in penultimate layer activations to tightly cluster around the correct class template compared to the model trained with standard cross entropy objective . We demonstrate this clearly in Figure 1 Observation 1 . With LS applied on the ResNet-50 model , we observe that the penultimate layer representations become much tighter . As a result , substantial information regarding the resemblances of these instances to those of other different classes is lost . This is referred to as the information erase in LS-trained network ( teacher ) ( Müller et al. , 2019 ) . Claim 1 : Information erase in LS-trained teacher cause LS and KD to be Incompatible ( Müller et al. , 2019 ) : Müller et al . ( 2019 ) are the first to investigate this compatibility , and they argue that the information erasure effect due to LS ( shown in Figure 1 Observation 1 ) can impair KD . Given the prominent successes in KD methods being largely attributed to dark knowledge/ inter-class information emerging from the trained-teacher ( Hinton et al. , 2015 ; Tang et al. , 2021 ) , the argument by Müller et al . ( 2019 ) that LS and KD are incompatible due to information loss in the logits is generally convincing and widely accepted ( Khosla et al. , 2020 ; Arani et al. , 2021 ; Tang et al. , 2021 ; Mghabbar & Ratnamogan , 2020 ; Shen et al. , 2021a ) . This is also supported by empirical evidence . Claim 2 : Information erase in LS-trained teacher provides distance enlargement benefits between semantically similar classes , resulting in LS and KD to be Compatible ( Shen et al. , 2021b ) : Recently an interesting finding by Shen et al . ( 2021b ) argue that LS and KD are compatible . Though they agree that information erasure generally happens with LS , their argument focuses more on the effect of LS on semantically similar classes . They argue that information erase in LStrained teacher can promote enlargement of central distance of clusters between semantically similar classes . This allows the student network to easily learn to classify semantically similar classes which are generally difficult to classify in conventional training procedures . We show this increased separation between semantically similar classes with LS in Figure 1 Observation 1 . It can be observed that the central distance between the clusters of standard poodle and miniature poodle increases with using LS on the ResNet-50 teacher . In our work , we further extend to show that this property is inherited by the ResNet-18 student as well in Observation 2 . We remark that this inheritance is not shown by Shen et al . ( 2021b ) . This finding by Shen et al . ( 2021b ) is largely supported by experiments and quantitative results . Though Shen et al . ( 2021b ) claim that the benefit derived from larger separation between semantically similar classes outweigh the drawbacks due to information erase , thereby making LS and KD compatible , their investigation does not address the contradictory findings and empirical results obtained by Müller et al . ( 2019 ) . Research Gap : Studied in isolation , both these contradictory arguments are convincing and are well supported empirically . This has caused serious perplexity among the research community regarding the combined use of LS and KD .

Summary: The paper provides a new perspective for the compatibility of label smoothing and knowledge distillation. A new definition named "systematic diffusion" is proposed to analyze the contradiction between the claims of [1] and [2]. By observing the visualization of the penultimate layer representations, the authors claim that KD is compatible with LS under the setting of lower temperature T. The paper conducts a large-scale evaluation of the impact of increasing temperature T in KD in the perspective of the systematic diffusion. The findings are interesting and attractive. The visualization looks good.  [1] Rafael Muller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in Neural Information Processing Systems, 2019 [2] Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang-Ting Cheng, and Marios Savvides. Is label smoothing truly incompatible with knowledge distillation: An empirical study. International Conference on Learning Representations, 2021