Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study

1 INTRODUCTION . Label smoothing ( Szegedy et al. , 2016 ) and knowledge distillation ( Hinton et al. , 2015 ) are two commonly recognized techniques in training deep neural networks and have been applied in many state-of-the-art models , such as language translation ( Vaswani et al. , 2017 ; Tan et al. , 2019 ; Zhou et al. , 2020 ) , image classification ( Xie et al. , 2019 ; He et al. , 2019 ) and speech recognition ( Chiu et al. , 2018 ; Pereyra et al. , 2017 ; Chorowski & Jaitly , 2017 ) . Recently a large body of studies is focusing on exploring the underlying relationships between these two methods , for instance , Müller et al . ( Müller et al. , 2019 ) discovered that label smoothing could improve calibration implicitly but will hurt the effectiveness of knowledge distillation . Yuan et al . ( Yuan et al. , 2019 ) considered knowledge distillation as a dynamical form of label smoothing as it delivered a regularization effect in training . The recent study ( Lukasik et al. , 2020 ) further noticed label smoothing could help mitigate label noise , they showed that when distilling models from noisy data , the teacher with label smoothing is helpful . Despite the massive and intensive researches , how to use label smoothing as well as knowledge distillation in practice is still unclear , divergent , and under-explored . Moreover , it is hard to answer when and why label smoothing works well or not under a variety of discrepant circumstances . View of incompatibility between label smoothing and knowledge distillation . Recently , Müller et al . proposed a new standpoint that teachers trained with label smoothing distill inferior student compared to teachers trained with hard labels , even label smoothing improves teacher ’ s accuracy , as the authors found that label smoothing tends to “ erase ” information contained intra-class across individual examples , which indicates that the relative information between logits will be erased to some extent when the teacher is trained with label smoothing . This rising idea is becoming more and more dominant and has been quoted by a large number of recent literatures ( Arani et al. , 2019 ; Tang et al. , 2020 ; Mghabbar & Ratnamogan , 2020 ; Shen et al. , 2020 ; Khosla et al. , 2020 ) . However , this seems reasonable observation basically has many inconsistencies in practice when adopting knowledge distillation with smoothing trained teachers . Thus , we would like to challenge whether this perspective is entirely correct ? To make label smoothing and knowledge distillation less mysterious , in this paper , we first systematically introduce the mechanism and correlation 1Project page : http : //zhiqiangshen.com/projects/LS_and_KD/index.html . Training w/o LS Training w/ LS Validation w/o LS Validation w/ LS 1 1 2 2 1 2 1 2 ! '' ! ! ! ! ! '' ! ! ! ! • What will actually determine the performance of a student in knowledge distillation ? From our empirical study , we observe if the student architecture is settled , the dominating factor in knowledge distillation is the quality of supervision , i.e. , the performance of a teacher network . A higher-accuracy teacher is particularly successful in distilling a better student , regardless it is trained with or without label smoothing . This observation is partially against the conclusion in ( Müller et al. , 2019 ) which stated “ a teacher with better accuracy is not necessary to distill a better student ” . • When will the label smoothing indeed lose its effectiveness for learning deep neural networks ? Long-tailed class distribution and increased number of classes are two scenarios we observed wherein label smoothing will lose or impair its effectiveness . We empirically verify the findings on iNaturalist 2019 ( Van Horn et al. , 2018 ) , Place-LT ( Liu et al. , 2019 ) and curated ImageNet ( Liu et al. , 2019 ) . 2 BACKGROUND . In this section , we first introduce the background of label smoothing and knowledge distillation through a mathematical description . Given a dataset D = ( X , Y ) over a set of classes K , X is the input data and Y is the corresponding one-hot label with each sample ’ s label y ∈ { 0 , 1 } K , where the element yc is 1 for the ground-truth class and 0 for others . Label smoothing replaces one-hot hard label vector y with a mixture of weighted y and a uniform distribution : yc = { 1− α if c = label , α/ ( K − 1 ) otherwise . ( 1 ) where α is a small constant coefficient for flattening the one-hot labels . Usually , label smoothing is adopted when the loss function is cross-entropy , and the network uses softmax function to the last layer ’ s logits z to compute the output probabilities p , so the gradient of each training sample with respect to z will be : ∇H ( p , y ) = p−y = K∑ c=1 ( Softmax ( zc ) − yc ) , whereH ( p , y ) = − K∑ c=1 yclogpc is the cross-entropy loss and zc is c-th logit in z . Effects of label smoothing on loss To further understand the effects of label smoothing on loss function , Fig . 3 illustrates correction effects of smoothing on the binary cross-entropy loss ( K = 2 ) . We can observe that the standard logistic loss ( α = 0 ) vanishes for large and confident positive predictions , and becomes linear for large negative predictions . Label smoothing will penalize confident predictions and involve a finite positive minimum as it aims to minimize the average perclass . Generally , larger α values will produce larger loss values rebounding at positive predictions . This is also the underlying reason that smoothed loss can flatten the predictions of a network . In knowledge distillation , we usually pre-train the teacher model Tw on the dataset in advance . The student model Sw is trained over the same set of data , but utilizes labels generated by Tw . More specifically , we can regard this process as learning Sw on a new labeled dataset D̃ = ( X , Tw ( X ) ) . Once the teacher network is trained , its parameters will be frozen in the whole distillation . The student network Sw is trained by minimizing the similarity between its output and two parts : the hard one-hot labels and the soft labels generated by the teacher network . Letting pTwc ( X ) = Tw ( X ) [ c ] , pSwc ( X ) = Sw ( X ) [ c ] be the probabilities assigned to class c in the teacher model Tw and student model Sw . The distillation loss can be formulated as λH ( pSw , y ) + ( 1 − λ ) H ( pSw/T , pTw/T ) where T is the temperature scaling factor and λ is the trade-off coefficient to balance the two terms . 3 THE “ ERASE INFORMATION ” EFFECT BY LABEL SMOOTHING . This section aims to explain the erasing information effect more thoroughly . We start by reproducing the visualization of penultimate layer ’ s activations using the same procedure from ( Müller et al. , 2019 ) . We adopt ResNet-50 trained with hard and smoothed labels on ImageNet . As shown in Fig . 1 , we obtain similar distributions as ( Müller et al. , 2019 ) . Since examples in training set are the ones used for distillation , we mainly analyze the visualization from the training data . The core finding in ( Müller et al. , 2019 ) is that if a teacher is trained with hard labels , representations of examples are distributed in broad clusters , which means that different examples from the same class can have different similarities ( D1 and D2 ) to other classes . For a teacher trained with label smoothing , they observed the opposite behavior . Label smoothing encourages examples to lie in tight equally separated clusters , so each example of one class has very similar proximities ( D1 is closer to D2 ) to examples of the other classes . Our re-visualization also supports this discovery . The authors derive the conclusion that a teacher with better accuracy is not necessarily to distill a better student . This seems reasonable as the broad clusters can enable different examples from the same class to provide different similarities to other classes , which contains more information for knowledge distillation . However , if refocusing on the two semantically similar classes , when label smoothing is applied , the clusters are much tighter because label smoothing encourages each example is to be equidistant from all other class ’ s templates , while , the tight cluster substantially promotes different class representations to be separate , i.e. , the distance of clusters Dc increases , which further indicates that different class examples obtain more distinguishable features . This phenomenon is crucial as these difficult classes are the key for boosting classification performance . Generally , it is not necessary to measure “ how much a poodle is a particularly similar to a tench ” since we have enough evidence to classify them , but it is critical to have information “ how different is a toy poodle to a miniature poodle ” . Visualizations of teacher predictions . We further visualization the mean distribution of different classes crossing examples , as shown in Fig . 4 . We average all the probabilities after softmax layer if the examples belong to the same category , and show the first 100 classes in ImageNet . Usually , the probabilities have a major value ( the bars in Fig . 4 ( 1 ) ) that represents model ’ s prediction on category and other small values ( i.e. , minor predictions in Fig . 4 ( 2 ) ) indicate that the input image is somewhat similar to those other categories , some discussions about minor predictions are given in Appendix F. Our purpose of this visualization is to make certain of what label smoothing really calibrates in a network and shed light on how it affects the network predictions . We can observe in this figure that a model trained with label smoothing will generate more softened distributions , but the relations across different classes are still preserved . We conjecture the softened supervision is also the reason why teachers with label smoothing produce larger training loss during knowledge distillation . Consequently , label smoothing will both decrease the variance ( verified by following stability metric ) and mean predictive values within a class , but will not impair the relations crossing different classes . 3.1 A SIMPLE METRIC FOR MEASURING THE DEGREE OF ERASED INFORMATION . Different from the visualization scheme ( Müller et al. , 2019 ) of finding an orthonormal basis of the plane that only studies this problem qualitatively , we further address the “ erasing ” phenomenon through a statistical metric that is simple yet effective , and can measure the degree of erasing operation quantitatively . Our motivation behind it is straight-forward : If label smoothing erases relative information within a class , the variance of intra-class probabilities will decrease accordingly , thus we can use such variance to monitor the erasing degree , since this metric evaluates the fluctuation of the representations , we can also call it the stability metric . The definition is as follows : SStability = 1− 1 K K∑ c=1 ( 1 nc nc∑ i=1 ||pTw { i , c } − p Tw { i , c } || 2 ) ( 2 ) where i is the index of images and nc is the # image in class c. pTw { i , c } is the mean of p Tw in class c. This metric utilizes the probabilities of intra-class variance to measure the stability of a teacher ’ s prediction . The results on various network architectures are shown in Sec . 5 and a PyTorch-like code for calculating this metric is given in Appendix C. Such metric has at least two advantages : 1 ) It can measure the degree of erased information quantitatively and further help discover more interesting phenomena , e.g. , we observe that data augmentation method like CutMix ( Yun et al. , 2019 ) together with longer training erases the relative information on logits dramatically and can further be reinforced by label smoothing . 2 ) We found that the proposed metric is highly aligned with model accuracy , thus such metric can be used as a complement for accuracy to evaluate the quality of teacher ’ s supervision for knowledge distillation .

Summary: The authors re-analyze and re-confirm the relationship between label-smoothing and knowledge distillation, which is firstly argued by Muller et al. (“When Does Label Smoothing Help?“, NerurIPS 2019.). This paper shows that the previous argument, "label smoothing is not helpful for knowledge distillation", does not always hold, and carefully re-visits the missing points of the previous analysis by Muller et al. Based on this analysis, label smoothing can be helpful for knowledge distillation and can be explained using the intra-class variation and between-class distance within similar classes. The authors have empirically verified the arguments of the paper with various experiments.